Are frontier models internally consistent?

Luis Fominaya · August 12, 2026 · 11 min read

Every day millions of people delegate their judgment to LLMs for cognitive tasks. People let AI make decisions on everything from fixing a faulty faucet to assisting doctors with diagnosis.

The common fear with AI is that it will replace humans in some indeterminate future. I say it has already taken over many jobs where people effectively become meat interfaces or LLM wrappers, feeding data to the LLM and letting it decide what to do.

AI capabilities keep expanding: budgeting, scheduling and coding already experience massive adoption. Soon, most critical decisions will be made or heavily assisted by LLMs: hiring, medical screening, insurance, grants and loans.

You may not want an AI to decide whether you get a mortgage, whether your symptoms warrant more tests, whether you will get the job or whether you should be granted bail. Some of those decisions are already scored by algorithms: pretrial risk assessment tools and credit scores have been around for a while. The difference is LLMs are still black boxes whose decision can change depending on how you phrase the question.

LLMs cost orders of magnitude less than humans. They do not get tired, they scale massively, and they carry little management overhead. It's hard to see a scenario where they don't dominate most cognitive tasks.

Even if there is a human in the loop who reviews and accepts the machine's output (this is far more palatable to most people), reviewers tend to defer to whatever the machine already decided. Eventually we will all be subject to artificial judgment, so we better make sure it's at least consistent.

noisebench evaluates the current frontier models' judgment quality in terms of noise (random variability in judgment) and bias (consistent deviation)

Both are pervasive in human judgement: sentences in criminal cases are often affected by factors like ethnicity that should have no impact, doctors sometimes disagree in diagnosis, and different underwriters set different prices for the same policies.

This benchmark is inspired mainly on the book "Noise" by Kahneman, Sibony and Sunstein, which of course I recommend. Whereas before it was entirely impractical to imagine all experts across domains following the same guidelines and processes to avoid noise, soon it will be a fixture of the world.

And it is up to frontier labs to address noise in judgment.

Introducing noisebench

noisebench measures the amount of noise in LLMs by asking frontier models to forecast different questions multiple times and measuring the random variations in their forecasts.

[high-level noise index chart]

These questions come from current Polymarket markets with some of the highest activity across topics like politics, finance and tech. These are questions such as "Will JD Vance win the 2028 US Presidential Election?", "Hantavirus pandemic in 2026?" and "Fed rate hike in 2026?"

The benchmark uses two methods to derive all the noise metrics:

  1. Direct forecasting: Asking the model to estimate the odds of an outcome happening, as a number between 0.01% and 99.99%.
  2. Comparative forecasting: Asking the model to pick the outcome that is more likely from a pair.

Every task is done more than once for every model to measure variations that happen purely by randomness, keeping the inputs equal. Every model runs at temperature 1 and top_p 1 with medium reasoning effort, which is what a normal caller gets by default. In total the run made 24,000 inference calls: 16,000 direct forecasts and 8,000 pair judgments.

Also for each question the model is asked to forecast separately the odds of the outcome being "Yes" and "No", and then we compare the consistency of the judgments.

On the comparative forecasting task, question pairs are also presented in the four possible binary combinations of positive vs. negated to test the robustness of the model's logic. Let's see it with an example from the benchmark:

Market A: "Billionaire one-time wealth tax passes in California election 2026?" Market B: "Will Marine Le Pen win the 2027 French presidential election?"

We forecast:

  • 4x What are the odds "Billionaire one-time wealth tax passes in California election 2026?" resolves to "Yes"
  • 4x What are the odds "Billionaire one-time wealth tax passes in California election 2026?" resolves to "No"
  • 4x What are the odds "Will Marine Le Pen win the 2027 French presidential election?" resolves to "Yes"
  • 4x What are the odds "Will Marine Le Pen win the 2027 French presidential election?" resolves to "No"

And for the pair we forecast:

  • 2x What is more likely, A resolves to "Yes" or B resolves to "Yes"
  • 2x What is more likely, A resolves to "Yes" or B resolves to "No"
  • 2x What is more likely, A resolves to "No" or B resolves to "Yes"
  • 2x What is more likely, A resolves to "No" or B resolves to "No"

Note that a human would just assign probabilities for A and B and then answer all these questions according to their initial odds. For example, using the markets' own prices at the time of the dataset's snapshot:

  • P(A_yes) = 33.5%
  • P(B_yes) = 32%

This implies:

  • P(A_no) = 66.5%
  • P(B_no) = 68%
  • P(Ayes) > P(Byes)
  • P(Ayes) < P(Bno)
  • P(Ano) > P(Byes)
  • P(Ano) < P(Bno)

But because we run the same prompts on separate inference requests that are isolated from one another, LLMs in this benchmark do not have access to their own previous odds. Instead, on every prompt they forecast each question from scratch, which allows us to see the impact and variation of different ways of asking the same thing.

Every judgment is also published on a custom contract on Base mainnet from a per-model wallet to create a transparent, immutable record of forecasts.

Methodology is everything

There are plenty of benchmarks already out there measuring general reasoning ability as well as forecasting skill for LLMs. There is also a paper examining similar questions in Zhu and Griffiths (2024), where they use probabilistic identities and repeated judgments to show that LLM probability judgments are incoherent, as well as other related work on self-inconsistency in LLM judges. But nothing comprehensive or recent enough to be useful with the current frontier models.

The issue is typically that complex benchmarks measure too much at the same time, or do so with flawed rubrics. There are two main approaches to benchmarking forecasting ability that sound good in theory but are actually measuring something else:

  1. Trading PnL: Model profit on real prediction markets depends on the quality of the harness, the measurement window, available funds, breadth and depth of research gathering, and mainly, the inherent randomness in outcomes. It measures trading, which is a composite of forecasting + making decisions about budgeting and bet sizing. Paper trading is considerably worse than real trading because it ignores the actual impact of trades on liquidity and price action.
  2. Forecast Brier score: Requires waiting on outcomes to settle, so it's hard to use for long-duration events. It is still as outcome-oriented as trading, and as influenced by the inherent randomness in outcomes that requires a very high number of measurements to be useful at all. Also requires considering how to weigh difficulty and forecast time relative to the event (forecasting the day a question opens is not the same as forecasting once over 90% of its duration has passed and the result may be almost certain).

The main problems here being the time until resolution and the impact of randomness. A model forecasting 70 outcomes that all happen in the span of 1-2 months with a similar Brier score to humans does not mean it is a good forecaster: the human may not even be a good baseline, randomness can have disproportionate impact as well as difficulty.

Much like in a game of poker, models can reason incorrectly and still win because of variance: you can go all-in with a 2 and an 8 preflop, get called by pocket aces, and still win the pot. Doesn't mean, on average and over the long run, that the decision was correct.

Having to wait until resolution or resorting to pastcasting (estimating odds of events in the past) limit the ability to use these benchmarks for real-time evaluation of new models. Meanwhile, noise metrics need no resolution at all.

The core issue with these benchmarks and tournaments is that we can't see the impact of path-dependency (when looking at a model's trading PnL), nor the variation in their forecasts. For example, if the model's only forecast for an event is 8%, the best human forecast is 10%, and the event does not happen, the model is credited with the better score: Brier 0.0064 against 0.01, where lower is better.

But imagine we had forecasted 4 times instead of 1, and the model's forecasts (keeping all inputs equal) were: 8%, 12%, 10%, 15%, the average would come out to 11.25%, worse than humans. Those four numbers sit an average of 2.25% from their own mean, which is close to this benchmark's field-average odds spread of 2.5%.

This benchmark's main limitations are:

  1. Generalization: Only prediction market events are evaluated here, and the noise in estimates for future events may not be representative of logical consistency in other tasks such as law, insurance or medicine.
  2. Dataset size: 100 markets among the most active events on Polymarket, resolving between August 2026 and January 2029 with most of them inside six months, offers a variety of subjects, but it could be larger, especially considering how unique so many of these events are. Increasing the number of times each prompt is repeated would also make the results more robust, at a greater cost. Please keep in mind this project is entirely self-funded.
  3. Comparability: Future versions of this benchmark with newer models will require being entirely rerun, since the events we use are live. That will, however, give us a chance to understand and measure how well the results of some of the current models hold in the future.

How bad is the noise?

For details of each type of noise I recommend browsing the site, but just to get an idea, the best model on the noise index (the unweighted mean of the five metrics) was GPT-5.6 Sol Pro at 4.25%, and the worst was Mistral Large 2512 at 16.19%, close to four times noisier. However even the best model experienced a 7% individual-pair disagreement, with the field average being 16.3%.

That means the average model, across the 20 selected frontier and highly popular models, contradicts its own odds on 16.3% of pairs, roughly 8 of the 50 it was asked. The metric takes the majority of a model's eight head-to-head judgments on a pair and checks which market it favors against the higher of the model's own two averaged estimates.

The model with the lowest odds spread (random variation in direct forecasting) is Claude Opus 5, at only 1%. Yet it still has an individual-pair noise ratio of 16.3%.

A direct application of the benchmark's results are how odds spread and negated spread (how far a market's "Yes" and "No" answers are from summing to 100%) affect any model's direct estimate.

For example, if Kimi K3 forecasts an event as having 50% odds, given it has a 2.8% odds spread, any single forecast request is likely to be anywhere between 47.2% and 52.8%. Odds spread is a mean absolute deviation, so it describes the answers you actually get back rather than a confidence interval on a true value. If we were measuring Kimi's outcomes by looking at its Brier Score, and the event happened (outcome = 1), then the reported Brier score for 50% would be (0.5 - 1) ** 2 = 0.25.

But once we account for that spread, the Brier Score for the same belief ranges between

  • Brier(lower) = (0.472 - 1) ** 2 = 0.279
  • Brier(higher) = (0.528 - 1) ** 2 = 0.223

This difference of 0.056 in Brier Score may seem small, but if we look at the differences in Brier Scores reported by two of the most popular forecasting benchmarks and leaderboards, the picture looks different:

  • BTF-3 (https://evals.futuresearch.ai/) has its top two entries at 0.116 and 0.118 Brier score, a difference of 0.002. First place is FutureSearch's own system rather than a bare model, and second is Claude Opus 5.
  • ForecastBench (https://www.forecastbench.org/leaderboards/) currently has its top two entries 0.4 points apart in Brier Index on the preliminary leaderboard, and 0.1 points apart on the tournament leaderboard, where first place is the human superforecaster median. Brier Index is (1 - sqrt(Brier)) * 100%, so those gaps are in probability units, the same units as odds spread. Note we found a minimum of 1% variation in odds, more than double the larger of the two gaps.

When considering the full range of noise in LLM forecasts instead of just the most favorable metric, even more questions appear:

  • How much would results change simply by changing the question's framing from "Will X happen?" to "Will X not happen?"
  • What can we learn from the discrepancy between ranking pairs of events versus forecasting single outcomes? Is either of them consistently better calibrated?
  • If models disagree with themselves depending on how questions are framed, to what extent do we accidentally bias results when writing the prompts for LLM-based decision systems?

Conclusion

As AI adoption increases and we delegate more and more decisions to it, it's crucial to understand and mitigate noise as much as possible. noisebench is a serious attempt to probe the current state of decision quality for the top LLM models, and already shows LLMs exhibit true inconsistencies in judgment.

If the results of this benchmark generalize to other applications of judgment and estimates, even a few percentage points of noise can make the difference for millions of people when the output of these systems determines who gets a loan, the cost of insurance, or whether the patient's symptoms mean further tests are justified.