Ask a large language model how confident it is and it will happily answer. 0.85. 0.92. Whatever number sounds right for the conversation so far. Plenty of agent designers route work based on those numbers: above 0.8, send it; below, escalate to a human. Amrut Joshi’s recent essay on LLM confidence takes that pattern apart, and the argument is worth internalizing if you build agents, because the number you are trusting was never a measurement of anything.
The number is a token, not a gauge
The core observation is mechanical. An LLM generates text one token at a time, and the string “0.85” is produced the same way as every other token: as a plausible continuation of the conversation. The model never reads a gauge inside itself. It learned what confidence scores look like in text, during training, from millions of documents where people wrote confident-sounding numbers next to confident-sounding claims.
That training makes the model calibrated for exactly one task, predicting the next token of text. It was never scored on whether the confidence it writes matches how often it is actually correct. Those are different objectives, and nothing in pretraining connects them.
The observable symptoms follow directly, and you can check them yourself. Ask the same question twice and the number moves, even though the information did not. Change the wording and the number moves again. Ask “are you sure?” and watch it wobble. If the model were applying a rule to the evidence, identical evidence would produce identical output. It does not, because there is no rule and no measurement, only continuation.
What a real confidence score requires
The essay’s framing device is weather forecasting, which is a fair comparison because forecasters genuinely are calibrated. When a forecaster says 70% chance of rain, you can check that claim: collect every day they said 70%, count how often it rained, and if it rained on about 70% of those days, the number means something. Calibration over many trials, not confidence in any single one.
Calibration alone is not enough, though. A forecaster who always says 30% in a season where it rains 30% of the time is perfectly calibrated and completely useless, because the number carries no information. A useful confidence score needs two properties: it reflects how much information the prediction is based on, and it has been checked against outcomes on the same kind of question. Both properties require a written record of predictions and results, which is the part everyone skips.
One thing this does not cover
None of this says the model has no internal uncertainty at all. The probability distribution over the next token is real and computable, and researchers use it. But a token-level logprob measures how likely the model thinks that particular word is in that particular context, which is a different quantity from how likely the answer is to be correct. A model can be extremely sure about which word to write next and be wrong about the underlying question. The logprob is also awkward to consume in practice, since it depends on decoding settings and says nothing about the many tokens that could have followed. The essay is right to leave it out of the recipe: if the observable you log is not connected to correctness by measurement, it does not belong in the rule.
Where confidence should come from instead
The constructive half of the essay is a five-step recipe for building confidence scores that actually mean something, with the LLM doing the task but not grading itself:
- Pick observables you can record without asking the model how it feels: did search find matching articles, how many, has the agent answered this type before, do two independent answers agree.
- Start from the base rate. If human review has approved 150 of 200 past replies, every reply starts at 75%.
- Record every reply with its observables and its eventual outcome.
- Group past replies by their observables and set the confidence for each group to its observed accuracy.
- Re-check calibration on a schedule, and fix the rule when a confidence level stops matching reality.
None of this is exotic. It is the same discipline any ML team applies to a classifier’s probability outputs, applied to the LLM as a component instead of an oracle. The rule-based score has the properties the model’s number lacks: same inputs give the same score, wording changes do not move it, and it is calibrated on your data, for your task, and improves as records accumulate.
Why this matters for agent routing
The pattern this kills is LLM-as-router with self-reported confidence. It also explains a lot of failures in the wild. Agents that confidently ship wrong answers and agents that escalate everything are both running on numbers that measure plausibility of phrasing rather than probability of correctness. If your escalation threshold sits on top of a number that shifts with rewording, your routing is theater.
The cheap version to build this week: log your agent’s task, the observables you can compute, and the ground-truth outcome whenever one arrives. Even a few hundred records turn the base rate into a real score, and the group-by-observables step can wait until the log exists. The essay’s closing point is the one to remember: a rule is only as good as its record, and if nobody checks outcomes, you are back to a number nobody has measured.