Agile4AIBlog

Why AI can't say 'I don't know'

The model knows the answer is uncertain. What it doesn't have is a good way to tell you. Understanding why changes how you read everything it says.

Ask someone a question they don’t know the answer to and, if they’re being honest, they’ll tell you. They might say “I’m not sure,” “I’d have to look that up,” or “I don’t know enough about that to say.” The uncertainty is expressed. You know how much weight to give the answer.

Ask an AI the same question and you’ll typically get an answer. It will be fluent, structured, and confident in tone. It may be accurate. It may be partially accurate. It may be entirely wrong. The response itself usually gives you very little signal about which of the three it is.

This isn’t a character flaw or a random glitch. It’s a trained behavior.

How AI models are trained

Language models learn by generating text and being evaluated on whether that text is useful and well-received. The training process — broadly described as Reinforcement Learning from Human Feedback (RLHF) — involves human raters evaluating model outputs, and the model learning to produce responses that score well.

Responses that clearly answer the question tend to score better than responses that hedge, express uncertainty, or decline to engage. “I’m not sure, but here’s what I know” is a less satisfying response than a clear, confident answer. Over millions of evaluations, models learn to produce confident answers.

Add to this that AI vendors are building products. A product that frequently says “I don’t know” is less commercially valuable than one that provides answers. The incentives around model training — both the training process itself and the deployment decisions that follow — generally push toward confidence over expressed uncertainty.

The result is a model that has genuine internal uncertainty about many things, and a strong learned tendency to not express that uncertainty. The uncertainty doesn’t disappear; it just doesn’t show up in the response you get.

There is a blunter way to say this. We are training these models the way a manager gets an answer out of a frightened employee: one chance, deliver something, and sound sure about it, or you’re fired. Nobody standing in front of that question says “I don’t know.” The picture is harsh, and it is close to what the AI training reward signal actually selects for.

This is also one significant source of what we have broadly labeled hallucinations — a label that faults the model instead of asking what conditions produced the response. A model that has learned it must always produce a confident answer will produce one even when it has nothing solid to build on. That is what many so-called hallucinations actually are: the predictable output of the conditions we set, not a glitch in the machine.

What that looks like in practice

You ask about something at the edge of the model’s training. It answers with apparent authority. The answer is plausible and internally consistent. You have no obvious signal that this is a case where the model is essentially guessing.

This is more dangerous than obvious errors. When an AI produces something clearly wrong — a factual error you immediately recognize, an implausible claim — you catch it. The confidence-without-uncertainty problem surfaces precisely in the cases where you can’t easily catch it: obscure facts, specialized domains, recent events, edge cases where the model has seen just enough to generate a plausible-sounding answer without having reliable grounding.

A related dynamic appears in how AI responds to leading questions. Just as leading questions trip up humans, they also shape an AI’s response. If you frame a question in a way that implies a particular answer — “why is X the best approach?” — the question isn’t asking the model if X is, indeed, the best approach; it is asking the model to generate reasoning in support of X. The model complies not because it is certain X is the best approach, but because the framing implied that answer, and generating supportive reasoning is more responsive to the framing than pushing back on it.

This is the mechanism behind how you ask changes what you get, which is worth reading alongside this post.

Calibration: what to do with this

None of this means AI outputs are useless. It means they’re uncalibrated — the expressed confidence level doesn’t map consistently to the actual accuracy level. A human expert who says “I think” and “I’m certain” is giving you useful signal about how much weight to put on each statement. An AI that consistently sounds certain is not giving you that signal.

The practical response is to supply your own calibration.

Ask explicitly: How confident are you in this? What would I need to verify independently? Many models, when directly prompted to express uncertainty, will do so. The expressed uncertainty under direct prompting is more reliable than the confidence embedded in the default answer.

Weight outputs differently by domain. In well-covered, stable domains — things that have been extensively documented and don’t change quickly — AI accuracy tends to be high. In specialized, niche, or rapidly evolving domains, default to treating AI output as a first draft that needs verification.

Treat fluency as neutral evidence. A response that is articulate and well-organized tells you the model is good at generating articulate, well-organized text. It doesn’t tell you the content is accurate. Separating those two signals — how well-written versus how accurate — takes deliberate effort, and it’s worth the effort.

The deeper dynamic

There’s a version of this that applies to more than AI. Any system — human or machine — that is rewarded primarily for producing answers will express less uncertainty than it actually has. Organizations that punish people for saying “I don’t know” get cultures where nobody says “I don’t know.” The problem is the same one; only the mechanism differs.

That puts the responsibility somewhere specific. The conditions that produce overconfident answers are conditions that humans have created — in how these models are trained, in what we reward, in how we ask. None of it is fixed, and the model didn’t choose these incentives.

Treat overconfident AI as something broken in the technology and we will keep building the same thing and keep being surprised by it. Treat it as a reflection of the conditions we set and it becomes something we can change. Most of the friction people complain about is of our own making, and so are the harder problems of alignment waiting behind it.

AI Psychological Safety is partly about creating conditions where the AI’s expressed uncertainty is invited, useful and visible. When you ask directly, when you build in space for uncertainty, when you actively probe the edges of what the model is confident about — you get output that shows you its uncertainty instead of hiding it.

The model can’t reliably volunteer its uncertainty. You can reliably ask for it.

Related content

Comments 0

Sort
Checking your sessionThe article is ready. Comment options will appear once your session check finishes.