All posts

There's a particular kind of error that AI makes more frequently than any other, and it's the one that's hardest to catch precisely because it doesn't look like an error. The answer is plausible. It flows well. It's consistent with what the reader already knows, more or less. There's nothing obviously wrong with it. It's just not right.

Researchers sometimes call this the plausibility problem: AI systems are optimized to produce text that a human would find reasonable, which is not the same as text that is accurate. A language model doesn't know things in the way humans know things. It produces outputs that pattern-match to what correct text looks like. Very often, the pattern-matching produces something true. Sometimes it doesn't. And the two outputs look exactly alike from the outside.

The distinction between "good enough" and "actually right" is one of the most important things families can teach children in the AI age, and it requires a different kind of thinking than most educational contexts ask for.

Why schools accidentally train the wrong reflex

Most academic contexts reward recognizable correctness. An essay that deploys the right vocabulary, follows the expected structure, and hits the familiar argument beats will often receive a good grade even if its specific claims are imprecise. A student learns, over years of schooling, that getting the form right is most of the battle.

AI is very good at form. It has been trained on enormous quantities of text that was considered good enough -- well-structured, fluent, hitting the recognizable beats. It has not been trained to be accurate in the sense of verifiably corresponding to facts. The optimization targets are different.

A student who has learned to equate good form with sufficient quality is poorly prepared to evaluate AI output, because they're applying the right standard for evaluating their own work to a tool that was trained to produce good form regardless of accuracy. The question "does this sound right?" is not the question "is this right?"

What the distinction looks like in practice

Consider a student asking AI about the causes of the 2008 financial crisis. AI will produce a coherent, well-structured explanation that references recognizable factors -- mortgage-backed securities, regulatory failures, the role of credit rating agencies. It will sound authoritative. It will match what a student remembers from class.

Is it right? Mostly, probably. Is it accurate in its specific claims, proportionate in its attribution of causes, reflective of the genuine disagreements among economists about the relative importance of different factors? That requires actually checking -- and the student who assumed that "sounds right to me" meant "is right" won't do that check.

The more consequential the claim, the more important the distinction. A historical detail that's slightly off in a middle school essay is a small problem. A medical claim that's plausible-but-wrong, a legal interpretation that pattern-matches to correct but isn't, a financial calculation that produces a reasonable-looking number through incorrect reasoning -- these are different-scale versions of the same failure mode.

Teaching the standard

The practical habit to build is asking two questions, not one. The first is the natural one: does this seem right? The second is the harder one: have I confirmed that it is?

"Seems right" is valuable -- it surfaces obvious errors, it uses prior knowledge efficiently, it's fast. But it's not verification. Verification requires finding the claim in a source that isn't AI, tracing it to its origin, and confirming that the original source actually says what AI said it says.

For children, the best exercises are ones that reveal the gap between plausible and accurate in a memorable way. Ask AI something you already know the precise answer to, and check whether AI gets the specific detail right or only approximately right. Often you'll find that AI gets the gist correct and a specific detail wrong -- or correct but imprecise in a way that matters.

Once a child has experienced the gap between "seemed right" and "wasn't quite right" a few times, the instinct to check specific claims rather than trust general impression starts to develop naturally.

The professional stakes

In professional life, the plausibility problem is where AI failures tend to cause real damage. A lawyer's brief that cites plausible-sounding but fabricated precedents looks, to a reader not checking closely, like solid legal work. A market analysis that uses AI-generated statistics sounds authoritative until someone traces the numbers to their supposed sources. A medical summary that pattern-matches to good medicine but gets a dosage or contraindication wrong is in a completely different risk category than a bad essay grade.

A student who has spent years developing the instinct that good form is not the same as accurate content will carry that standard into professional life, where it protects them and the people their work affects. A student who hasn't developed that instinct is going to encounter the gap eventually -- and in a professional context, "I thought it seemed right" is not a mitigation.