Orbrya
All posts
Orbrya2026-03-12

A Lawyer Lost $5,000 Trusting ChatGPT: What Real AI Failures Teach Our Kids

Three lawyers were sanctioned for submitting AI-fabricated case citations they never verified. What those cases reveal about how AI fails — and what kids need to learn before they're the ones who don't check.

A Lawyer Lost $5,000 Trusting ChatGPT: What Real AI Failures Teach Our Kids

A few years ago, a federal court brief arrived containing six case citations that looked completely normal. Each one had a proper name, a year, a court, a docket number. Opposing counsel tried to find them. None of them existed. The attorney who filed the brief, Steven Schwartz, had used ChatGPT to research the case and had submitted its output without checking whether the citations were real.

When opposing counsel challenged the citations, Schwartz did something that became the detail everyone remembers: he asked ChatGPT whether the cases were real. ChatGPT said yes. He included that confirmation in his response to the court.

The judge was not persuaded. Schwartz, his colleague, and their firm were jointly fined $5,000 — a single penalty shared among all three respondents, imposed jointly and severally. They were required to issue written apologies to the judges whose names had been falsely attached to the fabricated opinions. Schwartz had practiced law for more than 30 years.

This post is not about lawyers making mistakes. It's about what those mistakes reveal, and why the lesson transfers directly to what our kids are navigating right now — without the professional experience or the stakes that eventually forced adults like Schwartz to reckon with the limits of what AI actually does.

What AI Actually Did Here

It's worth slowing down on exactly what happened in the Schwartz case, because the failure is more specific than "the AI made things up."

ChatGPT didn't randomly hallucinate nonsense. It produced citations that were structured correctly, formatted correctly, and plausible enough that an experienced attorney didn't pause. The case names were realistic. The courts were real. The years were in the right range. Everything about the output signaled that verification had already happened — that these were real things retrieved from a real database — when in fact the model was generating text that fit the pattern of what a real citation looks like.

That distinction matters. The failure mode isn't noise. It's confident, well-formatted, plausible output on a topic where the AI has no actual access to ground truth. ChatGPT doesn't have a database of court records. It has patterns from training data, and it produces text that fits those patterns. When asked for case citations, it generates things that look like case citations. The looking like is seamless. The being isn't guaranteed.

Schwartz's second mistake — asking ChatGPT whether the cases were real — illustrates another layer of this. The model confirmed its own fabrications because it has no mechanism to distinguish between what it generated and what is true. It isn't lying. It doesn't know the difference. That's what makes it genuinely dangerous in contexts where accuracy matters, and why verification can't be delegated back to the source being verified.

The Pattern Held Across Multiple Cases

The Schwartz case wasn't isolated. It was the first one that became widely known, but the pattern repeated.

A California attorney named Amir Mostafavi submitted an appeal brief where 21 of 23 quotes attributed to cited cases were fabricated by AI. He received a $10,000 fine — the largest issued over AI fabrications by a California court at the time. The court's ruling stated directly that no brief filed in any court should contain citations that the responsible attorney has not personally read and verified.

Morgan and Morgan, one of the largest personal injury firms in the United States, was sanctioned after lawyers cited eight nonexistent AI-generated cases in a filing. In their response to the court, a firm spokesperson wrote: "This matter comes with great embarrassment and has prompted discussion about the training, implementation, and future use of artificial intelligence within our firm. This serves as a cautionary tale."

Three separate firms. Three separate cases. The same failure mode each time: an AI output that looked like verified information, accepted without the verification step that would have caught the error in minutes.

The question worth sitting with is not why these attorneys trusted ChatGPT. They trusted it for the same reason anyone trusts a confident, well-formatted answer: it didn't look like it needed checking. The question worth sitting with is what it takes to develop the habit of checking anyway — and whether we're teaching that habit to the people who will need it most.

The Jagged Frontier Problem

Harvard Business School researcher Ethan Mollick and colleagues ran a field experiment with 758 consultants at Boston Consulting Group, testing what happened when professionals used GPT-4 on real work tasks. The results were striking in both directions: on tasks that fell within the AI's capability range, the consultants using AI finished significantly more work faster and at higher quality. On tasks that fell outside that range, the AI users were about 19 percentage points less likely to produce correct solutions than those working without AI.

Mollick described this as the "jagged frontier" — a boundary between what AI does reliably and what it doesn't, but a boundary with no visible edge. From the outside, AI outputs look the same on both sides of it. The confident tone, the clean formatting, the plausible structure — none of that changes depending on whether the underlying content is accurate. The output that contains six real case citations and the output that contains six fabricated ones are indistinguishable unless someone checks.

The legal cases are examples of professionals working on the wrong side of that frontier without knowing it. Citation verification is exactly the kind of task where AI has no reliable access to ground truth, where its output is generated from pattern rather than retrieval, and where the cost of being wrong is measurable. The attorneys didn't know they were on the wrong side of the frontier. Nothing about the output told them.

This is the environment our kids are operating in every day, on assignments where the cost of being wrong is usually lower but the habit being formed is the same. A student who submits AI-generated claims without checking is practicing the same cognitive move Schwartz made. The fine isn't $5,000. But the habit is identical, and habits formed at twelve are harder to break at thirty.

What the Cases Teach That Classrooms Don't

These legal cases are useful for families because they make abstract concepts concrete in a way that statistics can't.

When we talk about AI hallucinations, we're describing a real phenomenon with a technical name that doesn't quite communicate the stakes. When we describe a 30-year attorney asking ChatGPT whether a case is real and getting a confident yes — and then submitting that confirmation to a federal judge — the phenomenon becomes visceral. The model doesn't know what it doesn't know. It can't flag its own errors. Asking it to verify itself is asking the source to audit the source.

When we talk about citation fabrication with kids who are writing history papers or science reports, the same logic applies at a smaller scale. The AI generates citations that look real. It formats them correctly. It will confirm them if asked. The only check that actually works is the one where you look the source up yourself — find the author's institution, confirm the publication, read the relevant section.

Most kids have never been told this explicitly. They've been told to cite their sources. They haven't been told that the source the AI gave them might not exist, and that the only way to know is to check before you include it.

The Empowerment Flip

The legal cases are cautionary, but they point directly toward a skill that is learnable, practical, and genuinely valuable.

Every one of the failures above would have been caught by a single step: looking up the citation before including it. That's not a complex skill. It's not technical. It doesn't require understanding how large language models work. It requires the habit of treating AI output as a claim rather than a fact — as something that might be right, rather than something that has been verified.

That habit, practiced consistently, is exactly what separates the professionals who use AI well from the ones who get sanctioned. And it's learnable at any age. A ten-year-old who does the Citation Hunt activity from our post on age-specific verification games is practicing the same cognitive move a junior associate at a law firm needs to make before submitting a brief. The scale is different. The underlying habit is not.

The families who build this habit at home now are giving their kids a specific and durable advantage. Not because the legal stakes will come up in middle school — they won't. But because the verification reflex, once formed, activates automatically across contexts. The twelve-year-old who learned to check citations for a history report is the twenty-five-year-old who checks before filing.

A Note on What's Changing

The legal profession has responded to these cases with binding guidance. Courts have issued standing orders requiring attorneys to certify that AI-generated citations have been verified. Bar associations have published ethics opinions on disclosure. The professional frameworks for responsible AI use in law are being built in real time, precisely because the failure mode became undeniable.

Schools are moving more slowly. As we covered in our post on the school policy gap, 55% of schools have no formal AI policy at all, and the guidance that does exist is almost entirely focused on detection and prohibition rather than on teaching the evaluation skills that make safe use possible.

That gap is a family responsibility right now. The professional world is building the guardrails. Schools are beginning to. In the meantime, the families teaching verification habits at home — through conversation, through games, through the consistent question of "how do you know?" — are building something the institutions haven't caught up to yet.

Next week we go deeper into the cognitive science behind why this skill is worth teaching systematically, including the three-part mental process researchers have identified as the foundation of everything we've been building toward in this series. It connects directly to why some students stay ahead of AI and others fall further behind.

If you'd like to be notified when that post goes live, and when we open access to the full curriculum launching August 2026, join the waitlist.

Orbrya builds AI literacy curriculum for K-12 families — homeschool and supplemental — focused on the verification and evaluation skills that separate effective AI use from dependent AI use. Curriculum launches August 2026.

Sources cited in this post: