The Epidemic of AI Hallucinations: Why "Trust, But Verify" is a Non-Delegable Duty

The cautionary tales started in 2023 and have not slowed down. Mata v. Avianca, Inc.¹ drew the first wave of attention when a federal judge in the Southern District of New York sanctioned two attorneys $5,000 for filing a brief built around case citations that ChatGPT had invented. The cases looked real. They were not.

By early 2026, researchers have catalogued hundreds of decisions worldwide in which courts found that a party submitted fabricated, AI-generated content.^{2, 3} The problem is not limited to attorneys using consumer chatbots. Stanford's empirical work on legal-specific AI tools, including closed-system products marketed to law firms, found that even those tools hallucinate at rates that should not be acceptable in a profession that depends on the accuracy of its citations.^{4, 5} Performance degrades sharply as legal questions get more complex, and the failure mode is often the most dangerous one: the model generates a citation that sounds authoritative, formatted correctly, attributed to a real reporter, and entirely made up.

Why LLMs Do This

Large language models do not reason in any sense a lawyer would recognize. They predict the next token in a sequence based on statistical patterns from their training data. When the right answer exists in their training and the question is framed unambiguously, the prediction can be quite accurate. When neither condition is met, the model produces something that pattern-matches what a real answer would look like. That is the hallucination. LLMs are also prone to sycophancy: they tend to confirm the user's premise even when the premise is wrong, generating supporting authority that does not exist.

The Courtroom Consequences

The Louisiana Fifth Circuit case of In Re: Sanctions Order of Kenney^{6, 7} is a useful local example. An attorney relied on AI-generated summaries provided by a paralegal and filed a memorandum containing fabricated cases. When opposing counsel flagged the problem, the attorney filed an errata that compounded the original error with additional false quotations. The court sanctioned the attorney, ordered her to pay opposing counsel's fees for the 7.6 hours spent confirming the citations were fake, and required AI ethics training.

In the Second Circuit, Park v. Kim⁸ went further. The court ordered the sanctioned attorney to provide a translated copy of the sanctions order to her own client, ensuring the client understood what had happened.

Under the Rules of Professional Conduct, the attorney is the ultimately responsible party for any work product, regardless of which tool produced the underlying text. Verification of AI-generated content is a duty under the rules, not a best practice.

Treat AI Output Like a First Draft from an Intern

The practical model that works inside law firms is straightforward. AI output gets treated the way an experienced partner treats a first draft from a bright but unreliable summer associate: useful as a starting point, never trusted on its face, every factual or legal assertion checked against primary sources before anything goes out the door. Firms that build that verification step into their workflow as a mandatory checkpoint avoid showing up in the next iteration of the hallucination case database.