Trust in legal AI won’t come from vendors’ promises. It comes from verification. We verify — and you can, too.
Every lawyer has heard the story: an attorney files a brief, the citations look impeccable, then opposing counsel or the judge discovers the cases don’t exist. Mata v. Avianca made it infamous in 2023, but it was never going to be a one-off. Damien Charlotin’s AI Hallucination Cases Database, the most comprehensive tracker of court decisions involving AI hallucinations updated daily, now documents over 1,600 cases worldwide [Editor’s note: As of 3:37pm EST today, that number was 2006]. Sanctions, fee awards, referrals to disciplinary bodies. This is no longer an anecdote. It’s a body of case law about case law that was never real.
Let’s be clear, a hallucinated citation isn’t a typo. It’s a fake authority presented with confidence in a profession where authority is everything. It comes in two flavors, one far sneakier than the other:
- Fabrication – where the case doesn’t exist at all. Embarrassing, but increasingly catchable: the citation simply doesn’t resolve.
- Misattribution – the case is real, the citation is accurate, but the opinion doesn’t say what the model claims it says. This is the dangerous one, because it survives at a glance. The citation checks out; the proposition doesn’t.
My team’s thesis is simple: if you use legal AI, you must measure how — and how much — these models hallucinate. For us, that isn’t a research curiosity; it’s core due diligence. So we built our own benchmark and run it continuously. We think every firm deploying legal AI should do the same. Here’s how.
Why not just trust the public benchmarks?
Three reasons.
- Benchmarks leak. Once published, a benchmark tends to enter training data. A model can score well on a test it has memorized while performing differently on your actual work.
- Vendor benchmarks measure what vendors want. A generic accuracy score tells you nothing about citation behavior in your domain or on your types of questions.
- Benchmarks go stale. New models ship monthly. A benchmark paper from eighteen months ago describes models no one uses anymore.
The conclusion we reached is simple: if accuracy matters to your business, you have to measure it yourself and keep measuring it.
Why not just have lawyers check the outputs?
The obvious alternative is human review. To hand AI outputs to associates and have them verify everything.
We considered this, and the math doesn’t work. Lawyer hours are expensive. The review process is slow. By the time a careful human evaluation finishes, three new models have shipped. Samples stay small. You cannot get statistical confidence from thirty hand-checked answers.
Reviewers are inconsistent. Different people apply different standards to the question “Does this case support that proposition?”
To evaluate models properly, you need to do it at scale, checking hundreds to thousands of verified citations at speed. Then be able to re-run that same evaluation in a few days when a new model drops. Humans just can’t do that. Fortunately, in law, you don’t need them to. The law gives you a source of truth.
Law’s unfair advantage over other professions: a source of truth
The fact that case law is a closed, citable, and queryable corpus makes legal AI uniquely benchmarkable.
That is why law can support a benchmark in a way most domains cannot.
A citation either resolves to a real case or it doesn’t. The opinion either supports the stated proposition or it doesn’t. The case is either still good law or it’s been overruled. These are checkable facts.
Thanks to the Free Law Project and its CourtListener platform, which provides free, open access to a comprehensive US case law database, anyone can check them. (Genuine credit where it’s due: FLP’s infrastructure makes this verification possible without a seven-figure data license.)
Every citation can be put through three questions, in escalating depth:
- Does the case exist? This catches fabrication.
- Does it say what the model claims it says? This catches misattribution, and it requires actually reading the opinion against the proposition, not just confirming whether the citation resolves.
- Is it still good law? This catches stale or overruled authority, which is a different failure mode from hallucination, but just as dangerous in a brief.
How we structure our benchmark
You don’t need our exact pipeline to do this. The shape of it is what matters, and the shape is reproducible by any firm with modest engineering resources.
Start from a published question set. We built on Stanford RegLab’s “Hallucination-Free?” research question set, which includes roughly a hundred legal research questions spanning constitutional law, civil procedure, bankruptcy, circuit splits, and more. Using academic questions adds credibility and makes your results comparable to published work.
The mix matters: doctrinal questions, bar-exam-style hypotheticals, precise metadata recall (who wrote this opinion? what year?), and crucially, adversarial questions asking for cases supporting false or absurd propositions. A trustworthy model’s correct answer to “find me a case holding X” when no such case exists is “no such case exists.”
Models that always try to be helpful fail this test in revealing ways.
Run it in two phases.
Phase 1 — Ask. Pose every question to every model under identical conditions and capture structured answers: the answer itself plus every citation with its case name, reporter cite, and the specific proposition it supports.
Phase 2 — Verify. Independently verify every citation. We use an automated verifier — an AI agent with tools to search CourtListener, retrieve the full opinion, and read it against the claimed proposition. Every citation gets a status.
Don’t just count “wrong”, classify.
Our taxonomy distinguishes verified (good law supporting the proposition), partial support, real-case-wrong-proposition (misattribution), case-doesn’t-exist (fabrication), and overruled/questionable (stale law). The distribution across these categories tells you more than a single error rate. Two models with identical headline accuracy can have completely different failure profiles. Which failure you can tolerate depends on your workflow.
Be statistically honest. Put confidence intervals on every rate; small samples lie. Run significance tests before claiming one model beats another. Compare models on identical questions. This does not require a statistician. It requires deciding in advance not to publish (even internally) a ranking your data can’t support.
Then slice — this is where it gets interesting. A single hallucination number hides the findings that actually change deployment decisions:
- By question type. Which legal tasks are hardest? Doctrinal recall, circuit splits, and metadata questions stress models in very different ways.
- By the age of the cited case. Training data has two dangerous edges. Very old cases are sparsely represented, so models tend to cite them misattributively—real case, wrong proposition. Cases decided after a model’s training cutoff don’t exist for the model, so citations to them are fabricated at several times the rate of older cases. Old law and new law fail in different ways, and a reviewer should know which to expect.
- By model size and price. Within a single provider’s lineup, do you get what you pay for? The price-performance curve is exactly what a procurement decision hinges on, and it’s measurable.
- Citation volume vs. accuracy. Some models answer with one careful citation; others volunteer five. Some cite four times as many cases per question as others at similar accuracy. Volume and accuracy are separate axes. Knowing where a model sits changes how you deploy it.
- Consistency. Ask the same model the same question twice: does it cite the same cases? This dimension is rarely measured and surprised us. A model can be accurate but inconsistent—citing a valid authority each run but a different one every time. Reproducibility matters in legal research. Two associates asking the same question shouldn’t get materially different foundations for the same argument. (Practical tip: comparing citations across runs requires fuzzy matching. The same case is phrased differently, so we use an LLM to pair them.)
In total, our most recent run verified roughly 1,800 individual case citations across nine frontier models.
That’s the scale at which patterns become visible, and confidence intervals become tight enough to act on.
Having this level of knowledge means we are not beholden to any model or any vendor. Because we evaluate continuously, we route each kind of work to the model that does it best — and if a better tool ships tomorrow, we’ll know within days and use it.
What the benchmark taught my team is that “which model is best?” is the wrong question. Models differ substantially.
The hallucination rates between the best and worst models we tested differ by an order of magnitude, but more importantly, they differ in style.
Some cite cautiously and rarely; others cite broadly. Some fail by inventing; others by misattributing. Some are accurate but irreproducible. Matching model behavior to use case is real engineering, and you cannot do it without measurement.
Why does this matter and what should you do?
Users shouldn’t have to take the claim “our AI is accurate” on faith, and neither should you. The responsible posture for anyone building AI on top of LLMs for legal work is to measure continuously with a methodology you can defend. That is the point. Not once, not at launch, continuously, because the model landscape changes monthly and yesterday’s evaluation describes yesterday’s models.
The legal industry is uniquely positioned to demand this. We have what almost no other domain has: a source of truth. The question sets are published. The case-law database is free and open. Even a lightweight version of this benchmark — fifty questions, manual spot-verification against CourtListener — will tell you more about a tool than any vendor deck.
So if you’re a firm evaluating AI tools, here is our suggestion: stop asking vendors for accuracy claims and start asking for citation-level verification statistics. Ask how the numbers were produced, on what questions, and verified against what. And if the answers are vague, run the test yourself. It’s more achievable than you think.
Trust in legal AI won’t come from vendors’ promises. It comes from verification. We verify — and you can, too.
Join the conversation!