The fix isn’t to swear off AI. It’s to stop using tools that guess when the job requires proof.
Judges have already caught legal teams citing cases that don’t exist. A brief goes in with a string of citations, the citations look right, the formatting is right and then someone tries to pull the actual opinion and finds nothing. The case was invented by a chatbot that had no idea it was making it up. These can be costly oversights.
That’s the part people miss. The AI wasn’t lying. It doesn’t know the difference between a real citation and a plausible-sounding one, because it was never working from a particular case law in the first place. It was trained on a slice of the internet and when it doesn’t have the answer, it produces something that reads like an answer instead of admitting it has nothing. These are often referred to as “hallucinations”, a cute anthropomorphism which tries to excuse the program’s bad behavior.
For most uses, that’s an annoyance. In litigation, it’s a career problem and it’s an ethics problem for the client sitting behind it.
The tool isn’t the issue. What it’s reading is.
The instinct after a few embarrassing headlines is to swear off AI entirely or to lock it down with a policy memo nobody reads twice. The impulse is understandable, but it’s aimed at the wrong target. The problem isn’t that a firm used AI. The problem is that the AI was answering questions about the law from general training data instead of from the record in front of it.
There’s a meaningful difference between an AI system that searches the open internet and guesses, and one that only answers from the documents you actually gave it: the filings, the discovery, the statutes and precedent you’ve pulled for the matter. Ask the second kind a question it can’t answer from that material, and it should say so. It shouldn’t reach for something that sounds confident instead.
That’s not a minor technical distinction. It’s the whole ballgame for legal work, where the cost of a wrong answer isn’t a bad recommendation; it’s a sanctions hearing.
Privilege doesn’t survive a trip to the cloud
Set hallucinations aside for a second. Here’s a second problem sitting right behind it: client data. Firms handle privileged material, sealed documents, or simply information that’s none of anyone else’s business. General-purpose AI chat programs (or LLMs, Large Language Models such as ChatGPT, Claude, and Gemini) were built to send whatever is given to them off to a server somewhere for processing. Without your consent, your confidential material might be used for training future versions of the program or, even worse, plagiarism.
Ask a partner whether they’d paste a client’s settlement strategy into a tool with unclear data retention practices and most will say no, correctly. But that same partner may have an associate doing exactly that at 11 p.m. before a filing deadline, because nobody gave them a private way to get a similar outcome in the timeframe that they need. Policy without an alternative just pushes the behavior underground.
What firms should actually be asking for
When evaluating AI tools for legal work, the questions are pretty simple, even if the vendors make them sound complicated.
- Does it answer only from the documents you gave it, or from the open web?
- When it doesn’t know, does it say so, or does it guess?
- Can you trace every answer back to a specific document, page or line?
- Where does your data go and who else can see it?
If a tool cannot show its work, it does not belong anywhere near a filing. If there’s doubt about the privacy of the data, then you must walk away.
None of this means AI has no place in a law firm. Quite the opposite. Reviewing thousands of pages of discovery, tracking down every prior filing that touches a given clause, summarizing depositions, that’s exactly the kind of work AI is good at and exactly the kind of work associates have too little time for. The technology is genuinely useful here. It just has to be built to work from your record, cite its sources, and keep the data where it belongs.
The lawyers who got sanctioned were not all trying to cut corners. Many were busy, under pressure and relying on tools that were never built to be trustworthy in this context. The fix isn’t to swear off AI. It’s to stop using tools that guess when the job requires proof.
Join the conversation!