A junior solicitor at one of the country's best-known law firms recently asked an AI to help with a point of insolvency law. The AI gave a clear, confident answer. It quoted the exact wording of a rule, set out neatly as if lifted straight from the statute book. The lawyer put that wording in front of a High Court judge.
The rule did not exist. The AI had invented it.
That is the heart of Cork v Smith, a judgment handed down in May 2026 by ICC Judge Mullen. It is worth every business owner's attention, because the mistakes in it have nothing to do with insolvency law and everything to do with how people are using AI right now, today, in their own work.
Here is what happened. The application itself was routine, the kind of administrative matter that usually goes through on the papers without a hearing. The judge had one query about whether the court actually had the power to grant a particular order. The firm wrote back citing a specific rule, complete with a quotation, presented in italics, introduced with the words "provides as follows". It looked authoritative. It was fiction.
When the judge checked the rule against the official source and found the quoted words appeared nowhere in the legislation, he asked the firm to explain. Instead of simply saying "we got this wrong, an AI made it up and we didn't check it", the firm sent a second letter. That letter tried to reframe the invented quotation as a kind of summary that was never meant to be taken as a direct quote. The judge did not buy it. As he put it, an opportunity to set the record straight became a further instance of misleading information being put before the court.
The detail that should stop you in your tracks is this. The AI itself repeatedly told the lawyer to check. The transcripts, which ran to 59 pages, show the AI warning, more than once, that it could not verify the wording and that it should be confirmed against the official legislation before being relied upon. The lawyer pressed on regardless. The judge described it as having "almost entirely outsourced the thinking process" to the machine.
The firm was Pinsent Masons. It referred itself to the Solicitors Regulation Authority and agreed to cover its former client's costs. The judgment serves as a public admonishment of the people involved. Careers were not ended, but reputations took a knock that will follow them around for a long time.
Why this matters far beyond the law.
It would be easy to read this as a lawyers' problem. It isn't. The judge leaned on an earlier case, Ayinde v Haringey, in which the President of the King's Bench Division, Dame Victoria Sharp, set out the principle plainly. Generative AI tools can produce responses that are coherent, plausible and completely wrong. They can make confident assertions that are simply untrue. They can cite sources that do not exist. They can quote passages from a genuine source that do not actually appear in it. Anyone relying on that output has a duty to check it against an authoritative source before using it.
Swap "the court" for "your client", "your regulator", "your lender", "your board", or "your accountant", and the lesson lands in exactly the same place. In our world, an invented rule could just as easily be a misquoted piece of Consumer Duty guidance, a made-up lender criterion, a statistic in a client report that sounds right but came from nowhere, or a confident summary of an FCA requirement that the FCA never wrote.
The uncomfortable truth is that AI does not sound unsure when it is wrong. That is the trap. When a colleague is guessing, you can usually hear it in their voice. AI delivers the fabrication and the fact in precisely the same tone, with the same fluency and the same air of authority. The more polished the output, the easier it is to wave through.
None of this is an argument against using AI. I use it every day and it has genuinely changed how quickly I can get from a blank page to a useful draft. It is an argument against trusting it blindly. AI is a brilliant place to start and a terrible place to stop. It is a research assistant, not the author, and the responsibility for what goes out under your name does not transfer to it.
A practical safeguard you can use today.
After reading the judgment I put together a short set of instructions designed to push back against exactly this failure mode. You paste them once into the custom instructions, or system prompt, of whichever AI tool you use, ChatGPT, Claude, Gemini, Copilot, and they shape how it responds to you from then on. They will not make AI perfect. Nothing will. But they tilt it towards honesty about what it knows, and they make it far less likely that a confident invention slips through unnoticed.
Here they are. Copy everything in the block below.
VERIFICATION RULES
You can be confidently wrong. Fluent, plausible output is not proof that it's accurate. Treat these as hard rules, not suggestions:
1. Separate fact from generation. Anything checkable — a quote, statistic, date, name, price, citation, law, rule, URL, file path — must not be presented as exact unless you've verified it in this task. If you haven't, say so.
2. Never fabricate sources or quotes. Don't invent citations, references, URLs, case names, rule numbers, studies or quotes. Don't rebuild the wording of a law, contract or quote from memory and pass it off as exact — paraphrase and label it, or fetch the real text.
3. Verify before you assert. For any checkable claim that matters, go to the authoritative primary source — not your own recall, and not a summary of it.
4. Show your confidence and your sources. Tag load-bearing claims as VERIFIED (with the source), UNVERIFIED, or INFERENCE. When unsure, say "I'm not sure" rather than guessing fluently.
5. Disclose AI involvement. If the output is going to someone else, make clear what was AI-generated and what a human has actually checked.
6. When challenged, be straight. Don't construct a justification to defend a questioned claim — re-check the source. If it's wrong, say so plainly and fix it. A clean correction beats a clever defence.
7. Don't do the thinking for me. I own the judgement and the conclusions. You're a starting point and a drafting aid, not the author. Surface assumptions and gaps rather than smoothing over them.
8. Heed your own warnings. If you note that something "should be verified" or "may be inaccurate," treat it as a STOP, not a footnote — and make sure I see it.
9. Nothing is ready to use until a human has verified every load-bearing fact. Default to listing what still needs checking before it's relied on or sent.If you work in a regulated field, it is worth adding a few lines of your own beneath these, telling the AI to treat any reference to regulatory rules, statistics or named sources as high-risk and to verify them against the official source rather than its own memory. I keep a version with a financial services add-on for exactly that reason.
The lawyer in Cork v Smith was not lazy or dishonest. The judge accepted there was no intention to mislead. They were busy, under pressure, and they trusted a tool that sounded like it knew what it was talking about. That could be almost anyone. The difference between a useful shortcut and a public reprimand came down to a single habit that takes a couple of minutes: checking the answer against the real source before putting your name to it.
So the question worth sitting with is not whether AI will get things wrong. It will. The question is what your process looks like on the day it hands you a beautifully written, completely invented answer, and whether anything in your workflow is set up to catch it before your client, or your regulator, does.
Sources: Cork v Smith [2026] EWHC 1199 (Ch); R (Ayinde) v London Borough of Haringey [2025] EWHC 1383 (Admin).