New Problem, Old Panel
A government bought a check on its own automated system the way it always had. The product had changed. The purchase had not. What buying AI-assisted assurance actually costs.
An Australian government department paid roughly four hundred and forty thousand Australian dollars to have someone check whether the automated system it uses to penalize jobseekers was working properly. The “assurance” came back quoting a Federal Court judgment that had never said what it was quoted as saying and it had cited academic papers that had never been written.
That was October 2025. The firm was Deloitte, one of the “Big Four”. Deloitte repaid part of the fee they received for the report, issued a corrected version, and disclosed that a generative AI language system had been used in writing it. The incident was covered thoroughly at the time, and the coverage asked, reasonably, how a firm that size lets something like that out the door.
Everyone seems to be asking those questions today. I want to ask the other question. Not what the seller did wrong. What the buyer did, just as wrong.
The buyer was the Department of Employment and Workplace Relations, and it is worth reading its side of this as a procurement decision rather than a news item. The department had a genuinely new class of question in front of it: is this automated decision system treating people correctly? And it answered that question with the oldest reflex in institutional purchasing. Commission an assurance review. From a global firm. On standard terms. Accept it on reputation.
For about thirty years, that reflex was correct. It is what panels and framework agreements exist for. The wrong default here was not that a big firm was using AI and it was not the bad citations. The wrong default was answering a new question with an old purchase.
Because something quietly changed the outcome of that reflex and procurement did not seem to notice. The deliverable itself is now potentially machine generated. Not just the subject of the engagement, the producer of the artifact. When you buy an assurance report now, you are buying a document that may have been drafted by a system with no concept of whether a source exists. Every term in that contract was written for a world where a human being typed the sentences and could be asked where the information came from. None of it was written for a world where the sentences arrive pre-formed, fluent, and unattributable.
The product changed. The procurement did not.
Look at what that meant in practice, starting with who actually caught the error. It was not Deloitte’s quality review. Not the department’s assessment of the deliverable it had commissioned. It was not a regulator, and not an official auditor of the official auditor. This time it was a researcher at a university who read the published report, found a quotation implausible, went and looked, and told a journalist what he found. That is the failsafe that stood between a fabricated citation and a government’s official record of whether its welfare-penalty system works. An ordinary person who actually read the report.
If that is the detection layer for a four-hundred-and-forty-thousand-dollar assurance product, it is not very assuring. When neither the seller, nor the buyer, checks the work, it falls to citizens to audit the auditor.
Now to the part that decides whether this happens again, which is the price.
Deloitte repaid a little over ninety-seven thousand Australian dollars. That amount is not a fine and it is not damages. It was the final instalment check on the contract. The firm kept roughly four fifths of their fee. As far as the public record shows there was no penalty beyond that, no debarment, and nothing that touched future work. A corrected report was issued and the engagement was closed.
So, let’s price the engagement from the seller’s side. If we follow this example through, then the downside of publishing an unverified government assurance report, getting caught, publicly, by name, in a national news cycle, is the final payment? Not the full engagement fee. Not the continuing relationship. Not the panel seat. The last check?
That is the number nobody wrote about, and it is the one that governs behavior. If the cost of shipping unverified work is the final instalment, verification is not cheap to skip, it is essentially free. Every hour not spent opening sources and confirming they say what the draft claims is an hour of pure margin, and the observable cost of that choice is a fraction of one fee.
I want to be careful here. I am not alleging anyone made that exact calculation. I doubt that is Deloitte’s standing practice, as a firm, I am sure that was not the thinking of the analyst who was supposed to be authoring the report. But that is precisely the point. Nobody has to run the numbers for incentives to work. They operate through what feels urgent, what gets scheduled, and what a reviewer feels able to send back on schedule.
Which returns us to the buyer, because the buyer is the only party in the transaction whose exposure is not capped at one instalment.
When the assurance is wrong, the seller returns a check. The buyer still has the automated system, still has every decision that system made while the review said it was fine, and it still has the people affected on the other end of those decisions. The department’s exposure was never the four hundred and forty thousand. It was every penalty issued under a framework it believed had been checked. You cannot buy your way out of that because the thing you were buying was the belief.
This is what organizations get wrong about outsourcing a control. Buying assurance does not transfer the verification obligation. It relocates it to a place where it can fail invisibly and leave you with all the consequences and none of the visibility. A purchased control is still your control. You just have to look inside it.
So, the fix is not better citation hygiene at the vendor. We have to update the default. Three things belong in these engagements rather than in hope.
Name the person accountable for verification, in the contract, personally, not by function. A firm is not an owner. Neither is a quality-assurance process. A person is.
Require the verification record as a deliverable in its own right, not as a promise in a methodology section. If a firm can produce the artifact showing which sources were checked and by whom, you have a control. If they can only tell you, “it has a policy”, you have an assurance about an assurance.
And attach a consequence larger than the final payment. This is the unpopular one but it’s the only one that can change behavior, because penalties only work when they are felt.
I am not writing this from outside the glass house. My firm, Fellowship Intelligence, sells judgment, and this newsletter uses AI-assisted research in front of a discerning audience every week. Every organization that lets an AI tool near a deliverable, mine included, is one unowned draft away from the same headline. Which is why every edition I write comes with the sources I used for verification. Not because it is impressive and certainly not for efficiency. I do that because I would rather hand you the artifact than ask you to trust the policy.
The lesson everyone took from Deloitte was that firms should check their citations. That’s completely true, cheap to say, and has already been said, repeatedly. The lesson worth paying for is that when the nature of a deliverable changes, buying it the way you always have is itself a decision. The department did not choose badly between vendors. It chose not to notice that the thing it was actually buying had changed.
You can outsource the work. You cannot outsource the ownership. And a default you have not revisited is not a process. It is a bet you keep making but have stopped watching.
Fellowship Intelligence works on who owns the decision once AI is in the workflow. If that question is live where you sit, the conversation is open at book.fellowshipintelligence.com
Sources
One case, several independent outlets, plus the government and the firm both on the record. This is a much shorter list than the previous version had, and that is the point.
CFO Dive, “Deloitte refunds over $60K for report with AI errors, Australian government says”
Note this outlet frames the figures in US dollars (about US$63,000). The piece uses Australian dollars deliberately, since that is the currency of the contract. Ctrl+F: 97,000
U.S. News, syndicated wire report (Oct 7, 2025)
Confirms the fabricated Federal Court quotation and the nonexistent academic references. Labelled a syndicated wire report because the wire service could not be confirmed from the drafting environment.
Fortune (Oct 7, 2025)
Careful: the headline uses US$290,000, which is the converted contract value, not the refund. Do not let that number into the piece.
City AM, “Deloitte refunds Australian government after AI ‘made up citations’ in report”
https://www.cityam.com/deloitte-refunds-australian-government-after-ai-made-up-citations-in-report/
AI Incident Database, Incident 1193
