Sincerity Is Not a Control

Disclosure: the author is founder and managing director of Fellowship Intelligence, which advises on AI governance and partners with other firms on enterprise engagements. He has a commercial interest in what follows. Read it accordingly.

Jacob Coxon did the expensive thing.

He resigned from Anthropic roughly two months before his equity vested, walking away from unvested shares reported to be worth millions, and posted a seven-part warning that the labs are “racing straight to self-improving superintelligence and gambling with our lives.” It reportedly drew close to 76 million views overnight. Whatever you think of it, that is a costly signal, and costly signals are the only kind worth much. The cheap version of this warning has been available for years and has cost its speakers nothing.

Then he went on Meet the Press and told Congress what to do about it: let the frontier labs regulate themselves until a proper regulatory body exists. The people running them, he said, are “completely genuine” and “would like to slow themselves down.”

In the same round of interviews, Coxon also said this: “If you’re under pressure to race, you have to cut corners” or “skip steps in the oversight process.”

Both of those cannot be true at once, or at least they cannot both be load-bearing. The first says the people racing should be the ones policing the race. The second says the racing is what stops anyone policing it properly. If competitive pressure reliably produces corner-cutting, then the people under competitive pressure are exactly the wrong people to leave in charge of the corners.

This is not a debating trick. It is the reason governance exists as a separate discipline from ethics. You build controls on the assumption that good intentions will run into a bad quarter and lose.

The pressure is not hypothetical, and it is priced

Coxon’s failure mode is not a thought experiment about some future race. The race has a budget.

The largest cloud companies are on track to spend somewhere between $725 billion and $800 billion on AI infrastructure in 2026, up from roughly $410 billion in 2025 and about $226 billion in 2024. Analysts expect it to pass a trillion dollars in 2027. Roughly three quarters of that goes to AI specifically: chips, servers, power, cooling, buildings.

That is the pressure Coxon named, in dollars. It roughly doubled in a year. The restraint we are being asked to trust has to hold against a spending commitment growing faster than any oversight attached to it.

There is a serious argument buried in his position, and it deserves its strongest form. Laws are slow and they sit still. Models ship in months. A law written against today’s systems may end up governing something that no longer exists. Coxon is right about that. It is a real problem.

But the answer he reaches is not the one that follows. If the worry is that the law goes stale, then write the law about what a company has to be able to prove, not about how the technology works. Proof requirements do not go stale when the model changes. Handing the job back to the companies does. Those are two different fixes for two different problems, and only one of them happens to be the fix the companies would pick.

We already ran this experiment

In July 2023, seven leading AI companies made voluntary safety promises at the White House. Eight more joined that September, and Apple signed on the following July. Test your models before release. Protect against insider threats and hacking. Label AI-generated content. Share what you learn about risks.

Researchers later went back and scored all sixteen companies on whether they actually did it. The study was peer reviewed and published. On average, companies kept just over half of what they promised.

On protecting model weights, the average was 17 percent. Eleven of the sixteen scored zero.

Model weights are the model itself. If you believe this technology could end the world, then who can steal a copy of it is not a side issue. It is the issue. Eleven of sixteen companies did nothing about it.

The scores split cleanly by whether anyone was watching. Companies in the industry’s own safety forum averaged 74.4 percent. Companies outside it averaged 40.7 percent. OpenAI scored highest, Anthropic next, Google third.

Now read that the way Coxon’s proposal asks you to. The companies he wants left in charge are the ones who kept about four fifths of a set of promises they helped write, that carried no penalty, that nobody was checking, and that they could have dropped at any time. Four fifths is the best score in the room. It is also a fifth of your own promises broken, under the easiest conditions you will ever get, before the money behind the race tripled.

The policy already bends to the competition

You do not have to guess how self-regulation behaves under pressure. It is written down.

Anthropic’s Responsible Scaling Policy is the most developed safety framework any lab has published. Anthropic publishes it, updates it in public, and logs its own changes, which is more than anyone else in the industry does. Version 3.0, from February 2026, includes an appendix called “Commitments Related to Competitors.” The policy calls these “competitor-contingent commitments.” They apply, in its words, “in scenarios where we can be confident that other relevant AI developers are doing the same.”

That is Coxon’s failure mode written into the rulebook. Some of the strongest promises only hold if rivals hold theirs. This is not hidden. It is stated openly and defended on reasonable grounds: holding back alone might not make anyone safer. Maybe not. But a promise that loosens when your competitors move is not a control. It is a preference with an escape clause, and the escape clause opens exactly when you would need the control most.

An earlier review supplies the sharper detail. In October 2024, looking back on its first year under the framework, Anthropic reported four times it had fallen short of its own rules. One of them: a required safety evaluation finished three days after the three-month deadline. Anthropic said the extra days made the evaluation better, and that the deadline had been written ambiguously. Its fix was to clarify the wording and change the deadline from three months to six.

Take that explanation at face value, because it is probably true. A rushed evaluation is worse than a late one. The problem is what happened next.

The rule moved to fit the behavior. When somebody outside is checking, missing a deadline gets written up, assigned to a person, and fixed. Nobody hands you the pen. When you are checking yourself, missing a deadline is a reason to move the deadline. Both get called governance. Only one of them stops you from doing anything.

An extinction estimate is not a control

Evan Hubinger, who runs alignment science at Anthropic, backed Coxon publicly. “We really do earnestly believe AI could kill all humans,” he wrote, putting his own estimate above 10 percent within the decade.

That number got most of the coverage. It is the least useful thing he said.

A better-than-one-in-ten guess at human extinction comes with no line you cannot cross, nobody’s name on it, nothing that happens when it gets worse, no way to prove it wrong, and no record anyone can go look at. You cannot build a control on it, because it does not tell you anything to control. You can only believe it or not, which is why it travels so far and changes so little.

The real disclosure was the next clause, and almost nobody quoted it. Anthropic, he wrote, does “not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

That is a different kind of statement. It is dated, it has a name attached, and it says a required safeguard does not exist and is behind schedule. Take the probability out and it still stands. You could prove it wrong tomorrow by producing the plan. Somebody owns it. And it points at what is supposed to happen next, because a missing safeguard is the exact situation a governance system exists to stop things over.

A missing safeguard is something you can check. A probability with no method behind it is something you either believe or you don’t. The first one puts somebody on the hook. The second one does not, which is part of why it is the one getting repeated.

What these warnings do, whatever anyone intends

Two things can be true at once. The people making these warnings mean them. And the warning lands on a conclusion that happens to suit the people making it.

Meaning it and benefiting from it are separate questions, and answering the first does not touch the second. That is why this argument never moves. One side hears an accusation of lying. The other answers by insisting it is not lying. Nobody said it was.

The framing works the same way every time. Say the risk is human extinction, and you have described something almost nobody can judge. Not a regulator. Not a member of Congress. Not the lawyer at a company putting this into production next quarter. If you make the subject that strange, the only people qualified to govern it are the people already inside it. Which is their argument: let us handle it for now.

It also moves the clock. An extinction estimate is a claim about the 2030s. It is not a claim about the decision a company made last week, and attention spent on the first is attention not spent on the second.

That trade has a cost, and it lands nowhere near a frontier lab. It lands on the hospital running a triage model with no one to escalate to, the insurer with no paper trail behind an automated denial, the bank that cannot say who signed off on the system now making its credit decisions. None of that is speculative. Those are systems with nobody’s name on them. They are also boring, which is why they lose the microphone to a number about the 2030s.

The bill is already arriving, just not where he is looking

Coxon asked Congress to wait. Three sets of American institutions did not, and none of them needed an extinction estimate to move.

Courts went first. A federal court in California is letting rejected job applicants sue Workday directly over the AI tool that screened them out. Not only the employers who used it. The software company that built it. The court’s reasoning is that the tool was doing the employer’s job for it, which makes it the employer’s agent, which puts it inside the discrimination laws. In 2025 the court let that proceed as a nationwide group claim on age. In 2026 it confirmed those protections cover applicants, not just people already on the payroll. Nobody has won anything yet and the case is still in discovery. That is not the point. The point is the question the court keeps asking: what did the system do, who was it acting for, and who can show it.

State legislatures are landing on the same question, and Colorado is the clearest case. Colorado passed a sweeping AI law in 2024, then repealed and replaced it in May 2026, weeks before it was due to take effect. Out went the risk categories, the duty of care on developers, the impact assessments, the documented risk program. In other words, all the paperwork about what might go wrong. What stayed: tell people before an automated system makes a decision about them, explain it within thirty days if the answer is no, give them a route to a human, and keep the records for three years. The Attorney General enforces it at up to $20,000 a violation.

Read that as a stress test, because that is what it was. A legislature with every reason to produce something impressive looked at the whole apparatus under real pressure and kept only the parts you can prove. A record shows what happened. A risk rating shows what somebody expected.

Insurers moved last and hardest. In January 2026 the body that writes the standard insurance forms published three new clauses that let insurers cut AI-related losses out of ordinary business liability cover. Insurers started filing to use them immediately. Berkshire Hathaway, Chubb and W.R. Berkley were among those seeking AI exclusions, and regulators approved most of what was filed. Travelers got its approvals and then told regulators it would not use them after all. The industry had been quietly carrying AI risk inside policies written before the technology existed, and it is now deciding, company by company, what to do about that.

No regulator made any of this happen. Insurers price what they can see, and an insurer looking at a company’s AI setup today cannot establish which version of the model made a decision, whether a person reviewed it, what information it used, or whether any record of it still exists. Some are cutting the risk out. The rest are still deciding. Either way the customer is carrying it until they do.

Courts, a state legislature under real pressure, and the insurance market all arrived at the same question on their own, and it is not a question about 2030. It is whether the company running the system can prove what it did. That is what it looks like when a risk has actually been taken in hand: a line you cannot cross, a person who owns it, something that happens when it goes wrong, and a record of all three. Without those, calling it governance is generous. It is a risk nobody has put a price on.

The timing is the whole point. This is not about explaining yourself after something goes wrong. It is about writing things down while they are still true. Which version made the call, who approved putting it there, whether anyone checked the output, what it drew on. Nobody assembles that afterward. It gets written when the system goes in or it never gets written, and the only party in the room at that moment is the company doing the installing.

That obligation does not go stale when the model changes underneath it, which is the exact objection Coxon raised and the exact thing his own proposal fails to answer.

It is already landing on hospitals, insurers, banks and employers, whether or not Congress ever settles the superintelligence question. Which makes the extinction debate a luxury good. It is an argument about a future obligation, held most loudly by the only people not yet carrying a present one.

Who owes what

Coxon is wrong. Not cautious. Not naive. Wrong. He named the thing that breaks safeguards, then handed the safeguards to the people it breaks them for. That is not a hard call, and nobody needs to wait for a new agency to make it.

Congress. Colorado already wrote the rule and kept it when everything around it got cut. Tell people before a system makes a decision about them. Explain it when the answer is no. Give them a way to reach a human. Keep the record. Four things, and not one of them mentions a model, so not one of them expires when the model changes. Laws that describe technology go stale. Laws that describe what you have to be able to show do not.

The labs, as vendors. Your customers need a record and you are the only one who can give it to them. Not weights. Not training data. Which version they got, a heads-up when you change it, and logs they can export and keep. Right now companies are carrying the legal risk for decisions they cannot reconstruct, because the proof sits on your servers and nobody has made you hand it over. None of that is your intellectual property. You are simply not being asked for it.

Companies using this. Most of you are asking the wrong question. You ask whether you have an AI policy. You can have a policy and still have nothing.

Try this instead. Pick one decision your company made last quarter where AI was involved. Not a category. One decision, at random, about one person. Now answer four things. Which version made it. Who approved using it there. Whether a person checked it, and who. What it drew on.

If you cannot answer all four about a decision you picked at random, this is not a problem heading toward you. It is already here. And the next person to ask will not be a consultant. It will be a plaintiff, a regulator, or your insurer, and by then the record either exists or it does not.

Boards. Ask that question at the next meeting. Do not accept a description of a process. Make them run it in the room.

Coxon paid real money to tell us that the people building this are under pressure and not in control. He was right about both. Then he recommended we rely on their character anyway.

The question was never whether they mean it. The question is what happens the first time meaning it costs them the lead.

Their documents already answer that. In writing. In an appendix. Under a heading about competitors.

Sources

NBC News, “Anthropic whistleblower says Congress should let AI companies police themselves until a regulatory body is established.”

Axios, “Anthropic whistleblower gave up his equity to leave the company,” 9 September 2026.

TIME, “He Helped Build Powerful AI at OpenAI and Anthropic. Now He’s Afraid It Could Kill Us,” 9 September 2026.

Evan Hubinger, post on X, September 2026.

Anthropic, Responsible Scaling Policy v3.0 (effective 24 February 2026), Appendix A: Commitments Related to Competitors.

Anthropic, “Learning from Experience,” published with the RSP update of 15 October 2024, at anthropic.com/responsible-scaling-policy.

“Do AI Companies Make Good on Voluntary Commitments to the White House?”, AAAI/ACM Conference on AI, Ethics, and Society (arXiv:2508.08345).

White House voluntary AI commitments, July and September 2023; Apple joined 26 July 2024, bringing the total to sixteen.

Mobley v. Workday, Inc., N.D. Cal.; conditional collective certification on age claims, May 2025; ruling on coverage of applicants, March 2026.

Colorado SB 26-189, signed 14 May 2026, effective 1 January 2027, repealing and replacing SB 24-205.

ISO endorsements CG 40 47, CG 40 48 and CG 35 08, effective January 2026; carrier filings and state approvals through 2026.

Hyperscaler capital expenditure estimates for 2026, compiled from company guidance and analyst summaries.

A note on sourcing

Coxon’s equity forfeiture and the reported value of his unvested shares are press-reported and have not been independently confirmed. Quotations from Coxon are as reported in the outlets listed above rather than from a transcript.

Capital expenditure figures are analyst estimates derived from company guidance; they vary by source and by how “AI infrastructure” is scoped.

The compliance study reports slightly different headline averages in its abstract and its published proceedings; “just over half” is used here to avoid a distinction that does not affect the point. The 17 percent model-weight-security figure is consistent across both.

Anthropic’s stated reasons for the evaluation interval change are reproduced above as the company gave them.

Mobley v. Workday remains in active litigation with no determination on the merits; the rulings described are procedural.

Colorado SB 26-189 does not take effect until 1 January 2027 and carries a sixty-day cure period that sunsets in 2030.

Insurance filings, regulatory approvals, carrier adoption and attachment to a policy are four separate events. The paragraph above distinguishes them, and the position of any individual carrier may have changed since publication.