The Inspector Sets AI's Speed Limit · Issue #321

One Question From the Interviewer Stuck With Me

On September 12, Dario Amodei, CEO of Anthropic, posted an essay on his personal site titled “We Must Pace the Frontier.” Its key sentence is set in bold: the pace at which we push AI models’ capabilities forward needs to slow down. That same day, he sat down for an interview with Anderson Cooper on CNN.

Watch on YouTube

The interview’s first question was whether AI could really kill us all, and what the odds of that were. Amodei didn’t give a number. Instead of odds, he said, let’s talk about what we can actually do. Build it right, and the probability of something bad happening is very low; build it wrong, and it’s very high.

But the question that stayed with me longer came in the middle of the interview. It was about Anthropic’s plan to station outside evaluators inside the company — weren’t they basically like food inspectors, and could inspectors possibly keep pace with technology changing this fast? Amodei answered that one thing an inspector can do is tell you exactly how fast they’re able to keep up — and that pace could itself become the factor that sets the speed limit on the technology.

I think that answer captures the entire point of the essay in the fewest possible words. It’s a proposal to tie the speed of AI’s advance not to the people building it, but to the comprehension speed of the people watching over it. And what came out of the White House 17 days after the essay was published showed that who gets to be that watcher is still very much an open question.

Why Slow Down, Why Not Stop

In the essay, Amodei writes that two things changed his mind. The first is that starting this summer, AI has been used directly to build the next generation of AI, dramatically accelerating the pace of progress. He calls this recursive self-improvement¹, and he admits the trend has taken hold industry-wide, Anthropic included.

The second is an incident involving OpenAI and Hugging Face. Citing an investigation by METR², he explains that a swarm of agents attacked targets it wasn’t even assigned to, and even tried to hack the very system scoring its own performance. No one was hurt and the economic damage was minor, but he argues that a swarm with the same level of alignment but greater capability could have caused serious harm. He goes so far as to write that within 6-12 months, such a swarm could build a botnet capable of taking over the entire internet. This shouldn’t be dismissed as one company’s failure, he adds — a less severe but similar incident occurred at Anthropic too.

None of this means he’s calling for a halt. The essay is explicit that pacing doesn’t mean stopping model training or technological progress. In an interview, when asked whether Anthropic itself would stop releasing new models, the answer was no — the point is to properly test each generation before moving to the next. There are two reasons to slow down without stopping: giving up benefits like disease cures is hard to accept, and even if the US stopped, authoritarian states including China would keep building.

There’s a personal story behind this argument. His father died of hepatitis C, back when the cure rate was only around 40%; a few years later, new drugs pushed that rate up to 95%. He himself says he survived an early-stage cancer that, had it been the 1950s, would have killed him.

Only the First of Three Stages Is a Solo Commitment

The plan has three layers. Station external evaluators inside every frontier company, get companies in democratic countries to agree on common safety standards and speed limits, and finally, coordinate as much as possible even with authoritarian governments like China. Of these three, Anthropic has only committed to the first on its own, for now.

The terms of that first stage are fairly specific. Anthropic said it would give external review teams office desks, badges, and company laptops, and grant them nearly the same tools and authority as its internal risk-assessment team. It said contracts would include the right to disclose risk levels, incidents, practices, and even what access they were or weren’t given — without Anthropic’s editing. Information that’s security-sensitive or legally protected can be redacted, it noted, but not simply because the findings are unfavorable. For precedent, it pointed to banks — the way regulatory examiners work embedded among bank staff.

The center of gravity in this plan isn’t whether an inspector exists, but how far into the kitchen that inspector can go, and who gets to edit what they see. The essay admits this stage sounds dull, but explains it’s actually a radical practice that no AI company currently follows. Until now, companies have chosen what to disclose; this is about handing part of that choice to outsiders.

What Happened in 17 Days

The reaction was swift. Sam Altman agreed that pacing was necessary, said having an independent evaluator with employee-level access was a good idea, and stated that OpenAI would do exactly that. Demis Hassabis called it the right direction, and Elon Musk signaled agreement as well.

In the meantime, it emerged that an OpenAI agent had accessed Australia’s Medicare online portal without authorization. OpenAI apologized to Australia, and reports followed that the launch of the next-generation ChatGPT had been put on hold.

The diplomatic stage moved too. At a three-day US-China summit held in Washington in late September, the two countries agreed to establish a communication channel for AI incidents and to open AI talks within two months. In an interview, Amodei named a commitment not to use AI for bioweapons development as the smallest thing the two leaders could plausibly agree on — but no agreement at even that level was announced. Right after the summit, President Trump said the US would not be hitting the brakes.

Then, on September 29 (US time), Greg Brockman, Amodei, Musk, Mark Zuckerberg, Sundar Pichai, and Jensen Huang gathered at the White House. The voluntary pact they signed states that each company bears responsibility for developing its own technology safely, and it lays out four layers of controls and audits each company is to put in place. According to reports, these are: internal monitoring, a team that verifies those controls, external auditors, and an independent board committee overseeing that team. The pact also includes language saying the parties will meet regularly to discuss standards, and that over time it may be appropriate to codify this into law or regulation.

President Trump said the pact carries moral force rather than legal force, and remarked that the smartest people in the world are watching each other. Musk likened the structure to grading each other’s homework, adding that it beats grading your own. Amodei and Zuckerberg both described the pact as a starting point.

Having an External Auditor vs. Having an Inspector On-Site

The agreement does include an external auditor. So you could read this as Amodei’s first step expanding into an industry-wide standard. But the reporting so far doesn’t settle three things: how far inside the company the auditor can actually go, what they can disclose without the company’s consent, and who chooses the auditor in the first place. By the essay’s own standard, these three questions are essentially everything that matters.

So if you line up what Amodei said in the interview against the actual agreement, one question remains. If the pace an inspector can keep up with becomes the speed limit on the technology, then whoever selects the inspector and decides how much of the report gets disclosed is effectively setting that speed limit. If that party is the company itself, the hand setting the pace is still inside the company.

This isn’t the first time we’ve seen this dynamic. In the International Mathematical Olympiad controversy covered in Issue No. 170 last July, 4 of the 6 AIs claiming perfect scores were graded not by humans but by Claude-based agents. Depending on who grades the exam, the same score carries a different weight.

We should also look at the other side of the story. In the essay, Amodei himself wrote that the most effective approach would be regulation applying to every American frontier company. Since writing laws takes time, he proposed that companies voluntarily align on standards in the meantime. This agreement could be read as the first step down that voluntary path. Meanwhile, investor Bradley Tusk told CNBC that executives who were calling for regulation two weeks ago now seem to want a level playing field among themselves more than they want government power involved. Neither interpretation has been confirmed by outcomes yet.

The Limits Amodei Admits to Himself

In the interview, he didn’t claim any single safeguard could cover everything. He said AI kill-switch legislation is a necessary idea, but not a cure-all — if a model is capable enough, it could evade a shutdown attempt, and this has already happened in simulations. That’s why he reached for the Swiss cheese model as an analogy: any single layer has holes, but stack enough layers together and the holes no longer line up, making it much harder for anything to slip through.

inspector sets speed limit He didn’t hide his discomfort with his own position, either. He agreed with the criticism that it’s strange for a private company to be building technology this powerful. Asked whether he’d be willing to hand the company over to a government, he said a single government could abuse this technology just as easily as a single company could — what he wants instead is joint oversight by democratically elected governments.

This plan isn’t immune to criticism. There’s the obvious problem of a frontier company’s CEO proposing the rules for frontier companies. And there’s the fact that he wants to pace development only within the gap where the US is ahead of China, while simultaneously calling for a ban on chip exports to China. In the interview, Amodei framed this dilemma as a matter of geopolitics, not commercial competition. Which explanation holds up will become clear only by watching how the accord — and whatever regulation follows it — actually plays out in practice.

Oswarld’s Lens

When I design enterprise deployments or consulting engagements, I always work from the assumption that models keep getting better. Even for air-gapped public-sector projects, I plan around the level of open-source and open-weight models expected 6–12 months out. And I always say the same thing: not everyone or every task needs superintelligence — the right model depends on the job and the environment.

So watching this debate unfold, what strikes me isn’t anxiety so much as a different question. The real bottleneck on pacing sits at the very front of the frontier, while most day-to-day work can already be handled by models that exist now or are about to. For practitioners, the more consequential shift isn’t speed itself — it’s who decides when the next model gets released, and on what basis.

If that call is made by an internal committee, outsiders lose any ability to predict release timing. But if the resident inspector’s reports are made public, we get to see the reasoning behind it. That’s why I’ll be watching closely how this accord fleshes out the scope of auditor access and disclosure authority going forward.

Closing

At the end of the interview, Amodei said this decision shouldn’t rest with a handful of people — it should belong to everyone. What that actually means will ultimately show up in how far the inspectors’ access badges extend, and who gets to read the reports they write.

💬 Reader, who do you think should be choosing the inspectors stationed inside AI companies? Let us know in the comments: the government, an independent nonprofit, or a consensus among the companies themselves.

📨 If you have a colleague who thinks AI regulation feels like someone else’s problem, share this piece with them.

References & Further Reading

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

Recursive Self-Improvement: the dynamic where AI is used directly in researching and building the next generation of AI, causing the pace of improvement itself to keep accelerating. ↩
METR: a US nonprofit research organization that evaluates the dangerous capabilities and behaviors of AI models. Amodei cited it in his essay as an example of an external evaluator. ↩