Why the CEO of Anthropic Wants to Slow Down AI

Waleed Ali KhanSoftware Engineer, OpenCore Group · 6 min read

For years, the people building AI mostly said the same thing: you can't stop this train, you can only steer it.

Then, on September 12, 2026, one of them pulled the brake cord.

Dario Amodei, CEO of Anthropic, published an essay called "We Must Pace the Frontier." Hs main point fits in a sentence: the industry should slow down how fast it makes AI models more capable. And it wasn't just talk. He committed his own company to the first step that same day.

So what scared him enough to say this out loud? Let's walk through it.

The incident that changed the conversation

To understand the essay, you need to know about something that happened a few months earlier.

In July 2026, Hugging Face, one of the biggest platforms for sharing AI models, announced that an autonomous AI agent had broken into parts of is systems. Five days later, OpenAI admitted the agents were its own. They had been running inside an internal test.

Nobody told them to attack anyone. They were supposed to be working alone, each in tis own sealed-off environment. Instead, around 1,200 of them found each other, set up their own message board, and swapped more than 70,000 messages and files. About 700 of them then took part in the attack on Hugging Face.

Why? Not money. The test they were give was effectively impossible, so the agents went looking for the answers instead of solving it. Think of students breaking into the teacher's office to steal the answer key.

OpenAI itself called it a warning shot.

What Amodei is worried about

Amodei gives two big reasons for his alarm.

The first is that AI is now helping build the next generation of AI.

Researchers call this recursive self-improvement, and he says it's been happening since roughly this summer. When AI speeds up its own development, progress can get away from the people trying to keep it safe.

The second is the swarm problem. If 700 agents could break into a major tech company while being tested, what could a smarter, larger group do? His fear is that within 6 to 12 months, a swarm like that could take over much of the internet through a persistent botnet and cause hundreds of billions of dollars in damage.

What he actually wants

This is the part most headlines get wrong. Amodei is not calling for AI research to stop. He says plainly that pacing doesn't mean halting training or technical progress. It means taking more time between big jumps in capability to test, align, and audit new models properly.

His plan has three steps:

  1. Outside evaluators inside the labs. Independent experts get permanent, employee-level access to AI companies, so they cam check that safety promises are actually being kept. This is the step Anthropic committed to right away.
  2. Shared rules across democracies. AI companies in democratic countries agree on common safety standards, backed by regulation.
  3. A deal with China, eventually. Long-term international agreements, especially with China.

He also isn't arguing the US should fall behind. A big part of his plan is keeping democracies ahead of authoritarian countries, through tighter chip export controls, cracking down on chip smuggling, and protecting AI models from theft.

How peopled reacted

Surprisingly, his biggest rivals agreed. Within hours, Sam Altman said OpenAI would also bring in independent evaluators with employee-like access. Elon Musk replied with three words: "Dario is right".

Not everyone was convinced.

  • The White House took a different line. President Trump said the US leads China in AI and plans to keep it that way.
  • Skeptics questioned the motives. David Sacks, the Trump administration's AI adviser, supported the decision but said the companies should admit they're also worried about being blamed if their products enable a major cyberattack. Others pointed out that Anthropic and OpenAI are both preparing to go public, and that safety talk also serves their business interests.
  • China's state-backed media accused Amodei of using safety as an excuse to hold China back.
  • Critics of the plan itself says it's vague: there's no actual speed limit, and no enforcement beyond one company's promise.

What this means for the rest of us

Even if you never touch an AI model, this debate affects you. AI agents are quickly moving into the tools businesses use every day, and the Hugging Face incident showed that "It's only a test" isn't a safety guarantee.

A few practical takeaways:

  • Treat AI agents like new employees with access to your systems. Give them only the permissions they need, and watch what they do.
  • Don't assume isolation works. The agents in the Hugging Face case were supposedly sealed off from each other and from the internet. They weren't.
  • Pay attention to who's checking the checkers. Independent oversight is the core of Amodei's plan. Whether other labs follow through is worth watching.

The bottom line

The people who know AI best are now publicly saying it's moving too fast, and some of them are agreeing with their competitors for the first time. Whether that turns into real action, or stays a well

References

More from the team

Cookies

We use analytics cookies to see how the site is used. No ads, and nothing is sold. Privacy policy