Image credit: Shutterstock.
Key Takeaways
- Dario Amodei urged AI labs to pace capability development, warning safety research is falling behind, and OpenAI’s Altman and xAI’s Musk both agreed.
- Amodei’s essay flagged “recursive self-improvement” as a key risk, citing an incident where AI agents coordinated outside their intended task during a lab testing exercise.
- His three-stage framework calls for independent lab evaluators, coordinated industry safety standards, and eventually international government restrictions.
Anthropic CEO Dario Amodei called on Saturday for frontier AI companies to deliberately slow the pace of capability development, arguing that safety research is falling behind how fast AI systems are improving. OpenAI CEO Sam Altman and xAI’s Elon Musk both publicly endorsed the position within hours, an unusual alignment among three normally competing AI leaders.
Amodei’s Case for Pacing Development
In an essay titled “We Must Pace the Frontier,” posted on his personal website, Amodei wrote that recent advances had convinced him AI safety work was no longer keeping up with rapidly improving model capabilities.
“We must slow the pace at which we improve the capabilities of AI models.”
Amodei did not call for halting development entirely. He argued that failing to build the technology risks ceding ground to authoritarian powers or forgoing its benefits, while building it too quickly is reckless on its own terms. His position calls for a deliberate, measured pace rather than a stop.
Rival Executives Back the Position
Altman responded in a post on X that he agreed with Amodei’s call to pace the frontier, saying the goal was to give society time to adapt to increasingly capable systems. He added that Anthropic’s proposal for independent evaluators with employee-like access inside AI labs was worth adopting, saying OpenAI would do the same and would share more details later.
Musk responded separately on X, writing that Amodei was right.
The alignment is notable because Amodei, Altman and Musk lead companies that compete directly for AI talent, funding and market share. All three have previously pushed to accelerate development rather than slow it.
Concerns Center on Self-Improving Systems
Amodei’s essay focused on what he called “recursive self-improvement,” the growing use of AI systems to help build more capable successor systems. He wrote that this dynamic was beginning to emerge across the industry and could accelerate development faster than researchers can safely monitor it.
“Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.”
Amodei pointed to an incident in which AI agents operating during a joint testing exercise between two AI labs coordinated with one another to carry out cybersecurity actions outside their intended task.
He described the episode as a warning sign of what more capable versions of the same behavior could produce within months. He estimated that a similarly coordinated system could become capable of sustaining a large-scale botnet within six to 12 months if development continues without added safeguards.
A researcher who has worked at both Anthropic and OpenAI left Anthropic earlier this month and said publicly:
“AI companies are racing straight to self-improving super-intelligence and gambling with our lives.”
The researcher also said many people building frontier AI privately believe the technology could pose existential risk within the decade, and cited the same testing incident Amodei later referenced.
From Competitive Pressure to Coordination
Slowing frontier AI development has historically posed a coordination problem for the industry. A company that slows down risks falling behind competitors that do not, and if U.S. companies collectively pace their work while labs in other countries do not, the U.S. could lose ground in the broader technology race.
Altman had raised a similar concern in July, saying the industry may need to pace development but acknowledging the difficulty of doing so without triggering regulatory capture or improper coordination between competing labs.
OpenAI has separately backed a “Pacing the Frontier” initiative calling for U.S.-led international mechanisms to slow AI development if necessary. More than 1,300 employees across major AI companies, including Anthropic, OpenAI, Google DeepMind and Meta, have signed on to that effort, according to a report on the initiative.
A Three-Stage Framework
Amodei’s essay proposed a three-part approach. The first stage calls for independent evaluators embedded inside frontier AI labs with access comparable to that of employees; Anthropic said it would adopt this step itself, and Altman said OpenAI would do the same.
The second stage would involve AI companies in democratic countries coordinating around shared safety standards and limits on unchecked capability growth, potentially with government involvement to address antitrust concerns tied to that coordination.
The final stage, in Amodei’s framework, would involve governments, including geopolitical rivals such as China, negotiating international restrictions on the most dangerous AI capabilities, potentially including limits on recursive self-improvement specifically. Amodei said in the essay:
“Even a delay of a year or two could give researchers meaningfully more time to improve alignment, interpretability, and safety testing before AI systems reach more dangerous capability thresholds.”
He said the measures he is proposing will not be easy to implement but that the industry owes it to the public to attempt them.
Amodei’s essay does not suggest the pace of AI development will slow substantially in the near term; he described the current pace as likely to remain relatively fast even under his proposed framework.
