Breaking

Former Anthropic Researcher Jacob Coxon Resigns, Warning Labs Have Only a Few Years to Align Superintelligence

September 13, 2026 at 15:11 EDT

Foundation Models
Policy & Regulation
AI Agents

Jacob Coxon, a 27-year-old pretraining researcher who worked at OpenAI and then Anthropic, has resigned, warning that frontier labs are racing toward self-improving superintelligence on a timeline of only a few years and that no private company should be trusted to run what he called a “mini Manhattan Project.”

September 2026 · Anthropic

A private mini Manhattan Project

Jacob Coxon left Anthropic warning that frontier labs have only a year or two to keep self-improving AI under human control — and that no private company should be trusted to run the effort.

>10%

Hubinger personal estimate of AI causing human extinction this decade

1–2 yrs

Window colleagues call crunch time or endgame

4 mo

Coxon’s tenure at Anthropic before resigning, short of the six-month equity vest

The 10% floor, drawn to scale

One orange block in a stack of ten is the personal lower bound Anthropic’s alignment science lead put on human extinction from AI within a decade.

>10%

extinction floor

Evan Hubinger said researchers earnestly believe AI could kill all humans, while risk from present models remains low. The orange share is the future-superintelligence case he is worried about — not today’s chatbots.

Two clocks on the same race

1–2 yrs

Crunch time
colleagues cite

10 yrs

Hubinger
extinction window

Column height is years. Coxon says the next one to two years decide the path; Hubinger’s personal extinction estimate is framed over a decade.

The path Coxon wants delayed

Frontier race

Labs compete with OpenAI and China

→

Recursive self-improvement

AI used to build more capable AI

→

No alignment plan

Hubinger: not clearly on track for superintelligence

Inside the labs

The fear is treated as real

Hubinger backed the core claim. Colleagues use “endgame” language. Coxon still calls Anthropic the most responsible player — and says many there want regulation because they cannot trust themselves in a race.

Pushback

The alarm is contested

Hugging Face CEO Clement Delangue said asking a pretraining researcher about extinction risk is like asking an AC technician about climate change. Other observers call the warning overstated or a bid for tighter control.

Coxon’s test

No private company should run this project

He says Anthropic is not cutting corners yet, but a race without a government mandate will force safety trade-offs as systems speed up.

Mandate

Private lab, not a state nuclear program

Near term

OpenAI and Anthropic delay recursive self-improvement

Longer term

International pacing of compute, CERN-like oversight

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.