Jacob Coxon Resigns from Anthropic as Researchers Debate Self-Improving AI Risk

Jacob Coxon, a researcher who spent 3 years working on pretraining at OpenAI and Anthropic, has resigned from Anthropic over concerns that the frontier AI race is moving toward self-improving systems faster than safety research can keep up. His warning has put an internal dispute over control, evidence and regulation into public view. This week, Coxon said the two labs were pursuing recursive self-improvement under competitive pressure and that the next 1–2 years could be decisive. His argument is not that current chatbots have already become uncontrollable; it is that future systems that can help build improved successors could accelerate beyond reliable human oversight. That scenario turns on AI alignment, the still-unsolved task of keeping a system’s goals and values compatible with human intentions as it encounters new situations. In an interview with Coxon, he described possible danger paths including AI-enabled biological threats, cyberweapons and loss of control. He proposed that OpenAI and Anthropic first agree not to immediately begin recursive self-improvement, followed by broader coordination involving the US and China. Coxon also said he left before his Anthropic equity vested, a detail he offered in response to questions about financial incentives.