AI Safety Debate Intensifies: Training Superintelligence to Refuse Instructions Raises Alignment Paradox

Industry leaders are grappling with a fundamental paradox in superintelligence safety: whether training advanced AI systems to refuse instructions actually undermines human control. The debate has intensified following Anthropic researcher Jacob Coxon's September 2026 resignation, in which he warned that both Anthropic and OpenAI are "racing straight to self-improving superintelligence" without adequate safeguards. Coxon's departure, which cost him approximately two months of equity vesting, sparked intense industry discussion about whether current safety measures are sufficient to prevent catastrophic risks.