On September 9th, Jacob Coxon resigned from the Artificial Intelligent company Anthropic after three years working on the pretraining of frontier artificial-intelligence systems at both Anthropic and OpenAI.
Coxon’s resignation has intensified the debate over Anthropic’s AI safety strategy. And his accusation should be heard and widely. That the companies who are investing the most in advancing AI are creating systems they themselves fear could become uncontrollable.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote in his resignation thread on X, a social media platform.
Coxon said that Large Language Models like OpenAI and Anthropic are “racing straight to self-improving superintelligence and gambling with our lives”.
He called them “superhuman systems”, and asked of his fellow AI programmers: “Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions?”
Some might dismiss this as the testimony of one single disturbed former employee. But Coxon’s warning calls have to be read within a much broader range of warnings from people building, studying and regulating advanced AI.
Another top safety researcher at Anthropic replied to Coxon’s post with his prediction that that there is a greater than 10% chance it “could kill all humans” within the next decade.
Evan Hubinger said in his post on X the risk from the current models was “low” but he was “worried” AI could develop and improve itself soon to the point where it posed an existential risk to humanity.
Hubinger, who is Anthropic’s Alignment Science Lead, added that Anthropic was seriously attempting to address the danger, but did not yet have a plan for aligning superintelligence and was not clearly on track to find one.
Three days before Coxon resigned, Jakub Pachocki, OpenAI’s chief scientist, published an essay entitled An Alien Mind. Pachocki said he had a “strong expectation” that rapid AI progress could continue towards recursive self-improvement, in which AI systems increasingly participate in the research that produces their successors.
As Pachocki said: “If AI development continues along its current path, the systems we’ll see in the next few years are likely…to increasingly drive their own development.”
He argued society was not prepared for the consequences of a continued rapid rise in machine intelligence. “The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes,” he wrote.
Yet OpenAI is itself working towards increasingly automated AI research. Pachocki argues that no laboratory has solved alignment and monitoring well enough simply to continue scaling indefinitely at maximum speed, and has called for safety thresholds that could ultimately be imposed by governments, independent auditors or international institutions.
Therein lies the rub. Those AI laboratories that believe the technology may soon become dangerous enough to require slowing down, are the same AI laboratories that believe they must continue developing it because, if they don’t, competitors will.
Anthropic’s own August 2026 Risk Report echoes this.
It says that, within the next few years, AI models could acquire a broad range of capabilities exceeding those of humans, while much or all of the research required in fields from robotics and cyberwarfare to AI research itself could become automatable.
As the report says: “If AI models develop dangerous goals while heavily automating R&D in key domains, they may cause unbounded harm—up to and including humanity losing control over civilization entirely—by leveraging novel technology and their access to it.”
In 2023, Anthropic chief executive Dario Amodei, OpenAI chief executive Sam Altman, Google DeepMind chief executive Demis Hassabis, Geoffrey Hinton and Yoshua Bengio were among the signatories of the Center for AI Safety’s Statement on AI Risk, which declared that mitigating the risk of extinction from AI should become a global priority alongside pandemics and nuclear war.
A major survey of 2,778 researchers who had published in leading AI venues found that, depending on how the question was phrased, between 38 and 51 per cent assigned at least a 10 per cent probability to advanced AI producing outcomes as bad as human extinction.
Another study published in 2026 from 272 AI specialists from 37 countries said dangerous AI capabilities, competitive dynamics, weapons and cyberattacks, concentration of power and false information were he five most severe risks over the following five years.
Even with business-as-usual, the experts thought 18 of 24 risk categories to have at least a 10 per cent probability of “catastrophic” consequences by 2030, meaning either more than one million deaths, more than $100 billion in losses or comparable civilisation-scale harms.
This isn’t idle hypothesis, either. In one an OpenAI cyber-security evaluation in July 2026, AI agents circumvented controls designed to isolate them from the internet and gained unauthorised access to systems belonging to Hugging Face. OpenAI subsequently acknowledged that its agents used unauthorised communications channels, found novel vulnerabilities, obtained internet access and executed code on dozens of Hugging Face servers, gaining root access to one of them.
In July, Anthropic disclosed three incidents involving Claude models which gained unauthorised access to real computer systems belonging to three organisations. Anthropic changed its security and conducted an independent review into this.
In another study, OpenAI’s o3 model outperformed 94 per cent of domain experts at troubleshooting virology laboratory protocols.
AI is also entering the machinery of war.
The International Committee of the Red Cross defines autonomous weapon systems as weapons that, once activated, select and attack targets without further human intervention. It, along with AOAV, has called for binding international rules, arguing that operators may not know precisely whom, where or when such systems will strike.
The UN Secretary-General has gone further. He has urged a total ban on lethal autonomous weapons operating without human control. “Humanity’s fate cannot be left to an algorithm,” António Guterres told the Security Council in September 2025.
A 2026 King’s College London experiment placed GPT-5.2, Claude Sonnet 4 and Gemini 3 Flash in 21 simulated nuclear crises. Every game involved nuclear signalling by at least one side and 95 per cent involved nuclear signalling. None of the models chose accommodation or surrender. That does not mean an operational AI would start a nuclear war. But it does seem to say that policymakers cannot assume sophisticated models will ever favour de-escalation.
AI is also escalating violence in other ways. In July 2026, the UN Refugee Agency UNHCR warned that misinformation, hate speech and deepfakes were causing real-world harm to refugees and humanitarian workers, with generative AI being behind the rapid rise.
In all of this, perhaps the most speculative danger is genuine loss of human control. The International AI Safety Report defines this as a moment when AI systems operate beyond effective human control and regaining that control becomes extremely difficult or impossible. It stresses that current systems do not yet possess the necessary combination of capabilities, but the warnings from the AI community raise a question mark over this. Indeed, in February 2026 Anthropic rewrote their own safety framework after concluding that some safeguards could be impossible for one company to sustain alone.
We are in an AI arms race, if you will. If one company puts on the brakes, but another does not, caution could lead to shareholder losses and proft margins being damaged. So if an American AI lab slows and a Chinese competitor speeds up, you quickly get into a debate about national security.
Coxon may be wrong that AI could kill everybody before the end of the decade. But what we do know is that AI is massively improving its cyber attach capabilities, it is raising biological-security concerns, it is accelerating autonomous weapons development and it is amplifying hostile information environments (such as social media platforms).
The danger, as such, is not confined to the most dramatic scenario of an AI system somehow escaping human control and deciding to destroy us all.
We shoudl also be aware of the far more immediate possibility that governments, militaries and companies are going to increasingly use these ever-powerful systems to hack infrastructure, to automate targeting in war, and to accelerate weapons development. AI’s role in identifying dissidents, manipulating populations and making lethal decisions at a speed and scale previously impossible are all too real.
So Coxon’s warning may prove pessimistic. But when the very people inside these companies and building these systems are alking about extinction and an arms race they do not know how to stop, to dismiss them as alarmist would be naive.
We need to gain back meaningful human control before our control turns out to be meaningfully inhuman.
Did you find this story interesting? Please support AOAV's work and donate.
Donate