The AI industry is having its loudest existential reckoning yet. A young researcher who worked at two of the most powerful AI companies has resigned because he believes the technology is being built in a way that could end humanity. His warning was shared by Anthropic\u2019s alignment lead, who said the company genuinely believes AI could kill all humans and put the chance at greater than 10 percent in the next decade. The argument that followed is no longer just an internet debate about hypothetical futures. It has become a question about careers, corporate positioning, IPO filings, and what society should do now.<\/p>\n
When Jacob Coxon left Anthropic<\/a>, he did not leave quietly. The AI researcher, who previously worked at OpenAI<\/a> before moving to Anthropic, published a warning focused on self-improving AI<\/a> and accused the leading AI companies of \u201cgambling with our lives.\u201d His concern was not abstract. Coxon argued that systems capable of improving themselves could cross thresholds that their creators do not fully understand, and that continuing to develop them without stronger safeguards is an unacceptable risk.<\/p>\n The immediate reaction inside Anthropic made the situation more dramatic. Anthropic\u2019s alignment lead shared Coxon\u2019s thread on X and posted: \u201cWe really do earnestly believe AI could kill all humans!\u201d He added that he personally thought the chance was greater than 10 percent within the next decade.<\/p>\n The exclamation point became its own subplot. On a recent episode of the Equity podcast, Anthony Ha, Kirsten Korosec, and Sean O\u2019Kane wrestled with the warnings. Ha defended the punctuation: if you genuinely believe that AI could destroy humanity, an exclamation point is a reasonable choice. His critique was aimed at something else. The word \u201cwe,\u201d he argued, was doing a lot of work. Whose beliefs were being represented? The alignment lead\u2019s tweet framed a highly contested prediction as a professional consensus, when every part of it was disputed \u2014 the odds, the timeline, and the chain of events that could turn a useful tool into an extinction event.<\/p>\n Jacob Coxon is an AI researcher who worked at OpenAI and then at Anthropic, two of the most important companies in the frontier AI race. His public resignation was not merely a personnel change. It was a direct challenge to the direction of the industry, and in particular to the development of self-improving systems that could revise their own code and behavior without meaningful oversight.<\/p>\n Coxon\u2019s actions carried a kind of credibility that statements from executives usually lack. While figures like Sam Altman and Dario Amodei often warn about existential risk while continuing to scale their companies, Coxon did something different. He put his professional trajectory where his message is. The \u201cwhy are you still doing this?\u201d objection that has trailed many AI leaders does not apply to him in the same way. He left.<\/p>\n P(doom) is an informal estimate of the probability that advanced AI will lead to the extinction of humanity. It is usually expressed as a percentage, but it is not a scientifically derived measurement. The concept began in online AI safety communities as a way for individuals to state their own level of concern, and it has since leaked into the broader conversation. Anthropic\u2019s alignment lead was effectively sharing his own P(doom) when he said the chance was greater than 10 percent within ten years.<\/p>\n For some researchers, P(doom) is useful because it forces specificity. For skeptics, it is exactly the wrong kind of specificity. Ha said on Equity that the \u201cgreater than 10 percent\u201d figure was a made-up number. It is not based on experimental data, a statistical model, or a peer-reviewed framework. It is a guess, presented with the confidence of a measured probability. Ha acknowledged in retrospect that the alignment lead was probably alluding to P(doom), but he still thought the trend of throwing out percentages was silly.<\/p>\n The deeper problem with P(doom) is that it compresses a complicated web of technical, social, political, and human questions into a single scalar. The number cannot explain whether risk rises with open-source models, falls with new compute regulations, or changes when alignment research makes unexpected progress. It only communicates how worried the speaker is. And as the Anthropic episode showed, a number that is meant to clarify can quickly become a rhetorical cudgel.<\/p>\n That matters because the same people who throw out percentages are often the same people who insist the problem is too urgent for normal regulatory timelines. If confidence in the number is low, confidence in the policy response should be low as well. That does not mean existential risk is fake. It means the conversation needs to move from numerology to evidence.<\/p>\n The fight over AI doom is no longer confined to research blogs and X threads. Anthropic is preparing for an initial public offering, and its S-1 filing could become a matter of public record within weeks. That document is where a company must tell investors what could hurt the business. A senior employee\u2019s public statement that AI could kill all humans does not fit neatly into the usual list of risk factors.<\/p>\n Sean O\u2019Kane raised the stakes on Equity with a memorable image: junior lawyers scrambling to rewrite Anthropic\u2019s risk factors. What exactly should the filing say? Should it declare that Anthropic\u2019s official position includes a greater than 10 percent chance that the company develops something that could eradicate all of humanity, and that such an outcome would be \u201cmaterially bad for our business\u201d?<\/p>\n The sentence is absurd enough to be funny, but the underlying issue is not. Public companies already warn investors about wars, pandemics, cyberattacks, and climate change. An existential AI risk factor is strange but not legally impossible. The problem is that it cannot be substantiated the way other risk factors can. It is a philosophical position, not a measurement.<\/p>\n An S-1 is the securities filing a U.S. company submits before listing shares in an initial public offering. It includes detailed financial information, management discussions, and a risk factors section describing what could go wrong. Anthropic\u2019s S-1 matters because it will reveal, in carefully vetted legal language, how the company itself understands the danger of the technology it is pushing into the world.<\/p>\n Kirsten Korosec added a market interpretation. In a traditional investment environment, a company that says its product could kill everyone would scare investors away. But the AI market has not been traditional. A model that is dangerous enough to threaten humanity is, in a strange way, a model that is powerful enough to be worth billions. The danger becomes a sign of capability. Talking about catastrophic risk signals to investors that the people selling the technology believe their own marketing.<\/p>\n Ha pushed back on the idea that all of this is a conscious marketing ploy. Real concern and business interest are not mutually exclusive. People who spend years building AI want to believe their work is the most important on earth. An honest warning can also be a self-promotional act, even if that is not the intention. The two impulses live in the same person, sometimes in the same sentence.<\/p>\n There is also a live corporate context. The Equity episode was recorded before Anthropic\u2019s CEO, Dario Amodei, published his own plan for more cautious AI development. That timing shows how quickly an AI company\u2019s narrative moves: one day its alignment lead is declaring a real chance of human extinction<\/a>, and a few days later its CEO is publishing a roadmap meant to make the future sound manageable. Both messages cannot have equal weight, and an S-1 will force the company to pick one.<\/p>\nWho is Jacob Coxon?<\/h3>\n
What is P(doom) and why does it keep dominating AI debates?<\/h2>\n
The S-1 problem: what does extinction risk mean for an IPO?<\/h2>\n
What is an S-1 filing and why does Anthropic\u2019s matter?<\/h3>\n
The loss-of-control signal coming out of AI labs<\/h2>\n