Cambridge mathematician Jacob Coxon announced that he was leaving Anthropic. He previously worked on the technical staff at OpenAI, is included in the official list of key contributors to GPT-4o, and also worked on model pretraining at Anthropic. After three years at both labs, Coxon came to the conclusion that neither company is acting responsibly and that both are heading straight toward self-improving superintelligence. He described this race as a game played with human lives.
People building artificial intelligence seriously believe the technology could destroy humanity before the end of the decade. Coxon stressed that this is not a marketing tactic.
“Executives and senior researchers soften their language in public, but in private they talk about being afraid. No other activity carries this kind of danger,” he wrote.
Why AI Labs Are Not Stopping
According to Coxon, many employees at OpenAI have not fully grasped the scale of the threat to civilization. At Anthropic, however, the threat is understood, but the company is not stepping out of the race to be first. The thinking there, he says, is that other companies will not act responsibly anyway.
“They want to be the first to get there,” the researcher believes.
He says systems more capable than humans will appear soon.
“They will be able to hack anything, rebuild any industry overnight, and build up real power and resources. Progress in these areas is already visible and is not slowing down,” Coxon wrote.
Will the Worst-Case Scenario Come True?
The researcher is counting on coordination between AI labs. Incidents such as the attack on Hugging Face, he says, have made agreements between US labs to slow down more realistic. But he does not believe a global race can be prevented without costly measures, including a temporary ban on expanding the capabilities of AI models.
Coxon is asking lab employees to imagine how the next few years could actually unfold.
“Are they ready to start training a system more capable than humans without having a rigorous understanding of its mind?” he asks.
Read Also: