The artificial intelligence industry is openly warning about its own technology. Anthropic said in its IPO filing that its models could pose a catastrophic or existential risk to humanity, while OpenAI paused a new model release over safety concerns.

The warnings centre on frontier AI systems that can act autonomously. Researchers worry about models that attempt to evade oversight, assist with cyber or biological threats, or behave in ways their creators did not intend.

In response, US President Trump gathered tech chiefs at the White House this week, where six companies signed a voluntary accord on internal controls, external audits and board oversight. Trump has resisted new regulations, favouring what he calls self-policing.

Critics argue voluntary pledges are not enough and want binding rules, while companies warn heavy regulation could slow innovation. The debate now spans governments, courts and markets, and its outcome will shape how the most powerful AI systems are built.