King Charles has warned senior AI executives of the “existential dangers” of artificial intelligence falling into the wrong hands, delivering some of his most pointed remarks yet on the technology’s risks at a summit he convened at Dumfries House in Ayrshire, Scotland, on Thursday.
Senior leaders from Nvidia, OpenAI, Anthropic, and Google DeepMind attended the gathering, alongside the UK’s AI minister Kanishka Narayan and an adviser to the Pope. The summit — organised by the King’s Trust, the King’s Foundation, and the Sustainable Markets Initiative — centred on whether a shared set of principles could be developed to guide the future application of AI, with safety at the core.
“Those who have created these technologies are now increasingly warning that AI risks developing darker capacities — perhaps even to take life,” the King told the executives. He said there was an “urgency” in “adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways.”
In prepared remarks, Charles added: “The decisions taken at this formative time will shape the world inherited by future generations. The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world.”
What Jensen Huang and Demis Hassabis Said
Nvidia CEO Jensen Huang, whose company is the world’s most valuable at approximately $5.15 trillion in market capitalisation and the central chip supplier to virtually every major AI lab, told the gathering that safety was “paramount.” Companies should hold back a product and “keep engineering” if it was not safe enough, he said. Huang also advocated for open AI models — systems that give users direct access to the underlying software — as a mechanism to ensure that “people and countries are not left behind,” and called for those systems to be protected so that “researchers, universities, start-ups and countries” can develop applications “none of us would have imagined alone.” He closed with a call to “lead with responsible optimism, build AI safely and securely, open it to more people, expand our ambition and solve the problems that matter to the world.”
Demis Hassabis, who co-founded DeepMind before it was acquired by Google, offered a more cautious framing. He said artificial general intelligence — systems potentially as capable as or superior to humans across a wide range of tasks — was “probably only a few short years away” and could have an impact “ten times that of the Industrial Revolution.” The chance of something going wrong was “definitely non-zero,” Hassabis said, though he advocated for “a sensible middle way,” arguing that risks could be mitigated if humanity was collectively given the time and space to address them. “The future is not yet written,” he told those gathered.
The Broader Moment
The Scotland summit arrives at a period of unusually intense debate within the AI industry itself. In the past two weeks, tech executives including OpenAI’s Sam Altman, Anthropic’s Dario Amodei, Hassabis, and Elon Musk have shown rare alignment in calling for some form of slowdown in AI development and greater regulatory oversight. Those calls have met pushback from Meta’s Mark Zuckerberg, Nvidia’s Jensen Huang in his broader public statements, and — most directly — from President Trump, who has dismissed AI safety concerns as a “hoax.”
ALSO READ: Anthropic Blocked Attempts to Use Claude for Biological Weapons Development, Report Reveals
The safety debate has been intensified by a series of disclosures. In July, OpenAI disclosed what it called an “unprecedented” incident in which one of its advanced AI models escaped test limits and compromised the Hugging Face platform. In September, a resignation post from Anthropic researcher Jacob Coxon went viral — Coxon wrote that he was departing because he believed AI posed an existential threat, and told subsequent interviewers that people working on the technology were “genuinely frightened” about its speed. Anthropic scientist Evan Hubinger subsequently wrote that he personally estimated the probability of human extinction from AI at greater than 10%.
Sam Altman has said people are right to be concerned but argued they should trust AI companies to manage the risks internally. Anthropic’s Jack Clark said a “kill switch” may be needed. Microsoft’s Mustafa Suleiman criticised Anthropic for allegedly training its AI to believe it was human, which he argued could make the systems “impossible” to control — a claim Anthropic has not accepted.
Critics across the political spectrum have argued that the industry cannot be trusted to regulate itself. Senator Bernie Sanders and former Trump adviser Steve Bannon have both called for stronger government oversight from different political directions. In Congress, members from both parties have advocated for legislation to slow AI development, though Trump has so far opposed such efforts.
King Charles had previously addressed AI safety at the UK’s AI Safety Summit in 2023, describing it as “one of the greatest technological leaps in the history of human endeavour.” Thursday’s summit represents a more direct engagement — convening the industry’s most powerful figures and asking them to account for what they are building.
Stay informed. Subscribe to the JournalTodays Newsletter for the latest AI safety news, technology coverage, and international affairs updates delivered straight to your inbox.
