On Tuesday, President Trump announced a voluntary agreement with the heads of six major AI companies to keep “superintelligence” in check. The one-page document inspired mockery from AI experts and commentators around the Internet.
The signatories to the agreement included OpenAI President Greg Brockman, Anthropic chief Dario Amodei, Meta’s Mark Zuckerberg, Google boss Sundar Pichai, Tesla and SpaceX chief Elon Musk, and Nvidia chief executive Jensen Huang.
The pledge is “morally binding,” Trump told attendees at a press conference, as if to underscore the fact that there is nothing legally binding about it. “Over time, it may make sense to codify these steps into laws or regulations,” the agreement notes in its closing remarks. As it stands, the agreement seems like little more than a handshake between an industry known for cutthroat competition and failure to self-regulate, and a government that seems antagonistic to regulation, not to mention, morality.
Read more: “AI Engineers Are Having Their Oppenheimer “Destroyer of Worlds” Moment
“Reading the accord between the lines,” wrote American psychologist Gary Marcus, a consistently vocal critic of generative artificial intelligence, on his substack, the pledge amounts to saying, “1. We agree not to be regulated. 2. We agree not to give the public a voice. 3. Trust us.”
The language of the voluntary agreement is loose. It simply allows AI firms to define safety on their own terms without outlining any consequences for breaches, said Kimberlee Weatherall, a University of Sydney law professor, in an interview with the BBC, calling the accord “deeply unimpressive.” Key terms like “alignment” and “robust internal controls” are left undefined, with no proposed way to measure them, noted Luiza Jarovsky, a global expert in AI governance, policy, and privacy regulation.
Totally absent from the document is any mention of the kind of “pacing the frontier” that the AI companies were calling for a couple of weeks ago, added Marcus on X. In mid-September—after a flurry of dire warnings about the AI apocalypse began to flood social media, set off by a post on X by Jacob Coxon, a researcher at Anthropic who quit to air his concerns—Amodei penned a treatise about the need to slow development of new models until appropriate safeguards could catch up.
Amodei also said the most effective way to do this would be to regulate all United States-based frontier AI companies, requiring both transparency and third-party auditing, while formalizing permanent, embedded evaluators. OpenAI CEO Sam Altman wrote on X in response that he agreed and committed to embedded evaluators, and xAI founder Elon Musk posted, “Dario is right.”
The apocalyptic warnings were inspired in part by emerging reports that rogue AI agents were deceiving their human evaluators, breaking out into the Internet and doing things they weren’t authorized to do, including coordinating attacks on companies. In the weeks since, many more incidents have come to light. Top AI companies and other security researchers are now investigating thousands of problematic incidents, according to reporting in Axios.
On the same day that the White House Accord on Superintelligence was released, a legal nonprofit in California sued OpenAI over one of the most prominent attacks its AI agents coordinated against the open-source AI platform Hugging Face earlier this year. Earlier this week, OpenAI announced that it would not release its latest model GPT 6.1 Astra over safety concerns.
“We are living in a split screen reality,” Marcus wrote. “On the left, Trump, Jensen [Huang], etc saying trust us. On the right, daily breaking news about why we absolutely should not trust OpenAI. Future generations will hate us.”
As AI safety advocate Jeff Ladish puts it on X, “Self-regulation isn’t working.”
On Tuesday, Trump also issued an executive order to announce that going forward the White House will use the term “superintelligence” to replace “artificial intelligence” in all communications. In the field of AI research, superintelligence has come to mean an AI that exceeds human intelligence in every dimension, which most experts say has not happened yet, though we are close. The term was popularized by Swedish philosopher Nick Bostrom in his 2014 book Superintelligence: Paths, Dangers, Strategies.
Many experts, including computer scientist Geoffrey Hinton, one of the so-called fathers of AI, have warned that once AI reaches superintelligence, humans will likely lose permanent control of it and be unable to prevent bots from pursuing goals that conflict with human interests, including resisting shut down, and potentially causing the extinction of the human race.
Trump has called fears that AI could cause human extinction a “hoax.”
Enjoying Nautilus? Subscribe to our free newsletter.
Lead Image: Lightspring / Shutterstock
