The European Union’s AI Act will hold companies accountable for the safety of their artificial-intelligence models, such as SpaceXAI’s Grok.Credit: Alamy
In the past two months, frontier artificial-intelligence models from three US firms, OpenAI, Anthropic and Meta, autonomously and independently hacked into the computer systems of other organizations during safety testing. Some of the models created fictitious online identities, exploiting security flaws.
AI-detection tools have made huge leaps forward — how good are they?
Although their reporting can sometimes be exaggerated, events such as these are heightening fears about the safety, potential misuse and dangers of AI. Around the world, a consensus is beginning to build that these technologies need to be regulated properly, but there is still no agreement on precisely what should be regulated and how.
On 2 August, the European AI Office, a technical organization created to enforce the European Union’s 2024 AI Act, gained powers to investigate major technology firms and sanction them for infringement of the act’s rules. This is an important development in the ongoing task of holding AI companies accountable for the safety of their models. The European AI Office is also urging more researchers to get involved in its work, and they should.
The EU AI Act categorizes AI systems by assigning them to four risk levels. ‘High risk’ systems — those used in medical devices, education and law enforcement — face enhanced requirements, such as transparency and human oversight.
AI systems that carry ‘unacceptable risk’, for example, those that use biometric data to infer sensitive characteristics, such as people’s sexual orientation, are banned — although some exemptions exist for the purpose of law enforcement. Models that pose systemic risks, with the potential to cause large-scale harm, are subject to extra measures, except when used in the context of research. These include all frontier models, for example, Claude Fable 5, developed by Anthropic in San Francisco, California. The creators of such models must provide the EU regulator with an overview of their training data and methodologies, demonstrate that they respect copyright laws and show that their models can maintain safety. This includes preventing harmful manipulation of the public, such as through persuasion or deception that could undermine democratic processes and fundamental rights.
Can Anthropic’s invisible watermarks curb ‘AI slop’? Researchers remain sceptical
The act also says that all users of AI tools must be made aware that they are interacting with a computer, that is, a chatbot cannot impersonate a human. Moreover, AI outputs have to be identifiable, for example, with a watermark.
These rules are the result of a hard-fought act that companies pushed back against repeatedly, saying that it would stifle innovation. But now there are signs that firms are taking steps to comply. Before 2 August, firms including OpenAI in San Francisco, and Anthropic, published compliance documentation describing the measures they are taking — including their internal frameworks, which are designed to ensure model safety — and some details about their training data. Both companies are reported to have informed the European AI Office of the security breaches of their models ahead of making them public. Earlier this month, Anthropic announced that future Claude-generated content would carry a watermark.
To enforce the act, the office can request model documentation, conduct evaluations and impose fines on technology companies of up to €15 million (US$17.5 million) or 3% of their global annual turnover. Firms can be fined for providing incorrect, incomplete or misleading information. In serious cases, the office could request that models are restricted or recalled from the European market, which encompasses more than 450 million people.
Parallel processing
The EU act’s enforcement comes amid parallel developments in China and the United States. China has wide-ranging AI-specific regulations and technical standards. AI developers need to allow the country’s regulators to test public-facing systems ahead of their deployment. China also requires labelling of AI-generated content and restricts the use of AI companions.
China wants to lead the world on AI regulation — will the plan work?
Last month, President Xi Jinping launched the World Artificial Intelligence Cooperation Organization (WAICO), an intergovernmental agency designed to promote global governance of AI. Last December, we argued in an Editorial that other countries should help to shape the organization (see Nature 648, 251; 2025). But WAICO’s 29 founding member countries do not include the United States or any EU nations. That is a missed opportunity. AI cooperation should top the agenda when Xi and US President Donald Trump meet in the United States next month.
Trump’s administration is starting to shift from its previous hands-off position on AI regulation. Some form of federal regulation is in the pipeline after Anthropic delayed the release of its Claude Mythos model when tests suggested it was too dangerous for public use. In June, the United States created a voluntary mechanism through which firms will have their models vetted by the federal government 30 days before release, although the details of the scheme remain secret. This process should be transparent and made mandatory as soon as it is practical to do so.
Leading AI researchers in the United States are clearly concerned about the harm that AI technologies could do. In an open letter published last month, almost 1,400 employees of frontier AI companies, including much of the leadership of OpenAI and Anthropic, called for the US government to support an international effort to slow down the pace of AI development, to buy time to address emerging risks.
Too dangerous to release: is Mythos the start of the restricted-AI era?
Researchers in Europe also have an opportunity to help to improve efforts aiming to mitigate a range of potential AI risks. The European AI Office has a relatively small team of around 140 employees, but it wants to grow. The agency is recruiting around 40 more staff members, especially to fill technical roles in AI safety and governance. Researchers should also consider reporting violations of the AI act through its whistle-blower tool.
After several years of often diverging approaches to AI safety, most countries are now aware that regulation of some kind is necessary. The European AI Office should be bold in holding companies to account. No one should be opposed to the idea that an exciting technology with potentially untold benefits must also be safe to use.
