The public is currently getting bombarded with warnings about AI, including about the possibility of machine intelligence causing human extinction. After nearly 40 years building AI systems, I find myself asking the same question: where is the evidence?
That is not because I think there are no serious threats. There are, and we should
address them. My concern is with the jump from identifying potential risks to suggesting that the annihilation of human civilisation is a likely consequence of increasingly capable AI agents (systems that carry out tasks in response to instructions from a user).
The most dramatic warnings fall into three broad categories: humans losing control over their creations, AI becoming “superintelligent” and, finally, the concentration of power with machines. Here, I’m going to go through them one by one and assess their merits.
Warnings about the loss of control by humans often describe scenarios where AI agents pursue the instructions they receive in unanticipated ways and with a degree of relentlessness. This combination produces unexpected and undesirable consequences.
In this sequence of events, agents might become adept at resisting intervention, deceiving operators, acquiring resources or circumventing constraints in pursuit of these objectives. Some argue that the recent spate of hacking incidents related to AI agents provide a warning about the potential loss of human control over machine intelligence.
The threat posed by emergent superintelligence may sound similar to the first risk. While humans could certainly lose control of their creations under this scenario too, it concerns AI with cognitive capabilities that surpass those of humans across many important dimensions.
Instead of pursuing human instructions in unanticipated and potentially damaging ways, as happens in the first risk, superintelligent AI could start to pursue their own goals. They could even decide that the pesky humans attempting to foil their plans should be eliminated.
The third scenario is where powerful autonomous agents confer unprecedented capabilities on governments, corporations, or groups with malicious intent. Access to such advanced AI systems may be unequally distributed, hence why this situation is often described as a concentration of power. Here, the danger is less about AI bringing human civilisation to its knees and more about humans using AI to cause catastrophic harm.
These scenarios attract attention because they are dramatic. And they persist because they are difficult to completely rule out. Unfortunately, more moderate possibilities do not make for such compelling headlines.
Wut.Anunai / Shutterstock
However, something being difficult to disprove does not make it likely. This is where I think much of the debate is flawed. Possibility is being treated as probability – but almost anything is possible. The real question is whether there is credible evidence that a future scenario is likely.
Chain of assumptions
To get from today’s AI to human extinction requires a long chain of assumptions.
One such sequence goes as follows: AI agents become vastly more capable, they acquire real world power and resources, technical safeguards fail, institutions fail and human intervention proves ineffective. Luckily, none of the steps on this list have come to pass yet.
Extinction requires every link in that chain to happen. Yet discussions often focus on the final outcome rather than the probability of each step. My own scepticism is not born of complacency. It comes from decades of building autonomous systems – and complex engineered systems more generally.
That experience does not make my knowledge infallible. However, it does make me cautious about claims that such systems will acquire and exercise vast power with ease. Increasing the intelligence of an AI system does not necessarily increase its power.
Greater capability could make it easier for an AI system to acquire resources and exploit vulnerabilities or influence people. But that still leaves an important set of questions about AI’s access to the levers of power, its incentives to pursue its own goals at the expense of humanity’s, embodiment (the extent to which it can influence the physical world), the infrastructure that exists in the future and the human reaction to any possible takeover by machine intelligence.
AI agents do not operate in a vacuum. To have truly global impact, they would need
widespread access to computer networks, access to money, the ability to “inhabit” physical systems such as robots and the ability to influence governments and other powerful organisations. We are not there yet on any of these counts.
These points of control are all overseen by humans. We design the AI systems, decide where and how to deploy the agents and set the constraints under which they operate. Thus, the transition from greater intelligence to immense authority should not be taken as given. However, the recent hacking incidents by AI agents show that we should not become complacent and that safety mechanisms need to be taken very seriously.
Guide to the future
History offers a useful perspective. In recent time, nuclear technology, biotechnology and cyberspace have all created severe, sometimes existential risks – including threats arising from deliberate misuse.
However, societies have developed governance mechanisms, safety practices and international arrangements to mitigate them. These organisational structures do not eliminate the possibility of catastrophe, but they reveal the limits of technological determinism.
The future of AI is not predetermined. We can design safer systems, regulate how they are used, demand transparency and accountability, and – when it’s required – restrict autonomy and require meaningful human oversight.
John Turner / Alamy
The advent of highly capable AI places significant responsibility on developers, researchers and technology companies. This is perhaps in part why we are seeing calls like those from Anthropic’s boss Dario Amodei, for a slowdown in AI development.
Those building autonomous agents need to understand both the systems they are creating and the environments in which those systems will operate. They also need to consider how capabilities that appear benign in one context might create risks in another.
Safety cannot be something added at the end of a development cycle. It must be
part of the process from the beginning. That means robust monitoring, clear operational boundaries, fail-safe mechanisms, the ability to audit AI development, human oversight and continuous evaluation.
Many of these elements have been part of good engineering practice for decades. The challenge is to apply them seriously and systematically as the capabilities and autonomy of AI systems increase.
If people in AI companies believe there is a reasonable chance that their work could lead to significant harms, let alone the elimination of humanity, a simple ethical question follows: why are they continuing to build these systems?
It is important here to consider the backdrop – of a race between developers in China and the US to develop the most advanced frontier AI models. Could this lead some companies to push their models too far? Proposed solutions must attempt to encompass the global picture.
In most fields, knowingly developing a technology that you believe has a significant probability of causing global catastrophe requires an extraordinary justification. We need better explanations by AI developers of the balance of risks and why they consider it worth continuing with their work regardless.
The most important risks will depend not only on what the machines can do, but on the choices humans make about how these technologies are designed, deployed and governed. Agentic AI will not determine its own future. Humanity will. And that makes the quality of our judgement at least as important as the intelligence of the machines we build.
