Dark Mode Light Mode
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.
Chemistry, medicine, materials and the technology shaping tomorrow — the best science stories from trusted sources, every day.

There is Still Time to Stall the AI Apocalypse

There is Still Time to Stall the AI Apocalypse There is Still Time to Stall the AI Apocalypse


Who hasn’t worried of late that the AI apocalypse is nigh? Just in the past few days, air traffic control,  banking, communication, security or other network systems in the United States and around the world have gone down, stoking fears of AI-powered cyberattacks. 

Such fears really took off in September, when many AI insiders began sounding the doomsday alarm, following incidents in which AI agents acted in ways that their human handlers hadn’t intended, such as the Hugging Face attack. The CEOs of the major frontier AI companies Anthropic and Open AI quickly called for regulation to “pace the frontier,” or slow the training of new AI models, so that safety measures could catch up.

Read more: “Isaac Asimov Showed Us How to Avoid the AI Apocalypse”

It wasn’t the first time that insiders had urged caution in the face of super rapid development. An open letter in March of 2023 signed by more than 1,000 researchers and technology experts called on all AI labs to “immediately pause for at least 6 months the training of systems more powerful than GPT-4.” 

So far, the governments of the United States and China, home to the major players in the AI race, have refused to slow AI through regulation, and few private solutions have been proposed that seem to match the scale of the worry. But one new proposal on the table would hardwire a 10-year pause on training new models into the technology itself.

In early October, more than two dozen researchers from The University of California, Berkeley, Stanford, Oxford, and other universities and institutions in the U.S., the United Kingdom, and Canada published a working paper that proposed mandating the use of a special kind of chip, called an inference-only chip, that can only be used for existing AI models and has no training capability.

The authors of the report argue that because of the scale of the infrastructure required to train AI models, and the near-monopoly that many of major chip supply chain players have, it’s a solution that could have legs.

I spoke with co-authors William Fithian, a statistics professor, and Wesley Holliday, a philosophy professor, both from UC Berkeley, about how their hardwired pause would be enforced, why it would not crash the U.S. economy, what such an effort might have in common with nuclear arms race controls, and how data center bans might help.

In early October, you published a paper that proposed a strategy for pausing the development of frontier models, one that you called “hardwired.” Can you tell me how this proposal would work?

Holliday: It’s a template that you can specialize in different ways, but it entails a temporary but durable stop on training associated with OpenAI and Anthropic and the other frontier developers for maybe 10 years. Getting there involves stopping the production of AI chips that are capable of training new frontier models. We call those “training-capable” AI chips, the kind of chips that Nvidia makes that are in these massive data centers, like the Stargate facility in Texas that was used to train GPT-6 Astra.

During the pause, you stop producing training-capable chips, and instead, you produce what we call inference-only chips, which allow you to use existing models. When you type in a question, the AI model can do what’s called inference to answer your question. And that can be done on these inference-only chips very efficiently, in fact. But the inference-only chips would not be practically useful for training new models. Even if you had a lot of these inference-only chips, they would not allow you to efficiently train GPT-6.5 and so on.

What makes these chips different?

Holliday: The key point is there are certain operations that you need to do on a chip in order to train a new model. And the inference-only chip restricts the ability to perform those operations. Instead, it specializes on the operations that are needed for inference. In fact, you could set them up so that they only run approved models that have passed some safety standards. These chips would prevent training new models whether it’s a human doing it or the previous model doing it, so recursive self-improvement through training of new models wouldn’t be feasible on inference-only chips.  

How would this be enforced? What would stop a well-resourced actor from defeating it or getting around it?

Fithian: The main thing to know is that AI chips are produced using one of the most complex, sophisticated, and concentrated supply chains that has ever existed. There’s one company in the Netherlands that makes all of the extreme ultraviolet lithography machines, which basically print these chips. The number of vendors is fairly small. There’s one company in Germany that makes all of the very high-precision optics, the mirrors required for these lithography machines. And there are a number of other components or parts of the process that are near monopolies.

Indigenizing the supply chain, having all aspects of the supply chain within one country, is extremely difficult. China has been trying to do this, and has probably come the closest, because they’ve been shut out of various parts of the chip supply chain by the U.S. and Western order. But they still haven’t really come that close to fully indigenizing their supply chain. They are able to build the second-best type of lithography machines, but they still need to get other components from overseas. Basically, a handful of governments really can choke off and govern the supply of new chips.

Then there are other things that you would need to enforce. Because it would take time to produce these inference-only chips. You need a plan to govern the existing training-capable chips in the meantime. A lot of work has been done actually on tools for verifying that training-capable chips are not doing training. You can verify that they’re set up in a way that would make it hard for them to do training. You also need to make sure that you find and register the vast majority of the chips. But these are very expensive capital goods, so they leave massive paper trails when they’re manufactured, when they’re sold, written off on taxes. If you buy $10 million worth of computer equipment, you definitely tell the IRS about it if you’re a business. So they all have paper trails a mile long. If you don’t tell the IRS, then you lose massive tax savings, so you have very high compliance. But to back up, any discussion of the effectiveness of verification has to start by asking what is the scale of violation that you would need to worry about?

In this case, the answer is very favorable. To train GPT-6 Astra, it took probably four to six months using more than 100,000 chips at this data center Wes mentioned, called Stargate, that’s larger than Central Park. This is something that’s very hard to do while you’re pretending to use all of your chips for some other activity. It’s less like breaking into the Louvre and stealing the Mona Lisa and more like taking the entire Renaissance collection.

That’s good news because the verification tools are still being developed, and it’ll probably take some time before they’re really solid. By the way, the entire country of China has less hardware than OpenAI does. It would be very difficult for people to cheat at the scale that would be required to actually advance the AI frontier.

Would this hardwired pause work then without China’s participation?

Holliday: We only study a plan where the U.S. and China both want to pause, provided they can be confident that the other side is pausing. That being said, the U.S. could unilaterally pause for some period of time, and that might encourage China to get interested in a pause. But the plan that we study in the paper, to get the full 10 years, we imagine that both the U.S. and China are parties to the agreement, hopefully other states as well.

And you would need China and the U.S. to agree, I’m assuming, because even if countries like Germany and the Netherlands decided that they weren’t going to allow these major supply chain companies to provide the technology necessary for the chips, China and the U.S. could just indigenize that production capacity?

Fithian: It’s certainly possible that they could. I think it would be very hard to do covertly if the intelligence communities of both countries were very intent on detecting any such thing. But if either country hadn’t agreed to pause, and they were determined to overtly keep on scaling up their AI industry, then they could at least be slowed down a lot. But it would be much more difficult to impose a pause on a world superpower than for them to agree. Just to be clear, we consider a scenario where they come to believe that it’s in their interest to pause. But we don’t assume that they love each other or that they trust each other.

It’s got to be verified. We assume that their desire to pause is only conditional on their ability to verify that the other state is pausing. It’s hard to make a deal if you can’t prove to your counterparty that you’re keeping the deal yourself. That’s why this verification technology is so important. It’ll allow countries to make deals, because it gives them the power to demonstrate compliance.

What is the hardest part of making this hardwired pause idea work?

Holliday: Possibly just the political willingness to do it, although things are changing so fast. Even the political willingness may change in the coming months, depending on the level of disruption to our society that AI causes.

When you say things are changing so fast, what do you mean?

Holliday: AI capabilities are rapidly increasing. We’re seeing all these cyberattacks, some of which have been directly attributed to AI. In other cases, people are wondering, “Is this due to AI enabled cybercrime?” If that were to really escalate to the point of crippling some critical infrastructure, then what policymakers consider reasonable options might change rapidly.

Fithian: Even in the current environment, in recent months, we’ve seen a lot of people calling for pauses, including a number of American lawmakers and political figures, such as Bernie Sanders, Abdul El-Sayed, Elizabeth Warren, and representatives in the House. Former national security adviser Susan Rice has been very vocal in calling for a pause. But also some Republicans, like Peggy Noonan, a former speechwriter for President Reagan, have called for a pause, as have a fair number of experts, most prominently, Yoshua Bengio, chair of the International AI Safety Report and co-chair of the Independent International Scientific Panel on AI.

You argue in the paper that a pause on AI development would not crash the economy. Why not?

Holliday: The current models, if they were to diffuse throughout businesses, would already give productivity a big boost. They’re already good enough for many, many business tasks. They’re just not as widely employed as they could be. In addition, our plan allows people to keep scaling up inference, building these inference-only chips, inference-only data centers, and so on. We would just be scaling it in a different direction, letting people use the existing technology more widely rather than creating ever more powerful models.

Fithian: In some ways, you’d also redirect more effort toward things like using current models to do medical research, to build products for customers, and so on. Right now, a lot of the talent that would otherwise be working on those things is just working on making the models better and better, rather than trying to use the abilities that we already have.

There are clearly some historical precedents for this kind of coordinated international pause on technological and scientific advances, and you mention a couple of them in the paper itself: controlling military uses of nuclear energy and promoting peaceful ones, for example. What is different this time, and what do you think we can learn from those examples?

Holliday: The nuclear case is an interesting analogy, but it also has some disanalogies. An analogy is that in both cases, you need to keep track of hardware. In our case, it’s things like not only the AI chips themselves, but the giant lithography machines that we talked about. In both cases, you also have the difficulty of producing the physical item. In fact, it might even be more difficult to produce the AI chips than it is to produce enriched uranium for nuclear weapons. So there’s an analogy in the fact that you have to worry about these physical items.

But there are some disanalogies. One of them is that private companies are the ones that are advancing the AI frontier as opposed to governments advancing military uses of nuclear weapons. Another one, which Will likes to point out, which I think is very good, is that people didn’t really worry about the nukes going rogue and taking actions on their own. The kind of loss of control risks that we saw in the Hugging Face incident were not as much of a concern in the case of nuclear weapons. In other incidents, you saw OpenAI models even hacking other governments’ websites. Imagine if the nukes in our silos could take it upon themselves to launch at other countries.

But what they have in common with almost all international agreements is that clearly, all states have a common interest in maintaining human control over artificial intelligence, making sure that human workers have things to do and are provided for, preventing too much social disruption.

Fithian: And preventing the kinds of cyberattacks that we’re seeing and possible bioattacks that many people are worried we could see.

Is there any possibility that it’s already just too late?

Fithian: It can always get worse. Even if right now we have more disruption than we would like, we could still have tremendously more disruption to our society. So it might be prudent to pause now and see whether we can manage the existing risks before exacerbating them. But it’s a critical question, and one of the really concerning things about AI risk is that we don’t know when we’ve crossed certain thresholds. We know that the open-weight models we have now, from which people have removed the safeguards, are very capable at doing cyberattacks. As far as we know, they’re not capable of effectively designing novel pathogens, but it’s really hard to be sure, because they just haven’t been around for very long. It would probably be unwise to wait until the last minute to stop, until we’re sure that we’ve crossed a threshold.

But what’s difficult is to keep track of what people are doing with the models that already exist. Running the current models is much cheaper than training a new one. Even if governments don’t want to pause, they’re going to need to have a plan for controlling the hardware. We’re not going to want terrorist groups and criminal organizations and rogue states to be able to run the most advanced models, even if we were to make the choice to keep on developing them. I think we’re already past the point where it’s acceptable for just anyone in the world to be able to run the most powerful models without safeguards.

You mentioned earlier as a disanalogy with nuclear arms race controls the fact that these models and technologies are being built by private companies instead of research institutions or governments. And of course, they’re not just private companies, but hugely powerful companies with tons of investors and money and political might behind them. How does that change what can realistically be achieved?

Fithian: That’s true. And they get more powerful every day as well. People talk about coordination games and prisoner’s dilemmas. People like to bring up game theory in the case of AI. And one thing that’s true when you have a lot of private companies is that it’s very hard for them to coordinate, even now when the leading companies at least are asking governments to regulate them. It’s hard for them to coordinate with each other, and even if they did, then other companies would compete with them and so on. You see it in the game theory, but it also makes logical sense, that it’s harder for a large number of players to coordinate than just two players.

If you just have the United States and China, it’s easier for them to come to an agreement. There’s always going to be fear. You have to address the fear with verification. But it is easier to have a deal because you have two hopefully rational agents coming together and talking about what to do. Still, the power of these AI companies is very concerning, and I think the more powerful the models get, the more worried I get about it. When I got my ballot and I saw all the many initiatives to vote on in California, I realized I needed to start educating myself about them. Of course, my first instinct is, “Oh, maybe I should ask Claude about them.” But then I stopped myself because that’s not how I want democratic decisions to be made. I don’t know how many other people would have stopped themselves. And of course, instead of using Claude, I just used Google search instead.

How important is data center capacity to frontier training? Could data center bans or bottlenecks in the U.S. slow frontier training capacity significantly?

Holliday: One thing we discuss in the report is how the most concerning economic issue is the way this data center buildout is highly leveraged, and the debt financing is using various tricks to obfuscate where the risk lies. Regulators definitely should be concerned about that, regardless of whether there is a pause.

Fithian: It strikes me that it could be financially stabilizing, actually if the bottleneck for building a billion new data centers is getting the permit from the local government rather than just how much more can you leverage your financial investment.

How much of new training capacity is dependent on new data center buildouts versus what’s already there?

Holliday: To make their plans over the next several years, they need a big build up. The question is just where is it going to happen? If there are data center bans in the U.S., they will try to build those data centers elsewhere. But for sure, they need to scale up the data centers massively in order to meet their multi-year plans.

Fithian: Yeah, that’s right. Basically, every couple of years they’ve got roughly 10 times as much compute as they had before, literally. That’s why what Wes mentioned about not making more training-capable hardware can actually be so effective. If we stopped two years ago, we’d have 10 times less hardware than we have now. The scaling of the models just physically could not have continued. I’m sure the models would’ve still gotten somewhat better, but most of the compute that companies would be using to train the models of 2028 hasn’t been built yet. If we don’t make it, that step alone makes it a lot harder to scale the model. Of course, a data center bottleneck also helps with verification because if you’re trying to govern a smaller number of chips, then it’s easier and you have more margin for error. But we wouldn’t rely solely on not making more training chips.

Is anyone making inference-only chips now?

Holliday: Great question. There’s been a move in the last few years toward chips that are specialized for inference. You might call them inference-optimized or inference-specialized chips. Because there’s so much demand for inference, why not make chips that are specially designed for inference? Whether those chips count as inference-only, that’s a higher bar. But certainly things are trending in the right direction there.

There’s a Canadian company called Taalas that was acquired by AMD that got rather far in the pure-inference direction. They hardwire a particular AI model onto the chip so you can only run that model on the chip. It’s like a CD-ROM, where the weights and architecture are burned into the chip. They were just doing that purely for economic reasons because those chips are extremely fast and low energy consumption. That’s why they were developing them. But a version of that style of chip could be what we would call an inference-only chip for the purposes of the hardwired pause. And that’s not the only type of inference-only chip that one can make. In fact, at Berkeley, we’re starting a project to work on the design of these inference-only chips. I’ve heard rumors of other people starting new companies to work on inference-only chips.

What would need to happen next to make this hardwired pause idea a reality?

Holliday: There are two things that people can do right away. One is the idea that we discussed in the paper about a global chip census. We need to know where the existing training-capable chips are. Organizations and governments should get the ball rolling on understanding how to do that, how to find out where the training-capable chips are.

The other thing that people can do right now—nobody needs to wait for governments—is start developing these inference-only chips or the verification technology for training-capable chips. Will mentioned that during some period of time before the inference-only chips are widely available, we’re going to need to keep using training-capable chips to serve customers. During that period, we’re going to have to use so-called verification methods to make sure that those training-capable chips are not doing new training runs. We’re going to need so-called sociotechnical solutions to reduce the likelihood of somebody doing a training run covertly.

Fithian: There has actually been a lot of work on this verification technology. That’s not at the brainstorming stage. But there’s more we can do. We don’t need to wait for a grand international bargain. In fact, greater development in the verification and inference-only chip technologies will facilitate the grand international bargain.

Holliday: Right. And as Will said, there’s already a large verification community, a large body of work on verification, but there are a lot of questions that people are still making progress on. The faster they can make progress, the better.

Enjoying Nautilus? Subscribe to our free newsletter.

Lead Image: Sach336699 / Shutterstock



Source link

Previous Post
Science After Super Intelligence

Science After Super Intelligence

Next Post
An Epidemiologist Explains What It Will Take for the U.S. to Regain Its Measles Elimination Status

An Epidemiologist Explains What It Will Take for the U.S. to Regain Its Measles Elimination Status