Philosopher Nick Bostrom is surprisingly cheerful for someone who has spent so much time worrying about ways that humanity might destroy itself. In photographs he often looks deadly serious, perhaps appropriately haunted by the existential dangers roaming around his brain. When we talk over Zoom, he looks relaxed and is smiling.
Bostrom has made it his life’s work to ponder far-off technological advancement and existential risks to humanity. With the publication of his last book, Superintelligence: Paths, Dangers, Strategies, in 2014, Bostrom drew public attention to what was then a fringe idea—that AI would advance to a point where it might turn against and delete humanity.
To many in and outside of AI research the idea seemed fanciful, but influential figures including Elon Musk cited Bostrom’s writing. The book set a strand of apocalyptic worry about AI smoldering that recently flared up following the arrival of ChatGPT. Concern about AI risk is not just mainstream but also a theme within government AI policy circles.
Bostrom’s new book takes a very different tack. Rather than play the doomy hits, Deep Utopia: Life and Meaning in a Solved World, considers a future in which humanity has successfully developed superintelligent machines but averted disaster. All disease has been ended and humans can live indefinitely in infinite abundance. Bostrom’s book examines what meaning there would be in life inside a techno-utopia, and asks if it might be rather hollow. He spoke with WIRED over Zoom, in a conversation that has been lightly edited for length and clarity.
Will Knight: Why switch from writing about superintelligent AI threatening humanity to considering a future in which it’s used to do good?
Nick Bostrom: The various things that could go wrong with the development of AI are now receiving a lot more attention. It’s a big shift in the last 10 years. Now all the leading frontier AI labs have research groups trying to develop scalable alignment methods. And in the last couple of years also, we see political leaders starting to pay attention to AI.
There hasn’t yet been a commensurate increase in depth and sophistication in terms of thinking of where things go if we don’t fall into one of these pits. Thinking has been quite superficial on the topic.
When you wrote Superintelligence, few would have expected existential AI risks to become a mainstream debate so quickly. Will we need to worry about the problems in your new book sooner than people might think?
As we start to see automation roll out, assuming progress continues, then I think these conversations will start to happen and eventually deepen.
Social companion applications will become increasingly prominent. People will have all sorts of different views and it’s a great place to maybe have a little culture war. It could be great for people who couldn’t find fulfillment in ordinary life but what if there is a segment of the population that takes pleasure in being abusive to them?
In the political and information spheres we could see the use of AI in political campaigns, marketing, automated propaganda systems. But if we have a sufficient level of wisdom these things could really amplify our ability to sort of be constructive democratic citizens, with individual advice explaining what policy proposals mean for you. There will be a whole bunch of dynamics for society.
Would a future in which AI has solved many problems, like climate change, disease, and the need to work, really be so bad?
Interactive therapeutic robot Paro keeps a resident company at a nursing home in Japan.Credit: Noriko Hayashi/Panos Pictures
Clara Berridge, an ethicist at the University of Washington, Seattle, recalls a story told by a colleague to a group of health-care and social-work students.
Nature Outlook: Robotics and artificial intelligence
An older man in a nursing home was given a robot that looked like a stuffed animal for companionship. He became attached to it and when he later fell ill and died, the nursing-home staff found him clutching his robot companion.
When the class was asked to offer impressions of the scenario they were split: either they thought it was beautiful that he wasn’t alone in his last moments, or they felt it was tragic to die without a human connection.
Robots are an increasingly popular form of therapy for older people with dementia. It’s been suggested that social robots, on which much of the research has been based, can improve people’s moods, increase social interaction, reduce symptoms of dementia and give carers some much-needed relief.
But some researchers are beginning to question if these devices are ready for widespread use with this population. The research proving robots’ worth is sparse, and there are ethical concerns — especially around the idea that their use might reduce human contact in a population that is dearly in need of it.
“If we’re going to invest resources in elder care, I want more staff in the facility so they don’t die alone,” Berridge says. Her grandmother passed away on her own in an understaffed nursing home during the COVID-19 pandemic. Berridge’s grandmother didn’t have the option of a robot companion, but Berridge would not have wanted that for her anyway. “There are so many other things I would choose for her before a robot,” she says.
Early adopters
The robots being used in therapy for older people and people with early cognitive decline fall into two broad categories: service robots and social robots. Service robots are designed to help people in their daily lives, such as by assisting with household tasks or mobility. Developing a robotic assistant that can navigate the home and safely interact with people and objects in its environment remains technologically challenging, however.
Paro robots, shown charging, are among the most common examples of social robots.Credit: FRANCK ROBICHON/EPA-EFE/Shutterstock
As a result, social robots intended to offer companionship and provide cognitive stimulation are a more common sight in care settings for older people. Some are humanoid in design and intended to act as ‘intelligent’ companions, holding rudimentary conversations and leading games and activities. Other social robots aim to mimic pets that can respond in some way to a person’s voice and touch.
Animals (often dogs) have been used in care settings to help residents to become more social and less agitated, and to improve their quality of life. But animals require a lot of care, whereas a robot pet does not. The most common example of this kind of social robot is Paro. Rather than a dog, the robot, which was designed in Japan, looks like a baby harp seal. A seal was chosen because it would be familiar and approachable, but it is not a common pet so a person would not immediately spot differences between its behaviour and that of the real animal. Paro is typically used to provide a form of pet therapy to older adults in assisted-living facilities, offering companionship and encouraging interaction between residents of the facilities and staff during therapy sessions.
Lillian Hung, creator of the Innovation in Dementia care and Aging (IDEA) lab at the University of British Columbia in Vancouver, Canada, purchased the furry bot in 2017 to use with people with dementia who had been admitted to Vancouver General Hospital. Initially, she used it in group therapy for people with dementia, and to help people who were reticent to talk during admission and discharge. But over time, Hung found more uses for Paro.
In one case, the robot came to the aid of a patient in accident and emergency who was hitting staff who came near him, and who kicked a laboratory technician trying to take a sample of his blood. “He had a cardiac condition that needed diagnostics and we had two choices: physically or chemically restrain him, or leave him alone,” Hung says. “Both options were not good.”
Instead, Hung placed Paro in the man’s lap. Paro turned its head as if waking from a nap, opened its eyes, and looked up at the patient. The man asked the robot if it had eaten lately. When Paro started moving around, the man began petting it. While he was engaged with Paro, the staff were able to perform the tests that they needed.
“I hadn’t planned to use the robot for that reason, but in the moment it was useful,” Hung says. “The patient had quality care and safety, and the staff were able to get their work done.”
In 2019, Hung reviewed 29 studies of Paro’s use in older-person care settings around the world with people with dementia1. She found three main benefits of the bot: reduced negative emotions and behaviours among patients, better social engagement and improved mood and care experience. “For an older person who is frail and struggles with language, the robot doesn’t judge,” Hung says. “It offers an unconditional presence. Regardless of what they say, it is always happy to listen.”
Creators of other social robots think that they could also be beneficial for this population. In 2020, Mohammad Mahoor, an electrical and computer engineer at the University of Denver in Colorado, built the third iteration of a humanoid companion robot he calls Ryan, which he began working on in 2013. The robot can recognize speech and facial expressions, and is designed to help reduce social isolation among people with early-stage dementia or depression by engaging them in conversation. Ryan can also remind people to take medications and can lead mental and physical games.
“Mostly these people live alone, their mood is down. We want to improve their quality of life,” Mahoor says. “When you engage residents, they are happier and their family members are more satisfied.”
Mahoor has carried out research in assisted-living facilities with Ryan. In one study, six older people with early cognitive decline were given around-the-clock access to Ryan for 4–6 weeks2. The participants reported enjoying interactions and conversations with Ryan and feeling happier when it was there. However, they did not report feeling less depressed after talking to the robot, and said it was not the same as talking to a real person — a distinction Mahoor understands. “We’re not replacing human interaction, just filling in the gaps,” he says.
Ryan is currently being used in two assisted-living facilities near Denver. The robot stays in the common area. Residents have a card that they can tap on it to spend 30 minutes each day talking, playing games or doing other activities.
Arshia Khan, a computer scientist at the University of Minnesota Duluth, is also working to show that robots can improve the quality of life of people with dementia. Humanoid companion robots, she says, can engage and stimulate people and reduce levels of anxiety and depression. They can also provide respite for carers by leading bingo sessions and playing games with residents in assisted-living facilities.
In one study, Khan and her colleagues placed Pepper and NAO — two humanoid robots built by the firm Softbank Robotics in Tokyo, and then specially programmed by Khan — in eight nursing homes in Minnesota. Surveys were conducted before and after implementing the robots. Compared with nursing homes that didn’t deploy robots, residents of facilities that did felt happier, more cared for, and less tired and frustrated after engaging with the robots3.
Cautious attitudes
Hung expects some resistance from carers to the use of robots. “Not everyone is ready to have robots,” she says. “When we did interviews with organizational leaders, they said money wasn’t the issue — their staff weren’t willing to work with robots.” While running a focus group at one care home, Hung and her colleagues returned from lunch to find the robot that they had brought with them not only unplugged, but wearing a paper bag over its head. “They were worried it was secretly recording them,” she says.
Companion robot Ryan is currently being used in two assisted-living facilities in the United States.Credit: Loclyz
Some older people also have concerns. In a 2023 study led by Berridge, 29 people living with mild Alzheimer’s disease were asked how they felt about robots and other assistive technologies4. Their key concerns were privacy — they wanted to know if the technology was monitoring them — and loss of human connection. Most participants said that they would prefer visits and phone calls from friends and family, or social outings and activities, over what a robot could offer.
In a separate survey of adults who were generally tech-savvy, respondents described social robots as “creepy,” “manipulative” and “unethical”, and said that they offer only the illusion of intimacy5. Most thought that an artificial companion would not make them feel less lonely (see ‘A frosty reception’).
Ref. 4
They were also uncomfortable with the idea that a carer might let them believe that a robot was a real person if they lacked the cognitive ability to know for themselves. Berridge says that this is an issue on which ethicists are split. Some think that if the belief soothes people, then the deception, intentional or not, shouldn’t matter. Others see it as potentially taking advantage of extremely vulnerable people. “Concerns consistently arise over the possibility of withdrawing human interaction and touch, dishonesty, and potential for diminished dignity, which philosophy-trained ethicists will tell you needs to be protected — even and especially when people lack autonomy,” Berridge says.
Uncertain benefits
In addition to concerns that older people and their carers might not be comfortable with social robots, there are also questions about the utility of these devices.
Some users with cognitive decline showed higher stress levels after using the communication robot Chapit.Credit: Tomohiro Ohsumi/Getty Images
In a 2022 meta-analysis of 66 studies of companion robots6, ‘telepresence’ communication robots, assistive robots and multifunctional robots being used to support people with dementia, Clare Yu, who studies dementia prevention at University College London, and her colleagues found that many of the robots were generally liked by study participants and could feasibly be used in a nursing home. And most studies reported that the robots did what the authors anticipated: relieved loneliness and isolation, reduced anxiety, and improved quality of life. But the researchers noted significant difficulties as well. Paro, for instance, is heavy, expensive and noisy; humanoid companion robots tend to have speech-recognition issues; and telepresence and multifunction robots were difficult to use.
Yu and her colleagues also don’t think that the design of these studies was sufficient to provide compelling evidence of benefit to people with dementia. According to Yu, many studies didn’t compare robots with other forms of care, such as human interventions. Sample sizes were often too small to draw conclusions, some studies didn’t use well-validated outcomes, and many didn’t appropriately randomize their cohorts. As a result, despite seemingly positive results in many cases, Yu and her colleagues concluded that there was no clear evidence that robots improved people’s quality of life, cognition or behaviour.
“Before I did this meta-analysis, I was really excited,” Yu says. “I thought robots were something that could be used in the future for people with dementia.” She now has serious doubts. “I think they are something that can be used in the future, but not at this present moment. We need some time to do more research to be able to say they are definitely beneficial.”
Another 2022 review7 that analysed nine studies of Paro suggested that the seal robot could improve quality of life for people living with dementia and reduce their use of medications. However, the authors similarly tempered their conclusion by noting that the studies that they analysed were mostly of low to moderate quality, meaning the authors were “cautious to make positive comments on the role of Paro”.
A 2020 study8 from a team of researchers in Japan went so far as to suggest that communication robots might be detrimental for some people with dementia. Twenty-eight older people, 11 of whom had cognitive decline, received sessions with Chapit, a stuffed-toy robot with speech recognition that can play games with users. Measurements of electroencephalogram (EEG) activity and salivary cortisol levels, taken before and after sessions, showed higher levels of stress among the participants with cognitive decline after using Chapit, but not among those in the group without cognitive decline. People with cognitive decline also reported not enjoying their time with Chapit, whereas people without cognitive decline did enjoy it — a finding that matched the EEG results.
A study last year9 from the same group used EEG activity to determine whether Chapit activated participants’ posterior cingulate gyrus and precuneus — parts of the brain that affect reflection, self-consciousness, imagination and prediction. In people without cognitive impairment, these areas were activated by the use of Chapit. In people with cognitive impairment, however, there was no significant change in brain activity.
The allure of technology
With efficacy being questioned, and signs of resistance among carers and prospective users, widespread adoption of robots in older-person care settings faces clear obstacles. “I don’t feel we are ready to have large-scale implementation,” Yu says.
Arshia Khan uses a robot to run cognitive-stimuation quizzes with care-home residents.Credit: Devonna Palmer
She thinks that higher-quality studies and randomized control trials with the power to show clear benefits need to be done first. Then, the findings need to be weighed against the cost of the intervention. “I don’t think there is anyone doing economic evaluations looking to see if the money spent is worth the benefits we gain,” she says.
It took Mahoor upwards of US$6 million to get to the current iteration of Ryan. He has seven units for which he is looking for buyers. Most care homes he has worked with cannot afford to purchase Ryan, so the robot will also be provided through a lease of $1,200 a month for 10 users. Khan, meanwhile, says that the base price for the robots she has worked with is $37,000, not including software, maintenance, or training and support. These costs have fuelled concerns that, should the robots prove effective, they will be out of reach of all but the most well-funded and exclusive care homes.
Caleb Johnston, an anthropologist at Newcastle University, UK, who has studied the ethics of using robots with ageing populations, says that in many areas, including in the United Kingdom, social care is chronically underfunded, even as money pours into social robots. Although “these may help with social and emotional support” he says, the system will still rely on poorly paid carers, often from overseas, “to do the messy work”, he says.
Berridge also thinks it is important that the needs of the people whom the technology is supposed to help are not lost as it rapidly improves. “Are we designing robots with and for people living with dementia? Or are we designing to manage people living with dementia?” she says. “We risk undermining solutions with wider and deeper reach when we don’t do an honest assessment of the nature of the problem being targeted.”
“There’s a lot of hype,” she adds. “I would say that tends to squeeze out critical questioning.”
Las Borinqueñas explores how women sought control of their reproductive lives in the 1950s.Credit: Valerie Terranova
Las Borinqueñas Directed by Rebecca Aparicio Ensemble Studio Theatre, New York City 3 April – 5 May 2024
It’s the 1950s and two US scientists are looking for somewhere to test the first birth-control pill. Where better than Puerto Rico? The territory had an established network of family-planning clinics, and the use of contraception had been legal there since 1937. That wasn’t the case in much of the United States, including Massachusetts, where biologist Gregory Pincus and obstetrician-gynaecologist John Rock were developing the oral contraceptive.
Puerto Rican women were interested in a pill that could give them more control over their reproductive lives. But as they lined up outside a clinic in the outskirts of San Juan to receive the medication, many were unaware that it was an experimental drug and they were part of a clinical trial. When some of them started reporting debilitating side effects, these were dismissed as psychosomatic.
The play Las Borinqueñas, whose title means ‘the Puerto Rican women’, revisits the complicated history of the world’s first oral contraceptive. Mixing the excitement of scientific breakthrough with the shock of flawed research ethics and shadows of colonialism and exploitation, it puts the spotlight on the women who, after playing a key part in the pill’s development, were quickly forgotten.
Reboot contraceptives research — it has been stuck for decades
It’s a long-overdue tribute and, most importantly, a reminder to remain vigilant against abuse and disrespect in studies involving human participants. In a world where the fight for access to birth control is ongoing, it is bold and commendable to recognize that this significant advance was built on ethically problematic studies that harmed some of the very women they aimed to serve and empower.
Written by Nelson Diaz-Marcano, a Puerto Rican theatre-maker based in New York City, the show was developed by the Ensemble Studio Theatre and the Alfred P. Sloan Foundation, a research funder also based in the city. It had its world premiere on 3 April and is playing until 5 May at the Ensemble Studio Theatre in New York City.
Taking control
The play follows the intertwined lives of five women — Chavela, Yolanda, Fernanda, Maria and Rosa — as they cross paths with the researchers testing the pill. As the audience witnesses their love stories, aspirations, struggles and loyal friendships, the protagonists open a window on the lives of hundreds of Puerto Rican women who enrolled in the tests, and how the experience changed them.
Each character is affected in a different way. Chavela sees the trial as chance to slow down the growth of her family while maintaining a passionate marriage. Yolanda envisions it as the lifeline that might save other women from the fate of her sister, Fernanda, who dies as a result of an illegal abortion. For Maria, it’s about avoiding pregnancy to advance her dream of becoming a writer — and about honouring Fernanda, her soulmate, with whom she could never openly have a relationship because of societal norms. But the hope brought on by the pill slowly fades when the women start feeling unwell.
The play shows how researchers and trial facilitators played down side effects because of the pill’s ground-breaking implications.Credit: Valerie Terranova
Rosa, who was suspicious of the pill from the start, urges the others to stop taking it, while boasting about the benefits of the sterilization that she underwent after giving birth. The doctors who suggested the procedure, however, never told her it was irreversible. The heartbreaking scene when she learns she will never be able to have another baby signals that the clinical trial wasn’t the first instance of medical abuse these women endured. By 1953, a eugenics-based programme in Puerto Rico had led to the sterilization of nearly one-fifth of women on the island to address concerns about ‘surplus population’.
From rabbits to women
The birth-control pill was the result of the encounter of Pincus and Rock, who were both studying the effects of synthetic progesterone, but in different contexts. Pincus was looking into the anti-ovulatory effect of the hormone in rabbits, and Rock was exploring it as a means to treat his patients’ infertility. The play focuses on Pincus, portrayed as an ambitious scientist determined to carve his name into history by creating a revolutionary product.
I wouldn’t be a scientist without my abortion
When someone becomes pregnant, their levels of progesterone go up, signalling to the body to shut down the ovaries and not release new eggs. Whereas Pincus wanted to mimic this process for the purpose of contraception, Rock hoped that a pause in ovulation would allow his patients’ reproductive systems to reset, increasing their chances of pregnancy after the treatment.
The scientists came together to test the pill in humans. The play briefly refers to a couple of small trials done in the United States, but to get the pill approved, it had to be tested on a larger scale. Pincus sets his sights on Puerto Rico and seeks to partner with Edris Rice-Wray, who was then the medical director of the Family Planning Association of Puerto Rico.
Rice-Wray expresses her concerns about negative side effects that had been observed in previous tests, but is convinced to join the project by Pincus’s wife, who highlights the potentially revolutionary implications of the pill for women around the world.
Rice-Wray is portrayed as a responsible public-health official who is nonetheless persuaded to push the boundaries of ethics for the greater good. She launches the programme with fanfare in 1956 and, at the suggestion of Pincus, does not mention the potential side effects to participants, most of whom are poor women with little access to health care. Her discomfort with the omission increases as she hears that the trial is taking a toll on participants.
Are women in research being led up the garden path?
In one scene, Chavela is taking laundry from the line when she is struck by dizziness and nausea. Her sister Rosa warns her that the pill is to blame, but she prefers to continue taking it rather than to risk becoming pregnant again. Rice-Wray reports those concerns to Pincus, who minimizes them as minor inconveniences compared with the wider benefits of the drug. Because of his disregard for the Puerto Rican study participants, the real-life Pincus was later accused of colonialism and exploitation of women of colour.
The protagonists eventually stop taking the pill and don’t experience long-term consequences. But the play mentions that three Puerto Rican women died during the trial, and that their deaths were never investigated.
Trial and error
In reality, of around 800 women who enrolled in the study, only 130 took the pill for a year or more, most dropping out because of the side effects. To make the results look more impressive, Pincus described them by saying that no pregnancy had been registered “in the 1,279 menstrual cycles” during which the treatment had been followed. In the play, his character brushes off the accusation of data embellishment. For him, it was simply a matter of using a different metric.
The pill, branded as Enovid, went on to be approved by the US Food and Drug Administration as a contraceptive in 1960. The participants of the clinical trial didn’t have access to the product once it reached the market: the price was prohibitive for the Puerto Rican working class.
How centuries of sexism excluded women from science — and how to redress the balance
More than six decades later, the contraceptive pills available are much safer. But access is still an issue. In the United States, until last year, people still needed a prescription to buy oral contraceptives — a significant barrier for those without health insurance.
Las Borinqueñas concludes with the women refusing to be defined by the experience of being exploited by scientists and having their right to decide about their own reproductive lives stripped away. Rosa publicly denounces the pill’s side effects and the irreversibility of sterilization on a radio show; she also conveys her resilience and hope for the future. The women will continue to take care of their families, to work and to pursue their dreams. They celebrate life and laugh at adversity.
Some would argue that their suffering was a small price to pay for the wider impacts of pill. But by giving names to the study participants and telling their stories, Las Borinqueñas serves as a powerful reminder that such disregard and injustice was never acceptable.
In the conflict between Russia and Ukraine, video footage has shown drones penetrating deep into Russian territory, more than 1,000 kilometres from the border, and destroying oil and gas infrastructure. It’s likely, experts say, that artificial intelligence (AI) is helping to direct the drones to their targets. For such weapons, no person needs to hold the trigger or make the final decision to detonate.
The development of lethal autonomous weapons (LAWs), including AI-equipped drones, is on the rise. The US Department of Defense, for example, has earmarked US$1 billion so far for its Replicator programme, which aims to build a fleet of small, weaponized autonomous vehicles. Experimental submarines, tanks and ships have been made that use AI to pilot themselves and shoot. Commercially available drones can use AI image recognition to zero in on targets and blow them up. LAWs do not need AI to operate, but the technology adds speed, specificity and the ability to evade defences. Some observers fear a future in which swarms of cheap AI drones could be dispatched by any faction to take out a specific person, using facial recognition.
Warfare is a relatively simple application for AI. “The technical capability for a system to find a human being and kill them is much easier than to develop a self-driving car. It’s a graduate-student project,” says Stuart Russell, a computer scientist at the University of California, Berkeley, and a prominent campaigner against AI weapons. He helped to produce a viral 2017 video called Slaughterbots that highlighted the possible risks.
The emergence of AI on the battlefield has spurred debate among researchers, legal experts and ethicists. Some argue that AI-assisted weapons could be more accurate than human-guided ones, potentially reducing both collateral damage — such as civilian casualties and damage to residential areas — and the numbers of soldiers killed and maimed, while helping vulnerable nations and groups to defend themselves. Others emphasize that autonomous weapons could make catastrophic mistakes. And many observers have overarching ethical concerns about passing targeting decisions to an algorithm.
The issue of weapons equipped with artificial intelligence was discussed by the United Nations Security Council in July 2023.Credit: Bianca Otero/Zuma/eyevine
For years, researchers have been campaigning to control this new threat1. Now the United Nations has taken a crucial step. A resolution in December last year added the topic of LAWs to the agenda of the UN General Assembly meeting this September. And UN secretary-general António Guterres stated in July last year that he wants a ban on weapons that operate without human oversight to be in place by 2026. Bonnie Docherty, a human rights lawyer at Harvard Law School in Cambridge, Massachusetts, says that getting this topic on to the UN agenda is significant after a decade or so of little progress. “Diplomacy moves slowly, but it’s an important step,” she says.
The move, experts say, offers the first realistic route for states to act on AI weapons. But this is easier said than done. These weapons raise difficult questions about human agency, accountability and the extent to which officials should be able to outsource life-and-death decisions to machines.
Under control?
Efforts to control and regulate the use of weapons date back hundreds of years. Medieval knights, for example, agreed not to target each other’s horses with their lances. In 1675, the warring states of France and the Holy Roman Empire agreed to ban the use of poison bullets.
Today, the main international restrictions on weaponry are through the UN Convention on Certain Conventional Weapons (CCW), a 1983 treaty that has been used, for example, to ban blinding laser weapons.
Autonomous weapons of one kind or another have been around for decades at least, including heat-seeking missiles and even (depending on how autonomy is defined) pressure-triggered landmines dating back to the US Civil War. Now, however, the development and use of AI algorithms is expanding their capabilities.
The CCW has been formally investigating AI-boosted weapons since 2013, but because it requires international consensus to pass regulations — and because many countries actively developing the technology oppose any ban — progress has been slow. In March, the United States hosted an inaugural plenary meeting on the Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy, a parallel effort that emphasizes voluntary guidelines for best practice rather than a legally enforceable ban.
Part of the problem has been a lack of consensus about what LAWs actually are. A 2022 analysis found at least a dozen definitions of autonomous weapons systems proposed by countries and organizations such as the North Atlantic Treaty Organization (NATO)2. The definitions span a wide range and show a limited amount of agreement on, or even an understanding of, AI, says Russell.
The United Kingdom, for example, says LAWs are “capable of understanding higher-level intent and direction”, whereas China says such a weapon can “learn autonomously, expand its functions and capabilities in a way exceeding human expectations”. Israel declares: “We should stay away from imaginary visions where machines develop, create or activate themselves — these should be left for science-fiction movies.” Germany includes “self-awareness” as a necessary attribute of autonomous weapons — a quality that most researchers say is far away from what’s possible with AI today, if not altogether impossible.
“That sort of means that the weapon has to wake up in the morning and decide to go and attack Russia by itself,” says Russell.
Although a more comprehensive, specific and realistic definition for LAWs will need to be ironed out, some experts say this can wait. “Traditionally in disarmament law, although it’s counter-intuitive, actually they often do the definition last in negotiation,” Docherty says. A working definition is usually enough to start the process and can help to soften initial objections from countries opposed to action.
The AI advantage
According to a 2023 analysis published by the Center for War Studies at University of Southern Denmark in Odense3, the autonomous weapons guided by AI available to army commanders today are relatively crude — slow-moving and clumsy drones equipped with enough explosive to blow up themselves and their targets.
These ‘loitering munitions’ can be the size of a model aircraft, cost about $50,000, and carry a few kilograms of explosive up to 50 kilometres away, enough to destroy a vehicle or to kill individual soldiers. These munitions use on-board sensors that monitor optical, infrared or radio frequencies to detect potential targets. The AI compares these sensor inputs with predesignated profiles of tanks, armoured vehicles and radar systems — as well as human beings.
Observers say that the most significant advantage offered by these autonomous bombs over remote-controlled drones is that they still work if the other side has equipment to jam electronic communications. And autonomous operation eliminates the risk that remote operators could be traced by an enemy and themselves attacked.
Although there were rumours that autonomous munitions killed fighters in Libya in 2020, reports from the conflict in Ukraine have cemented the idea that AI drones are now being used. “I think it’s pretty well accepted now that in Ukraine, they have already moved to fully autonomous weapons because the electronic jamming is so effective,” says Russell. Military commanders such as Ukraine’s Yaroslav Honchar have said that the country “already conducts fully robotic operations, without human intervention”3.
It’s hard to know how well AI weapons perform on the battlefield, in large part because militaries don’t release such data. Asked directly about AI weapons systems at a UK parliamentary enquiry in September last year, Tom Copinger-Symes, the deputy commander of the UK Strategic Command, didn’t give much away, saying only that the country’s military is doing benchmarking studies to compare autonomous with non-autonomous systems. “Inevitably, you want to check that this is delivering a bang for a buck compared with the old-fashioned system of having ten imagery analysts looking at the same thing,” he said.
Although real-world battlefield data is sparse, researchers note that AI has superior processing and decision-making skills that, in theory, offer a significant advantage. In annual tests of rapid image recognition, for example, algorithms have outperformed expert human performance for almost a decade. A study last year, for example, showed that AI could find duplicated images in scientific papers faster and more comprehensively than a human expert4.
In 2020, an AI model beat an experienced F-16 fighter-aircraft pilot in a series of simulated dogfights thanks to “aggressive and precise manoeuvres the human pilot couldn’t outmatch”. Then, in 2022, Chinese military researchers said that an AI-powered drone had outwitted an aircraft flown remotely by a human operator on the ground. The AI aircraft got onto the tail of its rival and into a position where it could have shot it down.
The US Air Force’s X-62A VISTA aircraft has been used to test the ability of autonomous agents to carry out advanced aerial manoeuvres.Credit: U.S. Air Force photo/Kyle Brasier
A drone AI can make “very complex decisions around how it carries out particular manoeuvres, how close it flies to the adversary and the angle of attack”, says Zak Kallenborn, a security analyst at the Center for Strategic and International Studies in Washington DC.
Still, says Kallenborn, it’s not clear what significant strategic advantage AI weapons offer, especially if both sides have access to them. “A huge part of the issue is not the technology itself, it’s how militaries use that technology,” he says.
AI could also in theory be used in other aspects of warfare, including compiling lists of potential targets; media reports have raised concerns that Israel, for example, used AI to create a database of tens of thousands of names of suspected militants, although the Israeli Defence Forces said in a statement that it does not use an AI system that “identifies terrorist operatives”.
Line in the sand
One key criterion often used to assess the ethics of autonomous weapons is how reliable they are and the extent to which things might go wrong. In 2007, for example, the UK military hastily redesigned its autonomous Brimstone missile for use in Afghanistan when it was feared it might mistake a bus of schoolchildren for a truckload of insurgents.
AI weapons can fairly easily lock on to infrared or powerful radar signals, says Kallenborn, comparing them to a library of data to help decide what is what. “That works fairly well because a little kid walking down the street is not going to have a high-powered radar in his backpack,” says Kallenborn. That means that when an AI weapon detects the source of an incoming radar signal on the battlefield, it can shoot with little risk of harming civilians.
But visual image recognition is more problematic, he says. “Where it’s basically just a sensor like a camera, I think you’re much, much more prone to error,” says Kallenborn. Although AI is good at identifying images, it’s not foolproof. Research has shown that tiny alterations to pictures can change the way they are classified by neural networks, he says — such as causing them to confuse an aircraft with a dog5.
Another possible dividing line for ethicists is how a weapon would be used: to attack or defend, for example. Sophisticated autonomous radar-guided systems are already used to defend ships at sea from rapid incoming targets. Lucy Suchman, a sociologist at Lancaster University, UK, who studies the interactions between people and machines, says that ethicists are more comfortable with this type of autonomous weapon because it targets ordnance rather than people, and because the signals are hard to falsely attribute to anything else.
One commonly proposed principle among researchers and the military alike is that there should be a ‘human in the loop’ of autonomous weapons. But where and how people should or must be involved is still up for debate. Many, including Suchman, typically interpret the idea to mean that human agents must visually verify targets before authorizing strikes and must be able to call off a strike if battlefield conditions change (such as if civilians enter the combat zone). But it could also mean that humans simply program in the description of the target before letting the weapon loose — a function known as fire-and-forget.
Some systems allow users to toggle between fully autonomous and human-assisted modes depending on the circumstances. This, say Suchman and others, isn’t good enough. “Requiring a human to disable an autonomous function does not constitute meaningful control,” she says.
The idea of full autonomy also muddies the water about accountability. “We’re very concerned about the use of autonomous weapons systems falling in an accountability gap because, obviously, you can’t hold the weapon system itself accountable,” Docherty says. It would also be legally challenging and arguably unfair to hold the operator responsible for the actions of a system that was functioning autonomously, she adds.
Russell suggests that there be “no communication between the on-board computing and the firing circuit”. That means the firing has to be activated by a remote operator and cannot ever be activated by the AI.
There is at least one point in the LAWs discussions that (almost) everybody seems to agree on: even nations generally opposed to controls, including the United States and China, have indicated that autonomous agents, including those with AI, should play no part in the decision to launch nuclear weapons, says Russell.
However, Russia seems to be more circumspect on this issue. Moscow is widely thought to have resurrected a cold-war programme called Perimetr, which — in theory at least — could launch a first nuclear strike on the West with no human oversight6. The United States and China have raised this issue in various talks about autonomous weapons, which many say could put pressure on Russia to change its strategy.
Policing the system
Unfortunately, says Kallenborn, any ban on the use of LAWs would be hard to enforce through inspections and observations — the classic ‘trust but verify’ approach commonly used for other regulated weaponry.
With nuclear weapons, for example, there’s a well-established system for site inspections and audits of nuclear material. But with AI, things are easier to conceal or alter on the fly. “It could be as simple as just changing a couple lines of code to say, all right, now the machine gets to decide to go blow this up. Or, you know, remove the code, and then stick it back in when the arms-control inspectors are there,” says Kallenborn. “It requires us to rethink how we think about verification in weapons systems and arms control.”
Checks might have to switch from time-of-production to after-the-fact, Kallenborn says. “These things are going to get shot down. They’re going to be captured. Which means that you can then do inspections and look at the code,” he says.
All these issues will feed into the UN discussions, beginning at the General Assembly this September; a precursor conference has also been set up by Austria at the end of April to help to kick-start these conversations. If enough countries vote to act in September, then the UN will probably set up a working group to set out the issues, Docherty says.
A treaty might be possible in three years, adds Docherty, who had a key role in the negotiations of the UN’s 2017 Treaty on the Prohibition of Nuclear Weapons. “In my experience, once negotiations start, they move relatively quickly.”
Growing evidence indicates that insects such as bees show some forms of consciousness, according to a new scientific statement.Credit: Phil Savoie/Nature Picture Library
Crows, chimps and elephants: these and many other birds and mammals behave in ways that suggest they might be conscious. And the list does not end with vertebrates. Researchers are expanding their investigations of consciousness to a wider range of animals, including octopuses and even bees and flies.
Armed with such research, a coalition of scientists is calling for a rethink in the animal–human relationship. If there’s “a realistic possibility” of “conscious experience in an animal, it is irresponsible to ignore that possibility in decisions affecting that animal”, the researchers write in a document they call The New York Declaration of Animal Consciousness. Issued today during a meeting in New York City, the declaration also says that there is a “realistic possibility of conscious experience” in reptiles, fish, insects and other animals that have not always been considered to have inner lives, and “strong scientific support” for aspects of consciousness in birds and mammals.
As the evidence has accumulated, scientists are “taking the topic seriously, not dismissing it out of hand as a crazy idea in the way they might have in the past,” says Jonathan Birch, a philosopher at the London School of Economics and Political Science and one of the authors of the declaration.
The document, which had around 40 signatories early today, doesn’t state that there are definitive answers about which species are conscious. “What it says is there is sufficient evidence out there such that there’s a realistic possibility of some kinds of conscious experiences in species even quite distinct from humans,” says Anil Seth, director of the Centre for Consciousness Science at the University of Sussex near Brighton, UK, and one of the signatories. The authors hope that others will sign the declaration and that it will stimulate both more research into animal consciousness and more funding for the field.
Blurry line
The definition of consciousness is complex, but the group focuses on an aspect of consciousness called sentience, often defined as the capacity to have subjective experiences, says Birch. For an animal, such experiences would include smelling, tasting, hearing or touching the world around itself, as well as feeling fear, pleasure or pain — in essence, what it is like to be that animal. But subjective experience does not require the capacity to think about one’s experiences.
The consciousness wars: can scientists ever agree on how the mind works?
Non-human animals cannot use words to communicate their inner states. To assess consciousness in these animals, scientists often rely on indirect evidence, looking for certain behaviours that are associated with conscious experiences, Birch says.
One classic experiment is the mirror test, which investigates an animal’s ability to recognize itself in a mirror. In this experiment, scientists apply a sticker or other visual mark on an animal’s body and place the animal in front of a mirror. Some animals — including chimpanzees (Pan troglodytes)1, Asian elephants (Elephas maximus)2 and cleaner fishes (Labroides dimidiatus)3 — exhibit curiosity about the mark and even try to remove it. This behaviour suggests the possibility of self-awareness, which might be a sign of consciousness.
In an experiment with crows (Corvus corone)4, the birds were trained to make a specific head gesture whenever they saw a coloured square on a screen, a task they carried out with high accuracy. While the birds performed the task, scientists measured the activity in a region of their brain associated with high-level cognition. The birds’ brain activity correlated with what the birds were reporting, not with what they were actually shown. This suggests that they were aware of what they were perceiving, another potential marker of consciousness.
Invertebrate inner lives?
Another experiment showed that octopuses (Octopus bocki)5, when picking between two chambers, avoided one where they had previously received a painful stimulus in favour of one where they were given an anaesthetic. This suggests that they experience and actively avoid pain, which some researchers think indicates conscious experience.
Research shows that octopuses avoid pain, which some scientists take as a sign of consciousness.Credit: Brandon Cole/Nature Picture Library
Investigations of fruit flies (Drosophila melanogaster) show that they engage in both deep sleep and ‘active sleep’, in which their brain activity is the same as when they’re awake6. “This is perhaps similar to what we call rapid eye movement sleep in humans, which is when we have our most vivid dreams, which we interpret as conscious experiences,” says Bruno van Swinderen, a biologist at the University of Queensland in Brisbane, Australia, who studies fruit flies’ behaviour and who also signed the declaration.
Some suggest that dreams are key components of being conscious, he notes. If flies and other invertebrates have active sleep, “then maybe this is as good a clue as any that they are perhaps conscious”.
Animal minds
Other researchers are more sceptical about the available evidence on animal consciousness. “I don’t think there is basically any decisive evidence so far,” says Hakwan Lau, a neuroscientist at the Riken Center for Brain Science in Wako, Japan.
Lau acknowledges that there is a growing body of work showing sophisticated perceptual behaviour in animals, but he contends that that’s not necessarily indicative of consciousness. In humans, for example, there is both conscious and unconscious perception. The challenge now is to develop methods that can adequately distinguish between the two in non-humans.
Seth responds that, even in the absence of definitive answers, the declaration might still have a positive influence in shaping policies relating to animal ethics and welfare.
For van Swinderen, the time is right to consider whether most animals might be conscious. “We are experiencing an artificial-intelligence revolution where similar questions are being asked about machines. So it behoves us to ask if and how this adaptive quality of the brain might have evolved in nature.”
Of the nearly two billion people living in countries that are holding elections this year, some have already cast their ballots. Elections held in Indonesia and Pakistan in February, among other countries, offer an early glimpse of what’s in store as artificial intelligence (AI) technologies steadily intrude into the electoral arena. The emerging picture is deeply worrying, and the concerns are much broader than just misinformation or the proliferation of fake news.
As the former director of the Machine Learning, Ethics, Transparency and Accountability (META) team at Twitter (before it became X), I can attest to the massive ongoing efforts to identify and halt election-related disinformation enabled by generative AI (GAI). But uses of AI by politicians and political parties for purposes that are not overtly malicious also raise deep ethical concerns.
GAI is ushering in an era of ‘softfakes’. These are images, videos or audio clips that are doctored to make a political candidate seem more appealing. Whereas deepfakes (digitally altered visual media) and cheap fakes (low-quality altered media) are associated with malicious actors, softfakes are often made by the candidate’s campaign team itself.
How to stop AI deepfakes from sinking society — and science
In Indonesia’s presidential election, for example, winning candidate Prabowo Subianto relied heavily on GAI, creating and promoting cartoonish avatars to rebrand himself as gemoy, which means ‘cute and cuddly’. This AI-powered makeover was part of a broader attempt to appeal to younger voters and displace allegations linking him to human-rights abuses during his stint as a high-ranking army officer. The BBC dubbed him “Indonesia’s ‘cuddly grandpa’ with a bloody past”. Furthermore, clever use of deepfakes, including an AI ‘get out the vote’ virtual resurrection of Indonesia’s deceased former president Suharto by a group backing Subianto, is thought by some to have contributed to his surprising win.
Nighat Dad, the founder of the research and advocacy organization Digital Rights Foundation, based in Lahore, Pakistan, documented how candidates in Bangladesh and Pakistan used GAI in their campaigns, including AI-written articles penned under the candidate’s name. South and southeast Asian elections have been flooded with deepfake videos of candidates speaking in numerous languages, singing nostalgic songs and more — humanizing them in a way that the candidates themselves couldn’t do in reality.
What should be done? Global guidelines might be considered around the appropriate use of GAI in elections, but what should they be? There have already been some attempts. The US Federal Communications Commission, for instance, banned the use of AI-generated voices in phone calls, known as robocalls. Businesses such as Meta have launched watermarks — a label or embedded code added to an image or video — to flag manipulated media.
But these are blunt and often voluntary measures. Rules need to be put in place all along the communications pipeline — from the companies that generate AI content to the social-media platforms that distribute them.
What the EU’s tough AI law means for research and ChatGPT
Content-generation companies should take a closer look at defining how watermarks should be used. Watermarking can be as obvious as a stamp, or as complex as embedded metadata to be picked up by content distributors.
Companies that distribute content should put in place systems and resources to monitor not just misinformation, but also election-destabilizing softfakes that are released through official, candidate-endorsed channels. When candidates don’t adhere to watermarking — none of these practices are yet mandatory — social-media companies can flag and provide appropriate alerts to viewers. Media outlets can and should have clear policies on softfakes. They might, for example, allow a deepfake in which a victory speech is translated to multiple languages, but disallow deepfakes of deceased politicians supporting candidates.
Election regulatory and government bodies should closely examine the rise of companies that are engaging in the development of fake media. Text-to-speech and voice-emulation software from Eleven Labs, an AI company based in New York City, was deployed to generate robocalls that tried to dissuade voters from voting for US President Joe Biden in the New Hampshire primary elections in January, and to create the softfakes of former Pakistani prime minister Imran Khan during his 2024 campaign outreach from a prison cell. Rather than pass softfake regulation on companies, which could stifle allowable uses such as parody, I instead suggest establishing election standards on GAI use. There is a long history of laws that limit when, how and where candidates can campaign, and what they are allowed to say.
Citizens have a part to play as well. We all know that you cannot trust what you read on the Internet. Now, we must develop the reflexes to not only spot altered media, but also to avoid the emotional urge to think that candidates’ softfakes are ‘funny’ or ‘cute’. The intent of these isn’t to lie to you — they are often obviously AI generated. The goal is to make the candidate likeable.
Softfakes are already swaying elections in some of the largest democracies in the world. We would be wise to learn and adapt as the ongoing year of democracy, with some 70 elections, unfolds over the next few months.
Voice synthesis has come a long way since 1978’s Speak & Spell toy, which once wowed people with its state-of-the-art ability to read words aloud using an electronic voice. Now, using deep-learning AI models, software can create not only realistic-sounding voices but can also convincingly imitate existing voices using small samples of audio.
Along those lines, OpenAI this week announced Voice Engine, a text-to-speech AI model for creating synthetic voices based on a 15-second segment of recorded audio. It has provided audio samples of the Voice Engine in action on its website.
Once a voice is cloned, a user can input text into the Voice Engine and get an AI-generated voice result. But OpenAI is not ready to widely release its technology. The company initially planned to launch a pilot program for developers to sign up for the Voice Engine API earlier this month. But after more consideration about ethical implications, the company decided to scale back its ambitions for now.
“In line with our approach to AI safety and our voluntary commitments, we are choosing to preview but not widely release this technology at this time,” the company writes. “We hope this preview of Voice Engine both underscores its potential and also motivates the need to bolster societal resilience against the challenges brought by ever more convincing generative models.”
Voice cloning tech in general is not particularly new—there have been severalAI voice synthesis models since 2022, and the tech is active in the open source community with packages like OpenVoice and XTTSv2. But the idea that OpenAI is inching toward letting anyone use its particular brand of voice tech is notable. And in some ways, the company’s reticence to release it fully might be the bigger story.
OpenAI says that benefits of its voice technology include providing reading assistance through natural-sounding voices, enabling global reach for creators by translating content while preserving native accents, supporting non-verbal individuals with personalized speech options, and assisting patients in recovering their own voice after speech-impairing conditions.
But it also means that anyone with 15 seconds of someone’s recorded voice could effectively clone it, and that has obvious implications for potential misuse. Even if OpenAI never widely releases its Voice Engine, the ability to clone voices has already caused trouble in society through phone scams where someone imitates a loved one’s voice and election campaign robocalls featuring cloned voices from politicians like Joe Biden.
Also, researchers and reporters have shown that voice-cloning technology can be used to break into bank accounts that use voice authentication (such as Chase’s Voice ID), which prompted US senator Sherrod Brown of Ohio, the chair of the US Senate Committee on Banking, Housing, and Urban Affairs, to send a letter to the CEOs of several major banks in May 2023 to inquire about the security measures banks are taking to counteract AI-powered risks.
OpenAI recognizes that the tech might cause trouble if broadly released, so it’s initially trying to work around those issues with a set of rules. It has been testing the technology with a set of select partner companies since last year. For example, video synthesis company HeyGen has been using the model to translate a speaker’s voice into other languages while keeping the same vocal sound.
Even if you find a study or paper that you can’t access in one of those databases, consider reaching out to the study’s lead author or researcher. In many cases, they’re happy to discuss their work and may even share the study with you directly and offer to talk about their research.
Get a Good Filtering System
For journalist Paulette Perhach’s article on ADHD in The New York Times, she used Epic Research to see “dual team studies.” That’s when two independent teams address the same topic or question, and ideally come to the same conclusions. She recommends locating research and experts via key associations for your topic. She also likes searching via Google Scholar but advises filtering it for studies and research in recent years to avoid using old data. She suggests keeping your links and research organized. “Always be ready to be peer-reviewed yourself,” Perhach says.
When you are looking for information for a story or project, you might be inclined to start with a regular Google search. But keep in mind that the internet is full of false information, and websites that look trustworthy can sometimes turn out to be businesses or companies with a vested interest in you taking their word as objective fact without additional scrutiny. Regardless of your writing project, unreliable or biased sources are a great way to torpedo your work—and any hope of future work.
For Accuracy, Go to the Government
Author Bobbi Rebell researched her book Launching Financial Grownups using the IRS’ website. “I might say that you can contribute a certain amount to a 401K, but it might be outdated because those numbers are always changing, and it’s important to be accurate,” she says. “AI and ChatGPT can be great for idea generation,” says Rebell, “but you have to be careful. If you are using an article someone was quoted in, you don’t know if they were misquoted or quoted out of context.”
Audrey Clare Farley, who writes historical nonfiction, has used a plethora of sites for historical research, including Women Also Know History, which allows searches by expertise or area of study, and JSTOR, a digital library database that offers a number of free downloads a month. She also uses Chronicling America, a project from the Library of Congress which gathers old newspapers to show how a historical event was reported, and Newspapers.com (which you can access via free trial but requires a subscription after seven days).
When it comes to finding experts, Farley cautions against choosing the loudest voices on social media platforms. “They might not necessarily be the most authoritative. I vet them by checking if they have a history of publication on the topic, and/or educational credentials.”
When vetting an expert, look for these red flags:
You can’t find their work published or cited anywhere.
They were published in an obscure journal.
Their research is funded by a company, not a university, or they are the spokesperson for the company they are doing research for. (This makes them a public relations vehicle and not an appropriate source for journalism.)
And finally, the best endings for virtually any writing, whether it’s an essay, a research paper, an academic report, or a piece of investigative journalism, circle back to the beginning of the piece, and show your reader the transformation or the journey the piece has presented in perspective.
As always, your goal should be strong writing supported by research that makes an impact without cutting corners. Only then can you explore tools that might make the job a little easier, for instance by generating subheads or discovering a concept you might be missing—because then you’ll have the experience and skills to see whether it’s harming or helping your work.
It was in just the second article of more than 1,000 that Otto Kalliokoski was screening that he spotted what he calls a “Photoshop masterpiece”.
The paper showed images from western blots — a technique used to analyse protein composition — for two samples. But Kalliokoski, an animal behaviourist at the University of Copenhagen, found that the images were identical down to the pixel, which he says is clearly not supposed to happen.
Image manipulation in scientific studies is a known and widespread problem. All the same, Kalliokoski and his colleagues were startled to come across more than 100 studies with questionable images while compiling a systematic review about a widely used test of laboratory rats’ moods. After publishing the review1 in January, the researchers released a preprint2 documenting the troubling studies that they uncovered and how these affected the results of their review. The preprint, posted on bioRxiv in February, has not yet been peer reviewed.
Their work “clearly highlights [that falsified images] are impacting our consolidated knowledge base”, says Alexandra Bannach-Brown, a systematic-review methodologist at the Berlin Institute of Health who was not involved with either the review or the preprint. Systematic reviews, which summarize and interpret the literature on a particular topic, are a key component of that base. With an explosion of scientific literature, “it’s impossible for a single person to keep up with reading every new paper that comes out in their field”, Bannach-Brown says. And that means that upholding the quality of systematic reviews is ever more important.
Pile-up of problems
Kalliokoski’s systematic review examined the reliability of a test designed to assess reward-seeking in rats under stress. A reduced interest in a reward is assumed to be a proxy symptom of depression, and the test is widely used during the development of antidepressant drugs. The team identified an initial pool of 1,035 eligible papers; 588 contained images.
By the time he’d skimmed five papers, Kalliokoski had already found a second one with troubling images. Not sure what to do, he bookmarked the suspicious studies and went ahead with collating papers for the review. As the questionable papers kept piling up, he and his colleagues decided to deploy Imagetwin, an AI-based software tool that flags problems such as duplicated images and ones that have been stretched or rotated. Either Imagetwin or the authors’ visual scrutiny flagged 112 — almost 20% — of the 588 image-containing papers.
“That is actually a lot,” says Elizabeth Bik, a microbiologist in San Francisco, California, who has investigated image-related misconduct and is now an independent scientific-integrity consultant. Whether image manipulation is the result of honest error or an intention to mislead, “it could undermine the findings of a study”, she says.
Small but detectable effect
For their final analysis, the authors examined all the papers that met their criteria for inclusion in their review. This batch, consisting of 132 studies, included 10 of the 112 that the team had flagged as having potentially doctored images.
Journals adopt AI to spot duplicated images in manuscripts
Analysis of these 10 studies alone assessed the test as 50% more effective at identifying depression-related symptoms than did a calculation based on the 122 studies without questionable images. These suspicious studies “do actually skew the results”, Kalliokoski says — although “not massively”, because overall variations in the data set mask the contribution from this small subset.
Examples from this study “cover pretty much all types of image problems”, Bik says, ranging from simple duplication to images that showed evidence of deliberate alteration. Using a scale that Bik developed to categorize the degree of image manipulation, the researchers found that most of the problematic images showed signs of tampering.
The researchers published their review in January in Translational Psychiatry without telling the journal that it was based in part on papers that included suspicious images. The journal’s publisher, Springer Nature, told Nature that it is investigating. (The Nature news team is editorially independent of its publisher, Springer Nature).
When they published their preprint the following month, the researchers included details of all the papers with suspicious images. They also flagged each study on Pubpeer, a website where scientists comment anonymously on papers. “My first allegiance is towards the [scientific] community,” Kalliokoski says, adding that putting the data out is the first step.
Bring reviews to life
The process of challenging a study’s integrity, giving its authors a chance to respond and seeking retraction for fraudulent studies can take years. One way to clear these muddied waters, says Bannach-Brown, is to publish ‘living’ systematic reviews, which are designed to be updated whenever papers get retracted or new research is added. She has helped to develop one such method of creating living reviews, called Systematic Online Living Evidence Summaries.
Systematic-review writers are also keen to see publishers integrate standardized ways to screen out dubious studies — rather than waiting until a study gets retracted.
Authors, publishers and editorial boards need to work together, Bannach-Brown says, to “catch some of these questionable research practices before they even make it to publication.”
In 2023, OpenAI told the UK parliament that it was “impossible” to train leading AI models without using copyrighted materials. It’s a popular stance in the AI world, where OpenAI and other leading players have used materials slurped up online to train the models powering chatbots and image generators, triggering a wave of lawsuits alleging copyright infringement.
Two announcements Wednesday offer evidence that large language models can in fact be trained without the permissionless use of copyrighted materials.
A group of researchers backed by the French government have released what is thought to be the largest AI training dataset composed entirely of text that is in the public domain. And the nonprofit Fairly Trained announced that it has awarded its first certification for a large language model built without copyright infringement, showing that technology like that behind ChatGPT can be built in a different way to the AI industry’s contentious norm.
“There’s no fundamental reason why someone couldn’t train an LLM fairly,” says Ed Newton-Rex, CEO of Fairly Trained. He founded the nonprofit in January 2024 after quitting his executive role at image generation startup Stability AI because he disagreed with its policy of scraping content without permission.
Fairly Trained offers a certification to companies willing to prove that they’ve trained their AI models on data that they either own, have licensed, or is in the public domain. When the nonprofit launched, some critics pointed out that it hadn’t yet identified a large language model that met those requirements.
Today, Fairly Trained announced it has certified its first large language model. It’s called KL3M and was developed by Chicago-based legal tech consultancy startup 273 Ventures, using a curated training dataset of legal, financial, and regulatory documents.
The company’s cofounder Jillian Bommarito says the decision to train KL3M in this way stemmed from the company’s “risk-averse” clients like law firms. “They’re concerned about the provenance, and they need to know that output is not based on tainted data,” she says. “We’re not relying on fair use.” The clients were interested in using generative AI for tasks like summarizing legal documents and drafting contracts, but didn’t want to get dragged into lawsuits about intellectual property as OpenAI, Stability AI, and others have been.
Bommarito says that 273 Ventures hadn’t worked on a large language model before but decided to train one as an experiment. “Our test to see if it was even possible,” she says. The company has created its own training data set, the Kelvin Legal DataPack, which includes thousands of legal documents reviewed to comply with copyright law.
Although the dataset is tiny (around 350 billion tokens, or units of data) compared to those compiled by OpenAI and others that have scraped the internet en masse, Bommarito says the KL3M model performed far better than expected, something she attributes to how carefully the data had been vetted beforehand. “Having clean, high-quality data may mean that you don’t have to make the model so big,” she says. Curating a dataset can help make a finished AI model specialized to the task its designed for. 273 Ventures is now offering spots on a waitlist to clients who want to purchase access to this data.
Clean Sheet
Companies looking to emulate KL3M may have more help in the future in the form of freely available infringement-free datasets. On Wednesday, researchers released what they claim is the largest available AI dataset for language models composed purely of public domain content. Common Corpus, as it is called, is a collection of text roughly the same size as the data used to train OpenAI’s GPT-3 text generation model and has been posted to the open source AI platform Hugging Face.
The dataset was built from sources like public domain newspapers digitized by the US Library of Congress and the National Library of France. Pierre-Carl Langlais, project coordinator for Common Corpus, calls it a “big enough corpus to train a state-of-the-art LLM.” In the lingo of big AI, the dataset contains 500 million tokens, OpenAI’s most capable model is widely believed to have been trained on several trillions.
