Brain–computer interfaces decode users’ brain signals.Credit: Kevin Frayer/Getty
Human communication relies on more than just talking; we nod, shrug, move our hands and point with our fingers. For people with severe paralysis, brain–computer interfaces (BCIs) can convey intended speech or movement. Now, researchers have shown that a single brain implant can do both at once.
The device, described1 today in Nature Neuroscience, is the first to translate both verbal and non-verbal modes of communication, simultaneously and within seconds of the user’s intent. It uses artificial intelligence to convert electrical brain activity into text that appears on screen and to prompt a personalized animated avatar to move.
Recent BCI technology has enabled people with paralysis to do tasks such as talking2 and controlling a cursor3 or a robotic arm4. “But most of these studies have focused on restoring one of these functions at a time,” says study co-author Samantha Brosler, a PhD student in bioengineering at the University of California, San Francisco.
This system is a proof of concept but aims to be “a step towards more multifunctional BCIs that can capture more of the expressivity and communication that is possible” than with current technologies, she adds.
“This gives us hope for neural prosthesis that will enter day-to-day usage,” says Christian Herff, a computational neuroscientist at Maastricht University in the Netherlands.
Multitasking challenges
The difficulty of decoding gestures and speech at the same time is that some of the neural signals that are involved overlap. And the patterns of neural activity can be slightly different during tasks that involve simultaneous speech and gesturing, compared with during speech-only and gesture-only tasks.
To distinguish between the two, Brosler and her colleagues worked with two participants. The device — a surgically implanted array of 253 electrodes placed on the surface of the brain’s cortex — “covers a fairly large surface area of the sensorimotor cortex, so it can capture signals related to both speech and body movements,” explains Brosler.
The first participant had impaired speech and movement following a brainstem stroke. The researchers asked him to silently speak five phrases and attempt to wave, nod, shake his hands and clap.
The second participant had a type of motor neuron disease called amyotrophic lateral sclerosis, which caused progressive speech paralysis and motor impairments. At the time of the study, he was able to type messages on his computer and phone. The team asked him to vocalize ten phrases and imagine making ten gestures such as a shrug and thumbs up — without moving his body except for his facial muscles.
The BCIs recorded the participants’ neural activity when they attempted to speak or gesture, as well as when they used speech and gestures together such as pairing the word ‘hello’ with a wave, or pairing ‘yes’ with a head nod.
