New brain-computer interface allows man with ALS to 'speak' again

A new brain-computer interface (BCI) developed at UC Davis Health translates brain signals into speech with up to 97% accuracy -- the most accurate system of its kind.

The researchers implanted sensors in the brain of a man with severely impaired speech due to amyotrophic lateral sclerosis (ALS). The man was able to communicate his intended speech within minutes of activating the system.

A study about this work was published today in the New England Journal of Medicine.

ALS, also known as Lou Gehrig's disease, affects the nerve cells that control movement throughout the body. The disease leads to a gradual loss of the ability to stand, walk and use one's hands. It can also cause a person to lose control of the muscles used to speak, leading to a loss of understandable speech.

The new technology is being developed to restore communication for people who can't speak due to paralysis or neurological conditions like ALS. It can interpret brain signals when the user tries to speak and turns them into text that is 'spoken' aloud by the computer.

"Our BCI technology helped a man with paralysis to communicate with friends, families and caregivers," said UC Davis neurosurgeon David Brandman. "Our paper demonstrates the most accurate speech neuroprosthesis (device) ever reported."

Brandman is the co-principal investigator and co-senior author of this study. He is an assistant professor in the UC Davis Department of Neurological Surgery and co-director of the UC Davis Neuroprosthetics Lab.

The new BCI breaks the communication barrier

When someone tries to speak, the new BCI device transforms their brain activity into text on a computer screen. The computer can then read the text out loud.

To develop the system, the team enrolled Casey Harrell, a 45-year-old man with ALS, in the BrainGate clinical trial. At the time of his enrollment, Harrell had weakness in his arms and legs (tetraparesis). His speech was very hard to understand (dysarthria) and required others to help interpret for him.

In July 2023, Brandman implanted the investigational BCI device. He placed four microelectrode arrays into the left precentral gyrus, a brain region responsible for coordinating speech. The arrays are designed to record the brain activity from 256 cortical electrodes.

"We're really detecting their attempt to move their muscles and talk," explained neuroscientist Sergey Stavisky. Stavisky is an assistant professor in the Department of Neurological Surgery. He is the co-director of the UC Davis Neuroprosthetics Lab and co-principal investigator of the study. "We are recording from the part of the brain that's trying to send these commands to the muscles. And we are basically listening into that, and we're translating those patterns of brain activity into a phoneme -- like a syllable or the unit of speech -- and then the words they're trying to say."

Faster training, better results

Despite recent advances in BCI technology, efforts to enable communication have been slow and prone to errors. This is because the machine-learning programs that interpreted brain signals required a large amount of time and data to perform.

"Previous speech BCI systems had frequent word errors. This made it difficult for the user to be understood consistently and was a barrier to communication," Brandman explained. "Our objective was to develop a system that empowered someone to be understood whenever they wanted to speak."

Harrell used the system in both prompted and spontaneous conversational settings. In both cases, speech decoding happened in real time, with continuous systemupdates to keep it working accurately.

The decoded words were shown on a screen. Amazingly, they were read aloud in a voice that sounded like Harrell's before he had ALS. The voice was composed using software trained with existing audio samples of his pre-ALS voice.

At the first speech data training session, the system took 30 minutes to achieve 99.6% word accuracy with a 50-word vocabulary.

"The first time we tried the system, he cried with joy as the words he was trying to say correctly appeared on-screen. We all did," Stavisky said.

In the second session, the size of the potential vocabulary increased to 125,000 words. With just an additional 1.4 hours of training data, the BCI achieved a 90.2% word accuracy with this greatly expanded vocabulary. After continued data collection, the BCI has maintained 97.5% accuracy.

"At this point, we can decode what Casey is trying to say correctly about 97% of the time, which is better than many commercially available smartphone applications that try to interpret a person's voice," Brandman said. "This technology is transformative because it provides hope for people who want to speak but can't. I hope that technology like this speech BCI will help future patients speak with their family and friends."

The study reports on 84 data collection sessions over 32 weeks. In total, Harrell used the speech BCI in self-paced conversations for over 248 hours to communicate in person and over video chat.

"Not being able to communicate is so frustrating and demoralizing. It is like you are trapped," Harrell said. "Something like this technology will help people back into life and society."

"It has been immensely rewarding to see Casey regain his ability to speak with his family and friends through this technology," said the study's lead author, Nicholas Card. Card is a postdoctoral scholar in the UC Davis Department of Neurological Surgery.

"Casey and our other BrainGate participants are truly extraordinary. They deserve tremendous credit for joining these early clinical trials. They do this not because they're hoping to gain any personal benefit, but to help us develop a system that will restore communication and mobility for other people with paralysis," said co-author and BrainGate trial sponsor-investigator Leigh Hochberg. Hochberg is a neurologist and neuroscientist at Massachusetts General Hospital, Brown University and the VA Providence Healthcare System.

Read more at Science Daily

How artificial intelligence gave a paralyzed woman her voice back

Researchers at UC San Francisco and UC Berkeley have developed a brain-computer interface (BCI) that has enabled a woman with severe paralysis from a brainstem stroke to speak through a digital avatar.

It is the first time that either speech or facial expressions have been synthesized from brain signals. The system can also decode these signals into text at nearly 80 words per minute, a vast improvement over commercially available technology.

Edward Chang, MD, chair of neurological surgery at UCSF, who has worked on the technology, known as a brain computer interface, or BCI, for more than a decade, hopes this latest research breakthrough, appearing Aug. 23, 2023, in Nature, will lead to an FDA-approved system that enables speech from brain signals in the near future.

"Our goal is to restore a full, embodied way of communicating, which is really the most natural way for us to talk with others," said Chang, who is a member of the UCSF Weill Institute for Neuroscience and the Jeanne Robertson Distinguished Professor in Psychiatry. "These advancements bring us much closer to making this a real solution for patients."

Chang's team previously demonstrated it was possible to decode brain signals into text in a man who had also experienced a brainstem stroke many years earlier. The current study demonstrates something more ambitious: decoding brain signals into the richness of speech, along with the movements that animate a person's face during conversation.

Chang implanted a paper-thin rectangle of 253 electrodes onto the surface of the woman's brain over areas his team has discovered are critical for speech. The electrodes intercepted the brain signals that, if not for the stroke, would have gone to muscles in her, tongue, jaw and larynx, as well as her face. A cable, plugged into a port fixed to her head, connected the electrodes to a bank of computers.

For weeks, the participant worked with the team to train the system's artificial intelligence algorithms to recognize her unique brain signals for speech. This involved repeating different phrases from a 1,024-word conversational vocabulary over and over again, until the computer recognized the brain activity patterns associated with the sounds.

Rather than train the AI to recognize whole words, the researchers created a system that decodes words from phonemes. These are the sub-units of speech that form spoken words in the same way that letters form written words. "Hello," for example, contains four phonemes: "HH," "AH," "L" and "OW."

Using this approach, the computer only needed to learn 39 phonemes to decipher any word in English. This both enhanced the system's accuracy and made it three times faster.

"The accuracy, speed and vocabulary are crucial," said Sean Metzger, who developed the text decoder with Alex Silva, both graduate students in the joint Bioengineering Program at UC Berkeley and UCSF. "It's what gives a user the potential, in time, to communicate almost as fast as we do, and to have much more naturalistic and normal conversations."

To create the voice, the team devised an algorithm for synthesizing speech, which they personalized to sound like her voice before the injury, using a recording of her speaking at her wedding.

The team animated the avatar with the help of software that simulates and animates muscle movements of the face, developed by Speech Graphics, a company that makes AI-driven facial animation. The researchers created customized machine-learning processes that allowed the company's software to mesh with signals being sent from the woman's brain as she was trying to speak and convert them into the movements on the avatar's face, making the jaw open and close, the lips protrude and purse and the tongue go up and down, as well as the facial movements for happiness, sadness and surprise.

"We're making up for the connections between the brain and vocal tract that have been severed by the stroke," said Kaylo Littlejohn, a graduate student working with Chang and Gopala Anumanchipalli, PhD, a professor of electrical engineering and computer sciences at UC Berkeley. "When the subject first used this system to speak and move the avatar's face in tandem, I knew that this was going to be something that would have a real impact."

An important next step for the team is to create a wireless version that would not require the user to be physically connected to the BCI.

Read more at Science Daily