Researchers at UC San Francisco have developed the first “neuroprosthesis” that has enabled a man with severe paralysis to communicate in sentences, translating signals from his brain to the vocal tract directly into words that appear as text on a screen. Aided by this technology, the man is also able to move a robotic arm to manipulate objects.
This achievement builds on more than a decade of effort by Edward Chang, MD, the Joan and Sanford I. Weill Chair of Neurological Surgery at UCSF, Jeanne Robertson Distinguished Professor, and senior author of the study.
“To our knowledge, this is the first successful demonstration of direct decoding of full words from the brain activity of someone who is paralyzed and cannot speak,” said Chang.
Dr. Eddie Chang and postdoctoral scholar David Moses, PhD, confer at UCSF’s Mission Bay campus on Friday, June 7, 2019, in San Francisco.
Photo: Noah Berger
To investigate the potential of this technology in patients with paralysis, Chang partnered with colleague Karunesh Ganguly, MD, PhD, neurologist and associate professor of neurology at UCSF, to launch a study known as BRAVO (Brain-Computer Interface Restoration of Arm and Voice). The first participant is a man in his late 30s who suffered a brain stem stroke more than 15 years ago that severely damaged the connection between his brain and his vocal tract and limbs. The participant, who asked to be referred to as BRAVO1, worked with the researchers to create a 50-word vocabulary that Chang’s team could recognize from brain activity using advanced computer algorithms. The vocabulary was sufficient to create hundreds of sentences.
In this Q&A, Drs. Chang and Ganguly share more about this groundbreaking research.
Q: What is the goal of the BRAVO trial?
BRAVO clinical trial testing session with trial participant, Bravo1, and Chang Lab researcher David Moses, PhD (postdoctoral fellow). During the session, the participant’s cortical activity was recorded while he attempted to produce words and sentences. The activity is recorded via a ECog electrode array that was implanted on the surface on the participant’s brain. The team uses machine learning and language modeling to analyze the signals and translate them into text.
Photo: Todd Dubnicoff, UCSF
Chang: The goal of the study is to determine whether electrode implants can be used to restore a variety of movements and communication in patients with paralysis caused by stroke, neurodegenerative disease or traumatic brain injury.
Ganguly: The goal is restoration of speech and of the ability to reach, grasp and manipulate objects in the environment. BRAVO1 has indicated the importance of both speech and mobility and has been learning to move a robotic arm and control a computer cursor.
Q: Does this research help restore the ability to communicate for a person with paralysis?
Chang: This trial tells us that, yes, we can restore words to someone who’s lost speech from paralysis. It’s the very beginning, but it tells us that this is possible. In this trial we’ve focused on text as the form of communication, but we’re also actively working on restoring the actual voice through a synthetic generator of speech.
Ganguly: For a person with paralysis to communicate generally, there’s also a need to control devices like a computer cursor. I think the BRAVO trial is unique in that it’s trying to address the entire spectrum. This one device can restore words to a person with paralysis and allow that person to move a robotic arm to control a computer cursor.
Q: How is decoding speech different from reading thoughts?
Neurosurgeon Edward Chang performs brain surgery at UCSF.
Photo: Barbara Ries
Chang: Internal thoughts are very complex and distributed throughout the brain. We understand how the brain controls the muscles of the vocal tract when we speak, and that’s the signal we’re tapping into to restore communication.
Ganguly: This technology requires active, overt attention from the user; it is very easy for the user to disengage from the device.
Q: What does this research mean for brain-machine interfaces?
Chang: This represents an important step in the evolution of brain-machine interfaces. About 30 years ago, there was an important set of discoveries here at UCSF that allowed for one of the earliest and most successful brain-machine interfaces – the cochlear implant. It’s a device that has restored hearing to thousands of patients who were born deaf or lost their hearing. The device records sound signals and translates them into electrical impulses to stimulate the auditory nerve to restore hearing.
In many ways, what we’re doing is very related – but in the opposite direction. We’re trying to tap into the brain signals for what someone is attempting to say and translating that into intelligible text or verbal information.
Neurosurgeon Edward Chang performs brain surgery at UCSF. Rory Murphy, Clinical Fellow, Neurosurgery assists (L).
Photo: Barbara Reis
Ganguly: The key innovation is that the device enables us to record signals over days, months and years, allowing for complex control that’s intuitive and stable and doesn’t require a lot of relearning or recalibration. This is probably the most important aspect for daily use in the real world.
Q: What are the next steps?
Chang: This trial is just the beginning. On the hardware side, we need to build systems that have higher data resolution to record more information from the brain, and more quickly. On the algorithm side, we need to have systems that can translate these very complex signals from the brain into audible spoken words.
One of the most important priorities is to expand the vocabulary so that it’s not limited to the 50 words with which we started. We also need to make sure that what we see in this one participant can be seen with other people for a broader patient population.
Ganguly: BRAVO1 can pick up and manipulate objects by controlling the robotic arm. The next phase is for him to be able to perform daily activities, such as pouring water into a cup. We’re at the cusp of using machine learning to assist him to do these complex tasks.
Dr. Eddie Chang (right) and postdoctoral researcher David Moses, PhD, confer at UCSF’s Mission Bay campus in San Francisco.
Noah Berger
Q: What motivates you to do this work?
Chang: It’s an amazing scientific project to understand the basic mechanisms by which the brain allows us to do something that’s very uniquely human – to speak and have language. That alone has been a very fulfilling aspect of all this.
As a physician who takes care of patients with severe brain injuries, I often find it frustrating to be unable to restore lost quality of life. So it’s both the scientific knowledge and addressing a huge unmet need in medicine: to figure out how to restore communication to people who have been paralyzed.
Ganguly: As a neurologist, I find a lot of my clinical care involves working with patients who have been acutely paralyzed from stroke, spinal cord injury or brain injury. The current state of the art leaves about 50 percent of those patients permanently paralyzed, so it’s a very large unmet burden. The BRAVO trial represents some of the first steps toward restoring function, and that’s very gratifying and exciting. It offers a road map toward a broader impact.
Illustration showing placement of the eCOG electrode on the participant’s speech motor cortex and the head stages used to connect the electrode to the computer.
Illustration: Ken Probst, UCSF
Q: What is the future of this technology?
Chang: I envision that in the future, we’re going to have a fully implantable, wireless device with a high bandwidth of information-processing capability that will allow us to get far higher resolution than we currently have. I’m quite optimistic about that happening in the next couple of years.
This means that people who have been suffering because they’re not able to communicate with people they love and with their caregivers about their basic needs will be able to express some of those emotions and needs in a way that is important to their quality of life and, in many cases, their survival.
Ganguly: My hope is that in five to 10 years, this technology will be used daily. I also hope it will help to restore function for those with less severe paralysis.
UCSF Medical Center has been recognized as the nation’s No. 1 hospital for neurology and neurosurgery in the 2021-2022 Best Hospitals rankings by U.S. News & World Report. All neurology and neurosurgery research and treatment take place within the UCSF Weill Institute for Neurosciences.
To learn more or to refer a patient:
Neurology and Neurosurgery at UCSF
Phone: (800) 444-2559
