'Crying with joy': Brain-implant patient speaks flawlessly with AI voice

With the help of a brain implant, a man paralyzed by amyotrophic lateral sclerosis can speak fluently again using an artificial intelligence voice. What is particularly exceptional is the accuracy with which the nervous system translates the signals, which is unique until now.

Neurosurgeons at the University of California, Davis, have developed and tested a new brain-computer interface (BCI) that can translate brain signals into speech with up to 97 percent accuracy. According to a press release, it’s the most accurate system of its kind to date, and a study on the work was published in the New England Journal of Medicine.

In July 2023, scientists inserted a brain implant into 45-year-old Casey Harrell, who had been suffering from amyotrophic lateral sclerosis (ALS) for five years at the time, a devastating disease that destroys the nerve cells that control movement in the body.

According to Aerzteblatt.de, the patient's speaking rate dropped to 6.8 words per minute. For an English speaker, 160 per minute is normal. Harel could only communicate by slowly typing words on the screen using a gyroscopic mouse that transferred his head movements to the cursor.

Electrodes instead of neurons

Neurosurgeons placed four arrays on Harel's meninges. Using a total of 256 microelectrodes, these devices intercept motor signals to the muscles of the lips, jaw, tongue and larynx, which are normally transmitted via nerve cells. The signals are then transmitted to a computer, which interprets them using software. This happens almost in real time, meaning that the patient can speak almost fluently with certain pauses between words, according to Aerzteblatt.de.

The patient was able to operate the system within a short period of time after activation, the researchers wrote. During the first training on speech data, the BCI needed 30 minutes to reach 99.6 percent word accuracy with a vocabulary of 50 words.

“When we first tried the system, he cried with joy when the words he was trying to say correctly appeared on the screen,” says neuroscientist Sergei Stavisky. “Everyone in the room will cry.” By the second session, the potential vocabulary had increased to 125,000 words, with the BCI achieving an accuracy of 90.2 with just 1.4 additional hours of training data.

Precision never known before

Their work represents the most accurate speech neural prosthesis known to date, says David Brandman, Stavisky’s colleague. “At this point, we can correctly decode what Casey is trying to say 97 percent of the time. That’s better than many commercially available smartphone apps that try to interpret a person’s voice.” Brandman said the technology is groundbreaking because it gives hope to people who want to speak but can’t. “I hope that in the future, technology like this BCI will help patients talk to their families and friends.

This technology is not fundamentally new, as other patients can also speak again thanks to brain implants. However, previous speech BCI systems often had errors in words, Brandon explains. “This made it difficult for the user to be consistently understood and was a barrier to communication. Our goal was to develop a system that would allow the user to be understood whenever they wanted to speak.”

Your own voice through artificial intelligence

A special experience for Harrell is also his ability to use his voice. Because what he says isn’t just displayed on a screen. Scientists trained an AI model to produce a voice over computer speakers using a speech Harrell gave before he developed ALS.

You can see and hear how well this works in a video from the university. The sound is just right, says Harrell. “It makes people cry who haven’t heard me in a long time.” He hopes that soon, people like him will have the opportunity to use a device like this to help them communicate.

Harrell says he’s still able to work full-time in climate protection. And with this device, he can do it better and more efficiently. “It’s very frustrating and frustrating not being able to communicate,” Harrell says. “It’s like being trapped.” Something like this technology would help those affected get back into life and into society.