Millions of people worldwide have lost the ability to use their upper limbs or the ability to speak due to shots, paralysis, or Lou Gehrig’s disease. These people can no longer write words out by hand, but they can still think about doing so. Now, scientists have figured out how to turn those thoughts into a typed-out message using a new brain-computer interface (BCI). They could use the interface to make imagined handwriting “write” out complete sentences as fast as other types of assistive communication.
Study co-author Krishna Shenoy, Ph.D., the Hong Seh and Vivian W. M. Lim Professor at Stanford University and a Howard Hughes Medical Institute (HHMI) Investigator, said:
Just think about how much of your day is spent on a computer or communicating with another person. Restoring the ability of people who have lost their independence to interact with computers and others is extremely important, and that is what is bringing projects like this one front and center.
Some professionals are hailing this as an “important milestone” in developing the technologies required to improve life, even though the user interface is a proof-of-concept finding only tested in one individual.
John Ngai, Ph.D., the NIH BRAIN Initiative’s director, said:
This study represents an important milestone in developing BCIs and machine learning technologies that are unraveling how the human brain controls processes as complex as communication. This knowledge is providing a critical foundation for improving the lives of others with neurological injuries and disorders.
The ability to communicate is mainly lost without technological intervention for people living in “locked-in syndrome” or paralyzed. This is a significant global issue that needs to be addressed.
Debara L, Tucci, M.D., M.B.A, M.S., NIDCD’s director, said:
Communication is central to how we function in society. In today’s world of internet-based communication, people with severe speech and physical impairments can face significant communication barriers and, potentially, isolation. We hope these findings will encourage commercial development of this latest BCI technology.
A Stanford University-led team of scientists developed this new BCI setup, which involves two implanted arrays of electrodes that monitor the electrical activity of about 200 neurons in the brain’s motor cortex – the region of the brain responsible for fine movement, thus controlling hand motions. The motor cortex can produce the same signals related to conscious movement even when a person can’t move their limbs anymore. Other teams have developed similar BCI systems to restore motor function through devices like robotic arms.
The brain-to-text BCI system is part of an international collaboration called BrainGate2. The researchers implanted electrodes on the surface of a 65-year-old quadriplegic male volunteer‘s brain and recorded the complex patterns of neural activity employed when he visualized writing individual letters by hand. They had him imagine writing 26 letters of the alphabet in lower case and the punctuation symbols ~ and > which represented a period and space, respectively.
The team decoded the electrical activity of approximately 200 different neurons and recoded each one. The neurons responded differently when he mentally “wrote” each character – so each of the letters’ recording is unique unto itself. This formed their bank of predictions of what letter he was trying to make.
Shenoy said:
Basically, what that means is that when you’re making the shape of a letter, you get a very unique pattern of electrical activity that [co-author Frank Willett’s] algorithms that are based in machine learning can readily interpret.
This entire process was also being channeled to a computer training a machine-learning algorithm to identify neural patterns representing individual characters. After the training period, they used this algorithm to translate the participant’s thought of writing a letter into its corresponding symbol on the screen as text. The computer displayed the letters in real-time.
Shenoy continued:
The future really is — as we learn more and more about the brain — [that], we should be able to interact with it and help overcome dysfunction as well as to understand how it normally functions.
Right now, other investigators can achieve about a 50-word dictionary using machine learning methods when decoding speech. By using handwriting to record from hundreds of individual neurons, we can write any letter and thus any word which provides a truly ‘open vocabulary’ that can be used in almost any life situation.
The man could write out complete sentences one letter at a time using brain-to-text technology. The system proved to be more efficient and accurate even than existing communication BCIs. The volunteer composed sentences at around 90 characters per minute with a 94% raw accuracy and 99% accuracy with autocorrect. That’s about how fast a person of similar age could type on a smartphone.
To compare, “point-and-click” interfaces that require paralyzed users to move an on-screen cursor to compose messages mentally have only achieved about 40 characters per minute.
Frank Willett, a research scientist at Stanford, told Axios:
It’s cool to finally be able to get speeds that are comparable to normal handwriting or comparable to smartphone typing in this age group.
Jennifer Collinger, an associate professor at the University of Pittsburgh who was not involved in this study, said:
[These findings] are exciting and interesting [for several reasons.] From a practical communications standpoint, the new technique appears to double the current rate of assistive communications. But, more than that, I would not have thought to try to decode handwriting. It seems like a very challenging problem: How will it be better than accessing an on-screen keyboard that people have been working towards for decades? The fact they were able to achieve such a high level of performance is interesting. They were able to show if you use both direction information and temporal variability, you can get very responsive, very accurate performance.
The scientist’s next steps include adding more characters to the system (like numbers and capital letters) and testing the technology with a volunteer that has lost the ability to speak.
The video below presents the existing version of the system in use:
