New AI development can convert brain activity into text

New AI development can convert brain activity into text
New York, 20.5.2020
In an article for „Nature Neuroscience“, Dr: Joseph Makin, and colleagues from the University of California (San Francisco) report that they believe they have „laid the foundation for a speech prosthesis“.
They recruited four participants who had electrode arrays implanted in their brains to monitor epileptic seizures. The participants were asked to read aloud several times from 50 given sentences, including „Tina Turner is a pop singer“ and „These thieves stole 30 jewels“. The team tracked their neural activity as they spoke.
This data was then fed into a machine learning algorithm that converted the brain activity data for each spoken sentence into a sequence of numbers. To ensure that the numbers related only to aspects of speech, the system compared predicted sounds with actual recorded sound from small pieces of brain activity data. The sequence of numbers was then entered into a second part of the system, which converted it into a sequence of words.
At first the system produced nonsensical sentences . But after comparing each sequence of words with the sentences that were actually read out, it improved and learned how the sequence of numbers related to the words and which words tended to follow each other.
The team then tested the system and generated written text based solely on the brain activity when speaking. The result was not perfect. Among his mistakes, „These musicians harmonize beautifully“ was decoded as „The spinach was a famous singer“, and „A roll of wire was lying near the wall“ became „The robin is wearing a yellow lily“.
However, the team found that the accuracy of the new system had increased significantly over the previous ones. While the accuracy varied from person to person, one participant only needed to correct an average of 3% of each sentence – higher than the 5% word error rate for professional human typists. The team emphasizes, however, that the algorithm processes only a small number of sentences, unlike the latter.
„If you try to go beyond the [50 sentences used], the decoding becomes much worse,“ Makin said, adding that the system probably relies on a combination of learning certain sentences, recognizing words from brain activity, and recognizing general patterns in English.
The team also found that training the algorithm on a participant’s data meant that less training data was required from the end user – something that could make training less burdensome for patients.
Dr. Christian Herff, also an expert in this field from the University of Maastricht, but not involved in this study, described the progress as follows: this new system would require less than 40 minutes of training data for each participant and a limited collection of sentences, whereas the previous system would have required millions of hours. Nevertheless, the new system is characterised by an unprecedented level of accuracy.
In his opinion, the system is not yet usable for many severely disabled patients because it is based on the brain activity recorded by people who say a sentence aloud. „Of course this is fantastic research, but these people could also just use ‚OK Google‘,“ he said. „This is not a translation of thought [but of the brain activity associated with language]“.
Herff stressed that people should not worry about others being able to read their thoughts now, because the brain electrodes need to be implanted. Meanwhile, Dr Mahnaz Arvaneh, an expert on brain-machine interfaces at the University of Sheffield, urged his colleagues to consider ethical issues now. „We are still very, very far from the point where machines can read our minds, but that doesn’t mean we shouldn’t think about it.