Functional magnetic resonance imaging (fMRI) captures coarse, colorful snapshots of the brain during activity. This particular type of magnetic resonance imaging has revolutionized cognitive neuroscience, but it is not a mind-reading machine.
But scientists are gradually tackling a fundamental barrier to using brain images to translate internal experiences into words. This technology can help people who are unable to speak or communicate outwardly, such as those who have had a stroke or amyotrophic lateral sclerosis.Current brain-computer interfaces include devices in the brain. However, neuroscientists hope to use non-invasive techniques such as fMRI to decode internal voices without the need for surgery.
Now, researchers have taken a step forward by combining fMRI’s ability to monitor neural activity with the predictive power of artificial intelligence language models. Hybrid technology has created a decoder that can recreate the story a person hears or imagines in a scanner with amazing accuracy. The decoder was even able to infer the story behind the short film someone saw on the scanner, but with less accuracy.
“Brain data contains a lot more information than we originally thought,” Jerry Tang, a computational neuroscientist at the University of Texas at Austin and lead author of the study, said at a news conference. I was.A study published on Monday Nature Communications, This is what Tang describes as “a proof of concept that language can be deciphered from non-invasive recordings of brain activity.”
Decoder technology is still in its infancy. It requires extensive training for each person who uses it, and it does not produce accurate transcriptions of the words they hear or imagine. But it’s still a notable progress. Researchers know that an early relative of the model behind ChatGPT, an AI language system, could help make informed inferences about the words that triggered brain activity just by looking at fMRI brain scans. increase. Although current technical limitations prevent decoders from being widely used, for better or worse, the authors stress the need to enact aggressive policies to protect the privacy of internal mental processes. increase. “What we’re getting is a kind of ‘gist,’ or paraphrasing, of what the original story was,” said Alexander Huth, a computational neuroscientist at the University of Texas at Austin and senior author of the study. It’s like,” he said.
An example of what one research participant heard, as transcribed in the paper, was: Examining the person’s brain scan, the model continued to decipher.
Anna Ivanova, a neuroscientist at the Massachusetts Institute of Technology who was not involved in the study, said: “Overall, there is still a long way to go, but the current results represent what has been achieved so far in fMRI language decoding. It’s better than that,” he said.
This model misses a lot about the story it deciphers. They struggle with grammatical features such as pronouns. It cannot decipher proper nouns such as names and places. However, it achieves higher accuracy than the conventional method. Between 72% and 82% of the time in the story, the decoder decoded the meaning more accurately than would be expected from random chance.
Martin Schrimpf, a computational neuroscientist at the Massachusetts Institute of Technology, said: Previous attempts to decipher brain activity using AI models had some success, but eventually hit a wall. Here, Tang’s team used “a much more accurate model of the language system,” he says Schrimpf. That model was He GPT-1, which appeared in 2018, and was the original version of He GPT-4, the model that now underpins ChatGPT.
Neuroscientists have been working for decades to decode fMRI brain scans to connect with people who are unable to communicate outwardly. In a major 2010 study, scientists used her fMRI to pose yes or no questions to individuals who were seemingly unconscious and unable to control their bodies.
But deciphering whole words and phrases is a bigger challenge. The biggest obstacle is fMRI itself, which tracks slow changes in the blood flow that oxygenates neurons, rather than directly measuring rapid neuronal firing in the brain. Tracking these relatively slow changes leaves the fMRI scan temporarily ‘blurry’. Think of a long exposure photograph of a busy city sidewalk. Facial features are blurred by motion. Using an fMRI image to try to determine what happened in the brain at a particular moment is like trying to identify the person in the picture. This is an obvious problem in deciphering language, where a single fMRI image captures up to ~20 word responses and flies at high speed.
Now, the predictive power of AI language models seems to help. In the new study, three participants lay motionless in an fMRI scanner for a total of 16 hours over her 15 sessions. Through headphones, I listened to excerpts from podcasts and radio shows. moss radio hour and of New York Times’ modern love. Meanwhile, the scanner tracked blood flow in different areas of the brain associated with language. Using these data, he then trained an AI model to find patterns in how each subject’s brain activated in response to specific words and concepts.
After uncovering these patterns, the model took a new set of brain images and predicted what people were hearing at the time they were taken. He incrementally progressed through the story, comparing new scans to patterns of many candidate words predicted by the AI. To avoid having to check every word in English, the researchers used her GPT-1 to predict the words most likely to occur in a given context. This created a small pool of possible word sequences from which we could select the most probable candidates. GPT-1 then proceeded to the next set of words until it deciphered the entire story.
Researchers used the same method to decipher stories that participants only imagined. They asked participants to imagine themselves telling her detailed one-minute story. Although the decoder was less accurate, it performed better than expected compared to random chance. This indicates that similar brain regions are involved in imagining something and simply perceiving it. The ability to translate imagined speech into words is important for designing brain-computer interfaces for people who cannot communicate verbally.
Moreover, the findings transcend language. In the most surprising result, the researchers had people watch animated shorts without sound through a scanner. Despite being explicitly trained in spoken language, the decoder was able to decipher the story from brain scans of participants watching the silent film. I was more amazed by the video than by the speech above,” says Huth. “I think we’re deciphering something deeper than language,” he said at a press conference.
Still, it will be years before this technology is used as a brain-computer interface in everyday life. For one, scanning technology is not portable. fMRI machines occupy entire rooms in hospitals and research institutions and cost millions of dollars. But Huth’s team is working to adapt these findings to existing brain-imaging systems that can be worn like a cap, such as functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG).
New research techniques also require extensive customization, requiring hours of fMRI data for each individual. “This is different from earbuds that just put them on and work,” he says Schrimpf. For each user he has to train an AI model to “adapt and tune the brain,” he adds. Schrimpf speculates that in the future, as researchers uncover what people’s brains have in common, the technology will need less customization. In contrast, Huth believes that more accurate models are more detailed and require more precise customization.
The team also tested the technology to see what would happen if someone resisted or interfered with the scan. Research participants can disguise it simply by telling a different story in their heads. When researchers asked participants to do this, the results were gibberish, says Huth. says. “[The decoder] It just collapsed completely.
Even at this early stage, the authors emphasize the importance of considering policies that protect the privacy of our inner words and thoughts.
