Functional magnetic resonance imaging (fMRI) is one of the most advanced tools for understanding how we think. As a person completes various mental tasks on her fMRI scanner, the machine produces fascinating and colorful images of the brain.
By looking at someone’s brain activity in this way, neuroscientists can tell which brain regions the person is using but not using. what The individual thinks, sees, and feels. Researchers have been trying to crack that code for decades. And now they’re making serious strides in using artificial intelligence to process numbers. It converted a person’s brain activity into an image that was uncannily similar to what they saw during the scan. The original and recreated images can be viewed on the researcher’s website.
“Using this kind of technology, we can build potential brain-machine interfaces,” says Yu Takagi, a neuroscientist at Osaka University in Japan and one of the authors of the study. Such future interfaces could one day help people who are currently uncommunicative, such as individuals who may appear unresponsive on the surface but may still be conscious. It was recently approved for presentation at the 2023 Computer Vision and Pattern Recognition Conference.
The study has been making headlines online since it was posted as a preprint (i.e., not yet peer-reviewed or published) in December 2022. But that description exaggerates what the technology can do, experts say.
As Shailee Jain, a computational neuroscientist at the University of Texas at Austin, said: He was not involved in this new study. “At the moment, I think the technology is far from being used to actually help patients or for the worse. But we are getting better every day.”
The new study is far from the first to use AI for brain activity to reconstruct the images people see. In a 2019 experiment, researchers in Kyoto, Japan used a form of machine learning called deep neural networks to reconstruct images from fMRI scans. The result looked more like an abstract painting than a photograph, but a human judge was able to match the AI-generated image exactly to the original.
Since then, neuroscientists have continued this work with newer and better AI image generators. In a recent study, researchers used Stable Diffusion, a so-called diffusion model from London-based startup Stability AI. Diffusion models (a category that also includes image his generators such as DALL-E 2) are “the protagonists of the AI explosion,” says Takagi. These models learn by adding noise to the training images. Like noise on a TV, noise distorts the image, but the model starts learning in a predictable way. Ultimately, a model can build an image from “static” alone.
Launched in August 2022, Stable Diffusion has been trained on billions of photos and their captions. Now that it has learned to recognize patterns in photos, it can combine visual features on command to generate entirely new images. “Simply say ‘a dog on a skateboard’ and you get a dog on a skateboard,” says Iris Groen, a neuroscientist at the University of Amsterdam. “Once we got that model, the researchers said, ‘OK, can we connect it to brain scans in a smart way?'”
The brain scans used in the new study are from a research database containing results from a previous study in which eight participants regularly lay in an fMRI scanner and agreed to see 10,000 images over a one-year period. . The result is a vast repository of fMRI data that shows how the human brain’s visual center (or at least the brains of these eight human participants of hers) reacts to each image. . In a recent study, researchers used data from her four of the original participants.
To generate a reconstructed image, an AI model needs to process two kinds of information: the low-level visual characteristics of the image and its high-level meaning. For example, an airplane in the sky instead of an angular elongated object on a blue background. The brain also processes these two types of information and processes them in different areas. To connect brain scans and AI, researchers used linear models to combine the parts that deal with low-level visual information. I also did the same for the part that handled the high-level conceptual information.
“We were able to generate these images by basically mapping them to each other,” says Groen. The AI model can then learn which image features correspond to subtle patterns in a person’s brain activation. Once the model was able to recognize these patterns, the researchers fed it never-before-seen fMRI data and told it to generate corresponding images. Finally, the researchers were able to compare the generated images with the original images to check the performance of the model.
Many of the pairs of images the authors present in their study are strikingly similar. “What excites me is that it works,” said Ambuj Singh, a computer scientist at the University of California, Santa Barbara, who was not involved in the research. Still, scientists haven’t figured out exactly how the brain processes the visual world, he says, Singh. Stable diffusion models do not necessarily process images in the same way as the brain, even if they can produce similar results. The authors hope that comparing these models to the brain will shed light on the inner workings of both complex systems.
This technology may sound fantastic, but it has many limitations. Each model has to be trained and used on just her one person’s data. Lynn Lee, a computational neuroscientist at Radboud University in the Netherlands, who was not involved in the study, said: If you want AI to reconstruct images from brain scans, you need to train a custom model. To do so, scientists need large amounts of high-quality fMRI data from the brain. Unless he agrees to sit perfectly still inside the MRI tube and focus on thousands of images, existing AI models don’t have enough data to start deciphering brain activity. .
Even with these data, Jain explains, AI models are only suitable for explicitly trained tasks. A model trained on how to perceive an image will not work when trying to decipher the concepts you have in mind. However, some research teams, including Jain, are building a different model for that.
It is not yet known whether this technique will help reconstruct images that participants have only imagined but not seen with their eyes. That ability is necessary for many applications of this technique. For example, using brain-computer interfaces to help people who cannot speak or gesture communicate with the world.
“There is a lot to be gained neuroscientifically from building decoding technology,” says Jain. But with potential benefits come potential ethical challenges, and as these technologies improve, they will become even more important to address. It’s not a good enough excuse to downplay the harm,” she says. “I think now is the time to think about privacy and negative uses of this technology, but we may not be at the stage where it can happen.”
