Images in the bottom row were recreated from brain scans of the person viewing the images in the top row
Yu Takagi and Shinji Nishimoto/Osaka University, Japan
A popular text-to-image-generating artificial intelligence can be tweaked to convert brain signals directly into images. However, the system requires extensive training using bulky and expensive imaging equipment, making daily mind readings far from reality.
Several research groups have previously generated images from brain signals using energy-intensive AI models that require fine-tuning of millions to billions of parameters.
Now, Shinji Nishimoto and Masaru Takagi of Osaka University in Japan have developed a much simpler approach using Stable Diffusion, a text-to-image generator released by Stability AI in August 2022. Their new method involves thousands instead of millions of parameters.
After successfully using Stable Diffusion, the text prompt turns into an image. It starts with random visual noise and tweaks it to produce images similar to those in training data with similar text captions.
Nishimoto and Takagi created two add-on models to drive AI with brain signals. The pair used data from four participants in a previous study that used functional magnetic resonance imaging (fMRI) to scan their brains, looking at 10,000 different pictures of landscapes, objects and people. rice field.
The pair use about 90% of the brain’s image data to create a link between fMRI data from a brain region that processes visual signals, called the early visual cortex, and the images people see. I trained the model on .
They used the same dataset to train a second model, with textual descriptions of images made by five annotators in the previous study, and from a brain region that processes image meaning called the ventral visual cortex. formed a link between the fMRI data of
After training, these two models (which had to be customized for each individual) were able to transform the brain imaging data into a form that was directly input to the stable diffusion model. Then, without training on the original images, we were able to reconstruct about 1000 images that people saw with about 80% accuracy. This level of accuracy is similar to that achieved in previous studies that analyzed the same data with a much more tedious approach.
“I couldn’t believe my eyes. I went to the bathroom, looked in the mirror, went back to my desk and looked again,” says Takagi.
However, the study only tested the approach on four people, and the mind-reading AI works better for some people than others, says Nishimoto.
Additionally, the approach requires lengthy brain-scanning sessions and huge fMRI machines, says Sikun Lin of the University of California, as the model needs to be customized to each individual’s brain. “This is totally impractical for everyday use,” she says.
In the future, a more practical version of this approach could allow people to use their imagination to create art, alter images, and add new elements to gameplay, says Lin. says.
topic:
