Movie clip reconstructed by an AI reading mice’s brains as they watch

We used artificial intelligence tools to extract almost entirely black-and-white movies from mouse brain signals.

Mackenzie Mathis of the Swiss Federal Institute of Technology Lausanne and her colleagues examined brain activity data from about 50 mice when they watched a 30-second movie clip nine times. The researchers then trained her AI to link this data to a 600-frame clip. In this clip, a man runs to his car and opens the trunk.

This data was previously collected by other researchers who inserted metal probes that record electrical pulses from neurons into the primary visual cortex of mice, a region of the brain involved in processing visual information. Some brain activity data were also collected by imaging mouse brains using a microscope.

Matis and her team then tested the trained AI’s ability to predict the order of frames in a clip using brain activity data collected from mice watching the movie 10 times. .

This reveals that the AI ​​can predict the correct frame within 1 second 95% of the time.

Other AI tools designed to reconstruct images from brain signals work better when trained on brain data from individual mice to predict.

To test whether this applies to AI, researchers trained an AI on brain data from individual mice. It then predicted the frame of the movie being watched with 50-75% accuracy.

“Training the AI ​​on data from multiple animals actually makes predictions more robust, so there is no need to train the AI ​​on data from specific individuals,” says Mathis.

By revealing the links between brain activity patterns and visual input, the tool could ultimately reveal how visual sensations are generated in blind people, Matisse said. .

“You can actually imagine scenarios where a visually impaired person might want to help them see the world in interesting ways by playing with the neural activity that gives them the sense of sight,” she says. increase.

This progress could be a useful tool for understanding the neural code underlying our behavior, and should be applicable to human data, says Shinji Nishimoto of Osaka University.