Brain-grounding of semantic vectors improves neural decoding of visual stimuli
Authors:
Shirin Vafaei,
Ryohei Fukuma,
Huixiang Yang,
Haruhiko Kishima,
Takufumi Yanagisawa
Abstract:
Develo** algorithms for accurate and comprehensive neural decoding of mental contents is one of the long-cherished goals in the field of neuroscience and brain-machine interfaces. Previous studies have demonstrated the feasibility of neural decoding by training machine learning models to map brain activity patterns into a semantic vector representation of stimuli. These vectors, hereafter referr…
▽ More
Develo** algorithms for accurate and comprehensive neural decoding of mental contents is one of the long-cherished goals in the field of neuroscience and brain-machine interfaces. Previous studies have demonstrated the feasibility of neural decoding by training machine learning models to map brain activity patterns into a semantic vector representation of stimuli. These vectors, hereafter referred as pretrained feature vectors, are usually derived from semantic spaces based solely on image and/or text features and therefore they might have a totally different characteristics than how visual stimuli is represented in the human brain, resulting in limiting the capability of brain decoders to learn this map**. To address this issue, we propose a representation learning framework, termed brain-grounding of semantic vectors, which fine-tunes pretrained feature vectors to better align with the neural representation of visual stimuli in the human brain. We trained this model this model with functional magnetic resonance imaging (fMRI) of 150 different visual stimuli categories, and then performed zero-shot brain decoding and identification analyses on 1) fMRI and 2) magnetoencephalography (MEG). Interestingly, we observed that by using the brain-grounded vectors, the brain decoding and identification accuracy on brain data from different neuroimaging modalities increases. These findings underscore the potential of incorporating a richer array of brain-derived features to enhance performance of brain decoding algorithms.
△ Less
Submitted 22 March, 2024;
originally announced March 2024.
Voluntary control of semantic neural representations by imagery with conflicting visual stimulation
Authors:
Ryohei Fukuma,
Takufumi Yanagisawa,
Shinji Nishimoto,
Hidenori Sugano,
Kentaro Tamura,
Shota Yamamoto,
Yasushi Iimura,
Yuya Fujita,
Satoru Oshino,
Naoki Tani,
Naoko Koide-Majima,
Yukiyasu Kamitani,
Haruhiko Kishima
Abstract:
Neural representations of visual perception are affected by mental imagery and attention. Although attention is known to modulate neural representations, it is unknown how imagery changes neural representations when imagined and perceived images semantically conflict. We hypothesized that imagining an image would activate a neural representation during its perception even while watching a conflict…
▽ More
Neural representations of visual perception are affected by mental imagery and attention. Although attention is known to modulate neural representations, it is unknown how imagery changes neural representations when imagined and perceived images semantically conflict. We hypothesized that imagining an image would activate a neural representation during its perception even while watching a conflicting image. To test this hypothesis, we developed a closed-loop system to show images inferred from electrocorticograms using a visual semantic space. The successful control of the feedback images demonstrated that the semantic vector inferred from electrocorticograms became closer to the vector of the imagined category, even while watching images from different categories. Moreover, modulation of the inferred vectors by mental imagery depended on both the image category and time from the initiation of imagery. The closed-loop control of the semantic vectors revealed an asymmetrical interaction between visual perception and imagery.
△ Less
Submitted 7 November, 2021;
originally announced December 2021.
