Decoding visual brain representations from electroencephalography through Knowledge Distillation and latent diffusion models
Authors:
Matteo Ferrante,
Tommaso Boccato,
Stefano Bargione,
Nicola Toschi
Abstract:
Decoding visual representations from human brain activity has emerged as a thriving research domain, particularly in the context of brain-computer interfaces. Our study presents an innovative method that employs to classify and reconstruct images from the ImageNet dataset using electroencephalography (EEG) data from subjects that had viewed the images themselves (i.e. "brain decoding"). We analyze…
▽ More
Decoding visual representations from human brain activity has emerged as a thriving research domain, particularly in the context of brain-computer interfaces. Our study presents an innovative method that employs to classify and reconstruct images from the ImageNet dataset using electroencephalography (EEG) data from subjects that had viewed the images themselves (i.e. "brain decoding"). We analyzed EEG recordings from 6 participants, each exposed to 50 images spanning 40 unique semantic categories. These EEG readings were converted into spectrograms, which were then used to train a convolutional neural network (CNN), integrated with a knowledge distillation procedure based on a pre-trained Contrastive Language-Image Pre-Training (CLIP)-based image classification teacher network. This strategy allowed our model to attain a top-5 accuracy of 80%, significantly outperforming a standard CNN and various RNN-based benchmarks. Additionally, we incorporated an image reconstruction mechanism based on pre-trained latent diffusion models, which allowed us to generate an estimate of the images which had elicited EEG activity. Therefore, our architecture not only decodes images from neural activity but also offers a credible image reconstruction from EEG only, paving the way for e.g. swift, individualized feedback experiments. Our research represents a significant step forward in connecting neural signals with visual cognition.
△ Less
Submitted 8 September, 2023;
originally announced September 2023.
An intertwined neural network model for EEG classification in brain-computer interfaces
Authors:
Andrea Duggento,
Mario De Lorenzo,
Stefano Bargione,
Allegra Conti,
Vincenzo Catrambone,
Gaetano Valenza,
Nicola Toschi
Abstract:
The brain computer interface (BCI) is a nonstimulatory direct and occasionally bidirectional communication link between the brain and a computer or an external device. Classically, EEG-based BCI algorithms have relied on models such as support vector machines and linear discriminant analysis or multiclass common spatial patterns. During the last decade, however, more sophisticated machine learning…
▽ More
The brain computer interface (BCI) is a nonstimulatory direct and occasionally bidirectional communication link between the brain and a computer or an external device. Classically, EEG-based BCI algorithms have relied on models such as support vector machines and linear discriminant analysis or multiclass common spatial patterns. During the last decade, however, more sophisticated machine learning architectures, such as convolutional neural networks, recurrent neural networks, long short-term memory networks and gated recurrent unit networks, have been extensively used to enhance discriminability in multiclass BCI tasks. Additionally, preprocessing and denoising of EEG signals has always been key in the successful decoding of brain activity, and the determination of an optimal and standardized EEG preprocessing activity is an active area of research. In this paper, we present a deep neural network architecture specifically engineered to a) provide state-of-the-art performance in multiclass motor imagery classification and b) remain robust to preprocessing to enable real-time processing of raw data as it streams from EEG and BCI equipment. It is based on the intertwined use of time-distributed fully connected (tdFC) and space-distributed 1D temporal convolutional layers (sdConv) and explicitly addresses the possibility that interaction of spatial and temporal features of the EEG signal occurs at all levels of complexity. Numerical experiments demonstrate that our architecture provides superior performance compared baselines based on a combination of 3D convolutions and recurrent neural networks in a six-class motor imagery network, with a subjectwise accuracy that reaches 99%. Importantly, these results remain unchanged when minimal or extensive preprocessing is applied, possibly paving the way for a more transversal and real-time use of deep learning architectures in EEG classification.
△ Less
Submitted 4 August, 2022;
originally announced August 2022.
