Adaptive Modality Balanced Online Knowledge Distillation for Brain-Eye-Computer based
Dim Object Detection

EfficientNet [22], a well-designed convolutional neural network, has demonstrated outstanding advancements in accuracy and computational efficiency. This achievement is attributed to the effective balance of network depth, width, and resolution, resulting in enhanced recognition performance. In this study, EfficientNet-B0 was used because of its high efficiency, achieving comparable accuracy to ResNet-50 while requiring fewer parameters and floating point operations for the same input size.

Three main components constitute EfficientNet-B0: convolutional neural networks, batch normalization functions, and mobile inverted bottleneck convolution, inspired by the MobileNet [19] concept. MBConv consists of three types of layers: convolutional layers, depth-wise convolutional layers, and squeeze-and-excitation networks (SENet) [56]. The convolutional layer expands and compresses channels, whereas the depth-wise convolutional layer reduces the parameter count. However, SENet focuses on channel relationships by assigning importance weights to various channels.

The parameters of EfficientNet in AMBOKD were initialized using pre-training on the ImageNet dataset to ensure optimal performance and efficiency. This enables the leverage of knowledge gained from the ImageNet dataset for improved results in our specific task.

The effective representation of visual images from the visual encoder can be easily extracted with the help of EefficientNet. The representation of the visual domain is computed as follows:

MCGRAM [8] is a compact convolutional neural network specifically designed for EEG signals, which comprises three primary components: the frequency encoder, spatial encoder, and temporal encoder.

The frequency encoder can effectively learn frequency-related features and capture the spectral characteristics of EEG signals by employing specific kernels in the multi-scale convolution module. Spatial representations are learned by the spatial encoder using a graph convolution module, which exploits the inherent spatial relationships between various electrode channels, enabling the spatial patterns present in the EEG data to be captured by MCGRAM. To extract temporal features and obtain the final global representation of the EEG modality, and the temporal encoder uses a two-layer long short-term memory network and a self-attention module. This combination allows temporal dependencies to be effectively modeled and crucial temporal patterns in the EEG signals to be captured by the model.

In summary, MCGRAM can capture and encode relevant information from EEG signals by cascading frequency, spatial, and temporal blocks. This resulted in a comprehensive and informative representation for further analysis and classification tasks. The representation of the EEG modality is computed as:

It should be noted that alternative algorithms can be employed as substitutes for the visual and EEG encoders. For example, algorithms such as ResNet-50 [18] and MobileNetV2 [19] can be used in the visual encoder, whereas EEGNet [57] and TSception [58] are viable options in the EEG encoder. The superiority and generalization of the AMBOKD approach proposed in this study can be effectively demonstrated by incorporating these alternative algorithms and conducting subsequent experimental validations (see Section V-E).

In the online knowledge distillation training process, the KD loss from the teacher model encapsulates the distilled knowledge. Most existing methods [49, 61, 53] compute the total training loss by adding the cross-entropy loss to the knowledge distillation loss, typically setting $\alpha$ and $\beta$ to 1 in Eq. (9). However, due to modality differences, simply averaging fails to effectively capture the superior knowledge from the teacher model. Furthermore, manually setting a fixed distillation loss weight often entails extensive experimentation and significant time costs.

To address the challenges mentioned above, we introduce a weight modulation block that dynamically calculate the weights for KD losses from different teacher models. This block efficiently extract knowledge during multimodal mutual learning. The entire workflow is described in Algorithm 1 in detail. The saturation function $\text{Sat}\left(\cdot\right)$ is defined as follows for $a<b$: $\text{Sat}\left(x,a,b\right)=a$ if $x\geq a$, $\text{Sat}\left(x,a,b\right)=x$ if $a\geq x\geq b$, $\text{Sat}\left(x,a,b\right)=b$ if $x\leq b$. The parameters $\alpha_{\min}$ and $\alpha_{\max}$ denote the lower and upper bounds of $\alpha$, similar to $\beta_{\min}$ and $\beta_{\max}$ for $\beta$.

Current multimodal methods lack the consideration of the modality imbalance optimization. Thus, we propose a dynamic gradient modulation block to dynamically balance the modal optimization level in multimodal mutual learning process, ensuring full optimization of different modalities.

We further analysis the imbalance optimization phenomenon during mutual learning by calculating the backward gradient of each modality. In the backward phase of the training process, when the fusion modality acts as the student, the gradient depends on the CE loss can be calculated as $\frac{\partial\mathcal{L}_{CE}^{f}}{\partial G_{f}^{i}}$ using the following formulas. The CE loss of the fusion modality, denoted by $\mathcal{L}_{CE}^{f}$, is given by

where $G_{f\left(y_{i}=c\right)}^{i}$ is the predicted value when the true label $y_{i}$ equals class $c$, and $M$ is the total number of categories. $\operatorname{G}_{f}^{i}$ is the $i$-th logit output of the fusion model, calculated as

where $\phi_{v}(\theta_{v},\cdot)$ and $\phi_{e}(\theta_{e},\cdot)$ represent the visual and EEG modality encoders, respectively, with $\theta_{v}$ and $\theta_{e}$ as their parameters. $W_{f}^{e}$ and $W_{f}^{v}$ are the weight matrices that determine the relative significance of the features extracted from the visual and EEG modalities. $W_{f}$ and $b_{f}$ act as the parameters in the fusion process.

Combined with Eq. (10) and Eq. (11), the gradient can be expressed as

$\phi_{v}(\theta_{v},F_{v}^{i})$ and $\phi_{e}(\theta_{e},F_{e}^{i})$ are simplified as $\phi_{v}^{i}$ and $\phi_{e}^{i}$ for convenience. When a particular modality, such as the visual modality, exhibits classification performance, it contributes more to the gradient $\frac{\partial\mathcal{L}_{CE}^{f}}{\partial G_{f}^{i}}$ through the expression $W_{f}W_{f}^{v}\phi_{v}^{i}$, thereby leading to a lower loss globally. Therefore, the EEG modality, characterized by lower confidence in accurate predictions, will obtain limited optimization during parameter updates via backpropagation. This phenomenon shows that the unimodal model might be overtrained or underoptimized when the training of the multimodal model is about to converge.

In this imbalanced training phenomenon, the disparate learning efficiencies of unimodality hinder the performance of the fusion model, resulting in under-optimized and overfitting representations that constrain the overall model performance. Thus, our proposed dynamic modulation block aims to dynamically balance the learning levels of modalities until they reach their optimal performance.

The updating process is presented in Algorithm 2. Specifically, the ratios $R^{S}_{(t)}$, $R^{T_{a}}_{(t)}$ and $R^{T_{b}}_{(t)}$ are computed for indicating the current training optimization level of each modality. To ensure consistent training optimization progress and maximize the completeness and effectiveness of the optimization in multimodal mutual learning, the dynamic learning progress ratio in $t$-th iteration step is computed as:

where $R_{\min}$ and $R_{\max}$ represent the lower and upper bounds in the function, $\gamma$ is a hyper-parameter to control the degree of modulation.

The extensively used adaptive moment estimation (Adam) optimization approach is employed in the backpropagation process. The primary idea underlying the Adam algorithm is to maintain a running average of the first-order moment (mean) and the second-order moment (variance) of the gradients. This facilitates the estimation of adaptive learning rates for various parameters during optimization. Using the dynamic modulation ratio $R_{(t)}^{DG}$, the parameters $\theta^{S}$ in the student model are optimized adaptively based on the modality learning progress. The updating process is expressed as

where $\eta$ is the learning rate and $\epsilon$ represents a small constant added to the denominator for numerical stability. $D^{\rm{mean}}_{(t)}$ and $D^{\rm{var}}_{(t)}$ represent the exponentially decaying average of the past gradients’ first moment estimate (mean) and second moment estimate (variance) at time $t$.

Incorporating the AMB module into the loss computing process and the optimization process addresses the challenge of modality imbalance in the multimodal mutual learning process. This mechanism ensures that all modalities learn the effective knowledge and optimized at a comparable pace during mutual learning. Consequently, enhanced completeness, integration, and efficiency are achieved by the fusion model.

The visual encoder uses EfficientNet [22] as the basic approach for feature extraction from the input images. Furthermore, for ablation analysis, ResNet-50 [18] and MobileNetV2 [19] are used as alternative visual encoders. The EEG encoder is based on our previous study called MCGRAM [8], which is employed to extract features from the EEG data. In addition, TSception [58] and EEGNet [57] are used as alternative EEG encoders for the ablation analysis. The parameters of the visual and the EEG encoders are set to follow the configurations as described in the previous study, providing consistency and comparability with the existing studies. To align the features obtained from the visual and EEG encoders, FC layers are used to scale the lengths of both sets of features to a common length of 64. This ensures compatibility for subsequent processing and fusion of the multimodal features.

The number of heads is set to 2 in the multi-head self-attention module of the fusion model. The kernel sizes of the FC layers for alignment are configured as $1280\times 64$ and $256\times 64$ for visual and EEG features, respectively. Before computing the Q, K, and V matrices, the matrices in the FC layer have a size of $64\times 64$. The FC layers have a size of $128\times 2$ in the classifier of the Fusion domain.

The temperature coefficient $\tau$ is set to 4 in the OKD approach to control the softness of the KD targets. The hyperparameters $\alpha$ and $\beta$ are dynamically adjusted by the AMB module, which contributes to the regularization component of the overall loss function. The hyperparameter $\gamma$ is set to 3 to fine-tune the sensitivity of the AMB module. In addition, we impose lower and upper limits, $R_{\min}$ and $R_{\max}$ at 0.1 and 10, respectively, to confine the range of dynamic modulation ratio $R^{DG}_{(t)}$ during the training process.

The PyTorch framework is used to implement the AMBOKD model. The Adam optimizer with a learning rate of $10^{-4}$ is used to optimize all network parameters. The model is trained for 15 epochs with a batch size of 64. The optimization loss function employed comprised the CE loss and the KD loss. A cross-validation approach is used to assess the performance of the AMBOKD model. The data of each subject are chosen as the validation set, whereas the data of the remaining nine subjects are employed as the training set. We repeat this process for each fold with five different seeds, and we obtain the final experimental finding by averaging the results of all the folds and seeds.

A comprehensive set of metrics, including area under the receiver operating characteristic curve (AUC), accuracy (ACC), F1 score (F1) and Precision are used to evaluate the performance of classification models. Each metric is analyzed in terms of mean and standard deviation, offering insight into the reliability of the model and its ability to generalize across various datasets or conditions. Specifically, the AUC is emphasized as the main metric because of its effectiveness in measuring the discriminative ability of a model, especially in scenarios with unbalanced class distributions. To evaluate the object detection performance of the brain-eye-computer based system, AP@50:95 and AR@50:95 are used to measure the average precision and recall over Intersection Over Union (IOU) thresholds ranging from 0.50 to 0.95, while AP@50 and AR@50 are used for an IOU of 0.50. In addition, F1@50:95 and F1@50 provide a balanced view of accuracy by considering both precision and recall at these thresholds.

To verify the effectiveness of the proposed AMBOKD approach, we conduct a series of comparison experiments with the baseline and state-of-art models, including unimodal approaches EEGNet [57], MCGRAM [8], ResNet-50 [18], EfficientNet [22], multimodal fusion approach MLB [62], multimodal knowledge distillation approaches MKD [49], EMKD [51], CA-MKD [50], multimodal online knowledge distillation approaches DML [47], KDCL [52], EML [53]. In addition, the EEG and visual modal model trained in AMBOKD are named AMBOKD-E and AMBOKD-V.

As indicated in Table I, our proposed AMBOKD achieve impressive results in the dim object recognition task of subject-independent experiments. This outcome surpasses the performance of baseline and state-of-the-art approaches. Meanwhile, the mutual learning mode with AMB module enhances the performance of EEG and visual modality models: all three metrics of AMBOKD-E and AMBOKD-V surpass those of the original unimodality models MCGRAM and Efficient-Net. The AUC-based ANOVA statistical test findings demonstrate that our proposed AMBOKD, AMBOKD-E, and AMBOKD-V significantly outperform all comparison approaches in subject-independent experiments ($p<0.05$)

Table II shows the comparison results obtained from subject-dependent experiments. Notably, optimal results are achieved across all subjects using our approach, indicating significant advantages. This finding serves as a compelling demonstration of the effectiveness and adaptability of our method when applied to subject-specific experiments within practical scenarios.

These results lead to the conclusion that the AMBOKD method effectively combines the cognitive and visual domains, extracting fused global representations. Furthermore, it enables dynamic equilibrium mutual learning between unimodality models and the fusion modality during the training process, leading to improved performance in the fusion, EEG, and visual models.

We conduct experiments on CIFAR-100 dataset [63] according to previous work [64]. The baseline methods include offline knowledge distillation methods (Vanilla [15], PKT [65]), representation knowledge distillation methods (CRD [66], RKD [67]), and online knowledge distillation methods (DML [47], KDCL [52], FT-KD [64], PESF-KD [64]). In the experiment, the combination of student and teacher networks includes ResNet56-ResNet20, ResNet110-ResNet32, ResNet56-VGG8, and VGG13-VGG8.

We use the same training setup as described in [64], employing an SGD optimizer with a momentum of 0.9, a batchsize of 64, a weight decay of $5\times 10^{-4}$. The initial learning rate is set at 0.05, with a decay factor of 10 every 30 epochs starting from epoch 150. For the AMB module, we specify a learning rate of $1\times 10^{-4}$. The comparison results with three different random seeds, presented as acc(@1, mean/standard deviation), are shown in Table III. Our AMBOKD achieve the best performance among all compared methods, demonstrating its robustness and practicability.

The impact of incorporating OKD with mutual learning and AMB in multimodal learning is investigated through a detailed ablation analysis. Specifically, the role of each component is examined by progressively simplifying the proposed AMBOKD approach. MMOKD-DG and MMOKD-DK are the varients of our method, which keep the part of dynamic weights block and the dynamic gradients block in the AMB module, seprately. Then, MMOKD omits the entire AMB component, serving as a baseline to evaluate the underlying OKD and mutual learning mechanism. MKD further eliminates the online training method, representing a multi-teacher KD framework. The V-KD and E-KD approaches are further simplifications, using single-teacher KD from the visual and EEG modalities, respectively. Lastly, AMM is the most basic form of our model, which depends solely on a multi-head self-attention mechanism for modality fusion, without any KD. In addition, our analysis incorporates the unimodality encoders MCGRAM for EEG and EfficientNet for the visual domain.

The ablation results are presented in Table IV. As can be seen, these uni-modality encoders MCGRAM and EfficientNet are outperformed by the AMM model, which leverages multi-head self-attention for fusion, thereby validating the multi-head self-attention mechanism’s capacity to effectively integrate multimodal data. The results of E-KD, V-KD and M-KD further verify the effectiveness of the knowledge distillation mechanism to improve the performance of the multimodal fusion model. MMOKD outperform MKD by 1.37% and 1.9% in AUC, which demonstrate the effectiveness of mutual learning method for achieving fully modal interaction and improving multimodal model performance. The results of the MMOKD-DG and MMOKD-DK methods are further improved compared with the MMOKD method, which indicates that the adaptive balance of gradients and the interaction between different modal models can provide a more effective training scheme Our latest method, AMBOKD fuses the above two dynamic balancing schemes and achieves the optimal results, 2.08% and 2.05% higher than MMOKD, demonstrating the significant potential of modal balancing in multimodal mutual learning tasks.

The latent representations of the models in the ablation analysis are visualized using the t-distributed stochastic neighbor embedding (t-SNE) approach to further verify the feature extraction and fusion capability of our proposed approach. As depicted in Fig. 10, the feature distributions of all subjects are displayed with various colors of dots, with red dots representing the features in the target domain, and dots in other colors representing the features in the nontarget class.

Specifically, unimodality models such as EfficientNet and MCGRAM exhibit distinct visual and EEG modality characteristics in their feature representations, as illustrated in Fig. 10. The visual-based EfficientNet model exhibits a more discrete feature representation, with a large intra-class distance, indicating its superior feature extraction performance but poor generalization performance of the classifier. Conversely, the feature representation of the MCGRAM model based on EEG is more clustered, with a small intra-class distance, indicating weaker feature extraction ability but stronger classifier generalization. As a multimodal fusion model, AMM effectively combines the benefits of vision and EEG modalities, thereby further reducing the inter-class distance while maintaining the feature representation capability. By integrating single-teacher guidance mechanisms into the multimodal fusion model, V-KD and E-KD models exhibit enhanced classification performance and generalization, as reflected in larger inter-class and smaller intra-class differences in feature representation. In addition, by introducing a multi-teacher mechanism, OKD approach, and multimodal AMB approach, the discrimination of feature representation extracted by MKD, MMOKD, and AMBOKD models become more pronounced and the feature representation distances within the same class become closer. This indicates that these mechanisms enhance the capability of multimodal fusion and effectively improve the classifier’s accuracy and generalization.

In conclusion, the effectiveness of the AMBOKD framework and its modules is effectively verified by the ablation study. The proposed AMBOKD approach outperforms other methods in the multimodal fusion task, exhibiting robust discrimination between positive and negative samples, an increase in inter-class distance, a reduction in intra-class distance, and statistically significant experimental findings ($p<0.05$).

We use the dynamic learning progress ratio $R^{DG}_{(t)}$ to assess the effectiveness of the modality equilibrium. The first training data from a 10-fold cross-validation technique with a seed value of 1 was utilized.

As illustrated in Fig. 11, we draw the changes of $R^{DG}_{(t)}$ for each modality within MMOKD, MMOKD-DG, and AMBOKD. In the standard MMOKD approach, despite the intermodal learning capabilities, optimization is impeded by the inherent characteristics of each modality. Notably, the EEG modality progresses at a slower rate compared to the Visual and Fusion modalities, which exhibit quicker optimization speeds. This discrepancy in training speeds results in a suboptimal mutual learning effect among the modalities. The Dynamic Gradient (DG) block is able to addresses this issue by dynamically adjusting the training gradients according to each modality’s learning progression. With this adjustment, MMOKD-DG improves the optimization speed and the final optimization degree of each modality, especially EEG modality. AMBOKD further employs a dynamic weight modulation module tailored to KL loss, optimizing the faster-learning Visual and Fusion modalities initially, while also supporting sustained development in the EEG modality. This strategy ensures that all three modalities are optimally and uniformly enhanced throughout the training process, achieving superior overall performance.

The sensitivity control of the AMB module is based on the hyperparameter $\gamma$, which can be found in Eq. (14). As shown in Fig. 12, the AMBOKD method with $\gamma=3$ achieves the highest AUC in session 1 and second-highest AUC in session 2, resulting in an improved classification performance for imbalanced samples.