CSAKD: Knowledge Distillation with Cross Self-Attention for Hyperspectral and Multispectral Image Fusion ^††thanks: This study was supported partly by National Science and Technology Council (NSTC), Taiwan, under Grants NSTC 110-2636-E-006-026, 110-2222-E-006-012, 111-2634-F-007-002, 110-2218-E-006-026, and 111-2221-E-003-019-MY3. ^††thanks: (Corresponding author: Li-Wei Kang.) ^††thanks: C.-C. Hsu and C.-M. Lee are with the Institute of Data Science, and with Miin Wu School of Computing, National Cheng Kung University, Tainan, Taiwan (R.O.C.) (e-mail: [email protected], [email protected]). ^††thanks: C.-C. Ni and L.-W Kang are with the Department of Electrical Engineering, National Taiwan Normal University, Taipei, Taiwan (R.O.C.) (e-mail: [email protected], [email protected]).

In [26], a pioneer MSI Pan-sharpening framework was presented, where the goal is to fuse LR-MSI and HR-panchromatic image (with single band and high spatial resolution) of the same scene to generate an image with high spectral and spatial resolutions. Moreover, based on the sparse or low-rank image prior knowledge of HSIs, several sparse representation-based or low-rank-based image fusion frameworks were presented for the SR of HSI or HR-MSI/LR-HSI fusion [7, 8, 27, 28, 29]. For example, in [7], an SR method for LR-HSI was proposed, where the prediction of the HR-HSI is formulated as a joint derivation task of the HSI dictionary and the sparse codes relying on the spatial-spectral sparsity of HSIs. In addition, a group spectral embedding-based HR-MSI/LR-HSI fusion method was presented in [8], where the manifold structures of spectral bands and the low-rank structure of HR-HSIs were explored. A spatial and spectral fusion model was also proposed in [27] by using sparse matrix factorization to fuse remote sensing images of HR with low spectral resolution (similar to HR-MSI) and LR with high spectral resolution (similar to LR-HSI). An image fusion framework relying on spectral unmixing and sparse coding was similarly proposed in [28] to fuse HR-MSI and LR-HSI. Furthermore, a coupled sparse tensor factorization framework was presented in [29] for fusing HR-MSI and LR-HSI, where estimating the dictionaries and core tensor was formulated as a coupled tensor factorization problem. Since these traditional methods rely on some image priors, such as sparse or low-rank, some real-world scenarios not fitting these assumptions may introduce some performance degradation. While the optimization-based approach usually requires high-precision computation, and hard to deploy those algorithms into moderate AI-chip since it is hard to parallelize (e.g., eigendecomposition is often used in optimization-based methods), reducing the computational complexity with promising performance is highly desired.

DL-based strategy has shown promise in HR-MSI/LR-HSI fusion tasks. With the development of DL technology, such as the powerful representation learning ability of CNNs, several SR frameworks for LR-HSI or the fusion of HR-MSI and LR-HSI have been recently proposed. A 3-D CNN was used in [15] to fuse multispectral and hyperspectral images to generate an HR-HSI, where the dimensionality of the HSI was reduced prior to the fusion process to significantly reduce the computational complexity. A blind HR-MSI/LR-HSI fusion problem was formulated and solved based on DL in [16], where the estimation of the observation model and fusion process are optimized iteratively and alternatively during the SR reconstruction. In addition, an HSI reconstruction algorithm with a data-driven prior relying on an optimization-inspired DL was presented in [17], where the prior was learned based on both the local coherence and dynamic characteristics of HSIs. Moreover, an end-to-end DL network was proposed in [18] to jointly learn multi-scale spatial-spectral features for HR-MSI and LR-HSI fusion (denoted by MSSJFL). In addition, a lightweight deep model-based progressive zero-centric residual network (denoted by PZRes-Net) was presented in [19] for SR of HSI, where the spectral-spatial separable convolution operations with dense connections were used to efficiently learn the residual image. In [20], a dual-UNet-based architecture with a multi-stage details injection strategy was presented for fusing HR-MSI and LR-HSI, where a multi-scale spatial-spectral attention module was utilized. Furthermore, a deep hyperspectral image fusion network (denoted by DHIF-Net) was proposed in [21], where an end-to-end optimization strategy of iterative spatial-spectral regularization was implemented. On the other hand, an unregistered and unsupervised mutual Dirichlet-Net was presented in [22] for SR of HSI. An Interpretable deep neural network designed for HR-MSI/LR-HSI fusion was proposed in [23]. An interpretable deep model named by spatial–spectral dual-optimization model-driven deep network was also presented in [24] for HR-MSI/LR-HSI fusion.

However, considering that the lightweight models designed manually could be tedious and, thus, hard to guarantee their performance, we propose an effective network architecture (i.e., DTS Network) to ensure a promised performance and followed by applying the KD-based approach to reduce the computational and spatial complexity without significant performance degradation.

Directly deploying a sophisticated network into low-power devices is infeasible due to its extreme limitation of memory and computational resources. KD manner offers a solution by training efficient ”student” models guided by complex ”teacher” networks, aiming for the student to match or exceed the teacher’s performance. This process involves strategic knowledge transfer, which can be categorized into response-based [30], feature-based [31], and relation-based [32] KD schemes.

Response-based KD focuses on emulating the teacher model’s final output, enabling the student model to learn directly from these predictions, as seen in [30]. Feature-based KD expands on this by using outputs from both the final and intermediate layers of the teacher model, enriching the student’s learning with deeper insights, exemplified by [31]. Relation-based KD, on the other hand, transfers inter-layer relationships to provide a nuanced understanding of model behaviors, as detailed in [32].

The application of KD in HSI processing tasks, including segmentation and pan-sharpening [33, 34], showcases its potential for enhancing HSI and MSI fusion with lower computational and space complexity. Directing adopts response-based KD, which could be enough to distill the knowledge in the teacher network to that in the student one. In this paper, we would like to emphasize that the efficient and effective network architecture for teacher and student models is essential, while KD is a way to further reduce the complexity by knowledge transfer. Therefore, we do not focus on the selection of the KD framework in this study.

To design a lightweight network model for efficient LR-HSI/HR-MSI fusion, we aim at leveraging knowledge distillation to reduce the network complexity. However, the KD often requires the network architectures of the teacher and student to be identical, so the native network architecture should be efficient and effective to have enough space to be pruned. Inspired by conventional ensemble learning in the machine learning field, it is possible to improve the classification performance by leveraging multiple independent classifiers together, where each classifier could be simple enough. Similarly, our DTS inherits the advantages of the idea for ensemble learning but rather just simply aggregates the spectral and spatial information from the input LR-HSI/HR-MSI. Specifically, we fully exploit spectral and spatial information from the inputs to design the four different sub-networks for information aggregation to improve the fusion performance with lower computational and space complexity.

This Section illustrates the architecture design of our DTS network, consisting of spatial- and spectral-aware networks (SpaNet and SpeNet) for LR-HSI and HR-MSI, respectively, as shown in Figure 2. To effectively refer the LR-HSI and HR-MSI information jointly, the proposed SpaNet and SpeNet not only retrieve the respective feature representations from LR-HSI and HR-MSI, but also extract the joint features of LR-HSI and HR-MSI simultaneously by internal feature concatenation, as shown in the left part of Figure 2. In this way, we could easily integrate the spatial and spectral feature representation without increasing the model complexity. Then, a novel CSA fusion module is proposed to judiciously aggregate the spatial and spectral feature representations to obtain the final HR-HSI. The details can be found as follows.

Traditional knowledge distillation (KD) techniques often employ a feature map-based loss, where the student network is trained to mimic the intermediate feature representations of the teacher network. This method, while effective in some scenarios, imposes a stringent requirement on the student network to replicate the feature maps exactly as the teacher’s. Such a constraint can limit the learning capacity of the student network, particularly when the student’s architecture is much lighter, and may lead to difficulties in convergence due to the complex nature of the feature spaces involved.

The feature map-based KD loss assumes that a direct correspondence between the teacher and student feature maps is necessary for knowledge transfer. However, this can be overly restrictive, as the student network might benefit from develo** its unique feature representations that are more suited to its capacity, yet still retain the essential characteristics learned by the teacher. The forced alignment of feature maps can, therefore, be counterproductive, leading to a challenging training process and potentially suboptimal student performance.

To address these issues, the feature-map KD loss should be placed in the relatively rare layers instead of each layer to allow the student network to learn its unique feature representations in most layers, thereby improving the performance of the student network. Specifically, Sigmoid cross-entropy loss is used to approximate the feature map distributions of student and teacher networks, as follows:

where $f_{\text{s}}$ denotes the Sigmoid activation function, and ${{\bm{Z}}_{\text{fused}}^{s}}$ and ${{\bm{Z}}_{\text{fused}}^{t}}$ represent the fused feature maps from student and teacher networks. To leverage the good quality of the reconstructed HR-HSI, the reconstruction-relative loss functions should be involved to enhance the spectral and spatial quality. Traditionally, the $\ell$-1 norm distance metric aims to enhance the data fidelity, while the energy of each band in an HSI may vary significantly, leading to the that the traditional $\ell_{\text{L1}}$ distance could pay more attention to the bands whose energy is relatively large. However, the spectrum feature of HSIs is essential for different tasks since each band has its purposes. Therefore, we propose a Band-Energy-Balance-Aware (BEBA) loss $\ell_{\text{BEBA}}$ to judiciously facilitate the problem above, thereby improving the spectrum quality of the reconstructed HSI ${\bm{Y}}^{*}$.

where $\alpha$ and $\beta$ are regularization parameters, ${\bm{D}}$ denotes the squared absolute difference between the prediction and target $|{\bm{Y}}^{*}-{\bm{Y}}|^{2}$, $\epsilon$ is a small positive constant, and $f_{\text{m}}$ denotes the mean operator over spatial axis. Specifically, $\alpha=0.5$ and $\beta=1$ are chosen in our experiments. In this way, the $f_{\text{m}}({\bm{Y}}^{2}+\epsilon)$ captures the energy of each band, thereby dynamically adjusting the weights of each band according to its energy.

The parameters $\alpha$ and $\beta$ play crucial roles in balancing the sensitivity of the loss function towards small and large prediction errors. The term $\alpha$ primarily scales the mean squared error, enhancing the function’s reactivity to smaller deviations between the predicted ${\bm{Y}}^{*}$ and the ground truth HR-HSI ${\bm{Y}}$. This scaling is particularly significant when dealing with data that possess subtle variations, as it amplifies the importance of minor discrepancies.

The parameter $\beta$, on the other hand, serves as a thresholding value that delineates the boundary between small and large errors. When the squared difference ${\bm{D}}$ is less than $\beta$, the ReLU term $f_{\text{ReLU}}({\bm{D}}-\beta)$ becomes zero, and the loss function primarily operates in a quadratic regime dominated by $\alpha{\bm{D}}/\beta$. This regime is sensitive to smaller errors, thus ensuring precision in the predictions. Conversely, for larger errors where ${\bm{D}}$ exceeds $\beta$, the loss function transitions into a linear regime, mitigating the potential issues of gradient explosion typically associated with large errors in quadratic loss functions. This linear portion of the loss function is given by $f_{\text{ReLU}}({\bm{D}}-\beta)-\alpha\beta$, which acts as a safeguard against the disproportionate penalization of large errors, enhancing the robustness of the model against outliers and noise.

To enhance the spectral quality of the reconstructed HR-HSI further, Spectral Angle Mapper (SAM) loss $\ell_{\text{SAM}}$ is also proposed to guide our teacher and student networks, as follows:

where ${\bm{Y}}_{n}$ denotes the $n$-th spectral vector, and we calculate the negative cosine similarity between the reconstructed HR-HSI ${\bm{Y}}^{*}$ and the ground truth HR-HSI ${\bm{Y}}$ as the SAM loss. This measure effectively captures the angular difference between the spectral signatures in the hyperspectral data, making it a robust metric for assessing the spectral fidelity of the predicted image in comparison to the ground truth. The cosine similarity is computed as the dot product of the vectors, normalized by the product of their magnitudes, ensuring that the loss function focuses solely on the angular difference, independent of the magnitude of the spectral signatures.

Finally, the standard reconstruction loss, i.e., $\ell$-1 norm loss $\ell_{\text{L1}}$, is used to ensure the high quality of the reconstructed HSI. Thus, the total loss of the teacher network would be

where the superposition of ${\bm{Y}}^{t}$ denotes the reconstructed HR-HSI by our teacher network and $\lambda$ is the parameters to control the importance between the spatial fidelity and spectral quality terms. Likewise, the total loss of the student network is defined by the reconstruction loss and KD loss, as follows:

where $\ell_{\text{L1}}({\bm{Y}}^{s},{\bm{Y}}^{t})$ aims to relax the constraint of $\ell_{\text{L1}}({\bm{Y}}^{s},{\bm{Y}})$ since the outcome of lightweight student network might hard to approximate to the ground truth accurately. All the balance parameters, $\lambda_{1}$,$\lambda_{2}$,$\lambda_{3}$, and $\lambda_{4}$, are set to $0.1$ respectively.

We evaluated our approach against seven state-of-the-art HSI/MSI fusion methods, including four supervised methods: PZRes-Net [19], MSSJFL [18], Dual-UNet [20], and DHIF-Net [21], and three unsupervised methods: CUCaNet [39], USDN [40], and U2MDN [41]. The performance was objectively measured using the three metrics, including PSNR, SAM, and RMSE. Experiments were conducted with both 4 and 6 MSI bands. As more bands in the LR-HSI are available, richer spectral information can be exploited to potentially enhance the SR quality. The quantitative results are presented in Table I. This study investigates two variants of the proposed model, denoted by the postfixes L and L-Ext. The L model is constructed by stacking additional blocks, as described in Section III.E, resulting in a larger model architecture. On the other hand, the L-Ext model is obtained by extending the training process of the L model with more epochs and a reduced learning rate of $5\times 10^{-5}$.

The experimental results demonstrate that the proposed framework outperforms the compared state-of-the-art methods in terms of spectral reconstruction performance for both 4 and 6 bands of HR-MSI. Additionally, our method exhibits superior overall and pixel-level restoration capabilities compared to the state-of-the-art methods based on the obtained better performances in terms of the SAM, PSNR, and RMSE metrics, respectively. The outstanding performance of our method can be attributed to several factors. First, the proposed DTS network effectively integrates spatial-spectral feature representations, leading to the excellent performance of the teacher model. Second, the relatively shallow network architecture of the student model enables faster inference times while maintaining high-quality results, resulting in a higher performance-to-complexity ratio compared to previous methods. Third, the proposed response-based KD framework provides refined and strong guidance, facilitating the student network in learning nuanced representations with fewer parameters, thereby streamlining the architecture without compromising model performance. Furthermore, the proposed CSA fusion module and the distillation strategy enable our method to adaptively determine the optimal weights for HSI and MSI features, even in the presence of noise, resulting in improved robustness and stability. The effectiveness of the CSA and KD strategies will be further demonstrated in the subsequent experiments. On the other hand, the unsupervised learning approach, such as CUCaNet [39], USDN [40], and U2MDN [41], are hard to meet the requirements of real-time inference scenarios, even the relatively lower number of parameters and FLOPs (floating point operations).

Discerning differences in HSI images through visualization in the RGB color system is challenging. To better observe these differences, we compute residual images by subtracting each method’s fused image ${\bm{Y}}^{*}$ from the corresponding ground truth ${\bm{Y}}$ and enhance the contrast through logarithmic map**. The resulting residual images, which would be closer to black while being closer to the ground truths, are depicted in Figure 5. These visualized results not only corroborate the superior quantitative performance of our method but also highlight the effectiveness of the proposed CSA and KD strategies in preserving fine image details and producing visually appealing false-color representations.

In real-world scenarios, the quality of LR-HSI and HR-MSI would suffer from lossy transmission or physical distortions, leading to the presence of heavy noise in the input data. Such noise would cause significant degradation and potentially catastrophic restoration results in the LR-HSI/HR-MSI fusion task. Therefore, it would be crucial to evaluate the robustness of the considered fusion methods in the noisy scenarios.

To assess the robustness of each method, we introduced Additive White Gaussian Noise (AWGN) with varying Signal-to-Noise Ratios (SNR) ranging from 25 to 45. The AWGN noise and its impact on the HSI are formulated as:

where ${\bm{X}}_{i}$ represents the tensor-form input image and $N$ is the number of tensor elements. The noisy LR-HSI and HR-MSI can be obtained by ${\bm{X}}_{m}={\bm{X}}_{m}+{\bm{N}}_{\text{awgn}}$ and ${\bm{X}}_{h}={\bm{X}}_{h}+{\bm{N}}_{\text{awgn}}$, respectively.

As shown in Table II, the proposed method achieves the best performance under noisy scenarios. We further considered two scenarios: (1) adding AWGN to the LR-HSI only, and (2) adding AWGN to both the LR-HSI and HR-MSI. The results for these scenarios are presented in Figures 6, 7, 8, and 9. As expected, the performance of all models generally deteriorates in the presence of noise; however, the degradation patterns vary across methods. The restored results from Dual-UNet [20] collapse when heavy noise is added to both LR-HSI and HR-MSI. The other three methods are also affected to varying degrees. In contrast, our approach is notably resilient, maintaining higher PSNR and SAM values whether noise is added solely to LR-HSI or to both LR-HSI and HR-MSI.

The effectiveness of our proposed CSAKD framework in handling noise can be attributed to two key factors. First, the Cross Self-Attention (CSA) fusion module adaptively determines the optimal weights for features extracted from the LR-HSI and HR-MSI branches. By dynamically adjusting these weights based on the input data, the CSA module can effectively suppress the influence of noise and prioritize the more reliable information from each modality. Second, the Knowledge Distillation (KD) strategy enables the student network to learn robust feature representations from the teacher network. During training, the teacher network is exposed to noisy inputs and learns to extract noise-resilient features. Through the distillation process, this robustness is transferred to the student network, allowing it to maintain high-quality fusion results even in the presence of noise.

In addition to the restoration performance and robustness of the models, we also consider their computational complexity and lightweight nature for deployment on different hardware platforms. This is crucial because HSI processing needs to be compatible with various hardware constraints and SDG requirements, allowing for feasible applications without heavy computational burden. The results of the complexity analysis are shown in the last two columns of Table I and Figure 10.

The HR-HSI restored from the state-of-the-art supervised learning-based methods all exhibit high fidelity. However, the model complexity and hardware requirements may vary in different aspects. The proposed method demonstrates comprehensively competitive capabilities across parameter size, FLOPs, running time per input pair, and memory usage during inference, underscoring an optimized balance between computational efficiency and fusion quality. By employing the proposed knowledge distillation framework, we significantly reduce the model size, FLOPs, and memory usage, which is extremely valuable for lightweight hardware platforms.

The other methods face different challenges arising from their drawbacks in handling heavy noise or their substantial hardware requirements. DHIF-Net [21] and PZRes-Net [19] are limited in their applicability to lightweight hardware due to their iterative spatial-spectral-aware optimization strategy or residual learning-based approach, which result in heavy parameter and memory requirements. Dual-UNet [20] achieves low computational complexity but struggles to address highly noisy data. MSSJFL [18] strikes a balance between maintaining fusion quality in the presence of noise and computational complexity, but its performance is relatively limited compared to our method.

The proposed CSAKD framework demonstrates exceptional performance, with both the teacher and student networks surpassing other state-of-the-art models. To explore the limitations of the CSAKD framework, which primarily depend on the DTS network architecture and the CSA-fusion module for achieving high-quality fusion results, we enhanced both networks by incorporating additional CLRA units. This enhancement aimed to assess their scalability and identify the upper bound of fusion performance, as detailed in Table I.

Specifically, we augmented the teacher network by stacking the CLRA unit in the four branches (${\bm{Z}}_{h},{\bm{Z}}_{hm},{\bm{Z}}_{mh},{\bm{Z}}_{m}$) 8, 8, 6, and 6 times, respectively. In contrast, the student network received a more modest increase of 2, 4, 4, and 4 stacks. These versions, denoted as Proposed-Teacher-L and Proposed-Student-L, are presented in Table I. This strategy served two purposes: first, to preserve the lightweight nature of the student network, and second, to amplify the learning capacity of the teacher model. The results indicate a significant improvement in the SAM and RMSE metrics for the student network. However, a limitation was observed in the PSNR metric. These findings suggest that our approach is viable for achieving superior fusion results when computational complexity is not a primary concern.

To further enhance the learning ability of the CSA-Large model, we explored the potential of a deeper teacher model, which can provide richer feature information in the feature domain. To investigate the effectiveness of the KD-guided framework, we extended the training process for CSA-Large by an additional 40 epochs and reduced the learning rate to $5\times 10^{-5}$. These versions, denoted as Proposed-Teacher-L-Ext and Proposed-Student-L-Ext, are presented in Table I. The results show that extending the training process degraded the teacher’s performance, indicating that the model had reached its bottleneck. Conversely, the student model acquired more effective guidance, aligning its performance with the proposed KD framework. This demonstrates that the proposed feature-map knowledge distillation loss $\ell_{\text{KD}}$ can effectively enhance the student network when using a deeper teacher network and a richer feature space.

The scalability analysis highlights the flexibility of the CSAKD framework in accommodating varying network depths and architectures. By increasing the number of CLRA units, the fusion performance can be further improved, particularly in terms of the SAM and RMSE metrics. However, the limitations observed in the PSNR metric suggest that there may be a trade-off between network depth and certain aspects of fusion quality. This trade-off should be carefully considered when designing the network architecture for specific applications.

The extended training experiment demonstrates the effectiveness of the KD-guided framework in transferring knowledge from a deeper teacher network to a lightweight student network. By leveraging the richer feature space provided by the deeper teacher, the student network can learn more nuanced representations and achieve improved fusion performance. This finding underscores the importance of the feature-map knowledge distillation loss $\ell_{\text{KD}}$ in enabling effective knowledge transfer and enhancing the student network’s learning ability.

In summary, the model scalability and extended training analysis provide valuable insights into the flexibility and effectiveness of the CSAKD framework. These findings can guide future research in designing and optimizing network architectures for HSI/MSI fusion tasks, while also highlighting the potential for further performance improvements through extended training and knowledge distillation.

Based on Table I, we corroborated the superior performance of the student network under teacher guidance. Furthermore, to explore the impact of teacher model complexity on student learning, highlighting the need for distillation loss to bridge the output gap between teacher and student, we perform the ablation study on loss function analysis shown as follows. In addition, we also perform the ablation study on CLRA depth analysis to determine the optimal number of CLRA units to balance the SR performance and the computational complexity, shown as follows.