CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening

Abstract

Pansharpening aims to enhance remote sensing image (RSI) quality by merging high-resolution panchromatic (PAN) with multispectral (MS) images. However, prior techniques struggled to optimally fuse PAN and MS images for enhanced spatial and spectral information, due to a lack of a systematic framework capable of effectively coordinating their individual strengths. In response, we present the Cross Modulation Transformer (CMT), a pioneering method that modifies the attention mechanism. This approach utilizes a robust modulation technique from signal processing, integrating it into the attention mechanism’s calculations. It dynamically tunes the weights of the carrier’s value (V) matrix according to the modulator’s features, thus resolving historical challenges and achieving a seamless integration of spatial and spectral attributes. Furthermore, considering that RSI exhibit large-scale features and edge details along with local textures, we crafted a hybrid loss function that combines Fourier and wavelet transforms to effectively capture these characteristics, thereby enhancing both spatial and spectral accuracy in pansharpening. Extensive experiments demonstrate our framework’s superior performance over existing state-of-the-art methods. The code will be publicly available to encourage further research.

I Introduction

Given the inherent constraints of remote sensing technology, obtaining MS images with high spatial resolution directly from satellites is a significant challenge. As a solution, pansharpening has become a crucial technique. It merges low-resolution MS (LRMS) images with high-resolution PAN images to produce high-spatial-resolution MS (HRMS) images with superior spatial detail. This fusion technique effectively navigates around the limitations of sensor technology, offering invaluable data for RSI analysis.

Figure 1: (a) The modulation process in the field of communication. (b) The modulation process in coded aperture snapshot spectral imaging (CASSI). (c) Our Cross Modulation Multi-head Self-Attention (CM-MSA) modulation process.

Deep learning breakthroughs, led by Convolutional Neural Networks (CNNs), have significantly advanced the field of pansharpening [4], [12], [29]. CNNs have shown exceptional prowess in pansharpening, skillfully extracting and combining intricate features from various images to enhance both spatial and spectral quality. Furthermore, since their introduction, Transformers [21] have revolutionized numerous fields, including pansharpening [11], [28], [17] by their unparalleled ability to model long-range dependencies using self-attention mechanisms. This capability gives them a significant edge in effectively blending spatial and spectral features, outperforming traditional CNN-based approaches by capturing the complex dynamics between different types of images.

However, the field of pansharpening still faces several challenges. Firstly, Hyperspectral Images (HSI) show spatial sparsity and spectral self-similarity, complicating spatial dependency modeling and highlighting the importance of prioritizing inter-spectral over spatial correlations in Transformers. Secondly, RSI are marked by their rich spectral content, complex surface textures and edge details, posing challenges for fusion and necessitating tailored approaches for effective integration. Lastly, the potential of attention structures, specifically tailored for pansharpening within the Transformer framework, warrants further exploration. Current integration methods of PAN and LRMS images within frameworks like PanFormer [30] and Hyperformer [1] mainly involve linear projection of images into tokens for merging via attention computations. While these methods are efficient, they don’t fully exploit the intricate relationship between the spatial resolution of PAN and the spectral diversity of LRMS images.

Figure 2: Overall structure of the proposed method.

In response to these challenges, we have developed innovative solutions, notably the CMT and a hybrid loss function, to elevate the pansharpening workflow. Firstly, as illustrated in Fig. 1 (c), to harness spectral features for LRMS images and spatial correlation for PAN images, we independently compute the attention block for each spectral and spatial channel ensuring a targeted processing of the distinct characteristics inherent. Secondly, to address the nuanced complexities of RSI, characterized by their rich spectral content and complex surface textures, we’ve implemented a hybrid loss function that combines Fourier and wavelet transforms. This approach utilizes Fourier transforms for identifying widespread features and wavelet transforms for enhancing local texture details. Together, they effectively improve spatial detail and maintain spectral fidelity in the pansharpening process. Lastly, our approach innovatively blends PAN’s spatial details with LRMS’s spectral data by applying advanced modulation techniques to the pansharpening process. Specifically, rather than merely concatenating features, our approach leverages the features of the modulator to dynamically modulate the carrier, altering the weights of the Transformer’s value (V) matrix, which achieves a more sophisticated fusion of features. Our approach, inspired by signal processing techniques depicted in Fig. 1 (a), uses modulation to seamlessly integrate high and low-frequency signals, significantly improving signal fidelity and richness. In parallel, CASSI systems, illustrated in Fig. 1 (b), compress images by modulating high frequencies with masks, akin to our deep learning methods for image enhancement like MST by Cai et al [3], which also employ masks for recovery.

In conclusion, the CMT framework significantly advances pansharpening through the following contributions:

1. Our Cross Modulation module within the CMT framework significantly enhances the fusion of PAN and LRMS images through a novel modulation technique.

2. We introduce a pioneering hybrid loss function that combines Fourier and wavelet transforms, which is the first attempt in the field of pansharpening to the best of our knowledge.

3. The CMT framework delivers outstanding results on benchmark datasets, establishing a new benchmark for pansharpening performance.

Figure 3: (a) The CMAB module consists of a Double Feed-Forward Network (DFFN), a CM-MSA module, and two layers of normalization. (b) Components of the DFFN. (c) The hybrid loss between Predicted Images ($PI$) and ground truth ($GT$), which employs both the 2D Discrete Fourier Transform (DFT) and the 2D Discrete Wavelet Transform (DWT).

II Method

II-A Overall Architecture

The CMT architecture, depicted in Fig. 2, is structured into three primary phases: feature extraction, modulation, and feature aggregation. In the feature extraction phase, distinct extractors are designed to capture the unique characteristics of PAN and LRMS images. During the modulation phase, the Cross Modulation Attention Block (CMAB) modulator separately modulates PAN images to enrich LRMS images and vice versa, allowing for an effective blend of spatial and spectral information. Finally, to aggregate feature, we use a $3\times 3$ convolutional kernel and four ResNet blocks simply.

Initially, LRMS images are upscaled to dimensions $H\times W\times c$. Then, PAN and LRMS images are processed through convolutional and ResNet blocks to extract local and global spatial details, resulting in feature sets $F_{\text{pan}}$ and $F_{\text{ms}}$, respectively, with channel expansion to $C=32$ in implementation.

The extracted features $F_{\text{pan}}$ and $F_{\text{ms}}$ are then modulated within the CMAB module, as shown in Fig. 3 (a). This module comprises a CM-MSA, two layers of normalization and a DFFN, employing varied activation functions for enhanced modulation. Fig. 3 (b) illustrates the specifics of the DFFN. Post-modulation, the features are concatenated and merged through a $3\times 3$ convolution and four ResNet blocks, allowing for further integration of spatial and spectral information.

Ultimately, the aggregated features are combined with the upsampled LRMS images to produce the HRMS.

II-B Modulation Approach

In signal processing, Double Side-Band modulation (DSB) [9], [19] stands as a classic and effective technique wherein the amplitude of a carrier wave is varied in accordance with the instantaneous value of the message signal, thus encoding information within the carrier. The mathematical expression for an DSB signal can be accurately given by:

$$s(t)=m(t)\cdot A_{c}\cdot\cos(2\pi f_{c}t),$$ (1)

where $s(t)$ denotes the modulated signal, $A_{c}$ represents the constant amplitude of the carrier wave, $f_{c}$ is the carrier frequency, and $m(t)$ embodies the message signal intended for transmission. This foundational concept of modulation underscores its robust capability to encode and transmit complex information efficiently and accurately across different mediums. In a similar vein, depicted in Fig. 1 (c), our pansharpening framework leverages a cross modulation paradigm where high-resolution spatial features and spectral details are mutually modulated. This bilateral modulation mechanism considerably amplifies the model’s proficiency in capturing and amalgamating multi-dimensional information, marking a significant leap from traditional methods.

Firstly, for the carrier, the input feature $X_{\text{in}}$ $\in\mathbb{R}^{H\times W\times C}$ is reshaped into tokens $X\in\mathbb{R}^{HW\times C}$. A multi-head attention mechanism is employed to improve generalization and capture multi-dimensional information, splitting $X$ into $k$ heads:

$$X=[X_{1},X_{2},\ldots,X_{k}],$$ (2)

where $X_{i}\in\mathbb{R}^{HW\times d_{k}}$, $d_{k}=\frac{C}{k}$, and $i=1,2,\ldots,k$.

Each $X_{i}$ is then linearly projected into queries $Q_{i}$, keys $K_{i}$, and values $V_{i}$ using the following equations:

$$Q_{i}=X_{i}W^{Q}_{i},\quad K_{i}=X_{i}W^{K}_{i},\quad V_{i}=X_{i}W^{V}_{i},$$ (3)

where $W^{Q}_{i}$, $W^{K}_{i}$, and $W^{V}_{i}\in\mathbb{R}^{d_{k}\times C}$ are learnable parameters, and $i=1,2,\ldots,k$.

For the modulator, aligned with the carrier, the input feature is reshaped into tokens $M\in\mathbb{R}^{HW\times C}$ and split into $k$ heads. Then, the modulation is integrated into the self-attention calculation by element-wise multiplying $M_{i}$ with $V_{i}$:

$$M=[M_{1},M_{2},\ldots,M_{k}],\quad V^{\prime}_{i}=M_{i}\odot V_{i}$$ (4)

where $M_{i}\in\mathbb{R}^{HW\times d_{k}}$, $i=1,2,\ldots,k$.

For a single head, the modulation-attention computation is:

$$\begin{split}\text{MA}(Q_{i},K_{i},V^{\prime}_{i})&=V^{\prime}_{i}\cdot\text{softmax}\left(\frac{K_{i}^{T}Q_{i}}{\alpha_{i}}\right),\end{split}$$ (5)

where MA is modulation attention and $\alpha_{i}\in\mathbb{R}$ is a learnable parameter that adaptively scales the matrix multiplication, enhancing the model’s ability to adjust attention weights dynamically.

The results from multi-heads are concatenated together, and with the addition of position encoding, the final output $X_{\text{out}}$ is derived as follows:

$$X_{\text{out}}=\text{Fc}(\text{Concatenate}(\text{MA}_{1},\text{MA}_{2},\ldots,\text{MA}_{k}))+F_{p},$$ (6)

where Fc denotes a fully connected layer, and $F_{p}$ represents the position encoding.

TABLE I: Quantitative results on 20 reduced-resolution and 20 full-resolution samples of GF2. (Red: best; Blue: second best).

Method	Reduced-Resolution Metrics			Full-Resolution Metrics
	SAM↓	ERGAS↓	Q4↑	$D_{\lambda}$↓	$D_{s}$↓	HQNR↑
PNN[16]	1.048 $\pm$ 0.226	1.057 $\pm$ 0.235	0.960 $\pm$ 0.010	0.0317 $\pm$ 0.0286	0.0943 $\pm$ 0.0224	0.877 $\pm$ 0.036
PanNet[27]	0.997 $\pm$ 0.212	0.919 $\pm$ 0.191	0.967 $\pm$ 0.010	0.0179 $\pm$ 0.0110	0.0799 $\pm$ 0.0178	0.904 $\pm$ 0.020
DiCNN[10]	1.052 $\pm$ 0.231	1.081 $\pm$ 0.254	0.959 $\pm$ 0.010	0.0369 $\pm$ 0.0132	0.0992 $\pm$ 0.0131	0.868 $\pm$ 0.016
FusionNet[5]	0.973 $\pm$ 0.212	0.988 $\pm$ 0.222	0.964 $\pm$ 0.009	0.0350 $\pm$ 0.0124	0.1013 $\pm$ 0.0134	0.867 $\pm$ 0.018
DCFNet[25]	0.872 $\pm$ 0.169	0.784 $\pm$ 0.146	0.974 $\pm$ 0.009	0.0240 $\pm$ 0.0115	0.0659 $\pm$ 0.0096	0.912 $\pm$ 0.012
MMNet[26]	0.993 $\pm$ 0.141	0.777 $\pm$ 0.134	0.969 $\pm$ 0.020	0.0443 $\pm$ 0.0298	0.1033 $\pm$ 0.0129	0.857 $\pm$ 0.027
LAGConv[13]	0.786 $\pm$ 0.148	0.687 $\pm$ 0.113	0.980 $\pm$ 0.009	0.0284 $\pm$ 0.0130	0.0792 $\pm$ 0.0136	0.895 $\pm$ 0.020
HMPNet[20]	0.803 $\pm$ 0.156	0.564 $\pm$ 0.099	0.981 $\pm$ 0.030	0.0819 $\pm$ 0.0499	0.1146 $\pm$ 0.0126	0.813 $\pm$ 0.049
Proposed	0.722 $\pm$ 0.136	0.624 $\pm$ 0.107	0.992 $\pm$ 0.001	0.0202 $\pm$ 0.0103	0.0338 $\pm$ 0.0086	0.947 $\pm$ 0.012

Figure 4: Qualitative result comparison between representative methods on the GF2 reduced-resolution dataset. The first row presents RGB outputs, while the second row gives the corresponding QNR maps.

II-C Loss Function

To enhance the resolution and quality of RSI, as shown in Fig. 3 (c), our method improves upon traditional fusion techniques by leveraging both Fourier and wavelet transforms. Fourier transforms [18] are essential for map** images into the frequency domain, capturing widespread environmental features. Besides, wavelet transforms [14] excel in delineating images across multiple scales, adeptly enhancing local textures and detailing.

The Fourier transform loss function is defined as:

$$L_{\text{Fourier}}=\frac{1}{M}\sum_{i=1}^{M}\|\mathcal{F}(PI_{i})-\mathcal{F}(GT_{i})\|_{1},$$ (7)

which applies the L1 loss to the difference in Fourier-transformed Predicted Images ($PI_{i}$) and ground truth images ($GT_{i}$), averaged over all $M$ training samples.

The loss function for the wavelet transform is defined as:

$$L_{\text{Wavelet}}=\frac{1}{M}\sum_{i=1}^{M}D_{j,c}(i),$$ (8)

where $D_{j,c}(i)=\|\mathcal{W}{j,c}(PI_{i})-\mathcal{W}{j,c}(GT_{i})\|_{1}$ and $\mathcal{W}{j,c}$ captures wavelet coefficients at scale $j$ and orientation $c$, reflecting local variations and textures, averaged across all training instances.

Combining these components, our comprehensive loss function integrates spatial, frequency, and wavelet domain losses:

$$L_{\text{total}}=L_{\text{spa}}+\lambda_{1}L_{\text{Fourier}}+\lambda_{2}L_{\text{Wavelet}}$$ (9)

where $L_{\text{spa}}$ is the L1 loss in the spatial domain, $\lambda_{1}$ and $\lambda_{2}$ are set to 0.7 and 0.2 to balance the contributions of the different loss components as weighting coefficients in implementation.

III Experiment

III-A Datasets and Implementation Details

To validate our approach, we construct datasets following Wald’s protocol [6], [24] on data collected from the WorldView-3 (WV3) and GaoFen-2 (GF2) satellites. Our datasets and data processing methods are downloaded from the PanCollection repository [7]. The datasets consist of images cropped from entire remote sensing images, divided into training and testing sets. The training set comprises PAN/LRMS/GT image pairs obtained by downsampling simulation, with dimensions of 64×64, 16×16×C and 64×64×C. Besides, we evaluated our method on the commonly used metrics in the field of pansharpening, including SAM [2],  ERGAS [23] and Q8 [8] for reduced-resolution dataset, $Ds$, $D_{\lambda}$ and HQNR [22] for full-resolution dataset. The CMT was trained with an initial learning rate of 0.001, for 400 epochs, and a batch size of 32, using Adam optimizer [15] with learning rate halved every 100 epochs. As for other DL-based methods, we utilize the default settings in related papers or codes to train the networks.

TABLE II: Quantitative results of DL-based methods on 20 reduced-resolution samples acquired by WV3. (Red: best; Blue: second best).

Method	Q8↑	SAM↓	ERGAS↓
PNN	0.893 $\pm$ 0.092	3.677 $\pm$ 0.762	2.680 $\pm$ 0.647
PanNet	0.891 $\pm$ 0.093	3.613 $\pm$ 0.766	2.664 $\pm$ 0.688
DiCNN	0.900 $\pm$ 0.087	3.592 $\pm$ 0.762	2.672 $\pm$ 0.662
FusionNet	0.904 $\pm$ 0.090	3.324 $\pm$ 0.698	2.465 $\pm$ 0.644
MMNet	0.915 $\pm$ 0.086	3.084 $\pm$ 0.640	2.343 $\pm$ 0.626
LAGConv	0.910 $\pm$ 0.091	3.103 $\pm$ 0.558	2.292 $\pm$ 0.607
HMPNet	0.916 $\pm$ 0.087	3.063 $\pm$ 0.577	2.229 $\pm$ 0.545
Proposed	0.917 $\pm$ 0.086	3.001 $\pm$ 0.610	2.201 $\pm$ 0.522

TABLE III: Ablation study on loss components. (Red: best; Blue: second best).

Loss Components			Metrics
L${}_{\text{spa}}$	L${}_{\text{wave}}$	L${}_{\text{fft}}$	Q8↑	SAM↓	ERGAS↓
✓	✗	✗	0.912±0.086	3.043±0.618	2.238±0.530
✓	✓	✗	0.913±0.087	3.033±0.612	2.223±0.523
✓	✗	✓	0.916±0.086	3.006±0.609	2.208±0.521
✓	✓	✓	0.917±0.086	3.001±0.610	2.201±0.522

TABLE IV: Ablation study on Modulation Approach. (Red: best; Blue: second best).

Method	$D_{\lambda}$↓	$Ds$↓	HQNR↑
V1	0.0210±0.0074	0.0364±0.0125	0.9435±0.0180
V2	0.0249±0.0123	0.0355±0.0132	0.9406±0.0187
V3	0.0234±0.0079	0.0388±0.0155	0.9389±0.0210
CMT	0.0201±0.0074	0.0344±0.0135	0.9463±0.0188

III-B Results

The performance of the proposed CMT method is show cased through extensive evaluations on GF2 datasets. TABLE I present a comprehensive comparison of CMT with various state-of-the-art methods on the GF2 dataset. The quantitative results show our method consistently surpasses existing approaches in all metrics. The visual comparison results are provided in Fig. 3. TABLE II present the results on the WV3 datasets, and the proposed method obtains the best average results on all quality indexes.

III-C Ablation Experiment

Ablation on Hybrid Loss. We compare the outcomes after training with different loss functions. The results are displayed in TABLE III. This comparison aims to ascertain the impact of each loss component on the overall performance of the model. We perform experiments on 20 reduced-resolution samples acquired by WV3 satellite.

Ablation on Modulation Approach. To validate the effectiveness of our method, we create three variants of the CMT. In the first variant (V1), modulation is omitted, taining only the transformer to evaluate its baseline feature integration performance. The second variant (V2) involves exclusively using the PAN image to modulate the MS features. In the third variant (V3), we solely use LRMS modulation of the PAN image. We perform experiments on 20 full-resolution samples acquired by the WV3 satellite. The results in TABLE IV show CMT has the best overall performance, proving our method’s efficacy.

IV Conclusion

In this study, we introduce the CMT, a novel pansharpening method that synergistically merges PAN and LRMS images. Central to CMT is the application of signal modulation, innovatively incorporated into a Transformer-based architecture. This allows for precise modulation of the Transformer’s value (V) matrix, facilitating a superior integration of spatial detail and spectral depth. Our method is characterized by its unique CM-SMA modulation technique and a bespoke hybrid loss function that blends Fourier and wavelet transforms. This loss function adeptly captures both global patterns and local textures, thereby enhancing spatial resolution while maintaining spectral fidelity. The versatility of CMT suggests its applicability beyond pansharpening, offering promising enhancements in various fields that require intricate and spectrally accurate image fusion.