HSONet:A Siamese foreground association-driven hard case sample optimization network for high-resolution remote sensing image change detection

To obtain deep features of scene embedding and utilize scene embedding at multiple scales to enhance feature representation, we designed the variant feature pyramid network (V-FPN) for multiscale feature extraction and adaptation to multilayer scene information embedding. The V-FPN consists of a feature extraction branch and a scene information embedding branch. Both branches utilize a variant FPN network divided into multiscale feature layers. The FPN, originally proposed by Liu et al. [6] for object detection tasks, aims to capture strong semantic information in multiple feature layers. To further enhance the model’s multiscale feature extraction and fusion capabilities, we designed each layer of the FPN with ResNet as the backbone network for basic feature information extraction. As shown in Figure 2, given a pair of input images $\mathrm{T}^{m},m=1,2$, the size of the feature layers in the feature extraction backbone network decreases vertically. This results in a multilevel feature layer set $\mathrm{F}_{i}^{m},i=1,2,3,4;m=1,2$, with output strides relative to the input image of (4,8,16,32) pixels. Building on the standard FPN structure, we use feature layers from deep to shallow and lateral feature connections to generate a pyramid feature map** set $\mathrm{P}_{i}^{m},i=1,2,3,4;m=1,2$ with the same dimensions. This method fully links spatial detail information of shallow features with strong semantic information of deep features, aiding in the modelling of multiscale target context information. This process can be expressed by Equation 1.

where $\tau$ represents a learnable 1x1 convolutional layer used for lateral feature connections and $\sigma$ represents a nearest-neighbour upsampling operation with a dimension of 2.

In the final layer $\mathrm{F}_{4}$ of the pyramid feature set, we designed an important feature flow branch, namely, the foreground-scene information embedding branch. This structure is based on global context aggregation to obtain the spatial scene embedding vector SV, which models the dependency relationship between geographical scenes and interest targets, including interest targets in the foreground and hard case samples. The specific structure and principle of this process will be introduced in the next section. Additionally, in our experiments, we used two types of backbone networks for feature extraction: ResNet[44] and the pyramid vision transformer[45]. The former, with its ingenious residual modules, has strong feature extraction capabilities. Considering accuracy and efficiency, we chose ResNet50 as the first backbone network. Moreover, the PVT can consider both low-level local features and high-level global features. Moreover, the transformer can learn long-range dependencies between features, giving it strong feature representation capabilities at different scales. In the experiments, we used PVTv2 as the second backbone network.

HRRS is characterized by rich spectral features and complex scene information. Issues such as varying lighting conditions, seasonal changes, and shadows and occlusions caused by tall buildings commonly occur. This means that there is a significant difference in image spectral features and a large intraclass variance in the background, specifically manifested as a large number of hard case samples in the background information, leading to problems such as false positives, false negatives, and pseudochanges in CD. To address this, we propose a foreground-science association module that leverages the scene context of remote sensing space to model the feature representation of foreground targets positively and enhance the feature discrimination ability of background samples. The main principle is shown in Figure 3. The FS-relation module first explicitly models the foreground targets and the geographic scene information and then uses implicit geographic scenes to associate the foreground with the relevant context. This relationship is subsequently used to enhance the input feature map**, increasing the difference between foreground targets and background hard cases and reducing the likelihood of false alarms and misreporting.

As depicted in Figure 4, the FS-relation module generates a new feature map** set $\mathrm{R}_{i}^{m}$ based on the feature pyramid collection $\mathrm{P}_{i}^{m},i=1,2,3,4;m=1,2$ This module initially re-encodes each layer of $\mathrm{P}_{i}^{m}$ to preliminarily form the feature map** $\mathrm{Q}_{i}^{m}$. This map** is then reweighted according to the correlation map $\mathrm{r}_{i}^{m}$, ultimately producing the relation-enhanced feature map** set $\mathrm{R}_{i}^{m}$. The correlation map $\mathrm{r}_{i}^{m}$ refers to the similarity matrix between the geographical scene and the foreground representation. To achieve greater discerning $\mathrm{R}_{i}^{m}$, the model learns a feature map** function to align the feature input $\mathrm{P}_{i}^{m}$ with the scene vector (SV) into the same dimension, facilitating feature interaction. Here, $Q_{i}^{m}\in R^{d\times H\times W}$ is transformed from the feature pyramid layer $P_{i}^{m}\in R^{d\times H\times W}$ through the scale projection function $v(\cdot)$, as expressed in Equations 2 and 3.

We implement $v(\bullet)$ efficiently, first through a 1×1 convolution layer, followed by batch normalization and ReLU.

The similarity relation matrix set $\mathrm{r}_{i}^{m}$ results from the interaction between the scene embedding information and the foreground pyramid features. Therefore, to compute this set, the 1-D scene embedding information $\mathrm{SV}\in\mathrm{R}^{d}$ interacts with the foreground feature projection $Q_{i}^{m}$. Here, the scene embedding information SV is obtained by linearly projecting $P_{4}^{m}$, as represented by Equation 4.

$\varphi(\bullet)$ signifies a scene-focused projection function. For ease of computation, this process is executed by a learnable 1×1 convolution, with the output channel number set to $d$. For the same pair of input images, the remote sensing scene information should be identical, allowing each layer of features in $Q_{i}^{m}$ to share SV. At this stage, the similarity relation matrix set $r_{i}^{m}$ is derived using Equation 5.

where $\delta(\bullet)$ denotes the similarity measure function. To simplify the computation and improve the efficiency, the similarity measure is realized by the vector inner product.

The details of the foreground-scene correlation process are illustrated in Figure 4. The final relation-enhanced feature map** $R_{i}^{m}$ is computed according to Equation 6.

$\varepsilon(\bullet)$ is a feature re-encoding structure designed to re-encode the pyramid feature map** set $P_{i}^{m}$, similar to the aforementioned $v(\bullet)$ structure. In this equation, we use a polynomial containing $r_{i}^{m}$ to weight and highlight the re-encoded feature map** $\varepsilon(P_{i}^{m})$.

A large-scale building change detection dataset consisting of 637 pairs of Google Earth images, each with a size of 1024×1024 pixels and a resolution of 0.5 m, spanning from 2002 to 2018. It contains a variety of building change information, such as villas, large factories, high-rise apartments, and garages.

This dataset records changes in various types of buildings, including large factories, villages, and warehouses, in the suburbs of Guangzhou, China, from 2006 to 2019. The 19 pairs of images are sourced from Google Earth, with spatial resolutions of 0.55 m and image sizes ranging from 1006×1168 to 4936×5224.

This dataset contains 11 pairs of multisource remote sensing images collected in different seasons; 7 pairs have a resolution of 7425×2202 pixels, and 4 pairs have 1900×1000 pixels, with resolutions varying from 0.03 m to 1 m. The challenge of this dataset lies in accurately detecting changes in buildings, roads, farmland, and vehicles, regardless of the impact of seasonal changes.

This dataset includes 20,000 pairs of aerial images collected in Hong Kong, China, from 2007 to 2014; all the images have a size of 256×256 pixels and a resolution of 0.5 m. The main types of changes in the data include urban buildings, road construction, marine construction, and vegetation changes.

To evaluate the performance of the model, we used seven evaluation metrics for result presentation: precision (P), recall (R), F1 score (F1), intersection over union (IOU), mean intersection over union (mIOU), overall accuracy (OA), and kappa coefficient (Kappa). In the RS-CD tasks, higher precision indicates more correct detections in positive cases, higher recall means fewer losses in predicted positive cases, and larger F1, IOU, and mIOU indicate better CD performance. A higher kappa coefficient indicates better consistency between two sets of predictions, while the OA is an overall assessment of all the predictions being correctly classified. The calculation formulas for these metrics are as follows.

We conducted experiments using an NVIDIA 4080Ti graphics card and trained the model with a minibatch Adam optimizer. The PyTorch deep learning framework was used. The Epoch was set to 150, batch size to 16, learning rate to 5×10-4, weight decay to5×10-4, Step size to 50, and momentum to 0.9. For the CDD dataset, we cropped the original images into small images of 256×256 pixels and randomly performed simple operations such as rotation, flip**, and central crop**. We obtained 10,000/3,000/3,000 pairs of image patches for training, validation, and testing, respectively. For the LEVIR-CD dataset, we cropped the original images into nonoverlap** blocks of 256×256 pixels, similarly performing random rotations, flip**, and central crop** for data augmentation. We divided the images randomly into three parts: 8392/600/1200 pairs for training, validation, and testing. For the Google-CD dataset, we cropped the original images into blocks of 256×256 pixels and performed random rotations, flip**, and central crop** steps; we obtained 2661/157/312 pairs of images for training, validation, and testing. For the SYSU-CD dataset, we cropped the original images into small images of 256×256 pixels, similarly performing random rotations, flip**, and central crop**, and obtained 14,001/1,999/4,000 pairs of images for training, validation, and testing.

In this subsection, we introduce the benchmark methods used for comparison. To demonstrate the superiority of HSONet, we selected the most classic research achievements in each subdirection for comparison, including three pure convolution-based CD methods, FC-EF, FC-Siam-Di, and FC-Siam-Co; a method based on channel and spatial attention, DASNet; a method embedding ViT into the CNN backbone BiT; a pure Swin-T based method, SwinSUNet; a method based on cross-temporal joint attention, DMINet; a method based on cross-interaction attention and multiscale feature fusion, ICIF-Net; a CD method for building edge feature enhancement, EGCTNet; a method based on high-frequency information enhancement and spatial attention, HFA-Net; and a 3-D attention method, USSFC-Net, based on spatial-spectral feature synergy.

1. (a)

   FC-EF[23]: An early feature fusion CD method. It directly concatenates dual-temporal images in the channel dimension before feeding them into the network.

2. (b)

   FC-Siam-Co[23]: A weight-sharing dual-encoder Siamese network that acquires change information by fusing features at various levels.

3. (c)

   FC-Siam-Di[23]: Compared with FC-Siam-Co, this method changes the feature map fusion method in the network from concatenation to absolute difference.

4. (d)

   DASNet[11]: Based on VGG16 or ResNet50, this Siamese network introduces spatial and channel attention in the feature encoding phase to enhance the network’s focus on interest changes and resist pseudochange interference. The change map is obtained through a feature metric module.

5. (e)

   BIT[30]: This method embeds a transformer module in the convolutional network, converting dual-temporal feature map**s into compact semantic tokens to facilitate modelling of the spatiotemporal context in feature space, thereby optimizing feature representation in image space.

6. (f)

   SwinSUNet[35]: Based on Swin-T, this dual U-shaped CD network uses multiple pure transformer encoders to capture the spatiotemporal context of dual-temporal images. After feature fusion and linear map**, the change information is decoded using multiple Swin-T blocks combined with skip connections.

7. (g)

   DMINet[13]: Based on dual-branch ResNet18 for feature extraction, DMINet unifies self-attention and cross-attention in one module to guide the global feature distribution of each input. Two change information acquisition structures are also designed, namely, the fusion method based on subtraction and concatenation and a multilevel differential aggregation method based on incremental feature alignment.

8. (h)

   ICIFNet[31]: This method designs a four-branch cross-interaction feature extraction structure with parallel CNN and ViT to promote the mutual penetration of local and global features. Mask-based aggregation and spatial alignment (SA) schemes are also introduced for scale integration, achieving information integration at different resolutions.

9. (i)

   EGCTNet[46]: Focusing on the issue that local fine features and global information in CD cannot be obtained simultaneously, a fusion encoder is designed combining a CNN and transformers. Additionally, an edge detection branch is proposed that uses edge information to guide mask feature generation.

10. (j)

    HFA-Net[47]: Addressing the issue of insufficient high-frequency information acquisition for clearly defined targets, a high-frequency attention-guided module is proposed. It consists of two main stages: first, spatial attention is used to search for and focus on buildings; second, high-frequency enhancement is used to highlight the high-frequency information of features, which better represents the edges of changed buildings.

11. (k)

    USSFC-Net[21]: Simultaneously modelling spatial and spectral features, this method proposes an efficient collaborative network to generate three-dimensional attention for richer feature information. Additionally, a multiscale decoupled convolution is designed to flexibly capture the multiscale features of changing objects.

To accurately assess the effectiveness of HSONet, we conducted quantitative experiments on four public CD datasets. TABLE II, TABLE III, TABLE IV, and TABLE V show the accuracy of HSONet on the test sets of LEVIR-CD, Google-CD, CDD, and SYSU-CD, respectively. The quantitative results indicate that our method consistently outperforms the others on three datasets and demonstrates significant advantages on the SYSU-CD dataset. For instance, on the first three datasets, HSONet outperforms the other comparative methods in most accuracy metrics, with F1 scores exceeding those of the latest method, DMINet, by 1.16%, 1.15%, and 1.72%, respectively. Objectively speaking, our method has achieved certain breakthroughs in all seven accuracy metrics on all four datasets, proving the significance of introducing an equilibrium optimization loss function and a novel CD network architecture with a foreground-science association module. Additionally, Figures 8, 9, 10, and 11 show the visualization results of HSONet on each test set. To clearly observe the accuracy of each area classification, we used different colours to represent TP (blue), TN (light blue), FP (red), and FN (orange). To obtain a clearer view of the performance comparison between the various methods, we visualized the scores of the seven accuracy metrics using radar charts; as Figure 7 shows, our method has several advantages in all the metrics on the four datasets, and the homogeneity is good, with no poor accuracy results.

To evaluate the performance of HSONet, we compared the F1 score variation over epochs for USSFC-Net, EGCTNet, and HSONet on four datasets. Figure 12 shows the change line graphs for LEVIR-CD, Google-CD, CDD, and SYSU-CD from top to bottom and left to right. The graph shows that our model has higher accuracy, faster convergence, and a more stable training process than the other two methods, with very few large fluctuations in accuracy during training. This indicates that HSONet is more robust and stable than the other models and has stronger robustness. Additionally, the model converges after reaching a certain point, and training for more epochs does not significantly enhance the CD capability. Therefore, in our experiments, we set the epochs according to the dataset: 60 epochs for LEVIR-CD and 100 epochs for Google-CD, CDD, and SYSU-CD.

To explore the contribution of EO-loss to model performance, we compared the original loss function with the EO-loss. As shown in TABLE VI, in the control experiments using the PVT, the accuracy of the EO-loss on the three datasets is 0.18%, 1.13%, and 0.12% greater than that of the BCE, and 0.11%, 0.98%, and 0.31% greater than that of the SF-loss. These results indicate that EO-loss indeed has certain advantages. However, since hard case samples constitute a smaller proportion of the data samples, the improvement in accuracy is not very large. This finding demonstrates the effectiveness and rationality of the method.

To explore the contribution of the FS-relation module to the performance of HSONet, we conducted ablation experiments on the FS-relation module on three datasets. As shown in TABLE VI, when ResNet was used as the backbone network, the three sets of experiments introducing the FS-relation module were comprehensively superior to those without it. For example, when the loss function is BCE, the F1 scores of the former are 0.38%, 0.50%, and 4.77% greater than those of the latter on three datasets, and similar results are obtained for the SF-loss and EO-loss experiment sets. This finding suggests that enhancing the feature representation of hard case targets by modelling the association of foreground targets of interest with related context in latent remote sensing spatial scene information is a feasible feature modelling strategy.

To verify the role of the multiscale structure-designed backbone in the network, we compared the experimental results of PVT and ResNet on three datasets. As shown in TABLE VI, in all six sets of comparative experiments, PVT achieves more competitive accuracy results than ResNet. For instance, when the loss function is EO-loss, the F1 scores of the PVT group are 0.55%, 1.45%, and 0.88% greater than those of the ResNet group. This indicates that the PVT has a stronger feature extraction capability than ResNet and is more suitable for CD tasks. This might be due to ViT’s superior global modelling capability and the multiscale information of the pyramid feature layer, which allows the FS-relation module to obtain more representative scene embedding vector SVs, thereby enhancing the feature representation of background hard case targets.

Optimizing hard case samples in the later stages of model training is crucial for enhancing model performance. At this stage, background hard case samples should become the focus of model optimization. Therefore, we introduce dynamic weight functions to shift the learning focus of the model from the foreground to the background. We designed three types of dynamic weight functions and conducted experiments on three datasets. As shown in TABLE VII, in the experiments using ResNet as the backbone, the maximum F1 score improvement achieved by using dynamic weight functions compared to not using them was 0.52%, 0.85%, and 0.63%, respectively; similar results were obtained in experiments using PVT as the backbone, with increases of 0.34%, 0.80%, and 0.54%, respectively. This indicates that the method based on dynamic weighting reduces early training errors in predicting hard case samples, thereby enhancing the CD performance of the model; moreover, focusing on optimizing background hard case samples in the later stages of training is both reasonable and effective. According to both sets of experiments, although all three dynamic weight functions achieved some accuracy improvements, they had no absolute advantage. This might be due to the varying numbers and distributions of hard case samples in different datasets; hence, different dynamic weight functions are suitable for different datasets. We can choose the appropriate dynamic weight function based on data characteristics. For example, cosine dynamic weighting is more suitable for datasets with a clear gap between simple and hard case samples, allowing for stable adjustment of the loss distribution to achieve healthy convergence; exponential dynamic weighting is more suitable for situations where there are fewer foreground samples and more background samples.

Introducing the hard case awareness term $(1-p)^{\gamma}$ leverages the distribution of loss values to approximate the regions of hard case samples and then adjusts the attention weight distribution between hard case samples and foreground samples using the focus Factor $\gamma$. Generally, the larger $\gamma$ is, the greater the weight and the higher the degree of attention. We conducted comparative experiments with progressively increasing $\gamma$ on two datasets and observed similar experimental effects. As shown in TABLE VIII, starting from $\gamma$=0.0 and increasing $\gamma$, the CD accuracy continuously improved. On the LEVIR-CD and Google-CD datasets, when $\gamma$ was set to 2.0 and 1.0, respectively, the F1 scores were 0.19% and 1.20% higher than when $\gamma$ was 0.0, reaching absolute accuracies of 92.42% and 93.14%, respectively. However, as $\gamma$ continued to increase, the performance of the model began to decline. This may be due to two reasons: first, image noise was also given a high focus Factor $\gamma$, leading to its misclassification as hard case samples; second, an excessively large $\gamma$ caused the model to excessively neglect foreground targets, thus reducing its discriminative ability for foreground interest targets. From this, we can conclude that an appropriate focus Factor $\gamma$ can enhance the model’s CD performance, but different data characteristics often require different optimal focus factors.

To demonstrate how our method focuses on various remote sensing targets, we visualized the attention heatmap of the last layer of HSONet and overlaid it on the T2 image to obtain the image shown in (f). In (f), the attention distribution of the changing targets can be clearly observed, where red denotes higher attention values and blue denotes lower values. As (f) shows in Figures 13 and 14, areas with high heating values completely cover the areas of interest changes. The house change areas in LEVIR-CD and Google-CD both have high heating values, indicating that our method learned the feature representation of interest targets and is not sensitive to noninteresting changes. Notably, HSONet demonstrates strong robustness to building shadows, as shown in (2) and (3) of Figure 13. The shadows around buildings do not affect our method’s ability to accurately detect real change areas. Moreover, shadows around buildings may be a key factor in change recognition because our model can implicitly learn certain additional feature concepts to facilitate the identification of change areas.

In addition, to more intuitively reflect the degree of separation between the changed and unchanged samples, we used t-distributed stochastic neighbour embedding (t-SNE)[48] to downscale the distribution of the visualized samples and compare and analyse our method with the distribution of the ICIF-Net features. As shown in (g) and (h) of Figures 14 and 15, red represents changed samples, and blue represents unchanged samples. Notably, in the dimensionality reduction results of HSONet, the separation boundary between the red and blue samples is clearer than that between the other samples, with both sample categories being more distinctly clustered and with few instances of red and blue intermingling. This indicates that our method has more accurate change perception and anti-interference capabilities, which is why HSONet’s change confidence is greater and it has clear boundaries. In contrast, in the ICIF-Net dimensionality reduction results, a certain number of red samples appear in the blue samples, which indicates that some of the changed samples and the unchanged samples are indistinguishable in the feature space. This leads to some samples at the edge of the changed area with low confidence being misclassified by the network, resulting in ”blurred boundaries of the changed area”.

To validate the attention shift pattern of the model during the early and late training stages, we visualized the attention focus distribution on sample data at different epochs. Each epoch is equidistant, allowing the observation of the pattern of shifting optimization focus. Figures 15, 16, and 17 show the heatmaps of a particular data sample in LEVIR-CD, Google-CD, and CDD, respectively, visualized using Grad-CAM[49]. All three images exhibit the same pattern: in the early stages of training, the optimization focus is concentrated on interest targets in the foreground, which in this case are ’buildings’. By the middle stages of training, the model gradually shifts towards learning about background information, such as the areas around roads or less obvious buildings. In the later stages of training, the model’s attention increasingly turns to learning about background features, during which there is a varying degree of neglect for foreground targets.