SOOD++: Leveraging Unlabeled Data to Boost Oriented Object Detection

Abstract

Semi-supervised object detection (SSOD), leveraging unlabeled data to boost object detectors, has become a hot topic recently. However, existing SSOD approaches mainly focus on horizontal objects, leaving multi-oriented objects common in aerial images unexplored. At the same time, the annotation cost of multi-oriented objects is significantly higher than that of their horizontal counterparts. Therefore, in this paper, we propose a simple yet effective Semi-supervised Oriented Object Detection method termed SOOD++. Specifically, we observe that objects from aerial images are usually arbitrary orientations, small scales, and aggregation, which inspires the following core designs: a Simple Instance-aware Dense Sampling (SIDS) strategy is used to generate comprehensive dense pseudo-labels; the Geometry-aware Adaptive Weighting (GAW) loss dynamically modulates the importance of each pair between pseudo-label and corresponding prediction by leveraging the intricate geometric information of aerial objects; we treat aerial images as global layouts and explicitly build the many-to-many relationship between the sets of pseudo-labels and predictions via the proposed Noise-driven Global Consistency (NGC). Extensive experiments conducted on various multi-oriented object datasets under various labeled settings demonstrate the effectiveness of our method. For example, on the DOTA-V1.5 benchmark, the proposed method outperforms previous state-of-the-art (SOTA) by a large margin (+2.92, +2.39, and +2.57 mAP under 10%, 20%, and 30% labeled data settings, respectively) with single-scale training and testing. More importantly, it still improves upon a strong supervised baseline with 70.66 mAP, trained using the full DOTA-V1.5 train-val set, by +1.82 mAP, resulting in a 72.48 mAP, pushing the new state-of-the-art. Moreover, our method demonstrates stable generalization ability across different oriented detectors, even for multi-view oriented 3D object detectors. The code will be made available.

1 Introduction

Sufficient labeled data is essential to achieve satisfactory performance for fully-supervised object detection. However, the data labeling process is expensive and laborious. To alleviate this problem, many Semi-Supervised Object Detection (SSOD) methods [39, 64, 76, 28], aiming to learn from labeled data as well as easy-to-obtain unlabeled data, have been proposed recently. By leveraging the potentially useful information from the unlabeled data, the SSOD methods can achieve promising improvement compared with the supervised baselines, i.e., methods only trained from the limited labeled data.

The recent advanced SSOD methods [39, 64, 76, 28] mainly focus on detecting objects with horizontal bounding boxes in general scenes. However, in more complex scenes, notably aerial scenes, we need to utilize oriented bounding boxes to precisely describe the objects. The complexity of annotating objects with oriented bounding boxes significantly surpasses that of their horizontal counterparts, e.g., the annotation process for oriented bounding boxes incurs an approximate 36.5% increase in costs. Thus, considering the higher annotation cost of oriented boxes, semi-supervised oriented object detection is worth studying.

Figure 1: (a) The comparison of general scenes and aerial scenes. Different from the general objects, the objects from aerial scenes are arbitrary orientations, small, and dense (aggregation). Correspondingly, the annotation cost of aerial objects is higher than that of general scenes. (b) The proposed SOOD++ outperforms the supervised baseline and SOTA methods [35, 76] by a large margin on the DOTA-V1.5 benchmark (val set) under various settings.

Compared with the general scenes, the distinguishing characteristics of objects from aerial scenes (referred to as aerial objects) are mainly from three aspects: arbitrary orientations, small, and dense, as shown in Fig. 1(a). The mainstream SSOD methods are based on the pseudo-labeling framework [5, 55, 54, 64] consisting of the teacher and student models. The teacher model operates as an Exponential Moving Average (EMA) of the student model, aggregated over historical training iterations. It aims to generate pseudo-labels for images within the unlabeled dataset, thereby facilitating the student model’s learning from a combination of labeled and unlabeled data.

However, through pilot experiments, we find that extending the SSOD framework to oriented object detection is a highly non-trivial problem. We think the following two aspects need to be considered: 1) First, the geometry information (e.g., orientation and aspect ratio) is an essential cue for representing the multi-oriented objects. How to use the geometry information for the interaction between teacher and student in a semi-supervised framework is crucial. In other words, when applying SSOD methods in such a scenario, it is necessary to consider the geometry information of pseudo-labels and predictions. 2) Second, aerial objects are often dense and regularly distributed in an image. These properties lead to a challenge for obtaining accurate pseudo-labels from the teacher, resulting in inaccurate knowledge transfer towards the student and damaging the learning effectiveness within a teacher-student framework. A simple yet effective solution is to consider the global perspective. The global appearance of pseudo-labels and predictions can naturally reflect their patterns and suppress local noise, benefiting a more consistent and steady learning process.

Based on these observations and analysis, in this paper, we propose a practical Semi-supervised Oriented Object Detection method termed SOOD++. SOOD++ is built upon the dense pseudo-labeling framework, where the pseudo-labels are filtered from the pixel-wise predictions (including box coordinates and confidence scores). The key designs are from three aspects: a Simple Instance-aware Dense Sampling (SIDS) strategy, Geometry-aware Adaptive Weighting (GAW) loss, and Noise-driven Global Consistency (NGC). The SIDS is used to construct comprehensive dense pseudo-labels from the teacher, similar to pre-processing. GAW and NGC aim to measure the discrepancy between the teacher and student from a one-to-one and many-to-many (global) perspective, respectively.

Specifically, existing dense pseudo-labeling methods [76, 35] directly sample the fixed number of pseudo-labels from the predicted maps, which easily overlooks the dense, small, and ambiguous objects that usually exist in aerial scenes. To construct more comprehensive pseudo-labels, we propose a Simple Instance-aware Dense Sampling (SIDS) strategy to sample the dynamic number of easy and hard cases (e.g., objects intermingled with the background) from the teacher’s output. Second, considering the pseudo-label-prediction pairs are not equally informative, we propose the Geometry-aware Adaptive Weighting (GAW) loss. It utilizes the intrinsic geometry information (e.g., orientation gap and aspect ratio) of each teacher-student pair, reflecting the difficulty of the sample by weighting the corresponding loss dynamically. In this way, we can softly pick those more useful supervision signals to guide the learning of the student.

Furthermore, we innovatively treat the outputs of the teacher and student as two independent global distributions. These distributions implicitly reflect the characteristics of the overall scenario, such as object density and spatial layout across the image. This naturally inspires us to construct global constraints, leading to the proposal of Noise-driven Global Consistency (NGC). Specifically, we first add random noise to disturb the teacher and student outputs (global distributions). Then, We use optimal transport (OT) to align the original teacher’s distribution with the original student’s distribution and similarly align the disturbed teacher’s distribution with the disturbed student’s distribution. Besides, the transport plans from the former two alignments reflect the intricate relationships within the distributions, so we propose aligning them to further evaluate noise impact and provide auxiliary guidance for the model. Applying such multi-perspective alignments will increase the models’ tolerance to minor input variations, enabling them to better capture essential features of the data rather than relying excessively on noise or specific details in the unlabeled data. By promoting global consistency, our NGC not only relieves the negative effect of some inaccuracy pseudo-labels in the unsupervised training stage but also encourages a holistic learning approach, considering objects’ spatial and relational dynamics.

Extensive experiments conducted on four oriented object detection datasets in various data settings demonstrate the effectiveness of our methods. For example, as shown in Fig. 1(b), on the challenging large-scale DOTA-V1.5 (val set), we achieve 50.48, 57.44, and 61.51 mAP under 10%, 20%, and 30% labeled data settings, significantly surpassing the state-of-the-art SSOD method [35] (using the same supervised baseline) by 2.92, 2.39, and 2.57 mAP, respectively. More importantly, it can still improve a 70.66 mAP strong supervised baseline trained using the full DOTA-V1.5 train-val set by +1.82 mAP, resulting in 72.48 mAP on the DOTA-V1.5 (test set) with single-scale training and testing. Moreover, we also achieve consistent improvement on the DOTA-V1.0, HRSC2016, and even nuScenes dataset (a representative multi-view oriented 3D object detection dataset).

In conclusion, this paper presents an early and solid exploration of semi-supervised learning for oriented object detection. By analyzing the distinct characteristics of oriented objects from general objects, we customize the key components to adapt the pseudo-labeling framework to this task. We hope this work will establish a robust foundation for the burgeoning field of semi-supervised oriented object detection and offer an effective benchmark.

This paper is an extended version of our conference paper, published in CVPR 2023 [23], where we make the following new contributions: 1) We propose a Simple Instance-aware Dense Sampling strategy, which not only considers the easily obtained dense pseudo-labels from the foreground but also mine the hard case merged with the background. This design significantly improves the quality of dense pseudo-labels. 2) We propose Geometry-aware Adaptive Weighting (GAW) loss, which leverages the intrinsic geometry information (e.g., orientation gap and aspect ratios) of aerial objects to dynamic weight the importance of a given teacher-student pair. 3) We propose Noise-driven Global Consistency (NGC) to achieve multi-perspective global alignments, encompassing the alignments between the original teacher and original student, the disturbed teacher and disturbed student, and the transport plans generated from the former alignments. 4) Through the above substantial technical improvements, our SOOD++ significantly outperforms its conference version SOOD [23] by +1.85, +1.86, and +2.28 mAP on the DOTA-V1.5 dataset under the 10%, 20%, and 30% labeled settings, respectively. 5) We have made significant improvements in the quality of the manuscript in various aspects, e.g., conducted experiments on more benchmarks (even 3D scenes), more profound insights into performance improvements, more thorough ablation studies, added more meaningful illustrative figures, and reorganized the contents and tables to better illustrate the paper, etc.

2 Related works

Semi-Supervised Object Detection.

In recent years, semi-supervised learning (SSL)[52, 1] has demonstrated remarkable performance in image classification. These methods exploit unlabeled data through various techniques such as pseudo-labeling[27, 18, 61, 29], consistency regularization [55, 60, 1], data augmentation [51, 6], and adversarial training [44]. Unlike semi-supervised image classification, semi-supervised object detection (SSOD) necessitates instance-level predictions and the additional task of bounding box regression, thereby increasing its complexity. In [48, 77, 53, 25], pseudo-labels are generated by combining predictions from various augmentation.

Subsequent studies [39, 54, 64, 67] have incorporated EMA from Mean Teacher [55] to update the teacher model after each training iteration. ISMT [67] integrates current pseudo-labels with historical ones. Unbiased Teacher [39] addresses the class-imbalance issue by replacing the cross-entropy loss with focal loss [34]. Humble Teacher [54] utilizes soft pseudo-labels as the student’s training targets, enabling the student to distill richer information from the teacher. Soft Teacher [64] adaptively weights the loss of each pseudo-box based on classification scores and introduces box jittering to select reliable pseudo-labels. Unbiased Teacher v2 [40] employs an anchor-free detector and uses uncertainty predictions to select pseudo-labels for the regression branch. Dense Teacher [76] replaces post-processed instance-level pseudo-labels with dense pixel-level pseudo-labels, effectively eliminating the influence of post-processing hyper-parameters. To ensure both label-level and feature-level consistency, PseCo [28] proposes Multi-view Scale-invariant Learning, which aligns shifted feature pyramids between two images with identical content but different scales. Consistent Teacher [57] combines adaptive anchor assignment, a 3D feature alignment module, and a Gaussian Mixture Model to reduce matching and feature inconsistencies during training. MixTeacher [36] explicitly enhances the quality of pseudo-labels to handle the scale variation of objects. The Semi-DETR [75] method is specifically designed for DETR-based detectors, mitigating the training inefficiencies caused by bipartite matching with noisy pseudo-labels. To address assignment ambiguity, ARSL [35] proposes combining Joint-Confidence Estimation and Task-Separation Assignment strategies.

However, all the methods mentioned above are designed for the general scenes. How to unleash the potential of the semi-supervised framework in aerial scenes has yet to be fully explored. This paper aims to fill this blank and offer a solid baseline for future research.

Oriented Object Detection.

Unlike general object detectors [17, 50, 37, 49], representing objects with Horizontal Bounding Boxes (HBBs), oriented object detectors use Oriented Bounding Boxes (OBBs) to capture the orientation of objects, which is practical for detecting aerial objects. In recent years, numerous methods [70, 32, 43, 10, 65, 42, 14, 73, 7, 8] have been developed to enhance the performance of oriented object detection. For instance, CSL [69] addresses the out-of-bound issue by transforming the angle regression problem into a classification task. R³Det [70] improves detection speed by predicting HBBs in the first stage and then aligning features in the second stage to detect oriented objects. Oriented R-CNN [62] introduces a simplified multi-oriented region proposal network and uses midpoint offsets to represent arbitrarily oriented objects. ReDet [21] incorporates rotation-equivariant networks into the detector to extract rotation-equivariant features. Oriented RepPoints [31] incorporates a quality assessment module and a sample assignment scheme for adaptive point learning, which helps obtain non-axis features from neighboring objects while ignoring background noise. LSKNet [32] extends the large and selective kernel mechanisms to improve the performance. The recent method COBB [59] employs nine parameters derived from continuous functions based on the outer HBB and OBB area.

The above methods typically focus on the fully supervised paradigm, requiring expensive labeling costs. Thus, several weakly supervised object detectors have been proposed recently, such as point-based methods [42] or HBB-based methods [74, 71]. However, they are either inferior to their fully supervised counterparts or fail to leverage unlabeled data for performance improvement. In this paper, we explore semi-supervised oriented object detection, which reduces annotation costs and boosts detectors, even those well-trained on large-scale labeled data, by utilizing additional unlabeled data.

3 Preliminary

This section revisits the mainstream pseudo-labeling paradigm in SSOD and then introduces the concept of Monge-Kantorovich optimal transport theory [47].

3.1 Pseudo-labeling Paradigm

Pseudo-labeling frameworks [64, 40, 76, 35] inherited designs from the Mean Teacher [55], which consists of two parts, i.e., a teacher model and a student model. The teacher is an Exponential Moving Average (EMA) of the student. They are learned iteratively by the following steps: 1) Generate pseudo-labels for the unlabeled data in a batch. The pseudo-labels are filtered from the teacher’s predictions, e.g., the box coordinates and the classification scores. Meanwhile, the student makes predictions for labeled and unlabeled data in the batch. 2) Compute loss for the student’s predictions. It consists of two parts: the unsupervised loss $\mathcal{L}_{u}$ and the supervised loss $\mathcal{L}_{s}$. They are computed for the unlabeled data with the pseudo-labels and the labeled data with the ground truth (GT) labels, respectively. The overall loss $\mathcal{L}$ is the sum of them. 3) Update the parameters of the student according to the overall loss. The teacher is updated simultaneously in an EMA manner.

Based on the sparsity of pseudo-labels, pseudo-labeling frameworks can be further categorized into sparse pseudo-labeling [64, 40] and dense pseudo-labeling [76, 35], termed SPL and DPL, respectively. The SPL selects the teacher’s predictions after the post-processing operations (e.g., NMS and score filtering), shown in Fig. 3(b). It obtains sparse labels like bounding boxes and categories to supervise the student. The DPL directly samples the pixel-level post-sigmoid logits predicted by the teacher, which are dense and informative, shown in Fig. 3(c).

In this paper, we consider the limitations of both sparse and dense pseudo-label sampling strategies when dealing with small-scale, densely distributed remote-sensing objects. Specifically, the SPL may fail to detect small objects, resulting in insufficient supervisory information. Conversely, DPL, despite its abundance, may introduce substantial noise due to the illegibility between dense small objects and the background. Thus, we propose a Simple Instance-aware Dense Sampling (SIDS) strategy while sampling dense pseudo-labels from the predicted boxes of foreground and background, generating more comprehensive dense pseudo-labels.

3.2 Optimal Transport

The Monge-Kantorovich Optimal Transport (OT) [47] aims to solve the problem of simultaneously moving items from one set to another set with minimum cost. The mathematical formulations of OT are described in detail below.

Let $\boldsymbol{X}=\{\boldsymbol{x}_{i}|\boldsymbol{x}_{i}\in\mathbb{R}^{d}\}_{i=1}^{n}$ and $\boldsymbol{Y}=\{\boldsymbol{y}_{j}|\boldsymbol{y}_{j}\in\mathbb{R}^{d}\}_{j=1}^{n}$ denote as two $d$-dimensional vector space. And we assume $\hat{\boldsymbol{x}},\hat{\boldsymbol{y}}\in\mathbb{R}^{n}$ are two probability measures on $\boldsymbol{X}$ and $\boldsymbol{Y}$. The all possible transportation ways $\Gamma$ from $\boldsymbol{X}$ to $\boldsymbol{Y}$ are formed as:

$$\Gamma=\{\boldsymbol{P}~{}\in\mathbb{R}^{n\times n}|\boldsymbol{P}\mathbf{1}_{n}=\hat{\boldsymbol{x}},\boldsymbol{P}^{T}\mathbf{1}_{n}=\hat{\boldsymbol{y}}\},$$ (1)

where $\mathbf{1}_{n}$ is a $n$-dimensional column vector with all elements equal to 1. The constraints ($\boldsymbol{P}\mathbf{1}_{n}=\hat{\boldsymbol{x}}$ and $\boldsymbol{P}^{T}\mathbf{1}_{n}=\hat{\boldsymbol{y}}$) ensure the probability mass from each point in the source distribution $\hat{\boldsymbol{x}}$ is accurately transported to the target distribution $\hat{\boldsymbol{y}}$, maintaining the total mass invariant. The Monge-Kantorovich’s Optimal Transport (OT) cost between $\hat{\boldsymbol{x}}$ and $\hat{\boldsymbol{y}}$ is then defined as:

$$\mathcal{W}_{ot}(\hat{\boldsymbol{x}},\hat{\boldsymbol{y}})=\min_{\boldsymbol{P}\in\Gamma}\left\langle\boldsymbol{C},\boldsymbol{P}\right\rangle,$$ (2)

where $\boldsymbol{C}\in\mathbb{R}^{n\times n}$ represents the cost matrix between two sets, and $\left\langle\cdot\right\rangle$ represents inner product. In common, the OT problem is solved by the following dual formulation:

$$\begin{split}\mathcal{W}_{ot}(\hat{\boldsymbol{x}},\hat{\boldsymbol{y}})=\max_{\boldsymbol{\lambda},\boldsymbol{\mu}\in\mathbb{R}^{n}}\left\{\left\langle\boldsymbol{\lambda},\hat{\boldsymbol{x}}\right\rangle+\left\langle\boldsymbol{\mu},\hat{\boldsymbol{y}}\right\rangle\right\},\\ \mathrm{s.t.}~{}~{}\boldsymbol{\lambda}_{i}+\boldsymbol{\mu}_{j}\leq\boldsymbol{C}_{i,j},~{}\forall i,j\leq n,\end{split}$$ (3)

where $\boldsymbol{\lambda}$ and $\boldsymbol{\mu}$ are the solutions of the OT problem, which can be approximated in an iterative manner [9].

In this paper, we extend optimal transport to construct the global consistency for the teacher-student pairs, which not only enhances the robustness of the detection process against inaccuracies in pseudo-labeling but also encourages a holistic learning approach where both spatial information and relational context are taken into consideration.

4 Method

Fig. 2 illustrates the pipeline of our SOOD++. Specifically, the SOOD++ is a dense pseudo-labeling framework, which consists of three key components: a Simple Instance-aware Dense Sampling (SIDS) strategy is used to construct the high-quality dense pseudo-labels, the Geometry-aware Adaptive Weighting (GAW) loss and Noise-driven Global Consistency (NGC) are adopted to measure the discrepancy between the teacher and student from a one-to-one and set-to-set (global) perspectives, respectively. In the following contents, we describe the overall framework in Sec. 4.1, and introduce the key design of the proposed components, i.e., SIDS, GAW, and NGC, in the following Sec. 4.2, Sec. 4.3, and Sec. 4.4, respectively.

Figure 2: The pipeline of our SOOD++. Each training batch includes labeled and unlabeled images, omitting the regular supervised part for simplicity. For the unsupervised part, we use the Simple Instance-aware Dense Sampling (SIDS) strategy to create high-quality pseudo-labels and then pair them with the student’s predictions. The Geometry-aware Adaptive Weighting (GAW) loss dynamically weighs each pair’s unsupervised loss based on geometry information. Additionally, we treat the pseudo-labels and student predictions as global discrete distributions, measuring their similarity via Noise-driven Global Consistency (NGC). Yellow points indicate sampled dense pseudo-labels and predictions.

4.1 The Overall Framework

Since aerial objects are usually dense and small, the sparse-pseudo-labeling paradigm might miss massive potential objects. Thus, we choose the dense pseudo-labeling (DPL) paradigm. The training process includes the supervised and unsupervised parts. For the supervised part, the student is trained regularly with labeled data. For the unsupervised part, we adopt the following steps:

• First, given the output of the teacher, we utilize a Simple Instance-aware Dense Sampling (SIDS) strategy to generate the comprehensive dense pseudo-labels $\mathcal{P}^{t}$. We also select the predictions $\mathcal{P}^{s}$ at the same positions of the student. Therefore, we obtain the teacher-student pairs (i.e., $\mathcal{P}^{t}$ and $\mathcal{P}^{s}$).

• Next, we use the proposed Geometry-aware Adaptive Weighting (GAW) loss to dynamically weigh each teacher-student pair’s unsupervised loss by leveraging the intrinsic geometric information, including the orientation gap and aspect ratio.

• Then, our proposed Noise-driven Global Consistency (NGC) will construct a relaxed constraint by viewing the $\mathcal{P}^{t}$ and $\mathcal{P}^{s}$ as two global distributions. The random noise is used to disturb the global distributions of both the teacher and student. Following this, we achieve multi-perspective global alignments, encompassing the alignments between the original teacher and student, the perturbed teacher and perturbed student, and the transport plans generated from the former alignments.

We adopt the widely-used rotated version of FCOS [56] as the teacher and student. Note that at each pixel-level point, rotated-FCOS will predict the confidence score, centerness, and bounding box. The basic unsupervised loss contains classification loss, regression loss, and center-ness loss, corresponding to the output of FCOS. We adopt smooth $\mathcal{L}_{1}$ loss for regression loss, Binary Cross Entropy (BCE) loss for classification and center-ness losses. Based on these, our GAW and NGC will be used to measure the consistency of the teacher and student from different perspectives.

4.2 Simple Instance-aware Dense Sampling Strategy

Figure 3: (a) GT box. (b) Sparse pseudo-label (bounding box) [64, 40]. (c) Previous methods [76, 35] sample dense pseudo-labels from the entire score map using Top-K. (d) We sample dense pseudo-labels from the predicted instance (foreground) and background with dynamic numbers.

Constructing the pseudo-labels is a crucial pre-processing for the semi-supervised framework. As discussed in Sec. 3.1, since objects in aerial scenes are usually small and dense, using a dense pseudo-labeling paradigm can better identify potential objects in the unlabeled data compared with the sparse pseudo-boxes paradigm.

Previous dense pseudo-labeling methods [76, 35] typically extract a fixed number (i.e., Top-K) of dense pseudo-labels from the predicted score map (Fig. 3(c)). In contrast, we dynamically sample a variable number of dense pseudo-labels from the predicted instance (box) areas and background (Fig. 3(d)). Specifically, we categorize the pseudo-labels from unlabeled data into two types: the easy case obtained from the predicted results and the hard case, which often blends seamlessly into the background. Given the predicted results from the teacher after post-processing (e.g., NMS), considered as the foreground (easy case), we randomly sample pixel-level prediction results within the region indicated by these predicted boxes with a sample ratio $\mathcal{R}$, resulting in $\mathcal{P}^{e}$. Note that due to the dynamic nature of the number of instances, our sampling number varies across different scenes, better matching the distribution of the objects.

However, merely adopting the above strategy to sample dense pseudo-labels from the foreground often overlooks dense, small objects that blend seamlessly into the background. To address this issue, we propose an intuitive approach to extract potential dense pseudo-labels from the background. Specifically, we empirically find that those potential high-quality positive samples, despite their low confidence, align well with the target location. Consequently, we can use the Intersection over Union (IoU) score between the predicted box and ground-truth box as a criterion for discerning pseudo-labels amidst background noise. However, given the lack of ground truth in the unlabeled dataset, explicit IoU scores are unattainable. Therefore, we introduce an IoU estimation branch trained with the labeled data that predicts the IoU of each pixel for the unlabeled data. Then, we apply threshold $\mathcal{T}_{h}$ to choose the pixel-level points with high IoU scores from the background, resulting in dense hard pseudo-labels $\mathcal{P}^{h}$. We merge $\mathcal{P}^{h}$ and $\mathcal{P}^{e}$ to establish the final dense pseudo-labels $\mathcal{P}^{t}$, which will serve as the supervisory signal for the unsupervised training process. Based on $\mathcal{P}^{t}$, we sample the predictions from the student at the same coordinates, defined as $\mathcal{P}^{s}$. After obtaining $\mathcal{P}^{t}$ and $\mathcal{P}^{s}$, the semi-supervised framework aims to make $\mathcal{P}^{s}$ approach $\mathcal{P}^{t}$, and we will describe how to construct one-to-one and many-to-many constraints in the following sections.

Admittedly, IoU estimation is indeed a widely used technique. However, our objective fundamentally differs from previous methods [13, 26, 30], which primarily focus on box refinement. In contrast, our method aims to mine potential dense pseudo-labels from the background in a semi-supervised framework. It should be noted that this paper does not claim the novelty of the proposed SIDS but rather presents a simple and effective sampling strategy that yields high-quality dense pseudo-labels.

4.3 Geometry-aware Adaptive Weighting Loss

Orientation is one essential property of an oriented object. As shown in Fig. 1(a), their orientations remain clear even if objects are dense and small. Some oriented object detection methods [46, 68, 72] have already employed such property in loss calculation. These works are under the assumption that the angles of the labels are reliable. Strictly forcing the prediction close to the ground truth is natural.

However, this assumption is not true under the semi-supervised setting, as the pseudo-labels may be inaccurate. Simply forcing the student to align with the teacher may lead to noise accumulation, adversely affecting the model’s training. Additionally, the impact of orientation varies significantly for oriented objects with diverse aspect ratios. For instance, as illustrated in Fig. 4, an oriented object with a large aspect ratio is highly sensitive to rotation changes, and even slight misalignment can cause significant IoU change.

Based on the above observations, we propose to softly utilize orientation information and aspect ratios, enabling the semi-supervised framework to better understand the geometric priors of oriented objects. Specifically, The difference in orientation between a prediction and a pseudo-label can indicate the difficulty of the instance to some extent and can be used to adjust the unsupervised loss dynamically. Moreover, since oriented objects with different aspect ratios exhibit varying sensitivities to rotation changes, this can be leveraged to modulate the importance of orientation differences in indicating instance difficulty. Therefore, we propose a geometry-aware modulating factor based on these two critical geometric priors. Similar to focal loss [34], this factor dynamically weights the loss of each pseudo-label-prediction pair by considering its intrinsic geometric information (e.g., orientation difference and aspect ratio).

Specifically, the geometry-aware modulating factor $\omega_{i}^{geo}$ for the $i$-th pair is defined as:

$$\omega_{i}^{geo}=1+\sigma_{i},$$ (4)

$$\sigma_{i}=\psi\frac{\left|r_{i}^{t}-r_{i}^{s}\right|}{\pi}\frac{(a_{i}^{t}+a_{i}^{s})}{2},r_{i}^{t},r_{i}^{s}\in[-\frac{\pi}{2},\frac{\pi}{2}),a_{i}^{t},a_{i}^{s}\geq 1,$$ (5)

where $r_{i}^{t}$ and $r_{i}^{s}$ represent the rotation angles in radians of the $i$-th pseudo-label and prediction, respectively. $a_{i}^{t}$ and $a_{i}^{s}$ denote their aspect ratios. Adding a constant to $\sigma_{i}$ ensures that the original unsupervised loss is preserved when the orientations of the pseudo-label and prediction coincide. The modulating factor $\sigma_{i}$ considers both orientation differences and average aspect ratios. A toy example is that when the difference in orientation is marginal, but the object’s aspect ratio is large, the proposed factor can still efficiently highlight such a case as a possible hard example and help improve the learning process, and vice versa.

With the geometry-aware modulating factor, the overall Geometry-aware Adaptive Weighting (GAW) loss is formulated as:

$$\mathcal{L}_{GAW}={\textstyle\sum_{i=1}^{N_{p}}}\omega_{i}^{geo}\mathcal{L}_{u}^{i},$$ (6)

where $N_{p}$ is the number of pseudo-labels and $\mathcal{L}_{u}^{i}$ is the basic unsupervised loss of the $i$-th pseudo-label-prediction pair. Using the geometry-aware modulating factor, the GAW loss provides a clear direction for optimization, facilitating the semi-supervised learning process.

Figure 4: IoU vs. Rotation angle over different aspect-ratios.

4.4 Noise-driven Global Consistency

The objects in an aerial image are usually dense and regularly distributed, as shown in Fig. 1(a). We argue that such an arrangement is similar to the structured layout of the text within a document. The spatial configuration of the object set is implicitly referred to as the layout, conveying the inter-object relationships and the overarching pattern intrinsic to the image. Ideally, the layout consistency between the student’s and the teacher’s predictions will be preserved if each pseudo-label-prediction pair is aligned. However, such a condition is too strict and may hurt performance when there are noises in pseudo-labels. Therefore, it is reasonable to add the consistency between layouts as an additional relaxed optimization objective, encouraging the student to learn information from the teacher from a global perspective. Besides, by treating the predictions from students as a global distribution, the relations between these predictions can be regularized implicitly, which provides implicit constraint to the student to some extent.

Figure 5: The details of our proposed Noise-driven Global Consistency (NGC). The random noise is added to the output of the teacher and student, and then we use optimal transport to construct global consistency for the before and after disturbances. Furthermore, we make an alignment between the two transport plans.

The details of our Noise-driven Global Consistency (NGC) are shown in Fig. 5. Given the output of teacher and student, we will treat them as two global distributions. First, we add different random noises to disturb both the teacher and the student distributions. Then, we propose to make multi-perspective alignments from three aspects: 1) aligning distributions between the original teacher and original student; 2) aligning distributions between the disturbed teacher and disturbed student; 3) aligning the transport plans from 1) and 2). Applying the multi-perspective alignments is beneficial for maintaining consistency and stability during the learning process. In other words, this global constraint makes the output distribution of the student model as close as possible to that of the teacher model despite disturbance. It will increase the models’ tolerance to minor variations in inputs, i.e., allowing them to more effectively capture the essential features of the unlabeled data without depending on noise or specific details.

Specifically, we propose to leverage the optimal transport theory to measure the global similarity of layouts between the teacher’s and the student’s predictions. Specifically, let $K$ denote the number of classes. We define the global distributions of classification scores predicted by the teacher ($\mathbf{s}^{t}\in\mathbb{R}^{N_{p}\times K}$) and the student ($\mathbf{s}^{s}\in\mathbb{R}^{N_{p}\times K}$) as follows:

$$\mathbf{d}_{i}^{t}=e^{\mathbf{s}_{i,cls(i)}^{t}},\quad\mathbf{d}_{i}^{s}=e^{\mathbf{s}_{i,cls(i)}^{s}},$$ (7)

where $cls(i)\in\{0,1,\cdots,K-1\}$ is the class index with the largest score for the $i$-th pseudo-label. And $\mathbf{s}_{i,cls(i)}^{t}$ indicates the score of $cls(i)$-th class for the $i$-th pseudo-label. The $\mathbf{s}_{i,cls(i)}^{s}$ has a similar definition but refers to the predictions of the student. The exponential used here is empirical, aiming for numerical stability and ensuring the value is greater than 0.

To make the model generate robust consistency, we further add noises to disturb the distributions:

$$\widetilde{\mathbf{d}}^{t}=\mathbf{d}^{t}+\beta\boldsymbol{\epsilon}^{t},\quad\widetilde{\mathbf{d}}^{s}=\mathbf{d}^{s}+\beta\boldsymbol{\epsilon}^{s},$$ (8)

where $\boldsymbol{\epsilon}^{t},\boldsymbol{\epsilon}^{s}\sim\mathcal{N}(0,1)$. $\beta$ is a hyper-parameter used to control the noise intensity. The original and disturbed global distributions both reflect the many-to-many relationships between the teacher and the student, highlighting the characteristics of objects in the aerial scenario. To construct the cost map for solving the OT problem, we consider the spatial distance and the score difference of each possible pair. Specifically, we measure the transport costs between pseudo-labels and predictions as follows:

$$\boldsymbol{C}_{i,j}=\boldsymbol{C}_{i,j}^{dist}+\boldsymbol{C}_{i,j}^{score},$$ (9)

$$\boldsymbol{C}_{i,j}^{dist}=\frac{\|\mathbf{z}_{i}^{t}-\mathbf{z}_{j}^{s}\|_{2}}{\max_{1\leq a,b\leq N_{p}}\|\mathbf{z}_{a}^{t}-\mathbf{z}_{b}^{s}\|_{2}},$$ (10)

$$\boldsymbol{C}_{i,j}^{score}=\frac{\|\mathbf{s}_{i,cls(i)}^{t}-\mathbf{s}_{j,cls(j)}^{s}\|_{1}}{\max_{1\leq a,b\leq N_{p}}\|\mathbf{s}_{a,cls(a)}^{t}-\mathbf{s}_{b,cls(b)}^{s}\|_{1}},$$ (11)

where $\|\cdot\|_{1}$ is the $\ell^{1}$ norm of a vector and $\|\cdot\|_{2}$ represents the Euclidean distance. $\mathbf{z}_{i}^{t}$ and $\mathbf{z}_{j}^{s}$ are pixel-level 2D coordinates of the $i$-th sample in the teacher and the $j$-th sample in the student, respectively.

Here, we define the global consistencies between the original (resp. noisy) teacher and original (resp. noisy) student as follows:

$$\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s}) =\mathcal{W}_{ot}(\mathbf{d}^{t},\mathbf{d}^{s})$$ (12)

$$=\left\langle\boldsymbol{\lambda}^{*},\frac{\mathbf{d}^{t}}{\|\mathbf{d}^{t}\|_{1}}\right\rangle+\left\langle\boldsymbol{\mu}^{*},\frac{\mathbf{d}^{s}}{\|\mathbf{d}^{s}\|_{1}}\right\rangle,$$

$$\mathcal{L}_{GC}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s}) =\mathcal{W}_{ot}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s})$$ (13)

$$=\left\langle\widetilde{\boldsymbol{\lambda}}^{*},\frac{\widetilde{\mathbf{d}}^{t}}{\|\widetilde{\mathbf{d}}^{t}\|_{1}}\right\rangle+\left\langle\widetilde{\boldsymbol{\mu}}^{*},\frac{\widetilde{\mathbf{d}}^{s}}{\|\widetilde{\mathbf{d}}^{s}\|_{1}}\right\rangle,$$

where ($\boldsymbol{\lambda}^{*},\boldsymbol{\mu}^{*})$ and ($\widetilde{\boldsymbol{\lambda}}^{*},\widetilde{\boldsymbol{\mu}}^{*}$) are the solutions of Eq. 3. We solve the OT problem by a fast Sinkhorn distances algorithm [9], obtaining the approximate solution. Based on the defined $\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s})$ and $\mathcal{L}_{GC}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s})$, their gradients with respect to $\mathbf{d}^{s}$ and $\widetilde{\mathbf{d}}^{s}$ follows:

$$\frac{\partial\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s})}{\partial\mathbf{d}^{s}}=\frac{\boldsymbol{\mu}^{*}}{\|\mathbf{d}^{s}\|_{1}}-\frac{\left\langle\boldsymbol{\mu}^{*},\mathbf{d}^{s}\right\rangle}{\|\mathbf{d}^{s}\|_{1}^{2}},$$ (14)

$$\frac{\partial\mathcal{L}_{GC}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s})}{\partial\widetilde{\mathbf{d}}^{s}}=\frac{\widetilde{\boldsymbol{\mu}}^{*}}{\|\widetilde{\mathbf{d}}^{s}\|_{1}}-\frac{\left\langle\widetilde{\boldsymbol{\mu}}^{*},\widetilde{\mathbf{d}}^{s}\right\rangle}{\|\widetilde{\mathbf{d}}^{s}\|_{1}^{2}},$$ (15)

the gradients can be back-propagated to learn the parameters of the detector. In addition, there is an OT plan $\boldsymbol{P}\in\mathbb{R}^{N_{p}\times N_{p}}$ (resp. $\widetilde{\boldsymbol{P}}\in\mathbb{R}^{N_{p}\times N_{p}}$ between the original (resp. noisy) teacher and the original (resp. noisy) student (refer to the Eq. 2). The OT plans reflect the map** from the source distribution to the target one. We suggest aligning transport plans between without and with disturbance, which can evaluate the impact of noise and offer auxiliary guidance for the model (Fig. 5). Given the OT plan, we will multiply it with the student distribution to be better aware of the global prior and then utilize the $\mathrm{MSE}$ to measure the difference between the two types of OT plans:

$$\mathcal{L}_{plan}=\mathrm{MSE}(\boldsymbol{P}-\widetilde{\boldsymbol{P}})$$ (16)

Overall, we obtain the training objective of NGC:

$$\mathcal{L}_{NGC}=\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s})+\mathcal{L}_{GC}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s})+\mathcal{L}_{plan},$$ (17)

when we optimize such OT-based loss, the predictions converge closely to the pseudo-labels from a global perspective. We argue that promoting such multiple global consistency not only relieves the negative effect of inaccuracy pseudo-labels when detection processing, but also encourages a holistic learning approach, considering objects’ spatial and relational dynamics.

It should be noted that the feature representation is considered an ideal global distribution only when supported by a sufficient number of samples (e.g., treating a few sample points as a global distribution is unrealistic). If the sample number $N_{p}$ is smaller than a fixed number $\mathcal{T}_{g}$, we do not treat $\mathbf{s}^{t}$ and $\mathbf{s}^{s}$ as global distributions. Accordingly, the semi-supervised framework will not adopt the $\mathcal{L}_{NGC}$.

Although optimal transport theory has been explored in other methods [15, 45, 16, 66], our goal of using OT is significantly different. They mainly focus on utilizing OT to implement neural architecture search [66, 45], label assignment [16, 4], and image matching [38]. However, the goal of our NGC is to establish the many-to-many relationship between the teacher and the student under the semi-supervised setting, providing robust global consistency, which is complementary to the GAW loss.

4.5 Training Objective

SOOD++ is trained with the proposed GAW and NGC for unlabeled data as well as the supervised loss for labeled data. The overall loss $\mathcal{L}$ is defined as:

$$\mathcal{L}=\underbrace{\mathcal{L}_{GAW}+\mathcal{L}_{NGC}}_{\mathcal{L}_{u}}+\mathcal{L}_{s},$$ (18)

where $\mathcal{L}_{u}$ and $\mathcal{L}_{s}$ indicate the semi-supervised and supervised loss, respectively. Note that the supervised loss $\mathcal{L}_{s}$ is the same as defined in the supervised baseline (rotated-FCOS), and our designs only modify the unsupervised part.

5 Experiments

5.1 Dataset and Evaluation Metric

Following conventions in SSOD, for each conducted dataset, we consider two experiment settings, partially labeled data and fully labeled data, to validate the performance of a method on limited and abundant labeled data, respectively.

TABLE I: The detailed hyper-parameters of our method.

Hyper-parameter	Value
$\psi$	50
Sample ratio $\mathcal{R}$	0.25
IoU threshold $\mathcal{T}_{h}$	0.1
Noisy intensity $\beta$	0.3
Global number threshold $\mathcal{T}_{g}$	150

DOTA-V1.5 [58] contains 2,806 large aerial images and 402,089 annotated oriented objects. It includes three subsets: DOTA-V1.5-train, DOTA-V1.5-val, and DOTA-V1.5-test, containing 1,411, 458, and 937 images, respectively. The annotations of the DOTA-V1.5-test are not released. There are 16 categories in this dataset. Compared with its previous version, DOTA-V1.0, DOTA-V1.5 contains more small instances (less than 10 pixels), which makes it more challenging. For the partially labeled data setting, we randomly sample 10%, 20%, and 30% of the images from DOTA-V1.5-train as labeled data and set the remaining images as unlabeled data. For the fully labeled data setting, we set DOTA-V1.5-train as labeled data and DOTA-V1.5-test as unlabeled data. We report the performance with the standard mean average precision (mAP).

DOTA-V1.0 [58] contains the same images and data splits as DOTA-V1.5, including 1,411 training images, 458 validation images, and 937 test images. As an early version of the DOTA-series datasets, it contains 15 categories identical to DOTA-V1.5, except for annotating container-crane. For the partially labeled data setting, we use the same labeled and unlabeled splits as in DOTA-V1.5, including 10%, 20%, and 30%. For the fully labeled data setting, we set DOTA-V1.0-train as labeled data and DOTA-V1.0-test as unlabeled. The mAP is used as the evaluation metric.

HRSC2016 [41] is a widely-used dataset for arbitrarily oriented ship detection, consisting of 436 training images, 181 validation images, and 444 test images, with sizes ranging from 300×300 to 1,500×900. For the partially labeled data setting, we randomly sample 10%, 20%, and 30% images from the training set as labeled data and set the remaining images as unlabeled data. For the fully labeled data setting, we set the training set as labeled data and the test set as unlabeled data. We report the AP₈₅ and mAP_50:95 performance on the validation set.

The nuScenes Dataset [2] is a large-scale oriented 3D object detection dataset containing 1,000 distinct scenarios and 1.4M oriented 3D bounding boxes from 10 categories. These scenarios are recorded with a 32-beam LiDAR, six surround-view cameras, and five radars. The scenes are officially split into 700/150/150 scenes for the training/validation/test set. Each scene has 20s video frames and is fully annotated with 3D bounding boxes every 0.5 seconds. For the partially labeled data setting, we randomly sample 1/12, 1/6, and 1/3 annotated frames from the training set as labeled data and set the remaining training frames as unlabeled data. For the fully labeled data setting, we set all annotated frames from the training set as labeled data and unannotated frames as unlabeled data. We use the mAP and NDS as the primary metrics. Higher values indicate better performance, and the detailed information can be found in  [2].

5.2 Implementation Details

For the DOTA-V1.5, DOTA-V1.0, and HRSC2016 datasets, we take FCOS [56] as the representative anchor-free detector (the ResNet-50 [22] with FPN [33] used as the backbone), where BEVDet [24] is adopted for the nuScenes dataset. Following previous works [58, 20, 21], we crop the original images into 1,024$\times$1,024 patches with a stride of 824. This means the pixel overlap between two adjacent patches is 200 for the DOTA series datasets. We utilize asymmetric data augmentation for unlabeled data. We use strong augmentation (random flip**, color jittering, grayscale, and Gaussian blur) for the student model and weak augmentation (random flip**) for the teacher model. The models are trained for 180k iterations on 2 $\times$ RTX3090 GPUs (except the nuScenes dataset, using 8 $\times$ V100). With the SGD optimizer, the initial learning rate of 0.0025 is divided by ten at 120k and 160k. The momentum and the weight decay are set to 0.9 and 0.0001, respectively. Each GPU takes three images as input, where the proportion between unlabeled and labeled data is set to 1:2. Following previous SSOD works [76, 39], we use the “burn-in” strategy to initialize the teacher model. The detailed hyper-parameters are shown in Tab. I.

TABLE II: Experimental results on the DOTA-V1.5 (val set) with single-scale training and testing under 10%, 20%, and 30% partially labeled data. ^◆ and ^★ indicate implementations with rotated-Faster R-CNN and rotated-FCOS, respectively. Colored numbers show performance improvement over the previous conference version (also one of the SOTA methods).

Setting	Method	Publication	10%	20%	30%
Supervised-baseline	Rotated-Faster R-CNN [50]	NeurIPS 2016	43.43	51.32	53.14
	Rotated-FCOS [56]	ICCV 2019	42.78	50.11	54.79
Semi-supervised	Unbiased Teacher◆ [39]	ICLR 2021	44.51	52.80	53.33
	Soft Teacher◆ [64]	ICCV 2021	48.46	54.89	57.83
	Dense Teacher★ [76]	ECCV 2022	46.90	53.93	57.86
	ARSL★ [35]	CVPR 2023	47.56	55.05	58.94
	SOOD★ [23] (conference version)	CVPR 2023	48.63	55.58	59.23
	SOOD++★ (ours)	-	50.48(+1.85)	57.44(+1.86)	61.51(+2.28)

TABLE III: Experimental results on DOTA-V1.5 (val set) with single-scale training and testing under fully labeled data. Numbers before the arrow are the supervised baseline (^◆ for rotated-Faster R-CNN and ^★ for rotated-FCOS).

Method	Publication	mAP
Unbiased Teacher◆ [39]	ICLR 2021	66.12 $\xrightarrow{-1.27}$ 64.85
Soft Teacher◆ [64]	ICCV 2021	66.12 $\xrightarrow{+0.28}$ 66.40
Dense Teacher★ [76]	ECCV 2022	65.46 $\xrightarrow{+0.92}$ 66.38
ARSL★ [35]	CVPR 2023	65.46 $\xrightarrow{+1.22}$ 66.68
SOOD++★ (ours)	-	65.46 $\xrightarrow{+3.58}$ 69.04

6 Results and Analysis

In this section, we compare our method with the state-of-the-art SSOD methods [76, 64, 39, 35] on several challenging datasets. For a fair comparison, we re-implement these methods and have carefully tuned their hyper-parameters to obtain their best performance on oriented object detectors with the same augmentation setting.

6.1 Main Results

6.1.1 Results on the DOTA-V1.5 dataset

We first evaluate our method on the large-scale, challenging, and most convincing oriented object detection dataset, i.e., the DOTA-V1.5 dataset.

Partially labeled data. We conduct a comprehensive evaluation of our SOOD++ across various proportions of labeled data, as shown in Tab. II. We can make the following observations: 1) Our SOOD++ achieves state-of-the-art performance under all proportions, achieving 50.48, 57.44, and 61.51 mAP on 10%, 20%, and 30% proportions, respectively, significantly surpassing our supervised baseline rotated-FOCS [56] by 7.70, 7.33, and 6.72 mAP. 2) Compared with the state-of-the-art anchor-free method ARSL [35] and our conference version [23], we significantly outperform them by 2.92/1.85, 2.39/1.86 and 2.57/2.28 mAP, highly demonstrating the substantial improvements we made over the conference version. 3) Compared with Unbiased Teacher [39] and Soft Teacher [64], two representative anchor-based methods, on 10% and 20% proportions, our SOOD++ achieves higher performance even though our supervised baseline (rotated-FCOS) is weaker than they used (rotated-Faster R-CNN). Under 30% data proportion, SOOD++ surpasses Unbiased Teacher [39] and Soft Teacher [64] by a large margin, i.e., 8.18 and 3.68 mAP, respectively.

Fully labeled data. We further compare our SOOD++ with other semi-supervised object detection (SSOD) approaches under the fully labeled data setting, which is more challenging. The goal of this setting is to use additional unlabeled data to enhance the performance of a well-trained detector on large-scale labeled data. As shown in Tab. III, the results indicate that we surpass previous SSOD methods [76, 64, 39] by at least 2.36 mAP. Besides, compared with our baseline, even though it has been trained well in massive labeled data, we still achieve 3.58 mAP improvement, demonstrating our method’s ability to learn from unlabeled data. An interesting phenomenon is that the performance of Unbiased Teacher [39] drops after adding unlabeled data. We argue the reason might be that Unbiased Teacher does not apply unsupervised losses for bounding box regression, which is essential for oriented object detection.

Figure 6: Experimental results on the DOTA-V1.0 (val set) with single-scale training and testing under 10%, 20%, and 30% partially labeled data. All methods are implemented with rotated-FCOS.

Setting	Method	Publication	10%	20%	30%
Supervised-baseline	Rotated-FCOS [56]	ICCV 2019	46.20	53.88	57.30
Semi-supervised	Dense Teacher [76]	ECCV 2022	50.87	58.02	60.91
	ARSL [35]	CVPR 2023	52.89	59.84	62.92
	SOOD++ (ours)	-	54.17	60.53	64.93

Figure 7: Experimental results on DOTA-V1.0 (val set) with single-scale training and testing under fully labeled data. Numbers before the arrow are the supervised baseline (rotated-FCOS).

Method	Publication	mAP
Dense Teacher [76]	ECCV 2022	70.22 $\xrightarrow{+0.98}$ 71.20
ARSL [35]	CVPR 2023	70.22 $\xrightarrow{+1.41}$ 71.63
SOOD++ (ours)	-	70.22 $\xrightarrow{+2.18}$ 72.40

TABLE IV: Experimental results on HRSC2016 (val set) with single-scale training and testing under 10%, 20%, and 30% partially labeled data. All methods are implemented with rotated-FCOS.

Setting	Method	Publication	10%		20%		30%
			AP85(%)	mAP50:90(%)	AP85(%)	mAP50:90(%)	AP85(%)	mAP50:90(%)
Supervised	Rotated-FCOS [56]	ICCV 2019	7.30	37.49	30.20	52.57	37.40	58.78
Semi-supervised	Dense Teacher [76]	ECCV 2022	21.90	49.93	32.80	59.63	41.40	63.08
	ARSL [35]	CVPR 2023	21.50	49.17	33.60	59.76	41.10	63.35
	SOOD++ (ours)	-	22.90	52.62	40.30	62.14	47.80	65.27

6.1.2 Results on the DOTA-V1.0 dataset

We then conduct the experiments on the DOTA-V1.0 dataset, which has the same images as the DOTA-V1.5 dataset but provides coarse annotations (e.g., many tiny objects with fewer pixels are not annotated).

Partially labeled data. We evaluate our method under different labeled data proportions on the DOTA-V1.0 dataset, as shown in Tab. 7. It is similar to the comparison of the DOTA-V1.5 dataset, where our SOOD++ achieves state-of-the-art performance under all proportions. Specifically, it surpasses the SOTA method [35] by 1.28, 0.69, and 2.01 mAP under 10%, 20%, and 30% labeled data settings, respectively. These impressive results demonstrate our method handles the coarse annotation dataset well.

Fully labeled data. Under this setting, we can explore the potential of the proposed semi-supervised framework even if it has been trained with extensive labeled data. As shown in Tab. 7, although the supervised baseline has already achieved high performance, our SOOD++ still takes 2.18 mAP improvements on it, an observed performance gain when using the unlabeled data. In addition, our method performs better compared with other semi-supervised methods [76, 35] with the same baseline.

6.1.3 Results on the HRSC2016 dataset

This part evaluates the proposed method on another representative aerial scenes dataset, i.e., HRSC2016 [41]. Notably, the commonly utilized mAP$(07)$ metric fails to adequately show the performance discrepancies among different methodologies. Consequently, we use more stringent evaluation metrics to provide a more precise performance evaluation, including AP₈₅ and mAP_50:90.

Partially labeled data. As shown in Tab. IV, our SOOD++ consistently outperforms other semi-supervised methods [35, 76] in various settings. Especially under the rigorous metric (i.e., AP${}_{85})$, our method shows a more pronounced improvement, which surpasses ARSL [35] by 6.7 and 6.7 mAP under 20% and 30% labeled data settings, indicating the effectiveness of our method.

Fully labeled data. Similar to the above datasets, we also compare the performance under the fully labeled data setting. As shown in Tab. V, our method outperforms previous SSOD methods [35, 76] and surpasses our supervised baseline by a large margin with 8.90 AP₈₅.

Figure 8: Some typical examples from the DOTA-V1.5 dataset. From left to right, there are ground truth, supervised baseline (rotated-FCOS[56]), the SOTA method ARSL [35], and our proposed SOOD++. Row 2 refers to scenarios with extreme aspect ratios case. Row 3 and row 4 are two extremely dense scenes. The red dashed and solid red circles represent false negative and false positive, respectively.

TABLE V: Experimental results on the HRSC2016 (val set) dataset with single-scale training and testing under fully labeled data. Numbers before the arrow are the supervised baseline (rotated-FCOS).

Method	Publication	AP85
Dense Teacher [76]	ECCV 2022	49.50 $\xrightarrow{+3.20}$ 52.70
ARSL [35]	CVPR 2023	49.50 $\xrightarrow{+5.00}$ 54.50
SOOD++ (ours)	-	49.50 $\xrightarrow{+8.90}$ 58.40

TABLE VI: The generalization ability of our SOOD++. The results are reported with single-scale training and testing.

(a) The effectiveness of SOOD++ on different methods under DOTA-V1.5 (val set) fully labeled data.

Detector	Publication	Method	mAP
CFA [19]	CVPR 2021	Supervised	65.75
		SOOD++ (ours)	68.49
Oriented R-CNN [62]	ICCV 2021	Supervised	67.26
		SOOD++ (ours)	69.50

(b) Experimental results on DOTA-V1.5 (test set). * indicates the strong supervised baseline we implemented.

Setting	Method	Publication	mAP
Supervised SOTA	DCFL [63]	CVPR 23	71.03
	LSKNet [32]	ICCV 23	70.26
	PKINet [3]	CVPR 24	71.47
	Strong Oriented R-CNN*	-	70.66
Semi-supervised	SOOD++ (ours)	-	72.48

6.2 Visualization analysis

We further visualize the qualitative results in Fig. 8. These samples are selected from typical scenes on the DOTA-V1.5 dataset (validation set). Specifically, the first sample is a sparse and relatively simpler case, and most semi-supervised methods can work well. However, when facing challenging cases, e.g., the second sample is the large aspect ratio case, and the third and fourth samples are two dense and small cases, we can find that compared with the SOTA method [35], our method presents better visualization. We believe this is because SOOD++ can exploit more potential semantic information from unlabeled data by leveraging the geometry prior and global consistency. These qualitative results further highlight the effectiveness of our method.

6.3 The generalization of our method

This section explores the generalization capabilities of our method from two perspectives: 1) We evaluate our method on different detectors, e.g., CFA [19] and Oriented R-CNN [62]; 2) We conduct experiments on the semi-supervised multi-view oriented 3D object detection to demonstrate the generalization on 3D scenes.

6.3.1 The generalization on different detectors

To examine the adaptability of our method, we conduct experiments on a suite of representative oriented object detectors under the fully labeled data setting. Specifically, we choose the CFA [19] and Oriented R-CNN [62] for evaluation, as shown in Tab. VI(a). When trained with the full DOTA-V1.5 training data, the fully supervised Oriented R-CNN [62] achieves 67.26 mAP, which is indeed a superior performance. Even so, our method improves it by 2.24 mAP and reaches 69.50 mAP by leveraging the unlabeled data. In addition, our method also reports 2.74 mAP improvement on the CFA [19]. These results are significant, demonstrating our method’s ability to seamlessly integrate with diverse detectors and effectively leverage the intrinsic strengths of the proposed key designs.

We further implement a strong supervised baseline (oriented R-CNN with EVA pre-trained weight [12]) and combine it with our SOOD++ to compare with SOTA oriented detectors [32, 63, 3], as shown in Tab. VI(b). Despite the strong baseline achieving 70.66 mAP on the DOTA-V1.5 test set, our approach shows a 1.82 mAP improvement by leveraging unlabeled data. It outperforms the PKINet [3] by a notable margin, pushing the new state-of-the-art.

TABLE VII: The generalization on the oriented 3D object detection dataset (nuScenes [2] val set). All semi-supervised methods use the same detector (BEVDet [24]).

Setting	Method	1/12 data		1/6 data		1/3 data		Full data
		mAP	NDS	mAP	NDS	mAP	NDS	mAP	NDS
Supervised	BEVDet [24]	8.29	11.47	13.75	19.52	20.31	25.52	25.97	33.02
Semi-	ARSL [35]	8.39	11.90	15.58	20.91	21.56	27.13	27.00	34.57
	SOOD++ (ours)	8.63	13.26	17.42	22.03	23.24	28.24	29.66	36.32

TABLE VIII: The effectiveness of our key components: Simple Instance-aware Dense Sampling (SIDS) strategy, Geometry-aware Adaptive Weighting (GAW) loss, and Noise-driven Global Consistency (NGC).

Sampling strategy	GAW	NGC	mAP
			10%	20%	30%
From Dense Teacher [76]	-	-	46.90	53.93	57.86
From SOOD [23]	-	-	47.24	54.07	57.74
SIDS	-	-	48.21	55.54	58.69
SIDS	✓	-	49.27	56.45	60.11
	-	✓	49.32	56.63	59.76
	✓	✓	50.48	57.44	61.51

TABLE IX: The analysis of our Simple Instance-aware Dense Sampling (SIDS) strategy. The experiments are equipped with GAW and NGC.

(a) The effect of sample ratio $\mathcal{R}$.

Sample Ratio $\mathcal{R}$	mAP
0.125	50.13
0.25	50.48
0.5	49.92
1.0	49.11

(b) The effect of different filtering criteria adopted in the background.

Criteria	mAP
Classification	49.41
Centerness	49.43
IoU	50.48

(c) The effect of IoU threshold $\mathcal{T}_{h}$.

Threshold $\mathcal{T}_{h}$	mAP
0.05	50.39
0.1	50.48
0.2	50.25
0.3	50.17

Figure 9: Qualitative results of our method on semi-supervised multi-view oriented 3D object detection task. The above multi-view images show the 3D detection results (green boxes). We also draw the predicted results on the BEV view (the lower part) for better visualizations.

6.3.2 The generalization on semi-supervised multi-view oriented 3D object detection

We find that the multi-view oriented 3D objects are very similar to the objects from aerial scenes, as both of them have arbitrary orientations and are usually dense. In addition, the advanced multi-view oriented 3D object detection methods usually detect 3D objects by transforming the perspective from the camera view to the Bird’s Eye View (BEV), similar to aerial scenes captured from a BEV. Thus, to further demonstrate the generalization of our method, we conduct experiments on the most convincing large-scale multi-view oriented 3D object detection dataset, i.e., nuScenes [2]. As shown in Tab. VII, our method reports superior performance, consistently outperforming the supervised baseline and ARSL [35] by a large margin in terms of mAP and NDS metrics under both partially and fully labeled data settings.

In addition, we provide some visualization to present the effectiveness of our method, as shown in Fig. 9. These quantitative and qualitative results prove SOOD++’s ability to be generalized to semi-supervised multi-view oriented 3D object detection.

6.4 Analysis and Ablation Study

The following experiments are conducted on the large-scale DOTA-V1.5 dataset (val set). Unless specified, all the ablation experiments are performed using 10% of labeled data with single-scale training and testing.

6.4.1 The effectiveness of key components

We first study the effects of the proposed key components, including the Simple Instance-aware Dense Sampling (SIDS) strategy, Geometry-aware Adaptive Weighting (GAW) loss, and Noise-driven Global Consistency (NGC), the results listed in Tab. VIII.

There are various dense sampling strategies for constructing dense pseudo-labels. Dense Teacher [76] and our conference version [23] sample dense pseudo-labels from the entire predicted score maps and box area of the foreground, respectively. This work not only considers the easy pseudo-labels of the foreground but also mines the potential pseudo-labels of the background, leading to the proposal of SIDS. Although our SIDS is relatively simple, compared with the strategies in [76, 23], we demonstrate notable improvement, which indicates the constructed higher-quality pseudo-labels of our method.

Based on SIDS, when GAW or NGC is individually incorporated, an increase in mAP is observed. It demonstrates the effectiveness of constructing the one-to-one or many-to-many (global) consistency. Finally, when combining all our key components (NGC, GAW, and SIDS), the improvement will be amplified, achieving satisfactory 50.48, 57.44, and 61.51 mAP under the 10%, 20%, and 30% labeled data settings, respectively. It is reasonable, as the one-to-one and many-to-many relationships are highly complementary. These detailed ablation studies prove the consistent improvements and the value of each component.

6.4.2 The analysis of Simple Instance-aware Dense Sampling (SIDS) strategy

In this part, we discuss the design of our Simple Instance-aware Dense Sampling (SIDS) strategy, which is used to construct high-quality pseudo-dense labels.

The effect of sample ratio $\mathcal{R}$. Tab. IX(a) studies the effect of the sample ratio $\mathcal{R}$ used in the foreground. The larger the number, the more pseudo-labels are sampled, and vice versa. We can see that the best performance, 50.48 mAP, is achieved when the sample ratio $\mathcal{R}$ is set to 0.25. We hypothesize that this value ensures a good balance between positive and negative pseudo-labels. Increasing it will introduce more noise pseudo-labels that harm the training process, and decreasing it will lead to the loss of some positive pseudo-labels and failure in learning. When set to 1.0, all pixel-level predictions within the instance box are sampled, which inevitably introduces a lot of noise dense pseudo-labels, leading to poor performance.

The filtering criteria used in the background. As discussed in Sec. 4.2, there are many potential hard objects that merge with the background, which are hard to extract as pseudo-labels. Thus, to construct comprehensive dense pseudo-labels, we utilize the predicted IoU score as an auxiliary metric to help sample dense pseudo-labels from the background. We study the effect of using different auxiliary metrics to mine the potential labels, including classification and centeredness scores of default rotated-FCOS [56], as shown in Tab. IX(b). The results prove the necessity of using IoU as the filtering criteria for mining the hard case of background. We argue the main reason is that the IoU score directly reflects the alignment between the predicted and ground truth, making it a more comprehensive metric than the other two.

The effect of threshold $\mathcal{T}_{h}$. We also discuss the effect of the IoU threshold $\mathcal{T}_{h}$, as shown in Tab. IX(c). It is insensitive to the chosen value, and we empirically find that the threshold set as 0.1 can achieve a better performance.

TABLE X: The analysis of our Geometry-aware Adaptive Weighting (GAW) loss. The experiments are equipped with SIDS and NGC.

(a) The effect of used geometry information.

Orientation gap	Aspect ratio	mAP
-	-	49.32
✓	-	49.81
-	✓	49.76
✓	✓	50.48

(b) The effect of $\psi$.

$\psi$	mAP
w/o GAW	49.32
1	49.83
10	49.95
50	50.48
100	50.13

TABLE XI: The analysis of our Noise-driven Global Consistency (NGC). The experiments are equipped with SIDS strategy and GAW loss.

(a) The effect of different global alignments.

Components	mAP
-	49.27
$\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s})$	49.83
$\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s})+\mathcal{L}_{GC}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s})$	50.17
$\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s})+\mathcal{L}_{GC}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s})+\mathcal{L}_{plan}$	50.48

(b) The effect of different cost maps.

Distance $\boldsymbol{C}_{i,j}^{dist}$	Score $\boldsymbol{C}_{i,j}^{score}$	mAP
-	-	49.27
-	✓	49.80
✓	-	50.09
✓	✓	50.48

(c) The effect of noisy intensity $\beta$.

$\beta$	mAP
0.1	50.31
0.2	50.38
0.3	50.48
0.4	50.13
0.5	50.21

(d) The effect of $\mathcal{T}_{g}$.

$\mathcal{T}_{g}$	mAP
1	49.06
100	49.83
150	50.48
200	49.95
250	49.68

6.4.3 The analysis of Geometry-aware Adaptive Weighting (GAW) loss

The effect of geometry information in the GAW loss. In GAW, we utilize the intrinsic geometry information (e.g., orientation gap and aspect ratios) to construct the geometry-aware modulating factor (Eq. 5). To investigate their influence, we conduct experiments on different combinations of the GAW loss. As shown in Tab. X(a), without the GAW, we just achieve 49.32 mAP. By individually utilizing the orientation gap or aspect ratio to construct the GAW, they report similar performance gains. When we combine the two information, the 50.48 mAP performance is reached. This substantial improvement confirms that using both geometric information together provides a more thorough understanding of the oriented objects’ characteristics, leading to better detection performance.

The effect of hyper-parameter $\psi$. We also study the influence of the hyper-parameter $\psi$ in GAW. As shown in Tab. X(b), we set $\psi$ to 1 and get the performance of 49.83 mAP. As $\psi$ increases, the performance of our method improves when $\psi$ varies from 1 to 50. However, further increasing it to 100 slightly hurt the performance. Therefore, we set it to 50 by default. For this observation, we conjecture that increasing the weight $\psi$ will enlarge the influence of geometry information but also amplify the impact of the teacher’s inaccurate labels. Notably, GAW consistently provides performance gains regardless of the $\psi$ value.

6.4.4 The analysis of Noise-driven Global Consistency (NGC).

Our NGC aims to establish a relaxed global constraint for the teacher-student pair, and this part will ablate it.

The effect of different global alignments. As mentioned in Sec. 4.4, we treat the output of teacher and student as two global distributions and propose to use random noise to disturb the distributions. Then, we make alignment from multi-perspective, including the alignment ($\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s})$) between the original teacher and the original student, the alignment ($\mathcal{L}_{GC}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s})$) between the disturbed teacher and the disturbed student, and the alignment ($\mathcal{L}_{plan}$) between two OT plans generated by the two former alignments. Thus, we first study the effectiveness of using different alignments. As shown in Tab. XI(a), only using the $\mathcal{L}_{GC}(\mathbf{d}^{t},\mathbf{d}^{s})$, we achieve 49.83 mAP, 0.56 mAP gain compared with the method that equipped with SIDS strategy and GAW loss, indicating the necessary of many-to-many relationship. As expected, by further aligning the two disturbed distributions ($\mathcal{L}_{GC}(\widetilde{\mathbf{d}}^{t},\widetilde{\mathbf{d}}^{s})$) and two transport plans ($\mathcal{L}_{plan}$), we obtain a noticeable improvement, finally resulting in 50.48 mAP. We argue that using random noise to disturb the global distribution mitigates the detrimental effects of label noise and reinforces the model’s ability to learn robust features from the unlabeled data.

Figure 10: An intuitive illustration of the distribution of the absolute difference between the teacher and student’s classification scores. Lower values indicate that the teacher and the student have more consistent distributions.

The effect of different cost maps. To construct the cost map for solving the OT problem, we consider the spatial distance and the score difference of each possible matching pair. Thus, this part studies the effects of different cost maps of optimal transport in NGC. As shown in Tab. XI(b), individually utilizing spatial distance or score difference results in a maximum improvement of 0.82 mAP, indicating that relying on a single type of information is insufficient for capturing the comprehensive global prior necessary for effective learning. When both spatial distance and score difference are integrated into the cost maps, the improvement is amplified to 1.21 mAP. It highlights the mutually beneficial relationship between these two factors. With their help, we effectively model the many-to-many relationship between the teacher and the student, providing informative guidance to the model.

The effect of $\beta$ and $\mathcal{T}_{g}$. The former hyper-parameter $\beta$ controls the intensity of adding noise, and the latter $\mathcal{T}_{g}$ determines how long a sequence can be considered as a global distribution. As listed in Tab. XI(c) and Tab. XI(d), we empirically find that the best performance is achieved when the values of $\beta$ and $\mathcal{T}_{g}$ are set as 0.3 and 150, respectively.

Qualitative visualizations. We further provide qualitative visualizations to analyze the effect of the proposed NGC. Specifically, we visualize detection results with and without NGC in Fig. 4, along with the distribution of absolute difference values between the teacher and the student’s predictions. For the distribution map, lower values indicate that the teacher and the student’s distributions are more consistent. Fig. 4 clearly demonstrates that NGC can improve the consistency between the teacher and the student, leading to better prediction results.

6.5 Limitation and Related Discussion

This paper provides a simple yet solid framework for semi-supervised oriented object detection. Although our method achieves promising results, the usage of aerial objects’ characteristics is limited. Apart from orientation and global layout, many other properties of aerial objects should be considered, e.g., scale variations. In addition, this paper has explored the generalization of semi-supervised oriented 3D object detection tasks. However, some other oriented detection tasks, such as scene text detection, still need to be studied, and we leave these for the future work.

7 Conclusion

This paper proposes a practical solution named SOOD++ for semi-supervised oriented object detection. Focusing on oriented objects’ characteristics in aerial scenes, we customize Geometry-aware Adaptive Weighting (GAW) loss and Noise-driven Global Consistency (NGC). The former considers the importance of geometry information for oriented objects. The latter introduces the global layout concept to SSOD, measuring the global similarity between the teacher and the student in a many-to-many manner. Additionally, a Simple Instance-aware Dense Sampling (SIDS) strategy is proposed to construct high-quality pseudo-labels. To validate the effectiveness of our method, we have conducted extensive experiments on several challenging datasets. SOOD++ achieves consistent improvements under partially and fully labeled data settings, compared with SOTA methods. We hope this work will encourage the community to pay attention to semi-supervised oriented object detection and facilitate future research.