DTG-SSOD: Dense Teacher Guidance for Semi-Supervised Object Detection
Abstract
The Mean-Teacher (MT) scheme is widely adopted in semi-supervised object detection (SSOD).
In MT, the sparse pseudo labels, offered by the final predictions of the teacher (e.g., after Non Maximum Suppression (NMS)
post-processing), are adopted for the dense supervision for the student via hand-crafted label assignment.
However, the “sparse-to-dense” paradigm complicates the pipeline of SSOD, and simultaneously neglects the powerful direct, dense teacher supervision.
In this paper, we attempt to directly leverage the dense guidance of teacher to supervise student training, i.e., the “dense-to-dense” paradigm.
Specifically, we propose the Inverse NMS Clustering (INC) and Rank Matching (RM) to instantiate the dense supervision, without the widely used, conventional sparse pseudo labels.
INC leads the student to group candidate boxes into clusters in NMS as the teacher does, which is implemented by learning grouping information revealed in NMS procedure of the teacher.
After obtaining the same grouping scheme as the teacher via INC, the student further imitates the rank distribution of the teacher over clustered candidates through Rank Matching.
With the proposed INC and RM, we integrate Dense Teacher Guidance into Semi-Supervised Object Detection (termed “DTG-SSOD”), successfully abandoning sparse pseudo labels and enabling more informative learning on unlabeled data.
On COCO benchmark, our DTG-SSOD achieves state-of-the-art performance under various labeling ratios.
For example, under 10% labeling ratio, DTG-SSOD improves the supervised baseline from 26. to 35.9 mAP, outperforming the previous best method Soft Teacher by 1.9 points.
Code will be released at: https://github com ligang cs
DTG-SSOD
Introduction
Thanks to plenty of annotated data, deep learning based computer vision has achieved great success in the past few years [1–4].
However, labeling accurate annotations is usually labour-consuming and expensive, especially for dense prediction tasks [5–8], such as object detection, where multiple bounding boxes with their corresponding category labels need to be annotated extensively for each image.
Some works attempt to leverage easily accessible unlabeled data to promote object detection in a semi-supervised manner [9–12].
The model is first initialized by learning on the labeled data.
Then it is adopted for generating pseudo labels (i.e., pseudo bounding boxes and category labels) on the unlabeled data for training.
Recently, advanced Semi-Supervised Object Detection (SSOD) methods follow the Mean-Teacher paradigm [14], where a stronger teacher model is built from the student model via Exponential Moving
[Figure]
Average (EMA) to produce supervision signals over unlabeled data.
Nevertheless, all of the previous methods [9–11, 13, 12, 15] adopt sparse pseudo labels after the post-processing (e.g., NMS and score filtering) as the supervision, passed from the teacher to student.
To guarantee the precision of pseudo labels, a high score threshold (e.g., 0.9) is usually selected to filter coarse detection boxes and only a few pseudo boxes could survive.
Through the hand-crafted label assignment strategy [16–18], the detector transfers sparse pseudo labels into dense supervision for student training, which is termed the “sparse-to-dense” paradigm.
Such sparse pseudo labels neglect some important properties of object detection (e.g., dense predictions, contextual relations between neighbouring samples), and lack of informative and dense teacher guidance, leading to limited benefits for learning on the unlabeled data.
In Fig.
1, we illustrate the “sparse-to-dense” paradigm, where the sparse pseudo labels originated from the dense predictions of teacher are adopted for the dense supervision of student.
This process involves many handcrafted components (e.g., NMS, score thresholding, and label assignment), which inevitably introduce accumulated noise/bias to supervisory signals for student.
Therefore, it is more intuitive to adopt the “dense-to-dense” paradigm, i.e., directly employing dense predictions of teacher to guide the training of student, whilst abandoning the intermediate operations.
In this work, to implement the “dense-to-dense” paradigm, we propose to enforce the student to mimic the behavior of the teacher model.
Specifically, the behaviour is motivated from the procedure how dense samples perform NMS, an essential post-processing scheme for popular detectors.
To regularize the consistency on NMS behaviour between the teacher and student, we introduce Inverse
NMS Clustering (INC) and Rank Matching (RM).
INC leads the student to group the candidate boxes into clusters in NMS as the teacher does.
In the process of NMS, candidates are grouped into multiples clusters.
Within each cluster, teacher candidates are supposed to predict the identical object, which will assign training labels to the corresponding student samples in turn.
Different from the previous sparse-to-dense paradigm, where sparse pseudo labels are converted to student training labels via the hand-crafted label assignment, our dense-to-dense paradigm assigns dense training labels based on grouping information by reversing the NMS process of the teacher.
Based on INC where the student obtains the same grouping scheme of NMS with the teacher, we further introduce
Rank Matching to align the score rank over the clustered candidates between teacher and student.
NMS suppresses the duplicated detection and reserves only one box for each cluster through the rank of clustered candidates.
Therefore, the rank of the samples within each cluster could contain rich relationship learned by the teacher and it can behave as informative dense supervision which guides the student to reserve as the same candidates as the teacher during NMS.
Through the proposed INC and RM, the teacher model provides not only fine-grained and appropriate training labels, but also informative rank knowledge, realizing Dense Teacher Guidance for SSOD (DTG-SSOD).
To validate the effectiveness of our method, we conduct extensive experiments on
COCO benchmark under Partially Labeled Data and Fully Labeled Data settings.
Under the Partially
Labeled Data, where only 1%,2%,5%, and 10% images are labeled, the proposed DTG-SSOD achieves the state-of-the-art performance, for example, under 5% and 10% labeling ratios, DTGSSOD surpasses the previous best method Soft Teacher by +1.2 and +1.9 mAP, respectively.
Under the Fully Labeled Data, where the unlabeled2017 set is used as unlabeled data, the proposed DTG-
SSOD not only achieves the best performance (45.7 mAP), but also improves the learning efficiency, e.g., DTG-SSOD halves the training iterations of Soft Teacher but achieves even better performance.
Related Work
Semi-Supervised Image Classification (SSIC)
The pseudo labeling [19–21] is a popular pipeline, where unlabeled data is labeled by the model itself, and then combined with labeled data to act as training examples in the cycle self-training manner.
In the pseudo-labeling, hard pseudo labels are usually used with a confidence-based thresholding where only images with high confidence can be retained.
Recently, some improvements [22–24] on pseudo-labeling are proposed, for example,
Curriculum Labeling [22] proposes that restarting model parameters before each training cycle can promote self-training; Noisy Student Training [23] adopts an equal-or-larger model with noise injected to learn the pseudo labels, which are generated in the last self-training cycle.
Apart from hard pseudo labels, Noisy Student Training also tried soft pseudo labels, observing that for in-domain unlabeled images, both hard and soft pseudo labels perform well, while for out-of-domain unlabeled data, soft pseudo labels can work better.
On the other hand, based on the insight that the model should be robust to input perturbations, UDA [25] designs advanced data augmentations and enforces the model to predict the similar class distributions for the differently augmented inputs.
Here, soft labels (i.e., distributions) are superior in measuring the distance between different model predictions over the hard ones.
As another strong method, FixMatch [26] introduces a separate weak and strong augmentation and converts model predictions into hard labels, since the labels with low entropy are believed to be beneficial in their pipeline.
To conclude, previous SSIC works investigate various forms of supervisory signals (including hard and soft pseudo labels), and observe the form of supervision signals matters in semi-supervised learning, which inspires us to investigate the optimal supervision form for semi-supervised object detection.
Semi-Supervised Object Detection (SSOD) CSD [27] is an early attempt to construct consistency learning for horizontally flipped image pairs, where only the flip transformation can be used as the augmentation, which greatly limits its performance.
STAC [28] borrows the separate weak and strong augmentation from FixMatch [26] and applies it to SSOD, obtaining promising performance.
After that, [10, 11, 13, 9, 29] abandon the conventional multi-phase training and update the teacher model from the student via EMA after each training iteration.
Apart from the framework, previous works promote SSOD from two main perspectives: designing advanced data augmentation and improving the precision of pseudo labels.
For the former one, Instant-Teaching [10] introduces more complex augmentations for unlabeled data, including Mixup and Mosaic; MUM [15] designs a Mix/UnMix data augmentation, which enforces the student to reconstruct unmixed feature for the mixed input images.
As for the latter, through ensembling the teacher model on input images and prediction head, respectively, Humble Teacher [11] and Instant-Teaching [10] reduce confirmation bias of the pseudo labels; Soft Teacher [9] selects reliable pseudo boxes by averaging regression results from multiple jittered boxes; Rethinking Pse [12] converts the box regression to a classification task, making localization quality easy to estimate.
Different from these methods, which all adopt sparse pseudo boxes with category labels as the supervisory signal, our method investigates the new form of supervision for the first time and proposes to replace sparse pseudo labels with more informative dense supervision.
Method
The mechanism of weak-strong augmentation pairs shows promising performance in SSOD.
Based on it, we first build an end-to-end pseudo-labeling baseline, described in Sec.
3.1, through which we explain the task formulation of SSOD and the conventional sparse-to-dense paradigm.
Then in
Sec. 3.2, we introduce the proposed Inverse NMS Clustering (INC) and Rank Matching (RM), i.e., the key two components of the proposed DTG-SSOD.
3.1 Sparse-to-dense Paradigm
Task Formulation.
The framework of SSOD is illustrated in Fig.
The Mean-Teacher scheme is a common practice for previous works [9, 10, 13, 11] and realizes the end-to-end training, where the teacher is built from the student via EMA after each training iteration.
The teacher takes weakly augmented (e.g., flip and resize) images as input to generate pseudo labels, while the student is applied with strong augmentations (e.g., cutout, geometry transformation) for training.
Strong and appropriate data augmentations play an important role, which not only increase the difficulty of student tasks and alleviate the over-confidence issue [25], but also enable the student to be invariant to various input perturbations, leading to robust representation learning [23, 26].
In this work, we borrow the data augmentations from Soft Teacher [9], and more details can be found in Appendix.
Each training batch consists of both labeled data { x li , y li } N l i =1 and unlabeled data { x ui } N u i =1 , which are mixed according to the ratio r = N l N u , where N l and N u are the number of labeled and unlabeled images in each batch.
The overall loss function of SSOD is formulated as:
[Equation]
where α controls the contribution of losses on labeled data L l and unlabeled data L u .
Sparse-to-dense Baseline.
All previous SSOD approaches [9, 10, 13, 11] are based on the sparseto-dense mechanism, where sparse pseudo boxes with category labels are generated to behave as ground-truths for the student training.
It comes with confidence-based thresholding where only pseudo labels with high confidence (e.g., larger than 0.9) are retained.
This makes foreground supervision on unlabeled data much sparser than that on labeled data, therefore, the class imbalance problem is amplified in SSOD and impedes the detector training severely.
To alleviate this issue, we borrow some strengths of previous works: Soft Teacher [9] sets the mixing ratio r to 1/4 to sample more unlabeled data in each training batch, which makes the number of foreground samples on unlabeled data approach that on labeled data; Unbiased Teacher [13] replaces the cross-entropy loss with the Focal loss [16], reducing the gradient contribution from easy examples.
Both of these two improvements, i.e., the appropriate mixing ratio r (1/4) and Focal loss, are adopted in the sparse-to-dense baseline and our dense-to-dense DTG method.
Because the teacher only provides sparse pseudo labels, which are further converted into the dense supervision for the student training, these methods are termed “sparse-to-dense” paradigm.
In theory, our SSOD method is independent of the detection framework and can be applicable to both one-stage and two-stage detectors.
For a fair comparison with previous works, we use the Faster RCNN [17] as the default detection framework.
3.2 Our method: Dense-to-dense paradigm
Pseudo labels in SSOD contain both bounding boxes and category labels.
Considering that classification scores are not strongly correlated with localization quality [30, 3, 31, 32], although confidencebased thresholding can filter out many false positives, the localization quality of pseudo boxes is not guaranteed.
Poorly-localized pseudo boxes will mislead the label assignment and confuse the decision boundary.
Therefore, previous methods [12, 9] design two separate sets of indicators to measure the confidence of categories and box locations, which complicates the SSOD pipeline.
Moreover, the sparse-to-dense paradigm neglects some important properties of object detection and lacks of powerful direct, dense teacher guidance.
Motivated by these, we propose to directly employ teacher’s dense predictions to supervise the student’s training, termed “dense-to-dense” paradigm, abandoning the explicit sparse pseudo boxes.
Dense predictions widely exist in both RPN and RCNN stages.
Specifically, in the RPN stage, multiple anchors are usually tiled on each feature map grid [17, 16]; while in the RCNN stage, taking 2,000 proposals as training samples is a common practice [17].
Compared against the sparse pseudo labels, dense predictions of the teacher contain rich knowledge, and we attempt to pass it to the student by enforcing the student to mimic the dense behaviour of the teacher.
Specifically, we analyze how teacher’s dense samples perform to eliminate redundancy in the process of Non Maximum
Suppression (NMS).
Based on the analysis, we design Inverse NMS Clustering and Rank Matching to regularize consistency on the NMS procedure between the teacher and student, from which the student can obtain precise training labels and informative supervision signals.
However, implementing dense mimic is not trivial.
Due to the separate weak-strong augmentations, one-to-one correspondence between the samples of the teacher and student can not be easily identified.
We design a simple yet effective matching strategy to address this.
On the R-CNN stage, to keep the same proposals between the teacher and student, we directly take proposals produced by the teacher RPN as input for the student R-CNN, realizing one-to-one correspondence.
For the RPN stage, since the feature map sizes and pre-set anchors differ between teacher and student, we first transfer the teacher’s anchors to the space of student through box transformation.
Then we assign
[Figure]
1.One
[Figure]
to
[Figure]
NMS procedure of the teacher
Candidate Grouping Ranking & Suppressing Reserved Boxes clusterclustercluster① ② ③ ④ ①
Proposals
Assign labels ① ② ③ ④ ① ② ③ ④
Teacher cluster 1 Teacher cluster
Dense Student Predictions 2.
Inverse NMS Clustering 3.
Rank Matching
Rank Distribution
[Figure]
① ③ ④ ②
The corresponding student predictions ①
[Figure]
①
[Figure]
the teacher’s anchors to the corresponding FPN level based on the anchor box scale.
The target FPN level m i can be obtained by: m i = log 2 area i S , where area i refers to the area of the teacher anchor box t i , and S is a hyper-parameter and set to 32 .
Once the target FPN level m i is obtained, we calculate the Intersection-over-Unions (IoUs) between the teacher anchor t i and all student anchors { s j } j ∈ m i belonging to the FPN level m i , and choose the student anchor with the largest IoU as the corresponding one: s i = arg max IoU ( t i , s j ) , j ∈ m i .
Through the proposed matching strategy, one-to-one correspondence between training samples of the teacher and student is identified, making dense behaviour imitation possible.
Inverse NMS Clustering .
We first revisit the mechanism of NMS by taking RCNN stage as the example.
Before NMS, predicted bounding box offsets are applied on proposals to derive regressed boxes.
Then a pre-defined score threshold τ filters out clear background detection boxes.
For the remaining detection boxes (or candidates), NMS conducts clustering on them based on their categories and locations.
As a result, candidates are grouped into multiple clusters and in each cluster, candidates are supposed to predict the same object.
At last, only one candidate with the highest confidence can be retained and others will be suppressed.
The process is illustrated in Fig.
To regularize consistency on the NMS behaviour of the teacher and student, we enforce the student to perform the same candidate grouping as the teacher.
In the NMS procedure of the teacher, candidates will be grouped into multiple clusters.
The cluster indexed by j can be defined as
C tj = { ( b ti , c ti ) | 1 ≤ i ≤ N j } , where ( b i , c i ) refers to the i th candidate, b and c denote the bounding box and category, respectively.
N j is the number of candidates belonging to C j .
In each cluster of the teacher (e.g., C tj ), the candidates are highly-overlapped and own
the same category c (i.e., c t 1 = · · · = c ti = c ).
Therefore, they are supposed to predict the identical object.
Supposing that the k th candidate ( b tk , c ) is finally reserved in C tj , it is the most precise detection box for the object and can behave as the detection target for other candidates in C j .
Then the target ( b tk , c ) will inversely assign training labels to the corresponding student samples { ( b si , c si ) | 1 ≤ i ≤ N j } .
To be specific, the category label c is employed to supervise the classification predictions of the student: { c si } N j i =1 , and the classification loss can be formulated as:
[Equation]
where f cls refers to the Focal loss [16].
Besides, for those teacher candidates, whose scores are lower than the NMS threshold τ , they are clear background boxes, thus their corresponding student samples will obtain background labels.
For the regression task, we directly take the reserved box b tk as the regression target for { b si } N j i =1 .
We describe the regression loss as:
[Equation]
where b t ′ k is obtained by transforming the b tk to the student space; f reg refers to regression loss, e.g.,
Smooth L1 Loss [17].
There is a clear difference between our method and the traditional sparse-todense paradigm.
Our method calculates classification and regression labels according to the grouping information revealed in the NMS procedure of the teacher, whilst the sparse-to-dense paradigm employs some hand-crafted label assignment strategies [18, 16, 17] to convert sparse pseudo labels into student’s dense training labels, which loses rich teacher guidance.
Rank Matching.
Through the Inverse NMS Clustering, the student can obtain the same grouping scheme as the teacher.
Based on it, we further introduce Rank Matching to align the score rank over clustered candidates between the teacher and student.
In each NMS cluster, the candidate boxes are ranked according to the classification scores, then only one candidate will be retained regarding the candidate rank.
Therefore, the rank of candidates could contain rich relationship information.
In object detection, the candidate rank is validated to be a key element and can affect detection performance much [30, 3, 31, 33].
For example, GFL [30] proposes that taking IoUs between predicted boxes and ground-truths as classification targets contributes to a more precise rank.
Different from these methods, our Rank Matching models the score rank over clustered candidates to align the NMS behaviour of the teacher and student.
Exactly, we model the score distribution of the teacher and student over the cluster C j = { p i } N j i =1 as:
[Equation]
where p i indicates the logit from the target category of the i th candidate, and T is the temperature coefficient.
The Kullback-Leibler (KL) divergence is adopted to optimize the distance between D ( p ti ) and D ( p si ) :
[Equation]
Then L RM is averaged over all NMS clusters.
Rank Matching not only provides relationship information but also enables the student to reserve the identical candidates in each NMS cluster as the teacher.
The overall loss function on unlabeled data is defined as:
[Equation]
where β controls contribution of rank matching loss.
Experiments 4.1 Datasets and Evaluation Metric
We benchmark our proposed method on the challenging dataset, MS COCO [34].
Following the convention [28, 9, 13, 10], two training settings are provided, namely Partially Labeled Data and
Fully Labeled Data settings, which validate the method with limited and abundant labeled data, respectively.
The val2017 set with 5k images is used as the validation set.
We use AP 50:95 (denoted as mAP) as the evaluation metric.
We describe two training settings as follows:
Partially Labeled Data.
The train2017 set, consisting of 118k labeled images, is used as the training dataset, from which, we randomly sample 1%,2%,5%, and 10% images as labeled data, and set the remaining unsampled images as unlabeled data.
Following the practice of previous methods [28, 9, 13], for each labeling ratio, 5 different folds are provided and the final result is the average of these 5 folds.
Fully Labeled Data.
Here, the train2017 set is employed as the labeled data, and the additional unlabeled2017 set, consisting of 123k unlabeled images, is used as unlabeled data.
This setting is to validate SSOD methods can still promote the detector even when labeled data is abundant.
[Table]
4.2 Implementation Details
Faster RCNN [17] with FPN [37] performs as default detection framework in this paper, and ResNet50 [38] is adopted as the backbone.
Moreover, we use SGD as the optimizer.
At the beginning of
NMS, a score threshold τ is leveraged to filter clear background samples.
In this work, we set τ to a relatively high value to exclude noisy teacher samples, whose behaviour is ambiguous and hardly provides helpful guidance.
Exactly, we set τ to 0.9 in NMS of the RPN stage, and 0.45 in NMS of the R-CNN stage, empirically.
To stabilize the training, we also warm up the learning on unlabeled data for the first 4k iterations [13].
The training schedules for two training settings are a bit different:
Partially Labeled Data.
The model is trained for 180k iterations on 8 V100 GPUs with an initial learning rate of 0.01, which is then divided by 10 at 120k iteration and again at 160k iteration.
Mini-batch size per GPU is 5, with 1 labeled image and 4 unlabeled images.
The loss weight of unlabeled images α is set to 4.0.
Fully Labeled Data.
The model is trained on 8 GPUs with mini-batch size as 8 per GPU, wherelabeled images are combined with 4 unlabeled images.
We adopt 720k iterations for training, and the initial learning rate is set to 0.01, which is divided by 10 at 480k and 680k iterations.
We adopt α = 2 . 0 .
4.3 Main Results
We compare the proposed DTG-SSOD to the state-of-the-art methods on MS COCO val2017 set in
Tab. 1.
We first evaluate our method using the Partially Labeled Data setting, and experimental results demonstrate our method achieves the state-of-the-art performance under various labeling ratios.
Specifically, on 5% and 10% labeling ratios, DTG-SSOD improves the supervised baseline by +10.73 and +9.02 mAP, reaching 31.90 and 35.92 mAP, and it also surpasses the previous best method Soft Teacher [9] by +1.16 and +1.92 mAP, respectively.
When the labeling ratio is 2%,
our
DTG-SSOD yields 26.84 mAP, which is even comparable to the 26.90 mAP supervised baseline using 10% labeled data.
When the labeled data is extremely scarce, i.e., 1% labeling ratio, our DTG-SSOD achieves a comparable performance to the previous best method MUM [15], 21.27 vs. 21.88 mAP.
MUM [15] designs a strong data augmentation to expand data space, which mixes original image tiles to construct new images.
It is worth noting that when the number of labeled data increases (2%,5%, and 10% labeling ratios), our DTD-SSOD outperforms MUM by at least +2 mAP, moreover, our method is orthogonal to these advanced data augmentations [15, 10] and can benefit from them.
As Tab. 1 shows, under the Fully Labeled Data setting, our DTG-SSOD surpasses previous methods by a large margin, at least 1.2 mAP.
Following the practice of [9, 12], we also apply weak augmentations to labeled data and obtain a strong supervised baseline of 40.9 mAP.
Even based on the such strong baseline, DTG-SSOD still attains the biggest improvements of +4.8 mAP, reaching 45.7 mAP, which validates the effectiveness of our method when the amount of labeled data is relatively large.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
4.4 Ablation Experiments
In this section, we conduct extensive ablation studies to validate the key designs.
Unless specified, all experiments adopt a single data fold of the 10% labeling ratio as the training dataset.
Effects of Dense Teacher Guidance.
Instead of conventional pseudo boxes with category labels provided, our method offers dense supervisory signals.
We first validate that Dense Teacher Guidance (DTG) is a feasible learning paradigm for SSOD in Tab. 2(a), where we apply DTG to RPN and
R-CNN stages step by step.
On unlabeled data, when only RPN is optimized with the DTG and
R-CNN is excluded from training, it can bring 1.7 mAP gains to the supervised baseline.
Furthermore, we apply DTG to both RPN and RCNN, and it yields 36.3 mAP, outperforming the baseline by +9. mAP.
Significant improvements against the supervised baseline indicate dense teacher guidance can behave as effective supervision signals for learning on unlabeled data.
DTG consists of two key components, i.e., Inverse NMS Clustering (INC) and Rank Matching (RM), then we delve into these two parts with results reported in Tab. 2(b).
Through imitating the candidate grouping of the teacher in NMS, INC provides appropriate training labels for the student.
With the derived classification label, the model enjoys 4.4 mAP gains.
When regression labels are also applied, another 2.2 mAP improvement is found, reaching 35.1 mAP.
Based on INC, RM is introduced to provide teacher’s score rank over clustered candidates for the student, which enables the student to reserve the same candidate in each NMS cluster as the teacher.
As results show, RM can bring an obvious improvement, +1.2 mAP, demonstrating comprehensive alignment on NMS behaviour is beneficial.
Besides, the gains also indicate the candidate rank carries helpful relation information learned by the teacher.
Comparison of sparse-to-dense and dense-to-dense paradigms.
In this work, we replace the traditional sparse-to-dense (spa-den for short) paradigm with new dense-to-dense (den-den) paradigm.
Detailed comparisons are conducted in Tab.
2(c-d).
We first build a sparse-to-dense baseline, then incrementally substitute our den-den paradigm into each network stage (i.e., RPN, RCNN) in Tab. 2(c).
Thanks to the appropriate mixing ratio r = 1 / 4 between labeled and unlabeled images and Focal
Loss, our sparse-to-dense baseline achieves comparable performance (34.5 mAP) to the previous methods.
When we substitute the den-den paradigm into R-CNN, it yields 35.5 mAP, which is 1.
points higher than the spa-den R-CNN.
After that, we further apply den-den to RPN, which attains +0.8 mAP gains.
These experiments validate the proposed dense-to-dense paradigm is superior to the traditional sparse-to-dense paradigm in both RPN and RCNN stages.
On the other hand, we also compare two paradigms under various labeling ratios in Tab. 2(d).
Under 5% and 10% ratios, our method is obviously better than the sparse-to-dense baseline, and
Sparse-to
[Figure]
Dense Dense
[Figure]
Sparse
[Figure] [Figure]
Positive sample
Negative sample
[Figure]
Proposal
Negative sample
Proposal
Positive sample
Proposal
Positive sample
[Figure]
surpasses it by 0.8% and 1.8% mAP, respectively.
Nevertheless, when the labeling ratio is 1%, the performance gap shrinks.
This is easy to understand: When labeled data is extremely scarce, detection performance is poor.
Thus the predictions of the teacher are relatively inaccurate, resulting in noisy teacher guidance.
Especially for Rank Matching, the teacher model fails in modeling accurate candidate relations due to weak performance.
To extract accurate guidance from the very weak teacher model is left for future work.
Training efficiency.
Apart from the standard training schedule, we also attempt to halve the training iterations in Tab. 2(e).
Under the 10% labeling ratio, our DTG-SSOD yields 33.7 mAP with only 90k iterations, which is very close to the previous best method Soft Teacher with 180k iterations, 34. mAP.
Besides, under the Fully Labeled Data setting (100% labeling ratio), our method halves the training iterations of Soft Teacher and achieves even better performance, 44.7 vs. 44.5 mAP.
These results validate the high efficiency of DTG-SSOD.
Ablation study on hyper-parameters.
There are two hyper-parameters (i.e., T and β ) in Rank
Matching, where temperature T is used to control the flatness of score distribution over clustered candidates, and β performs as the loss weight of L RM .
In Tab 3(a), we ablate T with β set to 1.0.
As results indicate, the performance of RM is insensitive to T , and we adopt T = 1 . 0 for the following experiments.
A coarse search on β is shown in Tab 3(b), where β
= 3 . 0 achieves the best performance.
Experiments on one-stage detectors.
We also conduct experiments to verify the proposed denseto-dense paradigm can generalize to one-stage detectors well.
We take the GFL [30] as the default one-stage detector, because of its high efficiency and strong performance.
Comparisons between the sparse-to-dense with dense-to-dense paradigms are reported in Tab 3(c).
As results indicate, the dense-to-dense paradigm surpasses the counterpart by +1.3 mAP, demonstrating the proposed dense teacher guidance can work well on one-stage detectors.
4.5 Analyses
In this section, we provide some analyses on why dense teacher guidance is superior to sparse pseudo labels.
In the sparse-to-dense paradigm, some pseudo boxes are poorly localized, which will mislead the standard IoU-based label assignment and cause noisy training labels for the student.
Without the dependency on pseudo boxes, our method can effectively alleviate this problem.
Two examples are shown in Fig.
3, where the identical student proposals will obtain different training labels in the two paradigms.
Compared with the sparse-to-dense paradigm, our method obviously offers more precise training labels for the student.
In the sub-fig (1), the sparse-to-dense paradigm treats the poor proposal (in white) as a positive sample.
Because the proposal has a high IoU value (i.e., 0.6) with the coarse pseudo box (in red), therefore, it reaches the requirement of positive samples in the
IoU-based label assignment.
In contrast, our Inverse NMS Clustering takes teacher predictions into consideration and succeeds in suppressing this misleading positive.
Concretely, the teacher model predicts low confidence (i.e., 0.3) for this proposal, thus it will be a clear background sample in
NMS.
Another example is shown in the sub-fig (2).
Due to the poor localization of the pseudo box, a relatively precise proposal is wrongly assigned as a negative sample in the sparse-to-dense paradigm.
However, our INC can avoid this problem by taking the teacher’s NMS behavior into consideration.
In Fig.
4, we also provide an example to illustrate that the teacher is better at modeling candidate rank.
Between the two candidates shown in the figure, the teacher ranks the one with more precise localization first, while the student fails.
Therefore, distilling the capacity of modeling candidate rank from the teacher to student is beneficial, which is missed in sparse pseudo labels.
Conclusion
In this work, we analyze the traditional semi-supervised object detection methods, where sparse pseudo boxes with the corresponding category labels are adopted for dense supervision of the student, formulating the popular “sparse-to-dense” paradigm.
We point out that the sparse-to-dense paradigm could accumulate noise in the hand-crafted NMS and label assignment process, and further lack dense and direct teacher guidance.
To address the problem, we propose a novel “dense-to-dense” paradigm, which integrates Dense Teacher Guidance into the student training.
Specifically, we introduce Inverse
NMS Clustering and Rank Matching to instantiate the dense guidance: INC leads the student to perform the same candidate grouping in NMS as the teacher does; RM passes the teacher’s knowledge of rank distribution over clustered candidates to the student.
Through INC and RM, the student can receive sufficient, informative, and dense guidance from the teacher, which naturally leads to better performance.
On COCO benchmark, extensive experiments under various labeling ratios validate that our DTG-SSOD can achieve SOTA performance in both accuracy and efficiency.
Acknowledgments and Disclosure of Funding
This work is partially supported by National Natural Science Foundation of China (Grant No.
62172225), the Fundamental Research Funds for the Central Universities (No. 30920032201), and the Young Scientists Fund of the National Natural Science Foundation of China (Grant No.62206134).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See abstract.
(b) Did you describe the limitations of your work?
[Yes] See Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section 4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A]
The data used in our work is open source and can be used for academic research.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
The data does not contain personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable
A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]