DTG-SSOD: Dense Teacher Guidance for Semi-Supervised Object Detection
Abstract
The Mean-Teacher (MT) scheme is widely adopted in semi-supervised objectdetection (SSOD).
In MT, the sparse pseudo labels, offered by the final predictionsof the teacher (e.g., after Non Maximum Suppression (NMS)
post-processing), areadopted for the dense supervision for the student via hand-crafted label assignment.
However, the “sparse-to-dense” paradigm complicates the pipeline of SSOD, andsimultaneously neglects the powerful direct, dense teacher supervision.
In thispaper, we attempt to directly leverage the dense guidance of teacher to supervisestudent training, i.e., the “dense-to-dense” paradigm.
Specifically, we proposethe Inverse NMS Clustering (INC) and Rank Matching (RM) to instantiate thedense supervision, without the widely used, conventional sparse pseudo labels.
INC leads the student to group candidate boxes into clusters in NMS as the teacherdoes, which is implemented by learning grouping information revealed in NMSprocedure of the teacher.
After obtaining the same grouping scheme as the teachervia INC, the student further imitates the rank distribution of the teacher overclustered candidates through Rank Matching.
With the proposed INC and RM, weintegrate Dense Teacher Guidance into Semi-Supervised Object Detection (termed“DTG-SSOD”), successfully abandoning sparse pseudo labels and enabling moreinformative learning on unlabeled data.
On COCO benchmark, our DTG-SSODachieves state-of-the-art performance under various labelling ratios.
For example,under 10% labelling ratio, DTG-SSOD improves the supervised baseline from 26.9to 35.9 mAP, outperforming the previous best method Soft Teacher by 1.9 points.
Introduction
Thanks to plenty of annotated data, deep learning based computer vision has achieved great successin the past few years [1–3].
However, labeling accurate annotations is usually labour-consumingand expensive, especially for dense prediction tasks [4–7], such as object detection, where multiplebounding boxes with their corresponding category labels need to be annotated extensively for eachimage.
Some works attempt to leverage easily accessible unlabeled data to promote object detectionin a semi-supervised manner [8–11].
The model is first initialized by learning on the labeled data.
Then it is adopted for generating pseudo labels (i.e., pseudo bounding boxes and category labels) onthe unlabeled data for training.
Recently, advanced Semi-Supervised Object Detection (SSOD) methods follow the Mean-Teacherparadigm [13], where a stronger teacher model is built from the student model via Exponential Moving
Average (EMA) to produce supervision signals over unlabeled data.
Nevertheless, all of the previousmethods [8–10, 12, 11, 14] adopt sparse pseudo labels after the post-processing (e.g., NMS and scorefiltering) as the supervision, passed from the teacher to student.
To guarantee the precision of pseudolabels, a high score threshold (e.g., 0.9) is usually selected to filter coarse detection boxes and only afew pseudo boxes could survive.
Through the hand-crafted label assignment strategy [15–17], thedetector transfers sparse pseudo labels into dense supervision for student training, which is termed the“sparse-to-dense” paradigm.
Such sparse pseudo labels neglect some important properties of object
[Figure]
In Fig.
1, we illustrate the “sparse-to-dense” paradigm, where the sparse pseudo labels originatedfrom the dense predictions of teacher are adopted for the dense supervision of student.
This processinvolves many handcrafted components (e.g., NMS, score thresholding, and label assignment), whichinevitably introduce accumulated noise/bias to supervisory signals for student.
Therefore, it is moreintuitive to adopt the “dense-to-dense” paradigm, i.e., directly employing dense predictions of teacherto guide the training of student, whilst abandoning the intermediate operations.
In this work, to implement the “dense-to-dense” paradigm, we propose to enforce the student tomimic the behavior of the teacher model.
Specifically, the behaviour is motivated from the procedurehow dense samples perform NMS, an essential post-processing scheme for popular detectors.
Toregularize the consistency on NMS behaviour between the teacher and student, we introduce Inverse
NMS Clustering (INC) and Rank Matching (RM).
INC leads the student to group the candidateboxes into clusters in NMS as the teacher does.
In the process of NMS, candidates are grouped intomultiples clusters.
Within each cluster, teacher candidates are supposed to predict the identical object,which will assign training labels to the corresponding student samples in turn.
Different from theprevious sparse-to-dense paradigm, where sparse pseudo labels are converted to student traininglabels via the hand-crafted label assignment, our dense-to-dense paradigm assigns dense traininglabels based on grouping information by reversing the NMS process of the teacher.
Based on INCwhere the student obtains the same grouping scheme of NMS with the teacher, we further introduce
Rank Matching to align the score rank over the clustered candidates between teacher and student.
NMS suppresses the duplicated detection and reserves only one box for each cluster through the rankof clustered candidates.
Therefore, the rank of the samples within each cluster could contain richrelationship learned by the teacher and it can behave as informative dense supervision which guidesthe student to reserve as the same candidates as the teacher during NMS.
Through the proposed INC and RM, the teacher model provides not only fine-grained and appropriatetraining labels, but also informative rank knowledge, realizing Dense Teacher Guidance for SSOD(DTG-SSOD).
To validate the effectiveness of our method, we conduct extensive experiments on
COCO benchmark under Partially Labeled Data and Fully Labeled Data settings.
Under the Partially
Labeled Data, where only 1%,2%,5%, and 10% images are labeled, the proposed DTG-SSODachieves the state-of-the-art performance, for example, under 5% and 10% labelling ratios, DTGSSOD surpasses the previous best method Soft Teacher by +1.2 and +1.9 mAP, respectively.
Underthe Fully Labeled Data, where the unlabeled2017 set is used as unlabeled data, the proposed DTGSSOD not only achieves the best performance (45.7 mAP), but also improves the learning efficiency,e.g., DTG-SSOD halves the training iterations of Soft Teacher but achieves even better performance.
Related Work
Semi-Supervised Image Classification (SSIC)
The pseudo labeling [18–20] is a popular pipeline,where unlabeled data is labeled by the model itself, and then combined with labeled data to act as training examples in the cycle self-training manner.
In the pseudo-labeling, hard pseudo labels areusually used with a confidence-based thresholding where only images with high confidence canbe retained.
Recently, some improvements [21–23] on pseudo-labeling are proposed, for example,
Curriculum Labeling [21] proposes that restarting model parameters before each training cyclecan promote self-training; Noisy Student Training [22] adopts an equal-or-larger model with noiseinjected to learn the pseudo labels, which are generated in the last self-training cycle.
Apart fromhard pseudo labels, Noisy Student Training also tried soft pseudo labels, observing that for in-domainunlabeled images, both hard and soft pseudo labels perform well, while for out-of-domain unlabeleddata, soft pseudo labels can work better.
On the other hand, based on the insight that the modelshould be robust to input perturbations, UDA [24] designs advanced data augmentations and enforcesthe model to predict the similar class distributions for the differently augmented inputs.
Here, softlabels (i.e., distributions) are superior in measuring the distance between different model predictionsover the hard ones.
As another strong method, FixMatch [25] introduces a separate weak and strongaugmentation and converts model predictions into hard labels, since the labels with low entropy arebelieved to be beneficial in their pipeline.
To conclude, previous SSIC works investigate various formsof supervisory signals (including hard and soft pseudo labels), and observe the form of supervisionsignals matters in semi-supervised learning, which inspires us to investigate the optimal supervisionform for semi-supervised object detection.
Semi-Supervised Object Detection (SSOD) CSD [26] is an early attempt to construct consistencylearning for horizontally flipped image pairs, where only the flip transformation can be used as theaugmentation, which greatly limits its performance.
STAC [27] borrows the separate weak andstrong augmentation from FixMatch [25] and applies it to SSOD, obtaining promising performance.
After that, [9, 10, 12, 8] abandon the conventional multi-phase training and update the teacher modelfrom the student via EMA after each training iteration.
Apart from the framework, previous workspromote SSOD from two main perspectives: designing advanced data augmentation and improvingthe precision of pseudo labels.
For the former one, Instant-Teaching [9] introduces more complexaugmentations for unlabeled data, including Mixup and Mosaic; MUM [14] designs a Mix/UnMixdata augmentation, which enforces the student to reconstruct unmixed feature for the mixed inputimages.
As for the latter, through ensembling the teacher model on input images and predictionhead, respectively, Humble Teacher [10] and Instant-Teaching [9] reduce confirmation bias of thepseudo labels; Soft Teacher [8] selects reliable pseudo boxes by averaging regression results frommultiple jittered boxes; Rethinking Pse [11] converts the box regression to a classification task,making localization quality easy to estimate.
Different from these methods, which all adopt sparsepseudo boxes with category labels as the supervisory signal, our method investigates the new formof supervision for the first time and proposes to replace sparse pseudo labels with more informativedense supervision.
Method
The mechanism of weak-strong augmentation pairs shows promising performance in SSOD.
Basedon it, we first build an end-to-end pseudo-labeling baseline, described in Sec. 3.1, through whichwe explain the task formulation of SSOD and the conventional sparse-to-dense paradigm.
Then in
Sec. 3.2, we introduce the proposed Inverse NMS Clustering (INC) and Rank Matching (RM), i.e.,the key two components of the proposed DTG-SSOD.
3.1 Sparse-to-dense Paradigm
Task Formulation.
The framework of SSOD is illustrated in Fig. 2(a).
The Mean-Teacher schemeis a common practice for previous works [8, 9, 12, 10] and realizes the end-to-end training, wherethe teacher is built from the student via EMA after each training iteration.
The teacher takes weaklyaugmented (e.g., flip and resize) images as input to generate pseudo labels, while the student isapplied with strong augmentations (e.g., cutout, geometry transformation) for training.
Strong andappropriate data augmentations play an important role, which not only increase the difficulty ofstudent tasks and alleviate the over-confidence issue [24], but also enable the student to be invariantto various input perturbations, leading to robust representation learning [22, 25].
In this work, weborrow the data augmentations from Soft Teacher [8], and more details can be found in Appendix.
Each training batch consists of both labeled data { x li , y li } N l i =1 and unlabeled data { x ui } N u i =1 , which aremixed according to the ratio r = N l N u , where N l and N u are the number of labeled and unlabeled
[Figure]
Dense Predictions
RPN 1.One-to-one correspondence
NMS procedure of the teacher
Candidate Grouping Ranking & Suppressing Reserved Boxes clusterclustercluster① ② ③ ④ ①
Proposals
Assign labels ① ② ③ ④ ① ② ③ ④
Teacher cluster 1 Teacher cluster
Dense Student Predictions 2.
Inverse NMS Clustering 3.
Rank Matching
Rank Distribution
[Figure]
① ③ ④ ②
The corresponding student predictions ①
The reserved
[Figure]
① [Figure] images in is formulated as
[Equation]
where α controls the contribution of losses on labeled data L l and unlabeled data L u .
Sparse-to-dense Baseline.
All previous SSOD approaches [8, 9, 12, 10] are based on the sparseto-dense mechanism, where sparse pseudo boxes with category labels are generated to behave asground-truths for the student training.
It comes with confidence-based thresholding where onlypseudo labels with high confidence (e.g., larger than 0.9) are retained.
This makes foregroundsupervision on unlabeled data much sparser than that on labeled data, therefore, the class imbalanceproblem is amplified in SSOD and impedes the detector training severely.
To alleviate this issue,we borrow some strengths of previous works: Soft Teacher [8] sets the mixing ratio r to 1/4 tosample more unlabeled data in each training batch, which makes the number of foreground sampleson unlabeled data approach that on labeled data; Unbiased Teacher [12] replaces the cross-entropyloss with the Focal loss [15], reducing the gradient contribution from easy examples.
Both of thesetwo improvements, i.e., the appropriate mixing ratio r (1/4) and Focal loss, are adopted in thesparse-to-dense baseline and our dense-to-dense DTG method.
Because the teacher only providessparse pseudo labels, which are further converted into the dense supervision for the student training,these methods are termed “sparse-to-dense” paradigm.
In theory, our SSOD method is independent ofthe detection framework and can be applicable to both one-stage and two-stage detectors.
For a faircomparison with previous works, we use the Faster RCNN [16] as the default detection framework.
3.2 Our method: Dense-to-dense paradigm
Pseudo labels in SSOD contain both bounding boxes and category labels.
Considering that classification scores are not strongly correlated with localization quality [28, 3, 29, 30], although confidencebased thresholding can filter out many false positives, the localization quality of pseudo boxes is notguaranteed.
Poorly-localized pseudo boxes will mislead the label assignment and confuse the decisionboundary.
Therefore, previous methods [11, 8] design two separate sets of indicators to measurethe confidence of categories and box locations, which complicates the SSOD pipeline.
Moreover,the sparse-to-dense paradigm neglects some important properties of object detection and lacks ofpowerful direct, dense teacher guidance.
Motivated by these, we propose to directly employ teacher’sdense predictions to supervise the student’s training, termed “dense-to-dense” paradigm, abandoningthe explicit sparse pseudo boxes.
Dense predictions widely exist in both RPN and RCNN stages.
Specifically, in the RPN stage,multiple anchors are usually tiled on each feature map grid [16, 15]; while in the RCNN stage, taking
2,000 proposals as training samples is a common practice [16].
Compared against the sparse pseudolabels, dense predictions of the teacher contain rich knowledge, and we attempt to pass it to thestudent by enforcing the student to mimic the dense behaviour of the teacher.
Specifically, we analyzehow teacher’s dense samples perform to eliminate redundancy in the process of Non Maximum
Suppression (NMS).
Based on the analysis, we design Inverse NMS Clustering and Rank Matchingto regularize consistency on the NMS procedure between the teacher and student, from which thestudent can obtain precise training labels and informative supervision signals.
However, implementing dense mimic is not trivial.
Due to the separate weak-strong augmentations,one-to-one correspondence between the samples of the teacher and student can not be easily identified.
We design a simple yet effective matching strategy to address this.
For the R-CNN stage, we directlycopy the teacher proposals to the student RCNN, realizing one-to-one correspondence.
For the RPNstage, since the feature map sizes and pre-set anchors differ between teacher and student, we firsttransfer the teacher’s anchors to the space of student through box transformation.
Then we assignthe teacher’s anchors to the corresponding FPN level based on the anchor box scale.
The index oftarget FPN level m i can be calculated by: m i = log 2 area i S , where area i refers to the area of theteacher anchor box t i , and S is a hyper-parameter and set to 32 .
Once the target FPN level m i isobtained, we calculate the Intersection-over-Unions (IoUs) between the teacher anchor t i and allstudent anchors { s j } j ∈ m i belonging to the FPN level m i , and set the student anchor with the largest
IoU as the corresponding one: s i = arg max IoU ( t i , s j ) , j ∈
m i .
Through the proposed matchingstrategy, one-to-one correspondence between training samples of the teacher and student is identified,making dense behaviour imitation possible.
Inverse NMS Clustering .
We first revisit the mechanism of NMS by taking RCNN stage as theexample.
Before NMS, predicted bounding box offsets are applied on proposals to derive regressedboxes.
Then a pre-defined score threshold τ filters out clear background detection boxes.
Forthe remaining detection boxes (or candidates), NMS conducts clustering on them based on theircategories and locations.
As a result, candidates are grouped into multiple clusters and in each cluster,candidates are supposed to predict the same object.
At last, only one candidate with the highestconfidence can be retained and others will be suppressed.
The process is illustrated in Fig. 2(b).
To regularize consistency on the NMS behaviour of the teacher and student, we enforce the studentto perform the same candidate grouping as the teacher.
In the NMS procedure of the teacher,candidates will be grouped into multiple clusters, and the cluster indexed by j can be defined as
C tj = { b ti , c ti } N j
i
=1 , where b and c refer to the bounding box and category, respectively, and N j isthe number of candidates belonging to C j .
In each cluster of the teacher (taking C tj = { b ti , c ti } N j i =
1as an example), the candidates own the same category c (i.e., c t 1 = · · · = c ti = c ), and highlyoverlapped boxes { b i } N j i =1 , thus they are supposed to predict the identical object.
Here, we takethe finally reserved box { b tk , c } to represent the detection target of C j .
Then the target { b tk , c } willinversely assign training labels to the corresponding student samples { b si , c si } N j i =1 .
To be specific, thecategory label c is employed to supervise the classification predictions of the student: { c si } N j i =1 , andthe classification loss can be formulated as:
[Equation]
where f cls refers to the Focal loss [15].
Besides, for those teacher samples, whose scores are lowerthan the NMS threshold τ , they are clear background ones and excluded from the NMS, thus weassign background labels to their corresponding student samples.
For the regression task, we directlytake the reserved box b tk as the regression target for { b si } N j i =1 .
We describe the regression loss as:
[Equation]
where b t ′ k is obtained by transforming the b t k to the student space; f reg refers to regression loss, e.g.,
Smooth L1 Loss [16].
There is a clear difference between our method and the traditional sparse-todense paradigm.
Our method calculates classification and regression labels according to the groupinginformation revealed in the NMS procedure of the teacher, whilst the sparse-to-dense paradigmemploys some hand-crafted label assignment strategies [17, 15, 16] to convert sparse pseudo labelsinto student’s dense training labels, which loses rich teacher guidance.
Rank Matching.
Through the Inverse NMS Clustering, the student can obtain the same groupingscheme as the teacher.
Based on it, we further introduce Rank Matching to align the score rankover clustered candidates between the teacher and student.
In each NMS cluster, the candidateboxes are ranked according to the classification scores, then only one candidate can be retainedregarding the candidate rank.
Therefore, the rank of candidates could contain rich relationshipinformation.
In object detection, the candidate rank is validated to be a key element and can affectdetection performance much [28, 3, 29, 31].
For example, GFL [28] proposes that taking IoUsbetween predicted boxes and ground-truths as classification targets contributes to a more precise rank.
Different from these methods, our Rank Matching models the score rank over clustered candidates toalign the NMS behaviour of the teacher and student.
Exactly, we model the score distribution of theteacher and student over the cluster C j = { p i } N j i =1 as:
[Equation]
where p i indicates the logits from the target category of the i th candidate, and T is the temperaturecoefficient.
The Kullback-Leibler (KL) divergence is adopted to optimize the distance between D ( p ti )and D ( p si ) :
[Equation]
Then L RM is averaged over all NMS clusters.
Rank Matching not only provides relationshipinformation but also enables the student to reserve the identical candidates in each NMS cluster asthe teacher.
The overall loss function on unlabeled data is defined as:
[Equation]
where β controls contribution of rank matching loss.
Experiments
4.1 Dataset and Evaluation Metric.
We benchmark our proposed method on the challenging dataset, MS COCO [32].
Following theconvention [27, 8, 12, 9], two training settings are provided, namely Partially Labeled Data and
Fully Labeled Data settings, which validate the method with limited and abundant labeled data,respectively.
The val2017 set with 5k images is used as the validation set.
We use AP 50:97 (denotedas mAP) as the evaluation metric.
We describe two training settings as follows:
Partially Labeled Data.
The train2017 set, consisting of 118k labeled images, is used as thetraining dataset, from which, we randomly sample 1%,2%,5%, and 10% images as labeled data,and set the remaining unsampled images as unlabeled data.
Following the practice of previousmethods [27, 8, 12], for each labelling ratio, 5 different folds are provided and the final result is theaverage of these 5 folds.
Fully Labeled Data.
Here, the train2017 set is employed as the labeled data, and the additionalunlabeled2017 set, consisting of 123k unlabeled images, is used as unlabeled data.
This setting is tovalidate SSOD methods can still promote the detector even when labeled data is abundant.
4.2 Implementation Details
Faster RCNN [16] with FPN [35] performs as default detection framework in this paper, and ResNet50 [36] is adopted as the backbone.
Moreover, we use SGD as the optimizer.
At the beginning of
NMS, a score threshold τ is leveraged to filter clear background samples.
In this work, we set τ to arelatively high value to exclude noisy teacher samples, whose behaviour is ambiguous and hardlyprovides helpful guidance.
Exactly, we set τ to 0.9 in NMS of the RPN stage, and 0.45 in NMS ofthe R-CNN stage, empirically.
To stabilize the training, we also warm up the learning on unlabeleddata for the first 4k iterations [12].
The training schedules for two training settings are a bit different:
Partially Labeled Data.
The model is trained for 180k iterations on 8 V100 GPUs with an initiallearning rate of 0.01, which is then divided by 10 at 120k iteration and again at 160k iteration.
[Table]
Mini-batch per GPU is 5, with 1 labeled image and 4 unlabeled images.
The loss weight of unlabeledimages α is set to 4.0.
Fully Labeled Data.
The model is trained on 8 GPUs with mini-batch as 8 per GPU, where 4 labeledimages are combined with 4 unlabeled images.
We adopt 720k iterations for training, and the initiallearning rate is set to 0.01, which is divided by 10 at 480k and 680k iterations.
We adopt α = 2 . 0 .
4.3 Main Results
We compare the proposed DTG-SSOD to the state-of-the-art methods on MS COCO val2017 set in
Tab. 1.
We first evaluate our method using the Partially Labeled Data setting, and experimentalresults demonstrate our method achieves the state-of-the-art performance under various labellingratios.
Specifically, on 5% and 10% labelling ratios, DTG-SSOD improves the supervised baselineby +10.73 and +9.02 mAP, reaching 31.90 and 35.92 mAP, and it also surpasses the previous bestmethod Soft Teacher [8] by +1.16 and +1.92 mAP, respectively.
When the labelling ratio is 2%,
our
DTG-SSOD yields 26.84 mAP, which is even comparable to the 26.90 mAP supervised baseline using10% labeled data.
When the labeled data is extremely scarce, i.e., 1% labelling ratio, our DTG-SSODachieves a comparable performance to the previous best method MUM [14], 21.27 vs. 21.88 mAP.
MUM [14] designs a strong data augmentation to expand data space, which mixes original image tilesto construct new images.
It is worth noting that when the number of labeled data increases (2%,5%,and 10% labelling ratios), our DTD-SSOD outperforms MUM by at least +2 mAP, moreover, ourmethod is orthogonal to these advanced data augmentations [14, 9] and can benefit from them.
As Tab. 1 shows, under the Fully Labeled Data setting, our DTG-SSOD surpasses previous methodsby a large margin, at least 1.2 mAP.
Following the practice of [8, 11], we also apply weak augmentations to labeled data and obtain a strong supervised baseline of 40.9 mAP.
Even based on the suchstrong baseline, DTG-SSOD still attains the biggest improvements of +4.8 mAP, reaching 45.7 mAP,which validates the effectiveness of our method when the amount of labeled data is relatively large.
4.4 Ablation Experiments
In this section, we conduct extensive ablation studies to validate the key designs.
Unless specified, allexperiments adopt a single data fold of the 10% labelling ratio as the training dataset.
Effects of Dense Teacher Guidance.
Instead of conventional pseudo boxes with category labelsprovided, our method offers dense supervisory signals.
We first validate that Dense Teacher Guidance(DTG) is a feasible learning paradigm for SSOD in Tab. 2(a), where we apply DTG to RPN and
R-CNN stages step by step.
On unlabeled data, when only RPN is optimized with the DTG and
R-CNN is excluded from training, it can bring 1.7 mAP gains to the supervised baseline.
Furthermore,we apply DTG to both RPN and RCNN, and it yields 36.3 mAP, outperforming the baseline by +9.5mAP.
Significant improvements against the supervised baseline indicate dense teacher guidance can
[Table] [Table]
[Table] [Table]
[Table] [Table]
behave as effective supervision signals for learning on unlabeled data.
DTG consists of two keycomponents, i.e., Inverse NMS Clustering (INC) and Rank Matching (RM), then we delve into thesetwo parts with results reported in Tab. 2(b).
Through imitating the candidate grouping of the teacherin NMS, INC provides appropriate training labels for the student.
With the derived classificationlabel, the model enjoys 4.4 mAP gains.
When regression labels are also applied, another 2.2 mAPimprovement is found, reaching 35.1 mAP.
Based on INC, RM is introduced to provide teacher’sscore rank over clustered candidates for the student, which enables the student to reserve the samecandidate in each NMS cluster as the teacher.
As results show, RM can bring an obvious improvement,+1.2 mAP, demonstrating comprehensive alignment on NMS behaviour is beneficial.
Besides, thegains also indicate the candidate rank carries helpful relation information learned by the teacher.
Comparison of sparse-to-dense and dense-to-dense paradigms.
In this work, we replace thetraditional sparse-to-dense (spa-den for short) paradigm with new dense-to-dense (den-den) paradigm.
Detailed comparisons are conducted in Tab.
2(c-d).
We first build a sparse-to-dense baseline, thenincrementally substitute our den-den paradigm into each network stage (i.e., RPN, RCNN) in Tab. 2(c).
Thanks to the appropriate mixing ratio r = 1 / 4 between labeled and unlabeled images and Focal
Loss, our sparse-to-dense baseline achieves comparable performance (34.5 mAP) to the previousmethods.
When we substitute the den-den paradigm into R-CNN, it yields 35.5 mAP, which is 1.0points higher than the spa-den R-CNN.
After that, we further apply den-den to RPN, which attains+0.8 mAP gains.
These experiments validate the proposed dense-to-dense paradigm is superior to thetraditional sparse-to-dense paradigm in both RPN and RCNN stages.
On the other hand, we also compare two paradigms under various labelling ratios in Tab. 2(d).
Under 5% and 10% ratios, our method is obviously better than the sparse-to-dense baseline, andsurpasses it by 0.8% and 1.8% mAP, respectively.
Nevertheless, when the labelling ratio is 1%,the performance gap shrinks.
This is easy to understand: When labeled data is extremely scarce,detection performance is poor.
Thus the predictions of the teacher are relatively inaccurate, resultingin noisy teacher guidance.
Especially for Rank Matching, the teacher model fails in modelingaccurate candidate relations due to weak performance.
To extract accurate guidance from the veryweak teacher model is left for future work.
Training efficiency.
Apart from the standard training schedule, we also attempt to halve the trainingiterations in Tab. 2(e).
Under the 10% labelling ratio, our DTG-SSOD yields 33.7 mAP with only 90kiterations, which is very close to the previous best method Soft Teacher with 180k iterations, 34.0mAP.
Besides, under the Fully Labeled Data setting (100% labelling ratio), our method halves the
[Figure] [Figure]
training iterations of Soft Teacher and achieves even better performance, 44.7 vs. 44.5 mAP.
Theseresults validate the high efficiency of DTG-SSOD.
Ablation study on hyper-parameters.
In Tab. 3(a-b), we conduct studies on the hyper-parameters Tand β .
When we ablate T , β is set to 1.0.
To conclude, we adopt T = 1 . 0 , β
= 3 in all experiments.
Ablation study on regression labels of INC.
In Inverse NMS Clustering, regression labels havetwo options: (1) The finally reserved teacher box in each cluster, as described in Sec. 3.2.
(2) Allteacher’s predicted boxes (denoted as “all boxes”).
As shown in Tab. 3(c), taking the reserved box asthe target achieves superior performance than “all bboxes”, 35.1 vs. 34.3 mAP.
4.5 Analyses
In this section, we provide some analyses on why dense teacher guidance is superior to sparse pseudolabels.
In the sparse-to-dense paradigm, some pseudo boxes are poorly localized, which will misleadthe standard IoU-based label assignment and cause noisy training labels for the student. Withoutthe dependency on pseudo boxes, our method can effectively alleviate this problem.
Two examplesare shown in Fig.
3, where the identical student proposals will obtain different training labels in thetwo paradigms. Compared with the sparse-to-dense paradigm, our method obviously offers moreprecise training labels for the student.
In the sub-fig (1), the sparse-to-dense paradigm treats thepoor proposal (in white) as a positive sample. Because the proposal has a high IoU value (i.e., 0.6)with the coarse pseudo box (in red), therefore, it reaches the requirement of positive samples in the IoU-based label assignment.
In contrast, our Inverse NMS Clustering takes teacher predictions intoconsideration and succeeds in suppressing this misleading positive.
Concretely, the teacher modelpredicts low confidence (i.e., 0.3) for this proposal, thus it will be a clear background sample in NMS.
Another example is shown in the sub-fig (2).
Due to the poor localization of the pseudo box, arelatively precise proposal is wrongly assigned as a negative sample in the sparse-to-dense paradigm.
However, our INC can avoid this problem by taking the teacher’s NMS behavior into consideration.
In Fig.
4, we also provide an example to illustrate that the teacher is better at modeling candidaterank.
Between the two candidates shown in the figure, the teacher ranks the one with more preciselocalization first, while the student fails.
Therefore, distilling the capacity of modeling candidate rankfrom the teacher to student is beneficial, which is missed in sparse pseudo labels.
Conclusion
In this work, we analyze the traditional semi-supervised object detection methods, where sparsepseudo boxes with the corresponding category labels are adopted for dense supervision of the student,formulating the popular “sparse-to-dense” paradigm.
We point out that the sparse-to-dense paradigmcould accumulate noise in the hand-crafted NMS and label assignment process, and further lack denseand direct teacher guidance.
To address the problem, we propose a novel “dense-to-dense” paradigm,which integrates Dense Teacher Guidance into the student training.
Specifically, we introduce Inverse
NMS Clustering and Rank Matching to instantiate the dense guidance: INC leads the student toperform the same candidate grouping in NMS as the teacher does; RM passes the teacher’s knowledgeof rank distribution over clustered candidates to the student.
Through INC and RM, the student canreceive sufficient, informative, and dense guidance from the teacher, which naturally leads to betterperformance.
On COCO benchmark, extensive experiments under various labelling ratios validatethat our DTG-SSOD can achieve SOTA performance in both accuracy and efficiency.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See abstract.
(b) Did you describe the limitations of your work?
[Yes] See Section 4.
(c) Did you discuss any potential negative societal impacts of your work?
[No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See Section4.2(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/