T OWARDS N OISE - RESISTANT O BJECT D ETECTION WITH N OISY A NNOTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Training deep object detectors requires large amounts of human-annotated images with accurate object labels and bounding box coordinates, which are extremely expensive to acquire.
Noisy annotations are much more easily accessible, but they could be detrimental for learning.
We address the challenging problem of training object detectors with noisy annotations, where the noise contains a mixture of label noise and bounding box noise.
We propose a learning framework which jointly optimizes object labels, bounding box coordinates, and model parameters by performing alternating noise correction and model training.
To disentangle label noise and bounding box noise, we propose a two-step noise correction method.
The ﬁrst step performs class-agnostic bounding box correction, and the second step performs label correction and class-speciﬁc bounding box reﬁnement.
We conduct experiments on PASCAL VOC and MS-COCO dataset with both synthetic noise and machine-generated noise.
Our method achieves state-of-the-art performance by effectively cleaning both label noise and bounding box noise 1 .
I NTRODUCTION
The remarkable success of modern object detectors largely relies on large-scale datasets with extensive bounding box annotations.
However, it is extremely expensive and time-consuming to acquire high-quality human annotations.
For example, annotating each bounding box in ILSVRC requires 42s on Mechanical Turk (Su et al., 2012), whereas the recent OpenImagesV4 Kuznetsova et al.
(2018) reports 7.4 seconds with extreme clicking (Papadopoulos et al., 2017b).
On the other hand, there are ways to acquire annotations at lower costs, such as limiting the annotation time, reducing the number of annotators, or using machine-generated annotations.
However, these methods would yield annotations with both label noise ( i .
e .
wrong classes) and bounding box noise ( i .
e .
inaccurate locations), which could be detrimental for learning.
Learning with label noise has been an active area of research.
Some methods perform label correction using the predictions from the model and modify the loss accordingly (Reed et al., 2015; Tanaka et al., 2018).
Other methods treat samples with small loss as those with clean labels, and only allow clean samples to contribute to the loss (Jiang et al., 2018b; Han et al., 2018).
However, most of those methods focus on the image classiﬁcation task where the existence of an object is guaranteed.
Several recent works have studied object detection with noisy annotations.
Zhang et al.
(2019) focus on the weakly-supervised (WS) setting where only image-level labels are available, and ﬁnd reliable bounding box instances as those with low classiﬁcation loss.
Gao et al.
(2019) study a semisupervised (SS) setting where the training data contains a small amount of fully-labeled bounding boxes and a large amount of image-level labels, and propose to distill knowledge from a detector pretrained on clean annotations.
However, these methods require access to some clean annotations.
In this work, we address a more challenging and practical problem, where the annotation contains an unknown mixture of label noise and bounding box noise.
Furthermore, we do not assume access to any clean annotations .
The entanglement of label noise and bounding box noise increases the difﬁculty to perform noise correction.
A commonly used noise indicator, namely the classiﬁcation loss, is incapable to distinguish label noise from bounding box noise.
Furthermore, it is problematic to correct noise directly using the model predictions, because label correction requires accurate
[Figure]
bounding box coordinates to crop the object, whereas bounding box correction requires accurate class labels to produce the regression offset.
To overcome these difﬁculties, we propose a two-step noise correction procedure.
In the ﬁrst step, we perform class-agnostic bounding box correction (CA-BBC), which seeks to decouple bounding box noise from label noise, and optimize the noisy ground-truth (GT) bounding box regardless of its class label.
An illustration of CA-BBC is shown in Figure 1.
It is based on the following intuition: if a bounding box tightly covers an object, then two diverged classiﬁers would agree with each other and produce the same prediction.
Furthermore, both classiﬁers would have low scores for the background class, i .
e ., high objectness scores.
Therefore, we directly regress the noisy GT bounding box to minimize both classiﬁer discrepancy and background scores.
CA-BBC also has the option to reject a bounding box as false positive if the objectness score is too low.
In the second step, we leverage the model’s output for label noise correction and class-speciﬁc bounding box reﬁnement.
It has been shown that co-training two models can ﬁlter different types of noise and help each other learn (Blum & Mitchell, 1998; Han et al., 2018; Yu et al., 2019; Chadwick & Newman, 2019).
Therefore, we distil knowledge from the ensemble of dual detection heads for noise correction, by generating soft labels and bounding box offsets.
We show that soft labels with well-adjusted temperature lead to better performance even for a clean dataset.
To summarize, this paper proposes a noise-resistant learning framework to train object detectors with noisy annotations.
The proposed framework jointly optimizes object labels, bounding box coordinates, and model parameters by performing alternating noise correction and model training.
We conduct experiments on two benchmarks: PASCAL VOC and MS-COCO, which contain different levels of synthetic noise as well as machine-generated noise.
The proposed method outperforms previous methods by a large margin.
We also provide qualitative results to demonstrate the efﬁcacy of the two-step noise correction, and ablation studies to examine the effect of each component.
R ELATED W ORK 2. C ROWDSOURCING FOR O BJECT D ETECTION
Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) have enabled the collection of large-scale datasets.
Due to the formidable cost of human annotation, many efforts have been devoted to reduce the annotation cost.
However, even an efﬁcient protocol still report 42.4s to annotate one object in an image (Su et al., 2012).
Other methods have been proposed which trade off annotation quality for lower cost, by using click supervision (Papadopoulos et al., 2017a), human-inthe-loop labeling (Russakovsky et al., 2015; Papadopoulos et al., 2016; Konyushkova et al., 2018), or exploiting eye-tracking data (Papadopoulos et al., 2014).
These methods focus on reducing human effort, rather than combating the annotation noise as our method does.
2. L EARNING WITH L ABEL N OISE
Deep Neural Networks (DNNs) can easily overﬁt to noisy labels in the training data, leading to poor generalization performance (Zhang et al., 2017).
Many works have addressed learning with label noise.
Some approaches correct noise by relabeling the noisy samples (Vahdat, 2017; Veit et al., 2017; Lee et al., 2018), but they rely on a small set of clean samples for noise correction.
Iterative relabeling methods (Tanaka et al., 2018; Yi & Wu, 2019) have been proposed which produce hard or soft labels using the model predictions.
Other approaches ﬁlter noise by reweighting or selecting training samples (Jiang et al., 2018b; Ren et al., 2018; Chen et al., 2019b; Arazo et al., 2019; Li et al., 2020).
Since DNNs learn clean samples faster than noisy ones, samples with smaller classiﬁcation loss are usually considered to be clean (Arpit et al., 2017).
To avoid error accumulation during the noise correction process, co-teaching (Han et al., 2018) trains two networks simultaneously, where each network selects small-loss samples to train the other.
Co-teaching + (Yu et al., 2019) further keeps the two networks diverged by training on disagreement data.
2. W EAKLY - SUPERVISED AND S EMI - SUPERVISED O BJECT D ETECTION
Weakly-supervised object detection aims to learn object detectors with only image-level labels.
Most existing works formulate it as a multiple instance learning (MIL) task (Dietterich et al., 1997), where each label is assigned to a bag of object proposals.
A common pipeline is to iteratively alternate between mining object instances using a detector and training the detector using the mined instances (Deselaers et al., 2010; Cinbis et al., 2017).
To address the localization noise in the object proposals, Zhang et al.
(2019) propose an adaptive sampling method which ﬁnds reliable instances as those with high classiﬁcation scores, and use the reliable instances to impose a similarity loss on noisy images.
Different from weakly-supervised object detection which assumes that the correct object label is given, our method deals with label noise and bounding box noise at the same time.
Semi-supervised methods train object detectors using training data with bounding box annotations for some images and only image-level labels for other images (Hoffman et al., 2014; Tang et al., 2016; Uijlings et al., 2018; Gao et al., 2019).
Gao et al.
(2019) propose an iterative training-mining framework consisting of detector initialization, box mining, and detector retraining.
To address the annotation noise of the mined boxes, they use a detector pretrained on clean annotations for knowledge distillation.
Different from all semi-supervised learning methods, our method does not need access to any clean annotations.
M ETHOD 3. O VERVIEW
Given a training dataset with images X , noisy object labels Y , and noisy bounding boxes B , our method aims to train an object detector parameterized by ⇥ , by jointly optimizing
Y , B and ⇥ .
We ﬁrst warm-up ⇥ where we train the detector in a standard manner using the original noisy annotations.
After the warm-up, we perform alternating optimization on the annotations and the model.
Speciﬁcally, for each mini-batch of data X = { x i } , Y = { y i } , B = { b i } , we ﬁrst keep ⇥ ﬁxed and perform noise correction to update Y and B , then we used the corrected annotations to update ⇥ .
An overview of the algorithm is shown in Algorithm 1.
We use a popular two-stage object detector ( i .
e .
Faster-RCNN (Ren et al., 2015)), which consists of a backbone feature extractor parameterized by ✓ cnn , a Region Proposal Network (RPN) ✓ rpn , a classiﬁcation head ✓ c , and a bounding box (bbox) regression head ✓ b .
Note that ✓ c and ✓ b have shared layers.
Let detection head with parameters ✓ d denote the union of the classiﬁcation head and the bbox regression head.
During training, we simultaneously train two detection heads ✓ 1 d = { ✓ 1 c , ✓ 1 b } and ✓ 2 d = { ✓ 2 c , ✓ 2 b } , which are kept diverged from each other by different (random) parameter initializations and different (random) training instance ( i .
e . RoI) sampling.
Due to the entanglement of an unknown mixture of label noise and bbox noise, it is difﬁcult to correct both types of noise in a single step.
Therefore, we propose a two-step noise correction method.
In the ﬁrst step, we perform class-agnostic bounding box correction (CA-BBC), which disentangles bbox noise from label noise.
In the second step, we utilize the outputs from dual detection heads for label noise correction and class-speciﬁc bbox reﬁnement.
Figure 2 shows an illustration of our framework.
Next we delineate the details.
Feature Extractor
Noisy GT BBox !
Class-Agnostic BBoxCorrection !
∗
PrROIPooling
Noisy GT Label #
Average
Sharpen “Cat” “Dog” # ∗
Detection Head A
Detection Head B
Detection Head A
Detection Head B
RPN
Sampling A !
∗∗
Sampling B !
∗∗
PrROIPooling $ ∗ =
argmax + (# +∗ )
Two-StepNoise
Correction
Model Training .
/+ ∗ .
+ ∗ 1 234/ (# ∗ )352/ (!
∗∗ ,$ ∗ ) 1 2340 (# ∗ ) 1 3520 (!
∗∗ ,$ ∗ ) 1 789 (! ∗∗ ) !
∗∗
3. C LASS - AGNOSTIC B OUNDING B OX C ORRECTION
We ﬁrst correct bounding box noise by updating B !
B ⇤ regardless of the label noise in Y .
As illustrated in Figure 1, CA-BBC uses two diverged classiﬁcation heads to produce two sets of class predictions on the same image region, and updates the bounding box to minimize classiﬁer discrepancy and maximize region objectness .
The intuition is: if a bounding box tightly covers an object, then two classiﬁers would agree with each other and produce the same predictions.
Moreover, both predictions would have low scores on the background class.
Speciﬁcally, given an image x 2 X , the backbone ﬁrst extracts a convolutional feature map.
For each noisy GT bounding box b 2 B , we perform a RoI-Pooling operation on the feature map to extract a ﬁxed-sized feature  ( x, b ) .
Then we give the RoI feature to the two classiﬁcation heads to produce two sets of softmax predictions over C + 1 classes (including the background class), p 1 (  ( x, b ); ✓ 1 c ) and p 2 (  ( x, b ); ✓ 2 c ) .
For simplicity we denote them as p 1 and p 2 .
The discrepancy between the two predictions is deﬁned as their L2 distance:
[Equation]
Minimizing the classiﬁer discrepancy w.r.t the bounding box will push it to a region where the two classiﬁers agree on its class label.
To prevent the bounding box from simply moving to a background region, we also minimize the classiﬁers’ scores on the background class, p bg1 and p bg2 .
In other words, we want to maximize the objectness of the region covered by the bounding box.
Therefore, we aim to ﬁnd the optimal b ⇤ that minimizes the following objective function:
[Equation]
where  controls the balance of the two terms and is set to 0.1 in our experiments.
For faster speed, we estimate b ⇤ by performing a single step of gradient descent to update b :
[Equation]
where ↵ is the step size.
Since RoI-Pooling (Ren et al., 2015) or RoI-Align (He et al., 2017) performs discrete sampling on the feature map to generate  ( x, b ) , L ( b ) is not differentiable w.r.t b .
Therefore, we adopt the Precise
RoI-Pooling method (Jiang et al., 2018a), which avoids any quantization of coordinates and has a continuous gradient on b .
In order to handle false positive bboxes that do not cover any object, we add a reject option which removes b from the ground-truth if both classiﬁers give a low objectness score (high background score), p bg1 > 0 .
9 and p bg2 > 0 .
3. D UAL -H EAD D ISTILLATION FOR N OISE C ORRECTION
In the second step, we perform class-speciﬁc self-distillation for label noise correction and bbox reﬁnement.
We simultaneously train two diverged detection heads which can ﬁlter different types of noise, and distil knowledge from their ensemble to clean the annotation noise.
Using the ensemble of two heads helps alleviate the conﬁrmation bias problem ( i .
e .
a model conﬁrms its own mistakes) that commonly occurs in self-training.
Soft label correction.
Given the RoI feature  ( x, b ⇤ ) , the two classiﬁcation heads produce two sets of softmax predictions over object classes, p ⇤ 1 and p ⇤ 2 .
Inspired by the bootstrapping method (Reed et al., 2015), we use the classiﬁers’ predictions to update the noisy GT label.
Let y 2 { 0 , 1 } C represent the GT label as a one-hot vector over C classes, we create the soft label by ﬁrst averaging the classiﬁers’ predictions and the GT label:
[Equation]
Then we apply a sharpening function on the soft label to reduce the entropy of the label distribution.
The sharpening operation is deﬁned as:
[Equation]
where ¯ y c is the score for class c .
The temperature T controls the ‘softness’ of the label and is set to 0.4 in our experiments.
A lower temperature decreases the softness and has the implicit effect of entropy minimization, which encourages the model to produce high conﬁdence predictions and allows better decision boundary to be learned (Grandvalet & Bengio, 2005; Berthelot et al., 2019).
Class-speciﬁc bounding box reﬁnement.
The two bbox regression heads produce two sets of perclass bounding box regression offsets, t 1 and t 2 .
Let c ⇤ denote the class with the highest score in the soft label, i .
e .
c
⇤ = arg max c y ⇤ c , c = 1 , 2 , ..., C. We reﬁne the bounding box b ⇤ by merging the class-speciﬁc outputs from both bbox regression heads:
[Equation]
where t c ⇤ 1 and t c ⇤ 2 are the bounding box offsets for class c ⇤ , and ⇢ controls the magnitude of the reﬁnement.
[Table]
3. M ODEL T RAINING
Let Y ⇤ and B
⇤⇤ denote a mini-batch of soft labels and reﬁned bounding boxes, respectively.
We use them as the new GT to train the model.
Speciﬁcally, we update ⇥ =
{ ✓ cnn , ✓ rpn , ✓ 1 d , ✓ 2 d } to optimize the following losses: (1) the loss function of RPN deﬁned in (Ren et al., 2015), L rpn ( B ⇤⇤ ) ; (2) the classiﬁcation loss for the two detection heads, L 1cls ( Y ⇤ ) and L 2cls ( Y ⇤ ) , deﬁned as the crossentropy loss
P i  y
⇤ i log( p i ) ; (3) the localization loss for the two detection heads, L 1loc ( B ⇤⇤ , Y ⇤ ) and
L 2loc ( B ⇤⇤ , Y ⇤ ) , deﬁned as the smooth L1 loss (Girshick, 2015).
E XPERIMENTS 4. D ATASETS AND I MPLEMENTATION D ETAILS
Since most available datasets for object detection have been extensively veriﬁed by human annotators and contain little noise, we created noisy annotations using two popular benchmark datasets,
PASCAL VOC (Everingham et al., 2010) and MS-COCO (Lin et al., 2014), First, we generated synthetic noise to simulate human mistakes of different severity, by corrupting the training annotation with a mixture of label noise and bounding box noise.
For label noise, we follow previous works (Jiang et al., 2018b; Arazo et al., 2019) and generate symmetric label noise.
Speciﬁcally, we randomly choose N l % of the training samples and change each of their labels to another random label.
For bounding box noise, we perturb the coordinates of all bounding boxes by a number of pixels uniformly drawn from [  wN b % , + wN b %] ( w is bbox width) for horizontal coordinates or [  hN b % , + hN b %] ( h is bbox height) for vertical coordinates.
We experiment with multiple combinations of label noise ranging from 0% to 60% and bounding box noise ranging from 0% to 40% .
Under 40% bbox noise, the average IoU between a noisy bbox and its corresponding clean bbox is only 0.45 .
For VOC, we use the union set of trainval2007 and trainval2012 as training data, and test2007 as test data.
We report mean average precision (mAP@ . 5 ) as the evaluation metric.
For
MS-COCO, we use train2017 as training data, and report mAP@ .
5 and mAP@ [ . 5 , .
95] on val2017 .
We also mined large amounts of free training data with noisy annotations by using machinegenerated annotations on unlabeled images.
We ﬁrst train a Faster R-CNN detector on 10% of labeled data from COCO train2017 , which has a validation mAP@ .
5 of 40.5.
Then we use the trained detector to annotate unlabeled2017 , which contains 123k unlabeled images.
We use COCO unlabeled2017 with machine-generated annotations as our noisy training data.
We use the common Faster-RCNN (Ren et al., 2015) architecture with ResNet-50 (He et al., 2016) and
FPN (Lin et al., 2017) as the feature extractor.
We train the model using SGD with a learning rate of 0.02, a momentum of 0.9, and a weight decay of 1 e  4 .
The hyper-parameters are set as  = 0 .
T = 0 . 4 , ⇢ = 0 . 5 , and ↵ 2 { 0 , 100 , 200 } , which are determined by the validation performance on 10% of training data with clean annotations (only used for validation).
We implement our framework based on the mmdetection toolbox (Chen et al., 2019a).
In terms of computation time, our method increases training time by ⇠ 24% compared to vanilla training.
During inference, we only use the ﬁrst detection head unless otherwise speciﬁed, which does not increase inference time.
4. E VALUATION ON CA-BBC
First, we evaluate the effect of the proposed CA-BBC method by itself.
We train a detector following the proposed learning framework, except that we only perform the ﬁrst step of noise correction ( i .
e .
CA-BBC).
Table 1 shows the results on VOC with different mixtures of label noise and bounding box noise.
Compared to vanilla training without any noise correction (Ren et al., 2015; Chen et al., 2019a), performing CA-BBC can signiﬁcantly improve performance, especially
[Table]
for higher level of bbox noise.
The improvement is consistent despite the increase of label noise, which demonstrates the ability of CA-BBC to disentangle the two types of noise and effectively correct bbox noise.
We also demonstrate the effect of the proposed discrepancy minimization by removing D ( p 1 , p 2 ) from the loss in Eq. 2, and only maximize the objectness of the bbox region, which leads to lower performance.
Figure 3 show qualitative examples of CA-BBC.
The noisy GT bboxes are shown in red whereas the corrected bboxes are shown in green.
CA-BBC can update the bounding boxes to more accurately capture the objects of interest.
4. C OMPARISON WITH THE S TATE - OF - THE - ART
We evaluate our full learning framework with two-step noise correction and compare it with multiple existing methods for learning with noisy annotations.
We implement all methods using the same network architecture.
Since previous methods operate in different settings as ours, we adapt them for our problem to construct strong baselines as described in the following:
• Co-teaching (Han et al., 2018) simultaneously trains two models where each model acts as a teacher for the other by selecting its small-loss samples as clean data to train the other.
It has been employed by Chadwick & Newman (2019) for training object detectors with noisy data.
We adapt co-teaching into our dual-head network, where each detection head selects box samples with small classiﬁcation loss to train the other head.
Note that the RPN is trained on all boxes.
• SD-LocNet
(Zhang et al., 2019) proposes an adaptive sampling method that assigns a reliable weight to each box sample.
Higher weights are assigned to samples with higher classiﬁcation scores and lower prediction variance over consecutive training epochs.
[Table] [Table] • NOTE-RCNN (Gao et al., 2019) uses clean seed box annotations to train the bbox regression head.
It also pretrains a teacher detector on the clean annotations for knowledge distillation.
Because we do not have clean annotations, we follow previous works (Han et al., 2018; Arazo et al., 2019) and consider box samples with smaller classiﬁcation loss as clean ones.
We ﬁrst train a detector in a standard manner to mine clean samples.
Then we utilize the clean samples following NOTE-RCNN (Gao et al., 2019).
Table 2 shows the comparison results on VOC, where the training data contains different mixtures of label noise and bbox noise.
Our method signiﬁcantly outperforms all other methods across all noise settings.
For high levels of noise ( N b = 40% , N l 2 { 40% , 60% } ), our method achieves ⇠ 20% improvement in mAP compared to vanilla training, and > 10% improvement compared to the state-of-the-art NOTE-RCNN (Gao et al., 2019)
On clean training data with 0% annotation noise, our method can still improve upon vanilla training by +1 .
9% , mostly due to the proposed soft labels.
Compared to the one-hot GT labels, soft labels contain more information about an image region in cases where multiple objects co-exists in the same bounding box.
Moreover, using soft labels has the effect of label smoothing, which could prevent overﬁtting and improve a model’s generalization performance (M¨uller et al., 2019).
Table 3 shows the results on COCO.
Our method outperforms all baselines by a large margin.
Under 40% of label and bbox noise, vanilla training results in a catastrophic degradation of  24 .
8% in mAP@ .
5 compared to training on clean data (oracle), whereas our method can reduce the performance drop to  7 . 1% .
The proposed method also achieves improvement under machine-generated noise , which validates its practical usage
to
train detectors by utilizing free unlabeled data. 4. A BLATION S TUDY
In Table 4, we add or drop different components in our framework to examine their effects.
Below we explain the results in detail.
More ablation study and qualitative results are shown in the appendix.
• In the ﬁrst row, we perform noise correction with only one detection head, by using its output to create soft labels and regress bounding boxes.
Compared with the proposed dual-head network where knowledge is distilled from the ensemble, using a single head suffers from conﬁrmation bias where the model’s prediction error would accumulate and thus degrade the performance.
• In the second row, we remove CA-BBC from the proposed framework and only perform the dual-head noise correction (Section 3.3).
Compared with the results using the proposed two-step noise correction (the third row), the performance decreases considerably for higher level (40%) of bounding box noise, which validates the importance of the proposed CA-BBC.
• The third row shows the results using the proposed method.
• In the last row, we use the ensemble of both detection heads during inference by averaging their outputs, which leads to further performance improvement.
C ONCLUSION
To conclude this paper addresses a new challenging research problem, which aims to train object detectors from noisy annotations that contain entangled label noise and bounding box noise.
We propose a noise-resistant learning framework which jointly optimizes noisy annotations and model parameters.
A two-step noise correction method is proposed, where the ﬁrst step performs classagnostic bbox correction to disentangle bbox noise and label noise and the second step performs dual-head noise correction by self-distillation.
Experiments on both synthetic noise and machinegenerated noise validate the efﬁcacy of the proposed framework.
We believe that our work is one step forward towards alleviating human from the tedious annotation effort