Towards Improving Calibration in Object Detection Under Domain Shift
Abstract
The increasing deployment of deep neural networks (DNNs) in real-world, including safety-critical applications, demands that in addition to displaying high accuracy they should also be well-calibrated.
We have witnessed some techniques addressing DNN calibration, however, they are only limited to visual classification applications and in-domain predictions.
Unfortunately, very little to no attention is paid towards addressing calibration of DNN-based visual object detectors, that occupy similar space and importance in many decision making systems as their visual classification counterparts.
In this work, we study the calibration of DNNbased object detection models, particularly under domain shift.
To this end, we first propose a new, plug-and-play, train-time calibration loss for object detection (coined as TCD).
It can be used with various application-specific loss functions as an auxiliary loss function to improve detection calibration.
Second, we devise a new implicit technique for improving calibration in self-training based domain adaptive detectors, featuring a new uncertainty quantification mechanism for object detection.
We demonstrate TCD is capable of enhancing calibration with notable margins (1) across different DNN-based object detection paradigms both in in-domain and out-of-domain predictions, and (2) in different domain-adaptive detectors across challenging adaptation scenarios.
Finally, we empirically show that our implicit calibration technique can be used in tandem with TCD during adaptation to further boost calibration in diverse domain shift scenarios.
Introduction
Owing to their high predictive power, deep neural networks (DNNs) are progressively becoming part of different safety-critical applications, including autonomous driving [5], healthcare [2, 32] and legal research [40].
In such high-stake applications, it is of paramount importance that the model predictions are not only correct, but also well-calibrated.
For a calibrated model, probability of being correct across all confidence levels and the predictions-confidence should be aligned [6, 35]
It is desired that the incorrect predictions have low-confidence, however, a poorly calibrated model is prone to have confident but incorrect predictions.
We have witnessed tremendous effort towards improving the predictive accuracy of models, however, considerably less attention is devoted to model calibration.
Among a few works addressing model calibration, the majority of them focus on classification tasks [6, 17, 20, 22, 35].
A dominant class of methods address calibration by proposing different post-hoc approaches [28, 6], The core idea is to transform model outputs by a single parameter which is optimized on a validation set, with an objective to improve the calibration of in-domain predictions.
Such methods involve limited parameters while calibrating the outputs of the models.
Further, they are restrictive, since in many real-world scenarios, a validation set is not always available.
To involve all model parameters, some methods propose train-time calibration techniques [17, 20, 22] by constraining that for a given sample, the predicted class confidence and likelihood for that class should be minimized.
The model could remain poorly calibrated for the classes with non-maximum prediction confidences [7].
Surprisingly, little to no attention has been paid towards addressing the calibration of deep learning based visual object detection methods, that form an important part of many decision making systems.
Also, most current efforts aim at improving model calibrations only for in-domain predictions.
However, in many real-world scenarios, owing to domain shift, the distribution of the data received by a deployed model can be very different to the distribution of its training data.
Therefore for several practical scenarios, a model should be well-calibrated for both in-domain and out-of-domain predictions.
Besides carrying scientific value, well-calibrated object detectors, especially under domain drift, will substantially improve overall trust in many vision-based safety-critical applications and will be of great value to industry practitioners.
In this paper, we study the calibration of object detection models for both in-domain and out-ofdomain detections.
We observe that: (1) detection models demonstrate poor calibration for in-domain and out-of-domain detections (see Fig. 1), and (2) unsupervised domain adaptive detection models are rather miscalibrated when compared to their predictive accuracy in a target domain.
Towards developing well-calibrated object detection models, for both in-domain and out-of-domain scenarios, we propose a new plug-and-play loss formulation, termed as train-time calibration for detection (TCD).
It can be used with task-specific loss functions during training phase and acts as a regularization for detections.
In addition, we develop an implicit calibration technique for self-training based domain adaptive detectors.
Finally, we empirically show that this technique is complementary to our loss function and they both can be utilized during adaptation to further boost calibration under challenging domain shift detection scenarios.
We validate the effectiveness of our loss function towards improving calibration of different DNN-based object detection paradigms and different domain adaptive detection models under challenging domain shift scenarios.
Related Work
Below we survey different techniques in literature aimed at addressing the calibration of deep neural networks.
Broadly, the existing calibration techniques can be divided into post-hoc and train-time calibration methods.
The former estimates a transformation through a hold-out data after training, whereas the latter involves model parameters during the training.
Further, there are methods that achieve implicit calibration through using model’s uncertainty or learning to reject OOD samples.
Post-hoc calibration methods: Post-hoc calibration methods are post processing methods that leverage a hold-out validation data to optimize the calibration parameters for transforming the outputs of trained deep neural networks.
A popular post-hoc calibration approach is temperature scaling (TS) [6] (an extension of Platt Scaling [28]) that re-scales the logits by an optimal temperature ( T ) parameter which is estimated on a validation set.
The overall accuracy of the model remains unaffected by temperature scaling, but the logits values are modulated in such a way that confidence scores are reduced.
Temperature scaling along with other methods like vector scaling and matrix scaling extends Platt scaling to multiclass setting.
[11] introduced bin wise temperature scaling that incorporates different temperatures for different confidence intervals.
To mitigate miscalibration under long-tailed distribution scenarios, Islam et al.
[10] used class frequency information.
Specifically, the model is calibrated using class distribution-aware temperature scaling and label smoothing.
The Dirichlet calibration (DC) [15] is derived from Dirichlet distributions and extends the beta calibration method [16] to multi-class setting.
It is achieved with a (extra) neural network layer on log-transformed class probabilities, which is learnt on a hold-out validation data.
Recently, [35] generalizes existing post-hoc calibration methods, to address the poor calibration of out-of-domain predictions, by transforming the validation set before performing the post-hoc calibration step.
In general, post-hoc methods for model calibration require hold-out validation set.
Also, the majority of post-hoc methods perfrom model calibration for in-domain predictions.
Train-time Calibration methods: Negative log-likelihood (NLL) is the most popular approach to train a DNN based classifier.
Recent work shows that NLL-based training results in overconfident scores [6].
To this end, various methods presented additional loss formulations that can be used with task-specific losses during the training procedure.
For instance, [27] penalized overconfident predictions by adding the negative of entropy to the loss function.
Similarly, Liang et al.
[20] improved calibration by proposing the difference of accuracy and confidence as an auxiliary loss to the cross-entropy loss.
Whereas, [17] proposed an auxiliary loss, for obtaining improved calibration, which is quantified with the help of reproducing kernel in a Hilbert space.
Recently, [7] developed an auxiliary loss formulation for calibrating non-predicted labels along with the predicted one.
It has been shown that methods which attempt to reduce overconfidence by modifying the hard-labels can achieve better model calibration.
Mukhoti et.
al.
[23], hypothesized that minimizing focal loss [21] increases the predicted distribution’s entropy while trying to minimize KL divergence between the predicted and ground-truth distribution.
Replacing cross entropy with Focal loss results in a well-calibrated classification model.
[24] empirically showed that label smoothing (LS), originally proposed by [33], helps in improving calibration.
[14] calibrated uncertainty by relating accuracy and uncertainty, that yields accurate predictions to be more certain while inaccurate predictions to be more uncertain.
Other relevant methods: A recent work [37] based on empirical results proposed a unified framework that includes training and post-hoc calibration for better calibration.
Hein et al.
[8] identified that ReLU activation (or similar piece-wise linear functions) is a core reason behind the overconfident predictions of DNNs far away from the training data.
To overcome this, the data augmentation is incorporated using adversarial learning.
[13] detected OOD samples by performing spectral analysis over initial CNN layers, resulting in much better calibrated model.
Finally, some works report implicitly calibrating DNNs via some loss function [24] or selecting pseudo-labels during self-training based on model’s uncertainty [25].
Note that, almost all aforementioned techniques for analyzing and improving DNN calibration focus on classification tasks.
There is little to no literature studying the calibration of DNN-based object detectors, including under domain shift.
Toward this aim, inspired by the train-time calibration methods, we propose a new auxiliary loss formulation (TCD) for DNN-based object detection calibration that can be integrated with any task-specific loss function while training.
It is capable of significantly calibrating in-domain and out-of-domain object detections.
In addition, we develop an implicit calibration technique, that is also complementary to our loss formulation, to boost the calibration of self-training based domain adaptive object detectors.
Improving Calibration in Object Detectors 3.1 Definitions of Calibration
Calibration for classification: Let D = ⟨ ( x i , y ∗ i ) ⟩ Ni =1 be a dataset of N image and ground-truth pairs from a joint distribution D ( X , Y ) .
Where x i ∈
R H × W × C is an image, and y ∗ i ∈ Y = 1 , 2 , ..., K is the associated ground-truth class label.
H , W , and C represent the width, height and the number of channels of an image respectively.
For a classification model F cls , that predicts a label ˆ y with a confidence score ˆ s , we can define a perfect calibration as [6]:
[Equation]
Where P (ˆ y =
y ∗ | ˆ s = s ) is the accuracy for a particular confidence score s .
For the model to be calibrated, this accuracy should match with the predicted confidence.
Calibration for object detection: For training an object detection model, the ground-truth annotations include object’s localization information along with their associated categories.
Let b ∗ ∈ B = [0 , 1] 4 be the bounding box annotation of the object and y ∗ be the corresponding class label.
We assume that an object detection model F det predicts the object location ˆb and the class label ˆ y with confidence ˆ s .
This prediction is accurate, if the Intersection-over-Union (IoU) between ˆb and b ∗ is larger than some threshold γ , and ˆ y is same as y ∗ , i.e. 1 [IoU( ˆb , b ∗ ) ≥ γ ]
[ˆ y = y ∗ ] .
Then, for object detection, a perfect calibration can be expressed as [19] 2 :
[Equation]
Where U = 1 denotes a correctly classified prediction.
Expected calibration error for classification and object detection: For a classification model, the absolute difference between the confidence score and the accuracy corresponding to that confidence score (eq. 2) is considered Calibration Error .
The miscalibration of the model is quantified by computing the expectation of calibration error over predicted confidence ˆ s [26, 6, 18].
[Equation]
The continuous confidence space of ˆ s is divided into M equally spaced bins to approximate ECE:
[Equation]
where I ( m ) is the set of samples in m th bin, and |D| is the total number of samples.
acc( m ) and conf( m ) denote the average accuracy and average confidence in m th bin, respectively.
Along similar lines, ECE for the object detection (D-ECE) can be defined as the expected deviation of precision from the predicted confidence [19]:
[Equation]
As in classification ECE, confidence space is divided into M equally spaced bins to approximate
D-ECE :
[Equation]
where prec( m ) denotes the average precision in a bin.
3.2 Train-time Calibration for Detection: TCD
In this section we detail our proposed train-time calibration mechanism for object detection.
It features a novel auxiliary loss function that involves both the classification and localization components.
DNNbased object detectors predict a bounding box and corresponding class confidences for a detected region.
In order to achieve detection calibration, it is imperative to take into account both of them.
Our core idea is to jointly calibrate the estimated (class-wise) confidences and predicted bounding boxes.
To achieve this, we propose to compute two quantities over a mini-batch during training: (1) the difference between the accuracy and the confidence, and (2) the deviation between the predicted bounding box overlap and the predicted class confidence.
Specifically, there are two components in our loss formulation.
Inspired by the train-time calibration techniques for classification, we adapt the confidence calibration loss [7] and develop the first component d cls .
It measures the absolute difference between the average confidence and average accuracy.
[Equation]
Where R is the number of locations in the output class confidences channel map and L is number of images in mini-batch.
q l,r [ k ] = 1 if label k is the ground-truth label for sample l in location r , and 0 otherwise.
s l,r [ k ] denotes k th class confidence for r th location in sample l .
The second component d det computes the mean of the absolute difference between the bounding box overlap (with the ground-truth/pseudo ground-truth) and the confidence of the predicted class over all the positive regions.
Let N lpos denotes the number of positive regions in the sample l .
The loss is defined as:
[Equation]
The d cls component in the proposed loss is capable of not only calibrating the confidence of the predicted class but also for the non-predicted classes.
It penalizes the model, if for a given class k , the average confidence across mini-batch samples and possible output locations deviates from the average occurrence across mini-batch of this class.
On the other hand, the d det component penalizes the deviation between the IoU score (computed between the ground-truth bounding box and predicted one) and its corresponding predicted confidence for positive regions.
It explicitly forces the object detector to match the confidence, of the predicted class, with the tightness of the predicted bounding-box over the detected object.
Since above components are not based on binning, like Eqs.(4,6), as such, it avoids the non-differentibility issue
Note that, both loss components, d det and d cls , operate over the mini-batch constructed during training and so the L TCD can be used as an auxiliary train-time calibration loss for calibrating object detectors, including domain-adaptive ones, in conjunction with various task-specific loss functions (secs.
Further, we show that the proposed loss formulation can also be used with the pseudolabels used in self-training based unsupervised domain-adaptive detection algorithms for improving calibration in target domain (sec.4.2).
Finally, we show that our loss formulation is complementary to implicit calibration techniques and so can be deployed to further enhance calibration (sec.4.3).
3.3 Implicitly Calibrating Self-Training based Domain Adaptive Detectors
We propose a technique aimed at implicitly improving the calibration of self-training based domain adaptive detectors.
Mostly, self-training based domain adaptive detectors involve constructing pseudo instance-level labels (termed pseudo-labels thereafter) corresponding to detections in a target domain.
Based on these pseudo-labels, pseudo-targets are formed for computing (classification) loss during the adaptation phase.
We observe that, these pseudo-targets are constructed as one-hot encoded channels, and so they struggle to reflect the predictive confidence or uncertainty of detections.
As a result, the adaptation model fails to account for the noise in detections, mostly prevalent under domain shift, and could inadvertently learn to make overconfident predictions, hence negatively affecting calibration.
To this end, we first present a new uncertainty quantification mechanism for object detection and then leverage this to modulate the one-hot encoded pseudo-targets.
Quantifying uncertainty in object detection: To quantify model’s uncertainty for detections, we follow a three-step process [25].
First, given an arbitrary image, we perform N stochastic forward passes (inferences) over the one-stage object detector using Monte-Carlo dropout [3].
Specifically, we apply spatial MC-dropout [36] over the convolutional filters after the feature extraction layer [25].
Let ˆ b n,m ∈ R 4 be the m th predicted bounding box during n th inference, s n,m be the corresponding confidence vector, and ˆ c n,m be the predicted class corresponding to the highest confidence
ˆ s n,m ∈ s n,m .
We define ˆ z n,m = ( ˆ b n,m , ˆ c n,m ) be the prediction pair.
Second, we identify and group the detections corresponding to ˆ z n,m across the inference space that have the same predicted class and an overlap with its bounding box greater than a certain threshold.
Specifically, we create a set A n,m for each ˆ z n,m .
This set consists of all predictions ˆ z k,l such that ˆ c n,m = ˆ c k,l and
IoU between ˆ b n,m and (cid:98) b k,l is larger than some threshold γ (set to 0.5 throughout experiments) [25].
Here k ̸ = n and l is an arbitrary detection in k th MC forward pass.
[Equation]
Finally, A n,m is utilized to estimate the uncertainty in ˆ z n,m .
We aim to capture uncertainty in predicted confidences and localization.
So, we first estimate the variance in predicted (class) confidences, center x, center y, and aspect ratio of the bounding boxes predictions in A n,m and later aggregate them to construct a joint measure of uncertainty.
Let { ψ } Jj =1 and { Ψ } Jj =1 be the vectors ( J denotes the length) containing variances and means of predicted confidences, center-x, center-y, and aspect ratio of the predicted bounding boxes in A n,m , respectively.
Let Ψ agg be the combined mean, computed as Ψ agg
= 1 J (cid:80)
Jj =1 Ψ j .
Then, the combined variance representing a single, joint measure of uncertainty is computed as:
[Equation]
Uncertainty-guided soft pseudo-targets: We leverage (joint) uncertainty to modulate the one-hot encoded pseudo-targets, formed according to (selected) pseudo-labels, to account for the entropy in object detections under target domain.
Let H ki
be i th location in the class k one-hot encoded channel map corresponding to a detected bounding box, with uncertainty u ki and (pseudo) class label k .
The uncertainty-guided soft pseudo-target is constructed as:
[Equation]
Where ¯ s ki = 1 T (cid:80)
j ˆ
s ji is the mean (class) confidence of the predicted bounding boxes in A i .
κ 1 is threshold for ranking between highly confident and relatively less confident detections, and κ 2 is the confidence threshold below which there is no detection considered.
[Table]
Experiments
Datasets: Cityscapes dataset [1] consists images of road and street scenes with bounding box annotations of following object categories: person, rider, car, truck, bus, train, motorbike, and bicycle .
KITTI dataset [4] offers images of road scenes, captured from different viewing angle than
Cityscapes and encompass wide-view of the area.
Foggy Cityscapes dataset [31] is constructed by simulating foggy weather, using depth maps, over the images in Cityscapes dataset.
From the available three levels of fog, per the norm, we use the one with most dense fog.
Sim10k dataset [12] is a collection of 10K images, synthesized from Grand Theft Auto V, and corresponding bounding box annotations of cars in those images.
Following prior works, we consider car class for experiments when adapting from KITTI or Sim10k.
BDD100k dataset [39] contains 100k annotated images with bounding boxes and category labels.
70k images are used for training and 30k are used for validation.
Following [38] we only consider daylight images to create a training subset of 36.7k images and a validation subset of 5.2k images.
Note that, the validation subset is used as an evaluation set.
Evaluation and implementation details: We report calibration performance using detection expected calibration error (D-ECE) and also report test accuracy.
Further, we also plot reliability diagrams for visualizing calibration.
Our TCD loss is developed to be used with the task-specific loss of
DNN-based object detectors, including the SOTA domain-adaptive ones.
For instance, the task specific losses for single-stage detectors use Focal loss [21] and IoU loss.
Similarly Smooth L1 loss and Cross-Entropy are used in training the two-stage detectors.
Let L D be the task-specific loss, then the total loss in our method is computed as: L = L D + L TCD .
For further implementation details, refer to the supplementary.
Considered object detectors: Most current state-of-the-art (SOTA) object detectors are built on either a single-stage or a two-stage backbone.
To include a representative detector models, from single-stage detectors we consider a state-of-the-art detector FCOS [34] and from multi-stage detectors we choose
Faster-RCNN [29].
Our choice is partially motivated by the fact that the same backbones are also used in the current SOTA domain-adaptive object detectors.
For the domain adaptive detectors we include EPM [9] and also SSAL [25] which are among the best performing single-stage domain adaptive detectors built on FCOS as source model.
For the two-stage domain adaptive variant we include SWDA [30] that is built on FasterRCNN.
4.1 Experiments with One-stage and Two-stage Detectors
In this setting, we train a DNN-based object detector (1) with its task-specific loss functions and (2) after adding our TCD auxiliary loss function.
For comparison we also include a post-hoc calibration technique based on temperature scaling.
The temperature parameter T is obtained using a hold-out validation set to perform temperature scaling at inference time in the target domain.
We measure the performance and calibration errors in two settings.
Here we measure the performance on the test set belonging to the same domain from which training dataset was sampled In-domain Detection: Here we measure the performance over the test-set partition of same dataset from which training set was extracted and Out-of-domain Detection: where we test it on an unseen target domain i.e., a different dataset.
Table 1 reports results with a one-stage detector (FCOS [34]) trained with its task-specific losses i.e. Focal loss and IoU loss.
We report calibration errors (D-ECE) and performance of the model.
[Table] [Table]
We show the impact on calibration after adding and training with the proposed auxiliary loss TCD.
For comparison we include post-hoc temperature scaling based calibration method.
For in-domain detections, we see that our proposal, single-stage detector trained after adding TCD, achieves the best
D-ECE score across all datasets.
Furthermore, our proposal allows boosting the detection performance by notable margins compared to a single-stage detector in all datasets.
For out-of-domain detections, we significantly improve the calibration performance over baselines in all domain shift scenarios.
For instance, it decreases the D-ECE score by 12.9% and 7.1% compared to the single-stage detector in CS → CS-foggy and CS → BDD100K, respectively.
At the same time, it provides notable gains in detection performance over single-stage detector e.g., a 4.8% gain (AP@0.5) in Sim10K → CS.
We report results with a two-stage detector Faster-RCNN [29], trained with a task-specific loss, taking a pre-trained two-stage detector and applying post-hoc temperature scaling, and a two-stage trained with task-specific loss and our TCD loss (Table 2).
For both in-domain and out-of-domain detections, we observe that our proposal, two-stage detector trained after adding TCD, delivers the best D-ECE score across all datasets.
4.2 Experiments with Domain-adaptive Detectors
In this setting, we incorporate our TCD auxiliary loss in the recent SOTA domain-adaptive detectors and observe impact on calibration performance.
In single-stage paradigm, we chose EPM [9], which accounts for pixel-wise centerness and objectness for target domain images in an adversarial alignment framework.
Further, we also choose a method from the line of self-training based domain-adaptive detectors.
Particularly, we select uncertainty-guided pseudo-labelling (UGPL) from SSAL [25].
In two-stage paradigm, we choose SWDA [30], which proposes strong local alignment and weak global alignment in an adversarial alignment framework.
Table 3 reports results with two single-stage domain-adaptive detectors: EPM [9], UGPL from [25].
After adding our proposed loss TCD, the calibration performance for both domain-adaptive detectors is significantly improved.
In Sim10K → CS, both EPM+TCD and SSAL(UGPL)+TCD reduce ECE score by 6.1% and 5.1%, respectively.
Further, we observe that after adding our auxiliary loss (TCD), there are notable gains in AP@0.5.
We show the impact of our TCD loss on the calibration performance of a two-stage domain-adaptive detector (SWDA [30]) in Table 3.
Upon adding our loss TCD in SWDA, we observe that the calibration of SWDA is improved in both adaptation scenarios.
4.3 Experiments with Implicit Calibration Technique
We validate the effectiveness of our implicit calibration technique (ICT) (sec. 3.3) by integrating in a self-training based uncertainty-guided pseudo-labelling baseline (UGPL)) from [25].
UGPL is a strong baseline in terms of benchmarking model calibration because it leverages uncertainty to select pseudo instance-level labels in an unlabelled target domain for self-training.
Tablereports results for following methods: SSAL(UGPL) [25], SSAL(UGPL)+ICT, SSAL(UGPL)+TCD,
[Table]
SSAL(UGPL)+ICT+TCD.
We see that SSAL(UGPL)+ICT decreases D-ECE by 0.9% in Sim10 → CS shift.
Further, adding TCD shows significant improvement in calibration performance (D-ECE).
Finally, using both ICT and TCD achieves the best calibration scores in both Sim10K → CS and
CS → CS-foggy shifts, thereby revealing that ICT and TCD are complementary.
4.4 Ablation Study and Analysis
For ablation experiments with TCD, we use one-stage detector (FCOS [34].
For ICT ablation experiments, we use the UGPL component of SSAL [25], which is a domain-adaptive detector.
Mitigating Under/Over-Confidence: Fig.
2 shows that our proposal, task-specific loss in conjunction with proposed TCD loss, facilitates a one-stage detector in mitigating both under-confident and overconfident detections in two different multi-class shift scenarios.
Impact on test accuracy: We observe the impact on test accuracy after adding our TCD loss with the task-specific loss of one-stage detector in Table 5.
Our proposal is capable of providing gains in test accuracy with visible margins over higher IoU thresholds and over the spectrum of large, medium and small objects.
[Figure]
[Table]
Without detection component d det : Table 6 shows the impact on calibration performance without d det component in TCD.
We see that without d det , the calibration performance (D-ECE) degrades
[Table]
On different uncertainty quantification methods for ICT: Table 7 reports the calibration performance of ICT with different methods of quantifying uncertainty.
We use variances across: predicted confidences only [25] u conf .
; predicted confidences and center-x,center-y u conf .
, x , y ; predicted confidences and center-x,center-y, aspect-ratio (ours) u conf .
, x , y , ar .
Among different methods, our proposed technique of estimating detection uncertainty in ICT provides the lowest D-ECE score.
[Table]
Limitation: Current study does not explore the impact on calibration with respect to objects sizes.
Further, our train-time calibration loss could potentially result in less improvement in classes with relatively less instances.
Conclusion
In this paper
[Table]
we approached the challenging problem of calibrating DNN-based object-detectors for in-domain and out-of-domain detections and improving their calibration in the domain-adaptation context.
We proposed an auxiliary train-time calibration loss TCD that jointly calibrates the class-wise confidences and localization performance.
Further, we develop an implicit calibration technique (ICT) for self-training based domain adaptive detectors.
Results show that TCD loss is capable of improving calibration of both one-stage and two-stage detectors.
Finally, we show that ICT and TCD together results in well-calibrated domain-adaptive detectors.