C ONFIDENCE E STIMATION U SING U NLABELED D ATA
Anonymous authors
Paper under double-blind review
A BSTRACT
Overconfidence is a common issue for deep neural networks, limiting their deployment in real-world applications.
To better estimate confidence, existing methods mostly focus on fully-supervised scenarios and rely on training labels.
In this paper, we propose the first confidence estimation method for a semi-supervised setting, when most training labels are unavailable.
We stipulate that even with limited training labels, we can still reasonably approximate the confidence of model on unlabeled samples by inspecting the prediction consistency through the training process.
We use training consistency as a surrogate function and propose a consistency ranking loss for confidence estimation.
On both image classification and segmentation tasks, our method achieves state-of-the-art performances in confidence estimation.
Furthermore, we show the benefit of the proposed method through a downstream active learning task.
I NTRODUCTION
Besides accuracy, the confidence, measuring how certain a model is of its prediction, is also critical in real world applications such as autonomous driving (Ding et al., 2021) and computer-aided diagnosis (Laves et al., 2019).
Despite the strong prediction power of deep networks, their overconfidence is a very common issue (Guo et al., 2017; Nguyen et al., 2015; Szegedy et al., 2014).
The output of a standard model, e.g., the softmax output, does not correctly reflect the confidence.
The reason is that the training is only optimized regarding to the training set (Naeini et al., 2015), not the underlying distribution.
Accurate confidence estimation is important in practice.
In autonomous driving and computer-aided diagnosis, analyzing low confidence samples can help identify subpopulations of events or patients that deserve extra consideration.
Meanwhile, reweighting hard samples, i.e., samples on which the model has low confidence, can help improve the model performance.
Highly uncertain samples can also be used to promote model performance in active learning (Siddiqui et al., 2020; Moon et al., 2020).
Different ideas have been proposed for confidence estimation.
Bayesian approaches (MacKay, 1992;
Neal, 1996; Graves, 2011) rely on probabilistic interpretations of a model’s output, while the high computation demand restricts their applications.
Monte Carlo dropout (Gal & Ghahramani, 2016) is introduced to mitigate the computation inefficiency.
But dropout requires sampling multiple model predictions at the inference stage, which is time-consuming.
Another idea is to use an ensemble of neural networks (Lakshminarayanan et al., 2017), which can still be expensive in both inference time and storage.
To overcome the inefficiency issue, some recent works focus on the whole training process rather than the final model.
Moon et al.
(2020) use the frequency of correct predictions through the training process to approximate the confidence of a model on each training sample.
Geifman et al.
(2018) collect model snapshots over the training process to compensate for overfitting and estimate confidence.
However, most existing methods purely rely on labeled data, and thus are not well suited for a semisupervised setting.
Indeed, confidence estimation is critically needed in the semi-supervised setting, where we have limited labels and a large amount of unlabeled data.
A model trained with limited labels is sub-optimal.
Confidence will help efficiently improve the quality of the model, and help annotate the vast majority of unlabeled data in a scalable manner (Wang et al., 2022; Sohn et al., 2020;
Xu et al., 2021).
In this paper, we propose the first confidence estimation method specifically designed for the semisupervised setting.
The first challenge is to leverage the vast majority of unlabeled data for confi- dence learning.
For data without labels, our idea is to use the consistency of the predictions through the training process.
An initial investigation suggests that consistency of predictions tends to be correlated with sample confidence on both labeled and unlabeled data.
Having established training consistency as an approximation of confidence, the next challenge is that the consistency can only be evaluated on data available during training.
To this end, we propose to re-calibrate model’s prediction by aligning it with the consistency.
In particular, we propose a novel
Consistency Ranking Loss to regulate the model’s output after the softmax layer so it has a similar ranking of model confidence output as the ranking of the consistency.
After the re-calibration, we expect the model’s output correctly accounts for its confidence on test samples.
We both theoretically and empirically validate the effectiveness of the proposed Consistency Ranking Loss.
Specifically, we show the superiority of our method on real applications, such as image classification and medical image segmentation, under semi-supervised settings.
We also demonstrate the benefit of our method through active learning tasks.
Related work.
There are two mainstream approaches for confidence (uncertainty) estimation: confidence calibration and ordinal ranking.
Confidence calibration treats confidence as the true probability of making correct prediction and tries to directly estimate it (Platt, 2000; Guo et al., 2017; Jungo & Reyes, 2019; Zadrozny & Elkan, 2002; Naeini et al., 2015).
For any sample, the confidence estimate generated by a well-calibrated classifier should be the likelihood of predicting correctly.
Directly estimating the confidence may be challenging.
Instead, many works focus on the ordinal ranking aspect (Geifman et al., 2018;
Geifman & El-Yaniv, 2017; Mandelbaum & Weinshall, 2017;
Moon et al., 2020; Lakshminarayanan et al., 2017).
In spite of the actual estimated confidence values, the ranking of samples with regard to the confidence level should be consistent with the chance of correct prediction.
A model with well-ranked confidence estimate can be widely used in the field of selective classification, active learning and semi-supervised learning (Siddiqui et al., 2020; Yoo & Kweon, 2019; Sener & Savarese, 2018; Tarvainen & Valpola, 2017; Zhai et al., 2019; Miyato et al., 2018; Xie et al., 2020; Chen et al., 2021).
C ONSISTENCY - A N EW S URROGATE OF C ONFIDENCE
Our main idea is to use the training consistency, i.e., the frequency of a training datum getting the same prediction in sequential training epochs, as a surrogate function of the model’s confidence.
In this section, we first formalize the definition of training consistency.
Next, we use qualitative and quantitative evidences to show that consistency can be used as a surrogate function of confidence.
This justifies the usage of training consistency as a supervision for model confidence estimation, which we will introduce in the next section.
Definition: training consistency.
Assume a given dataset with n labeled and p unlabeled data,
D =
( X, Y, U ) .
Here X = { x 1 , . . .
, x n } , x i ∈ X is the set of labeled data with corresponding labels
Y = { y 1 , . . . , y n } , y i ∈ Y = { 1 , . . .
, K } .
The set of unlabeled data U = { x n +1 , . . .
, x n + p } , x j ∈
X cannot be directly used to train the model, but will be used to help estimate confidence.
We assume a simple training setting where we use the labeled set ( X, Y ) to train a model f
( x, y ; W ) :
X × Y → [0 , 1] .
The method can naturally generalize to more sophisticated semisupervised learning methods, where unlabeled data can also be used.
For any data, either labeled or unlabeled, x i ∈
X ∪ U , we have the classification ˆ
y i = arg max y ∈Y f ( x i , y ; W ) .
Note that traditionally the model output regarding the classification label, f ( x i , ˆ y i ; W ) = max y ∈Y f ( x i , y ; W ) , is used as the confidence.
Our definition involves the training process.
Denote by W t the model weights at the t -th training epoch, t = 1 , . . .
, T .
We use ˆ y ti = arg max y ∈Y f
( x i , y | W t ) to denote the model classification for sample x i at the t -th epoch.
For a sample x i , we define its training consistency as the frequency of getting consistent predictions in sequential training epochs during the whole training process ( T epochs in total):
[Equation]
Qualitative analysis shows consistency is a good surrogate of confidence.
We provide a qualitative example in Fig.
1 with the feature representations.
We observe that the data further from the
[Figure]
decision boundary have higher consistency, and the ones closer to decision boundary have lower consistency.
This is consistent with our intuition and seems to correlate with model confidence.
Although we do not have the ground truth of confidence, it is reasonable to believe that data further from the decision boundary will have higher confidence, whereas data near the decision boundary will have lower confidence.
The data representation is acquired by training a model (PreAct-ResNet110) on only 20% of CIFAR10 training set.
We extract the penultimate fully connected (FC) layer output as the feature representation.
Here we use t-SNE to visualize the feature representations of 80% unlabeled training images.
For ease of exposition, we only focus on class 0, class 1 and class 9.
As shown in
Fig.
1(a), since the model is trained with limited labeled data, the three classes are not completely separated in the representation space.
We expect lower confidence data to be closer to the decision boundary.
However, this is not well reflected by the model confidence output, i.e., the maximum softmax output, f ( x i , ˆ y i ; W ) .
As shown in Fig. 1(b), softmax output tends to be overconfident.
Meanwhile, we calculate consistency on unlabeled data and visualize them in Fig. 1(d).
The consistency map seems much more reasonable and better correlates with model confidence.
The consistency is high within each class, but slowly transits to low towards the decision boundary.
Since we do not have ground truth of confidence, as a reference, we also calculate and visualize the correctness of data, a known good proxy of confidence in fully-supervised setting (Moon et al., 2020).
The correctness of a datum x i is the frequency at which the model makes a correct classification on x i across different epochs, i.e., 1 T (cid:80)
Tt =1 1 { y i
=
ˆ y ti } .
As shown in Fig.
1(c), the correctness map is very similar to consistency.
This further justifies our proposal of using consistency.
Please note that this analysis does not justify correctness as a good confidence surrogate when training with limited labels.
When labeled data is limited, we can only calculate and use correctness on a limited amount of samples.
They cannot provide sufficient supervision for confidence learning.
This will be evident in our experiment section.
Quantitative analysis.
We further evaluate the quality of consistency as a confidence surrogate using a popular confidence evaluation metric, the area under the risk-coverage curve (AURC) and the excess-AURC (E-AURC).
Based on the ordinary ranking principle, AURC and E-AURC measure how well an estimated confidence is correlated with the chance of correct prediction; a sample with high confidence should have a high chance to be correctly predicted.
A low AURC or E-AURC indicates the estimation is a good approximation of the true confidence.
A formal definition of
AURC and E-AURC is provided in Sec. 3.2.
We evaluate AURC and E-AURC on the training data for consistency, correctness and softmax output of the model at different training epochs with regard to a model trained on limited labeled data.
The setting is the same as in the previous qualitative analysis.
The results are shown in
Fig.
We observe that overall all three confidence proxies improve their quality as the training continues (i.e., their AURCs/E-AURCs decrease as the training progresses).
Consistency is better than softmax output throughout the training (i.e., having a lower AURC or E-AURC).
Correctness has the best performance among the three (i.e., the lowest AURC or E-AURC).
However, correctness is unavailable for unlabeled data, thus does not help learning confidence in our problem.
L EARNING TO E STIMATE C ONFIDENCE : C ONSISTENCY R ANKING L OSS
[Figure]
We have established that consistency is a good approximation of the confidence.
However, consistency is only available for training data.
At the inference stage, we do not have such measure for test data.
To this end, we propose a novel Consistency
Ranking Loss during training to enforce the model’s maximum softmax output approximates the consistency well.
The hope is at the inference stage, the maximum softmax output will well approximate the consistency, and thus the confidence.
The consistency ranking loss is a pairwise ordinal ranking function to enforce the maximum softmax output to be consistent with the training consistency on all labeled and unlabeled training data X ∪ U .
Recall for the i -th training datum, x i ∈
X ∪ U , we denote by c
i its training consistency (Eq.
For convenience, we denote by κ i the model’s maximum softmax output for x i .
The loss function enforces the two to be consistent in terms of ordinary ranking, i.e., if c i < c s , then κ i <
[Equation]
s .
Formally,
[Equation]
The loss sums over all pairs of training data and penalizes a pair ( i, s ) when x i has a lower confidence than x s and the difference c
s − c
i is bigger than the difference κ s − κ i .
Through training, the confidence estimator κ is learned to have a similar gap as the consistency c ’s, i.e., κ s − κ i ≥ c s − c i , ∀ c s >
c i .
The loss seems expensive as it is quadratic to the sample size.
In practice, however, computation is not a major issue.
We evaluate the loss on samples within each minibatch (both labeled and unlabeled).
The loss is then only quadratic to the minibatch size.
As shown in the experiment section, this suffices in practice.
The details of pairing strategy is in Append.
A.2.
To fully exploit labeled training samples for confidence estimation, we also incorporate correctness into our training, using a ranking loss L corr .
Note L corr is only evaluated on labeled samples.
Our overall loss L is
[Equation]
Here L CE is cross entropy loss on labeled samples.
λ 1 and λ 2 are the weights of losses.
3. E XPERIMENT V ERIFICATION
In practice, it is intractable to compare all possible sample pairs even for small batch data.
We use experiment to show the empirical performance of consistency ranking loss on CIFAR-10 dataset.
Here we adopt the same experimental setting as Sec. 2.
All the results are collected from unlabeled training set (40000).
As shown in Tab. 1, our consistency ranking loss makes the AURC (E-AURC) of model confidence output (maximum softmax output) approximate the AURC (E-AURC) of training consistency very well.
The experimental results indicate that our consistency ranking loss can achieve very good performance even optimized on minibatch data with limited pairs.
3. T HEORETICAL G UARANTEE
In this section, we provide theoretical analysis of the proposed consistency ranking loss (Eq.
We show that the loss upperbounds the quality of the confidence estimator κ .
Informally, our main theorem (Thm. 1) states that the loss upperbounds the difference between the confidence estimation
[Table] [Table]
performance of κ and the consistency c , in terms of the popular metrics AURC and E-AURC.
In other words, as we minimize the loss, κ will have a similar performance as the consistency c in terms of confidence estimation.
Definition of AURC.
To evaluate if a confidence estimator κ is correlated with the chance of a sample being correctly predicted by classifier f , the area under the risk-coverage curve (AURC) is proposed.
Given a dataset with m samples D m = ( X m , Y m ) and a confidence estimator κ , the empirical selective risk is ˆ r ( f, g θ |D
[Equation]
) =
[Equation]
where ˆ y i = arg max y ∈Y f
( x i , y |D m ) .
We define indicator function l (ˆ y i , y i )
=
1 { ˆ y i ̸
= y i } .
Here, g θ ( x i ) =
1 { κ i ≥ θ } is a selection function thresholding on confidence scores κ .
The AURC is defined as AURC ( f, κ |D m ) =
[Equation]
|D m ) , where Θ is the set of all distinct values of confidence scores κ on X m , i.e., Θ = { κ i | i = 1 , . . .
, m } .
E-AURC was proposed as a normalized version of AURC.
κ ∗ is an optimal confidence score function of f on D m , yielding the correctly predicted samples having higher confidence scores than misclassified ones.
κ ∗ minimizes AURC ( f, ·
|D m ) .
Excess-AURC
E-AURC) is defined as,
E-AURC ( f, κ |D m ) =
AURC ( f, κ |D m ) − AURC
( f, κ ∗ |D m ) .
When E-AURC ( f, κ |D m )
=
and f is not an optimal classifier, then κ assigns higher confidence score to correct predictions than incorrect ones.
In such a way, we can filter out correct predictions by κ .
Next, we formally state and prove our main theorem.
Let T ( f |D m ) be the set constituted by all the correctly predicted samples, T ( f |D m ) =
{ x : ˆ y =
y, ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } .
Here we define R κ , a ranking function based on confidence score function κ on X m , R κ ( x ) ∈ { 1 , 2 , . . .
, m } , ∀ x ∈ X m .
If κ ( x i ) < κ ( x j ) than R κ ( x i ) >
R κ ( x j ) , ∀ x i , x j ∈ X m .
Here we assume both κ and R κ are one-to-one functions.
Given R κ , we define ranking set H κ
=
{ R κ ( x ) : ˆ y =
y, ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } .
Here we define c ( x i )
=
c i , x i ∈
X m .
Note that training consistency is a kind of confidence estimator and in the context below, we will use c to denote training consistency.
Here we define R c in certain way such that it is one-to-one function (More details are included in Append.
A.3).
Definition 3.1.
Given ranking function R c and R κ , we introduce a set D c = {| c ( R − 1 c ( i ))
− c ( R − 1 c ( j )) | ; i = γ m ( R κ ( x )) , j = γ m ( R κ ( x ))
+ sign ( R κ ( x ) − γ m
( R κ ( x )))) , x ∈ T, R − 1 c ( j ) ̸∈
T ( f |D m ) } , where γ m =
arg
min
γ ∈ τ ( H κ ,H c ) (cid:80) x ∈ T | R κ ( x ) − γ
( R κ ( x )) | , sign is a sign function, τ is the collection of the bijective function between the elements of H κ and H c .
Theorem 1.
Given dataset D m =
( X m , Y m ) and training consistency C m :
[Equation]
We notice that | E-AURC ( f, c |D m ) − E-AURC
( f, κ |D m ) | =
| AURC ( f, c |D m ) − AURC
( f, κ |D m ) | , thus the theorem above works for E-AURC too.
Proof.
Let T κ ( i ) be the set constituted by all the correct predicted samples with confidence less or equal than R − 1 κ ( i ) , T κ ( i ) =
{ x : R κ ( x ) ≤ i, ˆ y =
y, ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } .
T c ( i ) is the corresponding set introduced by training consistency c .
We define H c ( f, κ |D m )
=
(cid:80) mi =1 1 (# T κ ( i ) ̸
= # T c ( i )) and H kc ( f, κ |D m ) =
(cid:80) ki =1 1 (# T κ ( i ) ̸
= # T c ( i )) .
It can be proved:
[Equation] [Equation]
There is a fact that ∀ i
[Equation] [Equation]
[Equation] [Equation]
m } if
[Equation]
T κ
[Equation]
i
[Equation]
=
# T c
[Equation]
i )) , than ˆ
[Equation] [Equation]
We also have F r ( f, c |D m ) =
[Equation]
=
[Equation]
(cid:81) mj
=1 1 ( R
[Equation]
( x j ) ≤ i ) · 1 (ˆ y j = y j ) , where ˆ y j = arg max i ∈Y f ( x j ) i and ( x j ,
y j ) ∈
D m .
Here we define K ( f, c |D m ) =
# T ( f |D m ) − F r ( f, c |D m ) , which is the number of correct predicted samples with lower training consistency than at least one incorrect sample.
In the next part, we will prove that:
[Equation]
We first discuss the case when K ( f, c |D m ) = 0 .
This means that the correctly classified samples should have higher training consistency than the misclassified ones.
Let C t be the set constructed by the consistency of correctly classified samples, C t = { c ( x ) :
ˆ y
=
y, ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } and C f be the set constructed by the consistency of false classified samples, C f
= { c ( x ) : ˆ y ̸
= y, ˆ y = arg max i ∈Y f
x ) i , ( x, y ) ∈
D m } .
It is not difficult to tell that min D c = | min C t − max C f | under the case K ( f, c
|D m ) = 0 .
Here we have
[Equation]
|
[Equation]
− ( κ s − κ i ))
+ | min C t − max C f | ≥ H c ( f, κ |D m ) .
We should notice that the conclusion above does not rely on the data size of D m .
Here we define the ranking function R T ( · , κ |D m ) on T ( f |D m ) , R T ( x, κ |D m ) ∈ { 1 , 2 , . . .
, # T ( f |D m ) } , ∀ x ∈
T ( f |D m ) and if R κ ( x i ) <
R κ ( x j ) than R T ( x i , κ |D m ) <
R T ( x j , κ |D m ) , ∀ x i , x j ∈ T ( f |D m ) .
We assume the result stands when K ( f, c |D m ) = n .
Then when K ( f, c |D m ) = n + 1 , we start the discussion with the situation R − 1 T (# T ( f |D m ) , κ |D m ) =
R − 1 T (# T ( f |D m ) , c
|D m )
= x ∗ .
We have
[Equation] [Equation]
If R − 1 T (# T ( f |D m ) , κ |D m ) =
x 1 , R − 1 T (# T ( f |D m ) , c |D m ) =
x 2 and x 1 ̸ = x 2 , we can have one new confidence estimator κ t by switching the confidence value between x 1 and x 2 .
It is not difficult to see that consistency ranking loss introduced by κ t (Eq. 10):
L tcons ( f, X m |D m ) satisfying
L cons ( f, X m | C m ) ≥ L tcons ( f, X m | C m ) .
So the proof stands for K ( f, c |D m )
=
n + 1 under the assumption that it is true while K ( f, c |D m )
= n .
A detailed proof version is in Append.
A.3.
E XPERIMENT
To verify the effectiveness of our method in confidence estimation with limited labeled data, we start with evaluating on image classification and medical image segmentation tasks.
Active learning is usually used as a follow-up task of confidence estimation.
Here we implement active learning on image classification tasks to show that the highly uncertain samples picked up by our method will promote the learning process in a better manner.
4. I MAGE C LASSIFICATION
In this section, we show that our method can generate reliable confidence estimates with a small portion of labeled samples and majority of unlableled samples through classification tasks.
We evaluate our method on benchmark datasets, CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009).
Implementation details.
To show the superiority of our method in making use of unlabeled training samples, we compare our method with other baselines under different portions of labeled training data: 5% (2500), 10% (5000), 20% (10000), 100% (50000).
As for network architecture, we adopt
PreAct-ResNet110 (He et al., 2016) for CIFAR-10 and DenseNetBC ( k = 12, d = 100) (Huang et al., 2017) for CIFAR-100.
All methods are trained by SGD with a momentum of 0.9 and a weight decay of 0.0001.
We train our method for 300 epochs with the mini-batch size of 192, in which 64 are labeled, and use initial learning rate of 0.1 with a reduction by a factor of 10 at 150 and 250 epochs.
A standard data augmentation scheme for image classification is used, including random horizontal flip, 4 pixels padding and 32 × 32 random crop.
More details are in Append.
A.1.
For our method, we set the loss weights λ 1 , λ 2 in Eq.
3 as 0 .
5 respectively, and use maximum softmax as the model confidence output for both training and evaluation.
We compare our method with cross entropy loss only (Softmax), CRL (Moon et al., 2020), AES with 30 snapshot models (Geifman et al., 2018), Mcdrop with 50 stochastic ensembles (Gal & Ghahramani, 2016) and
Aleatoric+MCdropout (Kendall & Gal, 2017).
As for evaluation, besides the ordinal ranking evaluation metrics AURC and E-AURC, we also adopt three widely used calibration evaluation metrics,
[Table]
the expected calibration error (ECE) (Naeini et al., 2015), negative log likelihood (NLL) and the
Brier score (Brier et al., 1950).
The false positive rate at 95% true positive rate (FPR-95%-TPR) is also used to evaluate confidence estimation performances.
The experimental results are shown in Tab.
(The results of ‘full’ setting are included in Append. 5).
All the experiments are repeated for five times, and we report the means and standard deviations.
From the results, we can see that the proposed method achieves better performances in both confidence estimation and classification accuracy.
The improvement of our consistency ranking loss is even more salient under the scenarios of less labeled training samples.
This indicates that our method utilizes unlabeled samples to encourage neural network classifier to generate believable confidence estimates effectively.
It is also obvious that the improvement of our method is more significant on CIFAR-100.
This is because for CIFAR-100, the number of training samples in each class is much less than CIFAR-10, which worsens ‘the lack of samples’ effect in confidence estimation and classification.
This further demonstrates the necessity of utilizing the confidence information of unlabeled samples .
4. M EDICAL I MAGE S EGMENTATION
Even though deep neural network has achieved very impressive performance in medical image segmentation (Litjens et al., 2017), it is still important to estimate the model confidence on samples, so the expert can get involved in time to avoid misjudgement.
This emphasizes the importance of detecting and reacting to the failure of deep learning models.
Thus, confidence estimation is a promising way to manage this reliability concern (Jungo & Reyes, 2019).
On the other hand, image segmentation (dense label prediction) task is a good scenario to show the effectiveness of our
[Table]
consistency ranking loss, as the convergence of pairing loss is much more challenging.
Here we use a publicly available dataset to conduct the experiment: the international skin imaging collaboration (ISIC) lesion segmentation dataset 2017 (Codella et al., 2018), which consists 2000 training,validation and 600 testing annotated images.
Implementation details.
We conduct experiments with various portions of labeled training samples: 1/16 (125), 1/8 (250), 1/4 (500) and full (2000).
We use UNet (Ronneberger et al., 2015) with
ResNet34 backbone as the model architecture for all methods.
The loss weights λ 1 , λ 2 are set to 0 .
15 for segmentation task.
All models are trained by Adam with a learning rate 0.0001.
We train our method for 200 epochs with a mini-batch size of 96 (32 labeled and 64 unlabeled).
As for augmentation, a scheme with resizing ( 192 × 256 ), random scale, random crop ( 192 × 256 ), random horizontal and vertical flip is used.
Here we use mean Intersection-Over-Union (IOU) to evaluate the segmentation quality.
More details are in Append.
A.1.
The results are shown in Tab.
(The results of ‘full’ setting are included in Append. 4).
We observe that our method can produce more reliable confidence estimates than other baselines.
The lack of labeled training samples has little impact on the segmentation accuracy (mIOU), but deteriorates the confidence estimation quality severely.
Our method shows robustness to the shortage of labeled samples in terms of confidence estimation, because of the effective utilization of unlabeled samples.
When it comes to confidence calibration, our method has superior performances under various sizes of labeled samples.
[Figure]
λ 2 .
In Fig.
4, the qualitative results on ISIC2017 are showed, and we can observe that: 1) our method performs better for segmentation accuracy, 2) the maximum softmax output optimized on cross-entropy only is overconfident near the boundary of lesion region, 3) the confidence map generated by our method is more reasonable and consistent with training consistency.
Ablation study for λ 2 .
We also discuss the effect of weight changes on consistency ranking loss.
As shown in Fig.
3, the perturbation of weight λ 2 has limited influence on confidence estimation performances, and our choice ( λ 2 = 0 . 15 ) achieves the best outcome.
4. A CTIVE L EARNING
In practice, there are scenarios where unlabeled samples are cheap to acquire but manual annotations are costly.
To solve this problem, a promising direction is active learning, which requires learning algorithm actively queries experts for annotations.
The critical issue here is how algorithm can decide which unlabeled samples are valuable for the next stage learning.
A common opinion is that
[Figure]
the confidence levels of unlabeled samples denote the value for learning process.
Therefore, in this section, we adopt the strategy, which queries the samples with least model certainty.
Implementation details.
We evaluate active learning performance on CIFAR-10 and CIFAR-100.
ResNet18 architecture is used for all models.
For each stage
[Figure]
we train our model by SGD forepochs with initial learning rate of 0.1, which shrinks a factor of 10 at 120 and 160 epochs.
The rest hyper-parameters for our method are the same as Sec. 4.1.
Here we compare our method with
CRL, MC-dropout, core-set selection (Sener & Savarese, 2018), cross-entropy loss (using entropy as confidence output) and random sampling.
[Figure]
In the first training stage,labeled training samples and 49000 unlabeled samples are fed into training process.
After training, 1000 samples from the unlabeled set with least model confidence are selected and annotated.
We iteratively train and annotate for the next 9 stages.
All 10 stages construct a complete training procedure.
To make sure the experimental results are repeatable, we repeat each experiment for 5 times.
The results are shown in Fig.
We can see our method performs better in most stages for both
CIFAR-10 and CIFAR-100.
The reason is that our method can utilize the confidence information contained in unlabeled training samples by inspecting training consistency.
This advantage is more apparent on CIFAR-100 because the shortage of training samples for each class is more severe for CIFAR-100.
Through active learning experiment, we can conclude that the samples selected by confidence estimates generated by our method are more effective for the following active learning stages.
C ONCLUSION
In this paper, we study the validation of training consistency in estimating the model confidence of unlabeled samples.
Both t-SNE visualization and AURC (E-AURC) evaluation suggest that it generates reliable confidence estimates for both labeled and unlabeled samples with minor labeled samples.
We propose consistency ranking loss to make the model confidence output be consistent with training consistency on labeled and unlabeled samples.
We demonstrate the efficacy of consistency ranking loss by mathematical proof and empirical results.
The superiority of our method on image classification and medical image segmentation tasks suggest wide application prospect of our method.
The confidence estimates generated by our method also work for querying high quality training images and achieve better performance than other baselines in active learning.
Reproducibility Statement: We provide the necessary experimental details in Sec. 4.
More specifically, the implementation details of image classification tasks are provided in Sec. 4.1.
Sec. 4. contains the implementation details for image segmentation task.
The implementation details of active learning tasks are provided in Sec.
The details of baseline methods are described in Append.
A.1.
The details of pairing strategy are in Append.
A.2.
The details of evaluation metrics
AURC and E-AURC are included in Sec. 3.2.
A A PPENDIX
Append.
A.1 shows the implementation details of confidence estimation baselines.
Append.
A.2 shows the details of mini-batch data pairing strategy.
Append.
A.3 shows a detailed proof of Thm.
Append.
A.4 shows the full confidence estimation results on ISIC
Append.
A.5 shows the full confidence estimation results on CIFAR-10 and CIFARAppend.
A.6 shows the pseudo code of training model using our method.
Append.
A.7 shows the other possible ways for extracting consistency information.
A. I MPLEMENTATION DETAILS OF CONFIDENCE ESTIMATION BASELINES
For image classification, to make sure all baselines are fully converged, We train baseline models for 2700, 1200, 600, 300 epochs on 1/20 (2500), 1/10 (5000), 1/5 (10000), 1 (50000) labeled training images.
The mini-batch size is 128.
The initial learning rate is 0.1 and shrinks a factor of 10 at 1 / 2 × number of epochs and 5 / 6 × number of epochs .
As for medical image segmentation (ISIC2017), we train all baseline models for 1500, 700, 300, 200 epochs on 1/16 (125), 1/8 (250), 1/4 (500), 1 (2000) labeled training images.
The learning rate is 0.0001 all the time.
The mini-batch size of all baselines is 64.
A. D ETAILS OF MINI - BATCH DATA PAIRING STRATEGY
The mini-batch data are constructed by labeled samples D l = { ( x i , y i ) } bi =1 and unlabeled samples
D u = { ( x ui , y ui ) } di =1 .
We define x [ b ] = x 0 , x
[ i ] = x i +1 , i = 1 , . . .
, b − 1 and x u
[ d ] = x u 0 , x u
[ i ] = x ui +1 , i = 1 , . . .
, d .
In practice, we use empirical consistency ranking loss, which is defined by
L con =
b (cid:88) s
= sign( c s − c [ s ] )(( c s − c [ s ] ) − ( κ s − κ [ s ] ))
+ d (cid:88) s
= sign( c us − c u [ s ] ) (( c us − c u [ s ] ) − ( κ us − κ u [ s ] )) where c s = c ( x s ) , c [ s ] = c ( x [ s ] ) , κ s = κ ( x s ) , κ [ s ] = κ ( x [ s ] ) , c
us = c ( x us ) , c u [ s ] =
c ( x u [ s ] ) , κ us = κ ( x us ) , κ u [ s ] = κ
( x u [ s ] ) .
A. D ETAILED PROOF
Given R κ , we define ranking set H κ =
{ R κ ( x ) : ˆ y =
y, ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } .
To make sure the duplicate value in C m won’t affect the proof result, we stipulate that beside the rule above training consistency ranking function R c also need to minimize
[Equation]
where τ is the collection of the bijective function between the elements of H κ and H c .
Satisfying two conditions above, we can define R c as an one-to-one function.
Given R c , we introduce a set D c = {| c ( R − 1 c ( i ))
− c ( R − 1 c ( j )) | ; i = γ m ( R κ ( x )) , j = γ m ( R κ ( x ))
+ sign ( R κ ( x ) − γ m
( R κ ( x )))) , x ∈
T, R − 1 c ( j ) ̸∈ T ( f |D m ) } , where γ m = arg [Equation]
∈
[Equation] ) (cid:80) x ∈ T | R κ ( x ) − γ
( R κ ( x )) | , sign is a sign function and c ( x i )
=
c i , x i ∈
X m .
To answer this question, we prove that consistency training loss is an upper bound of the absolute difference between the E-AURC of training consistency and confidence estimator on given dataset
D m =
( X m , Y m ) and training consistency C m :
[Equation]
Proof.
Let T κ ( i ) be the set constituted by all the correct predicted samples with confidence less or equal than R − 1 κ ( i ) , T κ ( i ) =
{ x : R κ ( x ) ≤ i, ˆ y =
y, ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } .
T c ( i ) is the corresponding set introduced by training consistency c and T c ( i ) =
{ x : R c ( x ) ≤ i, ˆ y =
y, ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } .
We define H c ( f, κ |D m )
=
(cid:80) mi =1 1 (# T κ ( i ) ̸
= # T c ( i )) and H kc ( f, κ |D m ) =
(cid:80) ki =1 1 (# T κ ( i ) ̸
= # T c ( i )) .
It can be proved that :
[Equation]
There is a fact that ∀ i ∈ { 1 , 2 , . . .
, m } if 1 (# T κ ( i ) =
# T c ( i )) , than ˆ r ( f, g R − 1 κ ( i ) |D m )
=
ˆ r ( f, g R − 1 c ( i )
|D m )
, so
[Equation]
We also have F r ( f, c |D m )
= (cid:80) mi =1 (cid:81) mj
=1 1 ( R c ( x j ) ≤ i ) · 1 (ˆ y j = y j ) , where ˆ y j = [Equation] f ( x j ) i and ( x j , y j ) ∈
D m .
Here we define K ( f, c |D m ) =
# T ( f |D m ) − F r ( f, c |D m ) , which is the number of correct predicted samples with lower training consistency than at least one incorrect sample.
In the next part, we will prove that:
[Equation]
We first discuss the case when K ( f, c |D m ) = 0 .
This means that the correct classified samples should have higher training consistency than the misclassified.
Let C t be the set constructed by the consistency event count of correct classified samples, C t = { c ( x ) :
ˆ y
=
y, ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } and C f be the set constructed by the consistency event count of false classified samples, C f
= { c ( x ) : ˆ y ̸
= y, ˆ y = arg max i ∈Y f
x ) i , ( x, y ) ∈
D m } .
It is not difficult to tell that min D c = | min C t − max C f | under the case K ( f, c
|D m ) = 0 .
Because
D c = { min C t − max C f } when K ( f, c |D m ) = 0 .
Here we have:
[Equation]
The details of above formula please take a look at Theorem.
We should notice that the conclusion above does not rely on the data size of D m .
Here we define the ranking function R T ( · , κ |D m ) on
T ( f |D m ) , R T ( x, κ |D m ) ∈ { 1 , 2 , . . .
, # T ( f |D m ) } , ∀ x ∈ T ( f |D m ) and if R κ ( x i ) <
R κ ( x j ) than
R T ( x i , κ |D m ) <
R T ( x j , κ |D m ) , ∀ x i , x j ∈ T ( f |D m ) .
We assume the result stands when K ( f, c |D m ) = n .
Than when K ( f, c |D m ) = n + 1 , we start the discussion with the situation R − 1 T (# T ( f |D m ) , κ |D m ) =
R − 1 T (# T ( f |D m ) , c
|D m ) = x ∗ .
According to assumption here, we can get that
[Equation]
There is a fact that H min { R κ ( x ∗ ) ,R c ( x ∗ ) } c ( f, κ |D m / { x ∗ } ) =
H min { R κ ( x ∗ ) ,R c ( x ∗ ) } c ( f, κ |D m ) .
Since
H c ( f, κ |D m / { x ∗ } ) ≥ H min { R κ ( x ∗ ) ,R c ( x ∗ ) } c ( f, κ |D m / { x ∗ } )
We have m
[Equation] [Equation]
We begin below part with the scenarios R κ ( x ∗ ) ≥ R c ( x ∗ ) , than it is straight that there must exist | R κ ( x ∗ )
− R c ( x ∗ ) | samples from F ( f |D m ) constructing a set V = { x ∈ X m | R κ ( x ) <
R κ ( x ∗ ) , R c ( x ) > R c ( x ∗ ) , x ∈ F ( f |D m ) } .
It is easy to see that γ m
( R κ ( x ∗ ))
=
R c ( x ∗ ) and sign ( R κ ( x ∗ ) − γ m
( R κ ( x ∗ )))
= 1 .
So ∀ x ∈ V, | c ( x ∗ ) − c ( x ) | ≥ | c ( x ∗ ) − c
( R − 1 c ( R c ( x ∗ ) + 1)) | ∈
D c .
Therefore, we can get that
[Equation]
As for R κ ( x ∗ ) < R c ( x ∗ ) , the proof is in the same manner.
Summary the result above we get:
[Equation]
If R − 1 T (# T ( f |D m ) , κ |D m ) =
x 1 , R − 1 T (# T ( f |D m ) , c |D m ) =
x 2 and x 1 ̸ = x 2 , we can have one new confidence estimator by switching the confidence value between x 1 and x 2 :
[Equation]
It is not difficult to see that consistency ranking loss introduced by κ t : L tcons ( f, X m |D m ) satisfying
[Equation]
To see this fact, we pay attention to single sample.
First, we discuss sample x ∗ satisfying c ( x ∗ ) <
c ( x 2 ) , we can define a l = c ( x 2 ) − c ( x ∗ ) − ( κ ( x 2 ) − κ ( x ∗ )) , b l = c ( x 1 ) − c ( x ∗ ) − ( κ ( x 1 ) − κ ( x ∗ )) , a t = c ( x 2 ) − c ( x ∗ ) − ( κ t ( x 2 ) − κ t ( x ∗ ))
=
c ( x 2 ) − c ( x ∗ ) − ( κ ( x 1 ) − κ ( x ∗ )) , b t =
c ( x 1 ) − c ( x ∗ ) − ( κ t ( x 1 ) − κ t ( x ∗ ))
=
c ( x 1 ) − c ( x ∗ ) − ( κ ( x 2 ) − κ ( x ∗ )) .
Apparently, b l > a t , b l > b t , a t > a l , b t > a l and a l + b l − ( a t + b t ) = 0 .
Here we define function g ( x ) = max { 0 , x } .
It is straight that g ( a l ) + g ( b l ) ≥ g ( a t )
+ g ( b t ) .
We discuss this by different situations: 1) if b l ≤ 0 , then g ( a l ) + g ( b l ) =
g ( a t ) + g ( b t ) = 0 ; 2) if a t ≤ 0 , b t > 0 , since b l > b t , we have g ( a l ) + g ( b l ) =
g ( b l ) =
b l > b t = g ( b t ) =
g ( a t ) + g ( b t ) ; 3) if b l , a t , b t > 0 , a l < 0 , then g ( a l ) + g ( b l ) ≥ a l + b l ≥ a t + b t
=
g ( a t )
+ g ( b t ) .
As for other cases, they are easy to get.
Here we have l ( x ∗ )
=
g ( a l ) + g ( b l ) and l t ( x ∗ ) =
g ( a t ) + g ( b t ) , so l ( x ∗ ) − l t ( x ∗ ) ≥ 0 .
As for c ( x 1 ) ≥ c ( x ∗ ) ≥ c ( x 2 ) , we have a l = c ( x 1 ) − c ( x ∗ ) − ( κ ( x 1 ) − κ ( x ∗ )) , b l = c ( x ∗ ) − c ( x 2 ) − ( κ ( x ∗ ) − κ ( x 2 )) , a t =
c ( x 1 ) − c ( x ∗ ) − ( κ ( x 2 ) − κ ( x ∗ )) , b t =
c ( x ∗ ) − c ( x 2 ) − ( κ ( x ∗ ) − κ ( x 1 )) .
Apparently a l ≥ a t and b l ≥ b t , so we have g ( a l ) + g ( b l ) ≥ g ( a t )
+ g ( b t ) .
Also l ( x ∗ ) ≥ l t ( x ∗ ) .
As for c ( x ∗ )
> c ( x 1 ) , the proof is the same as the situation c ( x ∗ ) <
c ( x 2 ) .
If x ∗ = x 1 , than l ( x ∗ )
=
c ( x 1 ) − c ( x 2 ) − ( κ ( x 1 ) − κ ( x 2 )) and l t ( x ∗ ) =
c ( x 1 ) − c ( x 2 ) − ( κ ( x 2 ) − κ ( x 1 )) .
It is straight that l ( x ∗ ) ≥ l t ( x ∗ ) .
Summary the result above, we can know that L cons ( f, X m | C m ) − L tcons ( f, X m | C m )
=
(cid:80) x
[Equation]
≥ So the proof stands for case K ( f, c |D m )
= n + 1 , under the assumption it is true while
K ( f, c |D m )
= n .
Let T s be the set constituted by all the correct predicted samples,
T s = { x : l (ˆ y, y ) = 0 , ˆ y = arg max i ∈Y f
x ) i , ( x, y ) ∈
D m } .
T rank =
{ R κ ( t ) : t ∈ T s } is the corresponding ranking set.
We also define F s to be the wrong predicted set, which is F s = { x : l (ˆ y, y )
= 1 , ˆ y = arg max i ∈Y f ( x ) i , ( x
, y ) ∈
D m } and F rank =
{ R κ ( d ) : d ∈ F s } .
Theorem 2.
Given dataset with m samples D m , training consistency C m , confidence estimator κ and classifier f , we have
[Equation]
Proof.
In this proof, we don’t consider the situation when f is an optimal classifier or κ is a perfect confidence estimator (correctly predicted samples have higher confidence that the wrong ones).
Because for those two cases, The E-AURC difference will always be 0.
Here we define F r ( f, κ |D m )
=
min F rank , T r ( f, κ |D m ) =
max T rank .
It is straight
that H c ( f, κ |D m )
=
T r ( f, κ |D m ) − F r ( f, κ |D m )
when K ( f, c
|D m ) = 0 .
First we show that ∀ x i , x j ∈ X m , if κ i > κ s , c i < c s , l (ˆ y s , y s )
= 1 and l (ˆ y i , y i )
= 0 , than | c
s − c
i | ≥ (min C t − max C f ) .
It is apparent that c s ∈ C f and c i ∈ C t , so we have c s ≥ min C f
and c i ≤ max C t .
This inequation thus stands.
So far we have
[Equation]
Next, we prove that for any confidence score function κ , there is a fact that
[Equation]
Here we define K m = { x ∈ X m : R κ ( x ) > F r ( f, κ |D m ) , l (ˆ y, y ) = 0 } .
If # K m = 0 , than ∀ x i , x j ∈ X m , l ( ˆ y i , y i ) = 1 , l ( ˆ y j , y j ) = 0 , we have R κ ( x j ) <
R κ ( x i ) .
This property makes κ under this condition an optimal confidence estimator and T r ( f, κ |D m ) <
F r ( f, κ |D m ) .
It is straight that
I ( f, X m |D m ) ≥ 0 .
Therefore, we have I ( f, X m |D m ) ≥ T r ( f, κ |D m ) − F r ( f, κ |D m )
If # K m = 1 , we have K m =
{ x ∗ } , ( x ∗ , y ∗ ) ∈
D m and G m = { x ∈ X m : R κ ( x ) <
T r ( f, κ |D m ) , l (ˆ y, y )
= 1 } .
Since l (ˆ y ∗ , y ∗ ) = 0 , it is clear that R κ ( x ∗ ) ≤ T r ( f, κ |D m ) .
But if R κ ( x ∗ ) < T r ( f, κ |D m ) , we can get x t =
R − 1 κ ( T r ( f, κ |D m )) .
Apparently x t ∈ T s and
R κ ( x t ) >
R κ ( x ∗ ) > F r ( f, κ |D m ) , so x t ∈ K m , which conflicts the fact that # K m = 1 .
We have
R κ ( x ∗ )
=
T r ( f, κ |D m ) .
According to our assumption about training consistency, it is apparent that ∀ x # ∈ G m , ( x # , y # ) ∈ D m , l (ˆ y ∗ , y ∗ ) = 0 , l (ˆ y # , y # )
= 1 , κ ( x ∗ )
< κ ( x # ) and C ( x ∗ ) >
C ( x # ) , where ˆ y ∗ = arg max i ∈Y f ( x ∗ ) i and ˆ y # = arg max i ∈Y f
( x # ) i .
Here we also have # G m ≥
T r ( f, κ |D m ) − F r ( f, κ |D m ) .
To prove this, we assume # G m < T r ( f, κ |D m ) − F r ( f, κ |D m ) .
We have if x ∈ G m , than R κ ( x ) ≥ F r ( f, κ |D m ) , because F r ( f, κ |D m ) is the minimum value of
F rank .
Therefore, if # G m < T r ( f, κ |D m ) − F r ( f, κ |D m ) , there must exist x k ∈ X m such that
F r ( f, κ |D m ) <
R κ ( x k ) <
T r ( f, κ |D m ) and x k ∈ T s , which makes x k ∈ K m .
Apparently x k and x ∗ are two different elements of K m , which conflicts the fact that # K m = 1 .
This is a simple fact that I ( f, X
[Equation]
In the context below, we show that for every k ≥ 1 , if # K m =
k this inequation holds, than when # K m = k + 1 , this inequation also holds.
Otherwise, we assume this inequation holds for any confidence estimator satisfying # K m =
k , but there exists confidence estimator κ f such that I ( f, κ f , X m |D m ) <
T r ( f, κ f |D m ) − F r ( f, κ f |D m ) , K κ f m = { x ∈ X m : R κ f ( x ) >
F r ( f, κ f |D m ) , l (ˆ y, y )
= 0 } and # K κ f m = k
+ 1 .
Here we can have a confidence estimator
[Equation]
t
[Equation]
otherwise
It is apparent that we get κ t by swapping the confidence score of κ f at arg min x ∈ K κfm
R κ f ( x ) and R − 1 κ f ( F ( f, κ f |D m )) , so
# K κ t m =
k and I ( f, κ t , X m |D m ) ≥
T r ( f, κ t |D m ) −
F r ( f, κ t |D m ) .
It is also straight to see that T r ( f, κ t |D m )
=
T r ( f, κ f |D m ) , F r ( f, κ f |D m ) =
F r ( f, κ t |D m ) − 1 .
We also get that I ( f, κ f , X m |D m ) − I ( f, κ t , X m |D m ) ≥ 1 , because
[Table]
I
[Equation] ) ≥ 1 .
So far according to our assumption we have
[Equation]
, κ f , X m |D m ) <
T r ( f, κ f |D m ) − F r ( f, κ f |D m ) = ⇒ I ( f, κ t , X m |D m ) + 1 < T r ( f, κ f |D m ) − F r ( f, κ f |D m ) =
⇒
I ( f, κ t , X m |D m ) + 1 < T r ( f, κ t |D m ) − F r ( f, κ t |D m ) + 1 = ⇒
I ( f, κ t , X m |D m ) <
T r ( f, κ t |D m ) − F r ( f, κ t |D m )
Which leads to a conflict.
A. ISIC2017 CONFIDENCE ESTIMATION RESULTS IN T AB .A. CIFAR-10 AND CIFAR-100 CONFIDENCE ESTIMATION RESULTS IN T AB .A. P SEUDO CODE
Here we provide the pseudo code 1 for the training process of consistency ranking loss.
Here we need to notice that we normalize the training consistency for labeled samples and unlabeled samples in a mini-batch independently. The reason is that training consistency has a bias on labeled training samples, so the labeled samples have higher training consistency than unlabeled ones. There is a mismatch between the training consistency distribution of labeled samples and unlabeled samples. A simple way to overcome this issue is Min-max normalize the consistency of labeled and unlabeled samples separately.
In practice, this will achieve better performance.
A. A DDITIONAL QUALITATIVE RESULTS
In this paper, we use consistency events to quantify the sensitive levels of samples vibrating around the decision boundary in the training process.
It’s worth mentioning that this is not the only way to capture such information.
For example, we define label frequency r yi =
(cid:80) T − 1 t =0 1 { ˆ y ti = y } as a variable to count the number of sample x i being predicted as label y by classifier during training and it is also able to capture training consistency.
We can use the maximum label frequency (L-count) max y ∈Y r y i to quantify the training consistency on sample x i .
Apparently, the higher maximum size.
The best results of experiments are shown in bold.
The values of AURC & E-AURC are multiplied by 10 3 , FPR are multiplied by 10 2 and NLL is multiplied by 10 for clarity
[Table]
label frequency introduces less vacillation in the training process, leading to higher confidence.
Besides, we can also use the label frequency entropy (L-entropy) and margin (L-margin) to summarize consistency information.
We show that our consistency event has superior performance in estimating confidence in terms of the criterion of ordinal ranking.
As shown in Fig.
6, compared with maximum softmax output, training consistency estimate has much better performance in distinguishing correct predictions from wrong ones on unlabeled samples.
This indicates training consistency has the ability to extract valid confidence information without the necessity of ground truth labels.
Furthermore, we also find our consistency event captures the essential of training consistency in a better manner, resulting in better performances across all shown epochs.
Though Correctness also achieves good results, the demand of annotations restricts its application in scenarios with limited labeled data.
For the rest of this paper, training consistency is especially referring to our confidence estimate method.
Input: Dataloader for labeled and labeled training samples
Output: Trained deep model
Definition : uLoader and sLoader denote the dataloader for unlabeled and labeled samples;
D corr is the dictionary, storing the count of correctness for each labeled sample.
D con is the dictionary, storing the count of consistency for each sample.
1: for epoch in range(number of epochs) do 2: siterator = iterator( sloader ) 3: for (uninputs, unlabel-index ) in uLoader do 4: if siterator iteration end then 4: siterator = iterator( sLoader ) 4: update D corr , D con 4: inputs, targets, label-index = next( sLoader ) 5: else 5: inputs, targets, label-index = next( sLoader ) 6: end if 7: corr label =
Normalize ( D corr [ label-index ]) //
We use Min-max normalization here 8: con label =
Normalize ( D con [ label-index ]) //
Extracting training consistency for batch labeled samples 9: con unlabel =
Normalize ( D con [ unlabel-index ]) //
Extracting training consistency for batch unlabeled samples 10: con batch =
Concat ( con label , con unlabel ) 11: labeloutput = model (inputs) 12: unlabeloutput = model (uninputs) 13: output batch =
Concat(labeloutput, unlabeloutput) 14:
L CE =
L CE (labeloutput, targets) 15:
L corr =
L corr ( softmax(labeloutput), corr label ) 16:
L con =
L con (softmax( output batch ), con batch ) 17:
Loss total =
L CE + λ 1 L corr + λ 2 L con 18:
Update model weights 19: end for 20: end for
[Figure]