O N THE R EPRODUCIBILITY OF N EURAL N ETWORK P REDICTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Standard training techniques for neural networks involve multiple sources of randomness, e.g., initialization, mini-batch ordering and in some cases data augmentation.
Given that neural networks are heavily over-parameterized in practice, such randomness can cause churn – disagreements between predictions of the two models independently trained by the same algorithm, contributing to the ‘reproducibility challenges’ in modern machine learning.
In this paper, we study this problem of churn, identify factors that cause it, and propose two simple means of mitigating it.
We ﬁrst demonstrate that churn is indeed an issue, even for standard image classiﬁcation tasks (CIFAR and ImageNet), and study the role of the different sources of training randomness that cause churn.
By analyzing the relationship between churn and prediction conﬁdences, we pursue an approach with two components for churn reduction.
First, we propose using minimum entropy regularizers to increase prediction conﬁdences.
Second, we present a novel variant of co-distillation approach (Anil et al., 2018) to increase model agreement and reduce churn.
We present empirical results showing the effectiveness of both techniques in reducing churn while improving the accuracy of the underlying model.
I NTRODUCTION
Deep neural networks (DNNs) have seen remarkable success in a range of complex tasks, and signiﬁcant effort has been spent on further improving their predictive accuracy .
However, an equally important desideratum of any machine learning system is stability or reproducibility in its predictions.
In practice, machine learning models are continuously (re)-trained as new data arrives, or to incorporate architectural and algorithmic changes.
A model that changes its predictions on a signiﬁcant fraction of examples after each update is undesirable, even if each model instantiation attains high accuracy.
Reproducibility of predictions is a challenge even if the architecture and training data are ﬁxed across different training runs, which is the focus of this paper.
Unfortunately, two key ingredients that help deep networks attain high accuracy — over-parameterization, and the randomization of their training algorithms — pose signiﬁcant challenges to their reproducibility.
The former refers to the fact that
NNs typically have many solutions that minimize the training objective (Neyshabur et al., 2015;
Zhang et al., 2017).
The latter refers to the fact that standard training of NNs involves several sources of randomness, e.g., initialization, mini-batch ordering, non-determinism in training platforms and in some cases data augmentation.
Put together, these imply that NN training can ﬁnd vastly different solutions in each run even when training data is the same, leading to a reproducibility challenge.
The prediction disagreement between two models is referred to as churn (Cormier et al., 2016) 1 .
Concretely, given two models, churn is the fraction of test examples where the predictions of the two models disagree.
Clearly, churn is zero if both models have perfect accuracy – an unattainable goal for most of the practical settings of interest.
Similarly, one can mitigate churn by eliminating all sources of randomness in the underlying training setup.
However, even if one controls the seed used for random initialization and the order of data, inherent non-determinism in the current computation platforms is hard to avoid (see §2.3).
Moreover it is desirable to have stable models with predictions unaffected by such factors in training.
Thus, it is critical to quantify churn, and develop methods that reduce it.
In this paper, we study the problem of churn in NNs for the classiﬁcation setting .
We demonstrate the presence of churn, and investigate the role of different training factors causing it.
Interestingly our experiments show that churn is not avoidable on the computing platforms commonly used in machine learning, further highlighting the necessity of developing techniques to mitigate churn.
We then analyze the relation between churn and predicted class probabilities.
Based on this, we develop a novel regularized co-distillation approach for reducing churn.
Our key contributions are summarized below:
(i) Besides the disagreement in the ﬁnal predictions of models, we propose alternative soft metrics to measure churn.
We demonstrate the existence of churn on standard image classiﬁcation tasks (CIFAR-10, CIFAR-100, ImageNet, SVHN and iNaturalist), and identify the components of learning algorithms that contribute to the observed churn.
Furthermore, we analyze the relationship between churn and model prediction conﬁdences (cf. § 2).
(ii) Motivated from our analysis, we propose a regularized co-distillation approach to reduce churn that both improves prediction conﬁdences and reduces prediction variance (cf. §3).
Our approach consists of two components: a) minimum entropy regularizers that improve prediction conﬁdences (cf. §3.1), and b) a new variant of co-distillation (Anil et al., 2018) to reduce prediction variance across runs.
Speciﬁcally, we use a symmetric KL divergence based loss to reduce model disagreement, with a linear warmup and joint updates across multiple models (cf. §3.2).
(iii)
We empirically demonstrate the effectiveness of the proposed approach in reducing churn and (sometimes) increasing accuracy.
We present ablation studies over its two components to show their complementary nature in reducing churn (cf. §4).
1. R ELATED WORK
Reproducibility in machine learning.
There is a broad ﬁeld studying the problem of reproducible research (Buckheit & Donoho, 1995; Gentleman & Lang, 2007; Sonnenburg et al., 2007; Kovacevic, 2007; Mesirov, 2010; Peng, 2011; McNutt, 2014; Braun & Ong, 2014; Rule et al., 2018), which identiﬁes best practices to facilitate the reproducibility of scientiﬁc results.
Henderson et al.
(2018) analysed reproducibility of methods in reinforcement learning, showing that performance of certain methods is sensitive to the random seed used in the training.
While the performance of NNs on image classiﬁcation tasks is fairly stable (Table 2), we focus on analyzing and improving the reproducibility of individual predictions.
Thus, churn can be seen as a speciﬁc technical component of this reproducibility challenge.
Cormier et al.
(2016) deﬁned the disagreement between predictions of two models as churn.
They proposed an MCMC approach to train an initial stable model A so that it has a small churn with its future version, say model
B. Here, future versions are based on slightly modiﬁed training data with possibly additional features.
In Goh et al.
; Cotter et al.
(2019), constrained optimization is utilized to reduce churn across different model versions.
In contrast, we are interested in capturing the contribution of factors other than training data modiﬁcation that cause churn.
More recently.
Madhyastha & Jain (2019) study instability in the interpretation mechanisms and average performance for deep NNs due to change in random seed, and propose a stochastic weight averaging (Izmailov et al., 2018) approach to promote robust interpretations.
In contrast, we are interested in robustness of individual predictions.
Ensembling and online distillation.
Ensemble methods (Dietterich, 2000; Lakshminarayanan et al., 2017) that combine the predictions from multiple (diverse) models naturally reduce the churn by averaging out the randomness in the training procedure of the individual models.
However, such methods incur large memory footprint and high computational cost during the inference time.
Distillation (Hinton et al., 2015; Bucilua et al., 2006) aims to train a single model from the ensemble to alleviate these costs.
Even though distilled model aims to recover the accuracy of the underlying ensemble, it is unclear if the distilled model also leads to churn reduction.
Furthermore, distillation is a two-stage process, involving ﬁrst training an ensemble
and then distilling it into a single model.
To avoid this two-stage training process, multiple recent works Anil et al.
; Zhang et al.
Lan et al.
; Song & Chai (2018); Guo et al.
(2020) have focused on online distillation , where multiple identical or similar models (with different initialization) are trained while regularizing the distance between their prediction probabilities.
At the end of the training, any of the participating models can be used for inference.
Notably, Anil et al.
(2018), while referring to this approach as co-distillation, also empirically pointed out its utility for churn reduction on the Criteo Ad dataset 2 .
In contrast, we develop a deeper understanding of co-distillation framework as a churn reduction mechanism by providing a theoretical justiﬁcation behind its ability to reduce churn.
We experimentally show that using a symmetric KL divergence objective instead of the cross entropy loss for co-distillation (Anil et al., 2018) leads to lower churn and better accuracy, even improving over the expensive ensembling-distillation approach.
Entropy regularizer.
Minimum entropy regularization was earlier explored in the context of semi-supervised learning (Grandvalet & Bengio, 2005).
Such techniques have also been used to combat label noise (Reed et al., 2015).
In contrast, we utilize minimum entropy regularization in fully supervised settings for a distinct purpose of reducing churn and experimentally show its effectiveness.
1. N OTATION
Multi-class classiﬁcation.
We consider a multi-class classiﬁcation setting, where given an instance x P X , the goal is to classify it as a member of one of K classes, indexed by the set Y (cid:44) r K s .
Let W be the set of parameters that deﬁne the underlying classiﬁcation models.
In particular, for w P W , the associated classiﬁcation model f p¨ ; w q : X Ñ ∆ K maps the instance x
P X in the K -dimensional simplex ∆ K Ă R K .
Given f p x ; w q , x is classiﬁed as element of class ˆ y x ; w such that
[Equation]
This gives the misclassiﬁcation error (cid:96)
[Equation]
, where y is the true label for x .
Let P
X , Y be the joint distribution over the instance and label pairs.
We learn a classiﬁcation model by minimizing the risk for some valid surrogate loss (cid:96) of the misclassiﬁcation error (cid:96) 01 :
L `
[Equation]
E X , Y “ (cid:96) p Y , f p X ; w qq ‰ .
In practice, since we have only ﬁnite samples S P p
X ˆ Y q n , we minimize the corresponding empirical risk.
[Equation]
C HURN : MEASUREMENT AND ANALYSIS
In this section we deﬁne churn, and demonstrate its existence on CIFAR and ImageNet datasets.
We also propose and measure alternative soft metrics to quantify churn, that mitigate the discontinuity of churn.
Subsequently, we examine the inﬂuence of different factors in the learning algorithm on churn.
Finally, we present a relation between churn and prediction conﬁdences of the model.
We begin by deﬁning churn as the expected disagreement between the predictions of two models (Cormier et al., 2016).
Deﬁnition 1 (Churn between two models) .
Let w 1 , w 2 P W deﬁne classiﬁcation models f p¨ ; w 1 q , f p¨ ; w 2 q : X Ñ ∆ K , respectively.
Then, the churn between the two models is
[Equation]
where ˆ Y x ; w 1 (cid:44) arg max j P Y f p x ; w 1 q j and ˆ Y x ; w 2 (cid:44) arg max j P Y
f p x ; w 2 q j .
Note that if the models have perfect test accuracy, then their predictions always agree with the true label, which corresponds to zero churn.
In practice, however, this is rarely the case.
The following rather straightforward result shows that churn is upper bounded by the sum of the test error of the models.
See Appendix B for the proof.
We note that a similar result was shown in Theorem 1 of
Madani et al.
Lemma 1.
Let P Err , w 1 (cid:44) P X , Y r Y ‰ ˆ Y X ; w 1 s and P Err , w 2 (cid:44) P X , Y r Y ‰ ˆ Y X ; w 2 s be the misclassiﬁca- tion error for the models w 1 and w 2 , respectively.
Then, Churn p w 1 , w 2 q ď P Err , w 1 ` P Err , w 2 . [Table]
Despite the worst-case bound in Lemma 1, imperfect accuracy does not preclude the absence of churn.
As the best-case scenario, two imperfect models can agree on the predictions for each example (whether correct or incorrect), causing the churn to be zero.
For example, multiple runs of a deterministic learning algorithm produce models with zero churn, independent of their accuracy.
This shows that, in general, one cannot infer churn from test accuracy, and understanding churn of an algorithm requires independent exploration.
2. D EMONSTRATION OF CHURN
In principle, there are multiple sources of randomness in standard training procedures that make NNs susceptible to churn.
We now verify this hypothesis by showing that, in practice, these sources indeed result in a non-trivial churn for NNs on standard image classiﬁcation datasets.
Table 2 reports churn over CIFAR-10, CIFAR-100 and ImageNet, with ResNets (He et al., 2016a;b) as the underlying model architecture.
As per the standard practice in the literature, we employ SGD with momentum and step-wise learning rate decay to train ResNets.
Note that the models obtained in different runs indeed disagree in their predictions substantially.
To measure if the disagreement comes mainly from misclassiﬁed examples, we measure churn on two slices of examples, correctly classiﬁed (Churn correct) and misclassiﬁed (Churn incorrect).
We notice churn even among the examples that are correctly classiﬁed, though we see a relatively higher churn among the misclassiﬁed examples.
This raises two natural questions: (i) do the prediction probabilities differ signiﬁcantly, or is the churn observed in Table 2 mainly an artifact of the argmax operation in (1), when faced with small variation in prediction probabilities across models?, and (ii) what causes such a high churn across runs?
We address these questions in the following sections.
2. S URROGATE CHURN
Churn in Table 2 could potentially be a manifestation of applying the arg max operation in (1), despite the prediction probabilities being close.
To study this, we consider the following soft metric to measure churn, which takes the models’ entire prediction probability mass function into account.
Deﬁnition 2 (Surrogate churn between two models) .
Let f p¨ ; w 1 q , f p¨ ; w 2 q : X Ñ ∆ K be two models deﬁned by w 1 , w 2 P W , respectively.
Then, for α P R ` , the surrogate churn between the models is
[Equation]
α ››››  .
As α Ñ 8 , this reduces to the standard Churn deﬁnition in (3).
In Table 2 we measure SChurn α for α “ 1 , which shows that even the distance between prediction probabilities is signiﬁcant across runs.
Thus, the churn observed in Table 2 is not merely caused by the discontinuity of the arg max , but it indeed highlights the instability of model predictions caused by randomness in training.
2. W HAT CAUSES CHURN ?
We now investigate the role played by randomness in initialization, mini-batch ordering, data augmentation, and non-determinism in the computation platform in causing churn.
Even though these aspects are sources of randomness in training, they are not necessarily sources of churn .
We experiment by holding some of these factors constant and measure the churn across 5 different runs.
We report results in Table 1, where the ﬁrst column gives the baseline churn with no factor held constant across runs.
Model initialization is a source of churn.
Simply initializing weights from identical seeds (odd columns in Table 1) can decrease churn under most settings.
Other sources of randomness include mini-batch ordering and dataset augmentation.
To hold the former constant, we ensure that every model iterates over the dataset in the same order; to hold the latter constant, we remove all augmentation during training.
These two aspects contribute to randomness between training runs, but ﬁxing them does not decrease churn; rather, they appear to have a regularizing affect on our hardware platform.
Finally, there is churn resulting from an unavoidable source during training, the non-determinism in computing platforms used for training machine learning models, e.g., GPU/TPU (Morin & Willetts, 2020; PyTorch, 2019; Nvidia, 2020).
The experiments in Table 1 were run on TPU.
Even when all other aspects of training are held constant (rightmost column), model weights diverge within 100 steps (across runs) and the ﬁnal churn is signiﬁcant.
We veriﬁed that this is the sole source of non-determinism: models trained to 10,000 steps on CPUs under these settings had identical weights.
These experiments underscore the importance of developing and incorporating churn reduction strategies in training.
Even with extreme measures to eliminate all sources of randomness, we continue to observe churn due to unavoidable hardware non-determinism.
2. R EDUCING CHURN : R ELATION TO PREDICTION PROBABILITIES
We now focus on the main objective of this paper – churn reduction in NNs.
Many factors that have been shown to cause churn in § 2.3 are crucial for the good performance of NNs and thus cannot be controlled or eliminated without causing performance degradation.
Moreover, controlling certain factors such as hardware non-determinism is extremely challenging, especially in the large scale distributed computing setup.
Towards reducing churn, we ﬁrst develop an understanding of the relationship between the distribution of prediction probabilities of a model and churn.
Intuitively, encouraging either larger prediction conﬁdence or smaller variance across multiple training runs should result in reduced churn.
To formalize this intuition, let us consider the prediction conﬁdence realized by the classiﬁcation model f p¨ ; w q on the instance x P X :
[Equation]
where ˆ y x ; w is the model prediction in (1).
Note that γ x ; w denotes the difference between the probabilities of the most likely and the second most likely classes under the distribution f p x ; w q , and is not the same as the standard multi-class margin (Koltchinskii et al., 2001): it captures how conﬁdent the model f p¨ ; w q is about its prediction ˆ y x ; w , without taking the true label into account.
The following result relates the prediction conﬁdence to churn.
See Appendix B for the proof.
Lemma 2.
Let γ x ; w 1 and γ x ; w 2 be the prediction conﬁdences realized on x P X by the classiﬁcation models f p¨ ; w 1 q and f p¨ ; w 2 q , respectively.
Then,
[Equation]
[Equation]
Here, D L1 p
f p x ; w 1 q , f p x ; w 2 qq “ ř j P Y | f p x ; w 1 q j ´ f p x ; w 2 q j | measures the L 1 distance.
Lemmaestablishes that, for a given instance x , the churn between two models f p¨ ; w 1 q and f p¨ ; w 2 q becomes less likely as their conﬁdences γ x ; w 1 and γ x ; w 2 increase.
Similarly, the churn becomes smaller when the difference between their prediction probabilities, D L1 p
f p x ; w 1 q , f p x ; w 2 qq , decreases.
R EGULARIZED CO - DISTILLATION FOR CHURN REDUCTION
In this section we present our approach for churn reduction.
Motivated by Lemma 2, we consider an approach with two complementary techniques for churn reduction: 1) We ﬁrst propose entropy based regularizers that encourage solutions w P W with large prediction conﬁdence γ x ; w for each instance x . 2)
We next employ co-distillation approach with novel design choices that simultaneously trains two models while minimizing the distance between their prediction probabilities.
Note that the entropy regularizers themselves cannot actively enforce alignment between the prediction probabilities across multiple runs as the resulting objective does not couple multiple models together.
On the other hand, the co-distillation in itself does not affect the prediction conﬁdence of the underlying models (as veriﬁed in Figure 2).
Thus, our combined approach promotes both large model prediction conﬁdence and small variance in prediction probabilities across multiple runs.
3. M INIMUM ENTROPY REGULARIZERS
Aiming to increase the prediction conﬁdences of the trained model t γ x ; w u x P X , we propose novel training objectives that employ one of two possible regularizers based on: (1) entropy of the model prediction probabilities; and (2) negative symmetric KL divergence between the model prediction probabilities and the uniform distribution.
Both regularizers encourage the prediction probabilities to be concentrated on a small number of classes, and increase the associated prediction conﬁdence.
Entropy regularizer.
Recall that the standard training minimizes the risk L p w q .
Instead, for α P r 0 , 1 s and S “ tp x i , y i qu i Pr n s , we propose to minimize the following regularized objective to increase the prediction conﬁdence of the model 3 .
[Equation]
where H ` f p x ; w q ˘ “ ´ ř j Pr K s f p x ; w q j log f p x ; w q j denotes the entropy of predictions f p x ; w q .
Symmetric KL divergence regularizer.
Instead of encouraging the low entropy for the prediction probabilities, we can alternatively maximize the distance of the prediction probability mass function from the uniform distribution as a regularizer to enhance the prediction conﬁdence.
In particular, we utilize the symmetric KL divergence as the distance measure.
Let Unif P ∆ K be the uniform distribution.
Thus, given n samples S “ tp x i , y i qu i Pr n s , we propose to minimize
[Equation]
where SKL
[Equation]
As discussed in the beginning of the section, the intuition behind utilizing these regularizers is to encourage spiky prediction probability mass functions, which lead to higher prediction conﬁdence.
The following result supports this intuition in the binary classiﬁcation setting.
See Appendix B for the proof.
As for multi-class classiﬁcation, we empirically verify that the proposed regularizers indeed lead to increased prediction conﬁdences (cf.
Figure 2).
Theorem 3.
Let f p¨ ; w q and f p¨ ; w 1 q be two binary classiﬁcation models.
For a given x P X , if we have H p f p x ; w qq ď H p f p x ; w 1 qq or SKL p
f p x ; w qq ě SKL p f p x ; w 1 qq , then γ x ; w ě γ x ; w 1 .
Note that the effect of these regularizers is different from increasing the temperature in softmax while computing f p x ; w q .
Similar to max entropy regularizers (Pereyra et al., 2017), they are independent of the label, a crucial difference that allows them to reduce churn even among misclassiﬁed examples.
3. C O - DISTILLATION
As the second measure for churn reduction, we now focus on designing training objectives that minimize the variance among the prediction probabilities for the same instance across multiple runs of the learning algorithm.
Towards this, we consider novel variants of co-distillation approach (Anil et al., 2018).
In particular, we employ co-distillation by using symmetric KL divergence (Co-distill SKL ), with a linear warm-up, to penalize the prediction distance between the models instead of the popular step-wise cross entropy loss (Co-distill CE ) used in Anil et al.
(2018) (see § C).
We ﬁrst motivate the proposed objective for reducing churn using Lemma 2.
Recall from Lemma 2 that, for a given instance x , churn across two models f p¨ ; w 1 q and f p¨ ; w 2 q decreases as the distance between their prediction probabilities D L1 p
f p x ; w 1 q , f p x ; w 2 qq becomes smaller.
Motivated by this, given training samples S “ tp x i , y i qu i Pr n s , one can simultaneously train two models corresponding to w 1 , w 2 P W by minimizing the following objective and keeping either [Table] of the two models as the ﬁnal solution.
[Equation]
From Pinsker’s inequality, D L1 p f p x i ; w 1 q , f p x i ; w 2 qq ď a
[Equation]
p f p x i ; w 1 q|| f p x i ; w 2 qq .
Thus, one can alternatively utilize the following objective.
[Equation]
; w 2 q ˘ , (9) where SKL ` f p x ; w 1 q , f p x ; w 2 q ˘ “ KL ` f p x ; w 1 q|| f p x ; w 2 q ˘ ` KL ` f p x ; w 2 q|| f p x ; w 1 q ˘ denotes the symmetric KL-divergence between the prediction probabilities of the two models.
In what follows, we work with the objective in (9) as we observed this to be more effective in our experiments, leading to both smaller churn and higher model accuracy.
In addition to the co-distillation objective, we introduce two other changes to the training procedure: joint updates and linear rampup of the co-distillation loss.
We discuss these differences in § C.
Regularized co
distillation.
In this paper, we also explore regularized co-distillation, where we utilize the minimum entropy regularizes (cf. § 3.1) in our co-distillation framework.
We note that the best results for the combined approach are achieved when we use a linear warmup for the regularizer coefﬁcient as well.
Note that combining the Co-distill SKL objective with an entropy regularizer is not the same as using the cross entropy loss, due to the use the different weights for each method, and rampup of the regularizer coefﬁcients.
This distinction is important in reducing churn ( Table 2).
E XPERIMENTS
We conduct experiments on 5 different datasets, CIFAR-10, CIFAR-100, ImageNet, SVHN and iNaturalist 2018.
We use LeNet5 for experiments on SVHN, ResNet-56 for experiments on CIFAR10 and CIFAR-100, and ResNet-v2-50 for experiments on ImageNet and iNaturalist.
We use the
The plot clearly shows that the proposed variant achieves a better tradeoff, compared to Co-distill CE . [Figure] [Table]
[Figure]
[Table] [Figure] [Table] same hyperparameters for all the experiments on a dataset.
We use the Cross entropy loss and the Momentum optimizer.
For complete details we refer to the Appendix A.
Top k regularizer.
For problems with a large number of outputs, e.g. ImageNet, it is not required to penalize all the predictions to reduce churn.
Recall that prediction conﬁdence is only a function of the top two prediction probabilities.
Hence, on ImageNet, we consider the top k variants of the proposed regularizers, and penalize the entropy/SKL only on the top k predictions, with k “ 10 .
Accuracy and churn.
In Table 2, we present the accuracy and churn of models trained with the minimum entropy regularizers, co-distillation and their combination.
For each dataset we report the values for the best α and β .
We notice that our proposed methods indeed reduce the churn and
Schurn.
While both minimum entropy regularization and co-distillation are consistently effective in reducing Schurn, their effect on churn varies, potentially due to the discontinuous nature of the churn.
Also note that the proposed methods reduce churn both among the correctly and incorrectly classiﬁed examples.
Our co-distillation proposal, Co-distill SKL consistently showed a signiﬁcant reduction in churn, and has better performance than Co-distill CE (Anil et al., 2018) that is based on the cross-entropy loss, showing the importance of choosing the right objective for reducing churn.
We present additional comparison in Figure 2, showing churn and Top-1 error on ImageNet, computed for [Table] different values of β (cf.
Finally, we achieve an even further reduction in churn by the combined approach of entropy regularized co-distillation.
Interestingly, the considered methods also improve the accuracy, showing their utility beyond churn reduction.
Ensembling.
We also compare with the ensembling-distillation approach (Hinton et al., 2015) in
Table 2, where we use a 2 teacher ensemble for distillation.
We show that the proposed methods consistently outperform ensembling-distillation approach, despite having a lower training cost.
Ablation.
We next report results of our ablation study on the entropy ( top 10 ) regularizer coefﬁcient α , and the Co-distill SKL coefﬁcient β in Figure 1.
While both methods improve accuracy and churn, the prediction entropy shows different behavior between these methods.
While entropy regularizers improve churn by reducing the entropy, co-distillation reduces churn by reducing the prediction distance between two models, resulting in an increase in entropy.
This complementary nature explains the advantage in combining these two methods (cf.
Table 2).
We next present ablation studies on weight decay regularization in Table 3, showing that the proposed approaches improve churn on models trained both with and without weight decay.
Fixed initialization.
Earlier in Table 1, we showed that ﬁxing the initialization of ﬁve runs of the baseline model lowers churn from 5 .
81 ˘ 0 .
08 to 5 .
56 ˘ 0 .
12 for CIFAR-10 on ResNet-56.
However, our proposed Co-distill SKL model is more effective, reducing churn to 4 .
29 ˘ 0 .
14 (see Table 2).
Using ﬁxed initialization on this model does not signiﬁcantly affect churn ( 4 .
24 ˘ 0 . 07 ).
D ISCUSSION
Connection to label smoothing and calibration.
Max entropy regularizers, such as label smoothing , are often used to reduce prediction conﬁdence (Pereyra et al., 2017; Müller et al., 2019) and provide better prediction calibration.
The minimum entropy regularizers studied in this paper have the opposite effect and increase the prediction conﬁdence, whereas co-distillation reduces the prediction conﬁdences.
To study this more concretely we measure the effect of churn reduction methods on the calibration.
We compute Expected Calibration Error (ECE) (Guo et al., 2017) for methods considered in this paper and report the results in Table 4. We notice that the minimum entropy regularizers, predictably, increase the calibration error, whereas the propose co-distillation approach (Co-distill SKL ) signiﬁcantly reduces the calibration error. We notice that our joint approach is competitive with the ensemble distillation approach on CIFAR datasets but incurs higher error on ImageNet. Developing approaches that jointly optimize for churn and calibration is an interesting direction of future work.
A E XPERIMENTAL SETUP
Architecture:
For our experiments we use the standard image classiﬁcation datasets CIFAR and
ImageNet.
For our experiments with CIFAR, we use ResNet-32 and ResNet-56 architectures, and for
ImageNet we use ResNet-v2-50.
These architectures have the following conﬁguration in terms of p n layer , n ﬁlter , stride q , for each ResNet block:
• ResNet-32-
• ResNet-56- [(9, 16, 1), (9, 32, 2), (9, 64, 2)] • ResNet-v2-50- [(3, 64, 1), (4, 128, 2), (6, 256, 2), (3, 512, 2)],
where ‘stride’ refers to the stride of the ﬁrst convolution ﬁlter within each block.
For ResNet-v2-50, the ﬁnal layer of each block has 4 ˚ n ﬁlter ﬁlters.
We use L2 weight decay of strength 1 e - 4 for all experiments.
Learning rate and Batch size: For our experiments with CIFAR, we use SGD optimizer with a 0.
Nesterov momentum.
We use a linear learning rate warmup for ﬁrst 15 epochs, with a peak learning rate of 1.0.
We use a stepwise decay schedule, that decays learning rate by a factor of 10, at epoch numbers 200, 300 and 400.
We train the models for a total of 450 epochs.
We use a batch size of 1024.
For the ImageNet experiments, we use SGD optimizer with a 0.9 momentum.
We use a linear learning rate warmup for ﬁrst 5 epochs, with a peak learning rate of 0.8.
We again use a stepwise decay schedule, that decays learning rate by a factor of 10, at epoch numbers 30, 60 and 80.
We train the models for a total of 90 epochs.
We use a batch size of 1024.
Data augmentation: We use data augmentation with random cropping and ﬂipping.
Parameter ranges:
For our experiments with the min entropy regularizers, we use α P r 0 .
5 s , and report the α corresponding to best churn in Table 2.
For our experiments with the co-distillation approach, we use β P r 0 .
08 s , and report the β corresponding to best churn in Table 2.
For our experiments on entropy regularized co-distillation, we use a linear warmup of the regularizer coefﬁcient.
We use the same range for α as described above, and use only the best β value from the earlier experiments.
Finally, we run our experiments using TPUv3.
Experiments on CIFAR datasets ﬁnish in an hour, experiments on ImageNet take around 6-8 hours.
Hyper parameters for Table 2: Finally we list in Table 5 the best hyperparameters used to obtain the results in Table 2.
For reference we also include the accuracy and the churn again for all methods.
B P ROOFS
Proof of Lemma 1 .
Recall from Deﬁnition 1 that
[Equation]
[Equation]
where p i q and p ii q follow from the union bound and the fact that P r A s ď P r B s whenever A Ď B , respectively.
Proof of Lemma 2 .
Note that, for any j ‰ ˆ y x,w 1 ,
[Equation]
Similarly
[Equation]
for any j ‰ ˆ y x
[Equation]
w
[Equation]
we can establish that
[Equation]
[Equation]
Note that experiencing churn between the two models on x is equivalent to
[Equation]
where p i q follows from (11) and (12).
Now, (13) implies that
[Equation]
Proof of Theorem 3 .
Let the prediction for a given x be p “ f p x ; w q .
W.L.O.G. let p ě 1 ´ p .
The prediction conﬁdence is then
[Equation]
Entropy case.
Now we can write entropy in terms of the prediction conﬁdence (cf.
[Equation]
Now the gradient of g p¨q is ∇
[Equation]
¯ , which is less than 0 for γ x,w P r 0 , 1 s .
Hence, the function g p γ x,w q is a decreasing function for inputs in range r 0 , 1 s .
Hence, if g p γ x,w 1 q ě g p γ x,w q implies γ x,w 1 ď γ x,w .
Symmetric-KL divergence case.
Recall that
[Equation]
For binary classiﬁcation this reduces to,
[Equation]
By using (15) and (16), we can rewrite this function in terms of γ x,w as:
[Equation]
Now notice that both the above terms are an increasing function of γ x,w , as log is an increasing function, and g p γ x,w q is a decreasing function.
Hence SKL p f p x ; w q , Unif q ě SKL p f p x ; w 1 q , Unif q implies γ x ; w ě γ x ; w 1 .
C D ESIGN CHOICES IN THE PROPOSED VARIANT OF CO - DISTILLATION
Here, we discuss two important design choices that are crucial for successful utilization of codistillation framework for churn reduction while also improving model accuracy.
Joint vs. independent updates.
Anil et al.
(2018) consider co-distillation with independent updates for the participating models.
In particular, with two participating models, this corresponds to independently solving the following two sub-problems during the t -th step.
[Equation]
where w t ´
Tand w t ´ Tcorresponds to earlier checkpoints of two models being used to compute the model disagreement loss component at the other model.
Note that since w t ´ Tand w t ´ Tare not being optimized in (17a) and (17b), respectively.
Thus, for each model, these objectives are equivalent to regularizing its empirical loss via a cross-entropy terms that aims to align its prediction probabilities with that of the other model.
In particular, the optimization steps in (17) are equivalent to
[Equation]
where H ` f p x ; w 1 q , f p x ; w 2 q ˘ “ ´ ř j f p x ; w 1 q j log
f
p x ; w 2 q j denotes the cross-entropy between the probability mass functions
f p x ; w 1 q and f p x ; w 2 q .
In our experiments, we verify that this independent updates approach leads to worse results as compared to jointly training the two models using the objective in (9).
Deferred model disagreement loss component in co-distillation.
In general, implementing co-distillation approach by employing the model disagreement based loss component (e.g., SKL ` f p x ; w 1 q , f p x ; w 2 q ˘ in our proposal) from the beginning leads to sub-optimal test accuracy as the models are very poor classiﬁers at the initialization.
As a remedy, Anil et al.
(2018) proposed a step-wise ramp up of such loss component, i.e., a time-dependent β p t q such that β p t q ąiff t ą t 0 , with t 0 burn-in steps.
Besides step-wise ramp up, we experimented with linear ramp-up, i.e., β t “ min t c ¨ t, β u and observed that linear ramp up leads to better churn reduction.
We believe that, for churn reduction, it’s important to ensure that the prediction probabilities of the two models start aligning before the model diverges too far during the training process.
D C OMBINED APPROACH : CO - DISTILLATION WITH ENTROPY REGULARIZER
In this paper, we have considered two different approaches to reduce churn, involving minimum entropy regularizers and co-distillation framework, respectively.
This raises an important question if these two approaches are redundant.
As shown in Figures 2, 1, and 3, this is not the case.
In fact, these two approaches are complementary in nature and combining them together leads to better results in term of both churn and accuracy.
Note that the minimum entropy regularizers themselves cannot actively enforce alignment between the prediction probabilities across multiple runs as the resulting objective does not couple multiple models together.
On the other hand, as veriﬁed in Figure 2, the co-distillation framework in itself does not affect the prediction conﬁdence of the underlying models.
Thus, both these approaches can be combined together to obtain the following training objective which promotes both large model prediction conﬁdence and small variance in prediction probabilities across multiple runs.
[Equation]
where
[Equation]
E A DDITIONAL EXPERIMENTAL RESULTS
In this section we present additional experimental results.
Ablation on CIFAR-100.
Next we present our ablation studies, similar to results in Figure 1, for
CIFAR-100.
In Figure 3, we plot the mean prediction entropy, accuracy and churn, for the proposed
SKL regularizer (7) and the co-distillation approach (9), for varying α and β , respectively.
Similar to Figure 1, the plots show the effectiveness of the proposed approaches in reducing churn, while improving accuracy.
E. O N HARDWARE NONDETERMINISM : GPU BASELINE ABLATION RESULTS
We now repeat the experiments in Sec. 2.3 on GPUs to investigate the unavoidable nondeterminism introduced by different hardware platforms.
We report results in Table 6.
Compared to Table 1, there are two main differences.
First, these experiments are run on GPU with smaller ResNet-32 models.
Second, these results always use data
[Figure]
[Table]
augmentation , but control for the randomness differently by ensuring that all models within each run perform the same augmentations during training
; we call these “identical input data” ablations because they control for both minibatch ordering and data augmentation.
Again, the the ﬁrst column gives the baseline churn with no factor held constant across runs, and the last column has every possible source of churn except hardware nondeterminism held constant across runs.
Model initialization is a signiﬁcant source of churn.
Simply initializing weights from identical seeds (columns 1-2 in Table 1) can signiﬁcantly decrease churn, no matter what other aspects of training are held constant.
Holding input data constant across runs does reduce randomness and further reduces churn, but is less signiﬁcant compared to constant initialization.
The rightmost column eliminates all possible sources of churn except for unavoidable nondeterminism in the computing platform used for training machine learning models, e.g., GPU/TPU (Morin & Willetts, 2020; PyTorch, 2019; Nvidia, 2020).
The experiments in Table 6 were run on GPU, and even when all other aspects of training are held constant (rightmost column), model weights diverge within 100 steps (across runs) and the ﬁnal churn is signiﬁcant.
We veriﬁed that this is the sole source of non-determinism: models trained to 10,000 steps on CPUs under these settings had identical weights.
Comparing these results to the rightmost column of Table 1, it seems TPU platforms currently introduce a higher baseline amount of churn than GPUs.
However, as shown in this work, our Co-distill SKL method can alleviate this somewhat.
E. 2 E NSEMBLE
In Table 7 we provide the results for 2-ensemble models and compare with distillation and codistillation approaches.
We notice that the proposed approach with entropy regularizer and codistillation achieve similar or better churn, with lower inference costs.
However the the ensemble models achieve better accuracy on certain datasets, and could be an alternative in settings where higher inference costs are acceptable.
[Table]
F L OSS LANDSCAPE : E NTROPY REGULARIZATION VS . TEMPERATURE SCALING
Figure 4 visualises the entropy regularised loss (6) in binary case.
We do so both in terms of the underlying loss operating on probabilities (i.e., log-loss), and the loss operating on logits with an implicit transformation via the sigmoid (i.e., the logistic loss).
Here, the regularised log-loss is (cid:96) : r 0 , 1 s Ñ R ` , where (cid:96) p
p q “ p 1 ´ α q ¨ ´ log p ` α ¨ H bin p
p q , for binary entropy H bin .
Similarly, the logistic loss is (cid:96) : R Ñ R ` , where (cid:96)
p f q “ p 1 ´ α q ¨ ´ log σ p f q ` α ¨ H bin p σ p f qq , for sigmoid σ p¨q .
The effect of increasing α is to dampen the loss for high-conﬁdence predictions – e.g., for the logistic loss, either very strongly positive or negative predictions incur a low loss.
This encourages the model to make conﬁdent predictions.
Figure 5, by contrast, illustrates the effect of changing the temperature τ in the softmax.
Temperature scaling is a common trick used to decrease classiﬁer conﬁdence.
This also dampens the loss for high-conﬁdence predictions.
However, with strongly negative predictions, one still incurs a high loss compared to strongly positive predictions.
This is in contrast to the more aggressive dampening of the loss as achieved by the entropy regularised loss.
[Figure] [Figure]