C HURN R EDUCTION VIA D ISTILLATION
{heinrichj, hnarasimhan,
, acotter,
A BSTRACT
In real-world systems, models are frequently updated as more data becomes available, and in addition to achieving high accuracy, the goal is to also maintain a low difference in predictions compared to the base model (i.e. predictive “churn”).
If model retraining results in vastly different behavior then it could cause negative effects in downstream systems, especially if this churn can be avoided with lim - ited impact on model accuracy
In this paper , we show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn.
We then show that distillation performs strongly for low churn training against a number of recent baselines on a wide range of datasets and model architectures, including fully-connected networks, convolutional networks, and transformers
I NTRODUCTION
Deep neural networks (DNNs) have had profound success at solving some of the most challenging machine learning problems.
While much of the focus has been spent towards attaining state-of-art predictive performance, comparatively there has been little effort towards improving other aspects.
One such important practical aspect is reducing unnecessary predictive churn with respect to a base model.
We deﬁne predictive churn as the difference in the prediction of a model relative to a base model on the same datapoints.
In a production system, models are often continuously released through an iterative improvement process which cycles through launching a model, collecting additional data and researching ways to improve the current model, and proposing a candidate model to replace the current version of the model serving in production.
In order to validate a candidate model, it often needs to be compared to the production model through live A/B tests (it’s known that ofﬂine performance alone isn’t a sufﬁcient, especially if these models are used as part of a larger system where the ofﬂine and online metrics may not perfectly align (Deng et al., 2013; Beel et al., 2013)).
Live experiments are costly: they often require human evaluations when the candidate and production model disagree to know which model was correct (Theocharous et al., 2015; Deng & Shi, 2016).
Therefore, minimizing the unnecessary predictive churn can have a signiﬁcant impact to the cost of the launch cycle.
It’s been observed that training DNN can be very noisy due to a variety of factors including random initialization (Glorot & Bengio, 2010), mini-batch ordering (Loshchilov & Hutter, 2015), data augmentation and processing (Santurkar et al., 2018; Shorten & Khoshgoftaar, 2019), and hardware (Turner & Nowotny, 2015; Bhojanapalli et al., 2021)– in other words running the same procedure multiple times can lead to models with surprisingly amount of disagreeing predictions even though all can have very high accuracies (Bahri & Jiang, 2021).
While the stability of the training procedure is a separate problem from lowering predictive churn, such instability can further exacerbate the issue and underscores the difﬁculty of the problem.
Knowledge distillation (Hinton et al., 2015), which involves having a teacher model and mixing its predictions with the original labels has proved to be a useful tool in deep learning.
In this paper, we show that this surprisingly is not only an effective tool for churn reduction by using the base model as the teacher, it is also mathematically aligned with learning under a constraint on the churn.
Thus, in addition to providing a strong method for low churn training, we also provide insight into the distillation.
Our contributions are as follows:
• We show theoretically an equivalence between the low churn training objective (i
e. minimize a loss function subject to a churn constraint on the base model) and using knowledge distillation with the base model as the teacher.
• We show that distillation performs strongly in a wide range of experiments against a number baselines that have been considered for churn reduction.
• Our distillation approach is similar to a previous method called “anchor” (Fard et al., 2016), which trains on the true labels instead of distilled labels for the incorrectly predicted examples by the base model, but outperform this method by a surprising amount.
We present both theoretical and experimental results showing that the modiﬁcation of anchor relative to distillation actually hurts performance.
R ELATED W ORKS
Prediction Churn.
There are few works that address low churn training with respect to a base model .
Fard et al.
(2016) proposed an anchor loss which is similar to distillation when the base model’s prediction agrees with the original label, and uses a scaled version of the original label otherwise.
In our empirical evaluation, we ﬁnd that this procedure performs considerably worse than distillation.
Cotter et al. (2019a); Goh et al.
(2016) use constrained optimization by adding a constraint on the churn.
We use some of the theoretical insights found in that work to show an equivalence between distillation and the constrained optimization problem.
Thus, we are able to bypass the added complexity of using constrained optimization (Cotter et al., 2019b) in favor of distillation, which is a simpler and more robust method.
A related but different notion of churn that has been studied is where the goal is to reduce the training instability.
Anil et al.
(2018) noted that co-distillation is an effective method.
Bahri & Jiang (2021) proposes a locally adaptive variant of label smoothing and Bhojanapalli et al.
propose entropy regularizers and a variant of co-distillation.
We tested many of the baselines proposed in these papers and adapt them to our notion of churn and showed that they were not effective at reducing predictive churn w.r.t.
a base model.
Distillation.
Distillation (Ba & Caruana, 2013; Hinton et al., 2015), ﬁrst proposed to transfer knowledge from larger networks to smaller ones, has become immensely popular.
Applications include learning from noisy labels (Li et al., 2017), model compression (Polino et al., 2018), adversarial robustness (Papernot et al., 2016), DNNs with logic rules (Hu et al., 2016), visual relationship detection (Yu et al., 2017), reinforcement learning (Rusu et al., 2015), domain adaptation (Asami et al., 2017) and privacy (Lopez-Paz et al., 2015).
Our work adds to the list of applications in which distillation is effective.
The theoretical motivation of distillation however is less established.
Lopez-Paz et al. (2015) studied distillation as learning using privileged information (Vapnik & Izmailov, 2015).
Phuong & Lampert (2019) establishes fast convergence of the expected risk of a distillation-trained linear classiﬁer.
Foster et al.
(2019) provides a generalization bound for the student with an assumption that it learns a model close to the teacher.
Dong et al.
(2019) argued distillation has a similar effect as that of early stopping.
Mobahi et al.
(2020) showed an equivalence to increasing the regularization strength for kernel methods.
Menon et al.
(2020) establish a bias-variance trade-off for the student.
Our analysis provides a new theoretical perspective of its relationship to churn reduction.
D ISTILLATION FOR C ONSTRAINING C HURN
We are interested in a multiclass classiﬁcation problem with an instance space X and a label space
r m s “ t 1 , . . .
, m u .
Let D denote the underlying data distribution over instances and labels, and D X denote the corresponding marginal distribution over X .
Let ∆ m denote the p m ´
1 q -dimensional simplex with m coordinates.
We will use p : X Ñ ∆ m to denote the underlying conditional-class probabilities, where p y p x q “ P p Y “ y | X “ x q .
We assume that we are provided a base classiﬁer g : X Ñ ∆ m that predicts a vector of probabilities g p x q P ∆ m for any instance x .
Our goal is to then learn a new classiﬁer h :
X Ñ ∆ m , constraining its churn to be within an acceptable limit.
We will measure the classiﬁcation performance of a classiﬁer h using a loss function (cid:96) : r m s ˆ ∆ m Ñ R ` that maps a label y P r m s and prediction h p x q P ∆ m to a non-negative number (cid:96) p y, h p x qq , and denote the classiﬁcation risk by R p
h q : “ E p x,y q„ D r (cid:96) p
y, h p x qqs .
We would ideally like to deﬁne predictive churn as the fraction of examples on which h and g disagree.
For the purpose of designing a tractable algorithm, we will instead work with a softer notion of churn, which evaluates the divergence between their output distributions.
To this end, we use a measure of divergence d : ∆ m ˆ ∆ m Ñ R ` , and denote the expected churn between h and g by
C p h q : “ E x „ D X r d p g p x q , h p x qqs .
We then seek to minimize the classiﬁcation risk for h , subject to the expected churn being within an allowed limit (cid:15) ą 0 :
[Equation]
We consider loss and divergence functions that are deﬁned in terms of a scoring function φ : ∆ m Ñ R m ` that maps a distribution to a m -dimensional score.
Speciﬁcally, we will consider scoring functions φ that are strictly proper (Gneiting & Raftery, 2007;
Williamson et al., 2016), i.e. for which, given any distribution u P ∆ m , the conditional risk E y „
u r φ y p v qs is uniquely minimized by v “ u .
The following are general forms of the loss and divergence functions employed in this paper:
[Equation]
The cross-entropy loss and KL-divergence are a special case of this formulation when φ y p v q “ ´ log p v y q , and the squared loss and the squared L 2 distance can be recovered by setting φ y p v q “ ř i Pr m s p [Equation] [Equation] i q 2 .
3. B AYES - OPTIMAL C LASSIFIER
We show below that for the loss and divergence functions deﬁned in (2), the optimal-feasible classiﬁer for the constrained problem in (1) is a convex combination of the class probability function p and the base classiﬁer g .
Proposition 1.
Let p (cid:96), d q be deﬁned as in (2) for a strictly proper scoring function φ .
Suppose φ p u q is strictly convex in u .
Then there exists λ ˚ P r 0 , 1 s such that the following is an optimal-feasible classiﬁer for (1) :
[Equation]
Furthermore, if u ¨ φ p u q is α -strongly concave over u P ∆ m w.r.t.
the L q -norm, then λ ˚ ď b 2 (cid:15) {
[Equation]
1: Inputs: Training sample S “ tp x 1 , y 1 q , . .
, p x n , y n qu , Grid of mixing coefﬁcients Λ “ t λ 1 , . .
λ L u , Base classiﬁer g , Constraint slack (cid:15) ą2: Train a classiﬁer h k for each λ
P Λ by minimizing the distilled loss
4): h k P argmin h P H p
L λ k p
h q 3: Find a convex combination of h 1 , . .
, h L by solving following convex program in L variables:
[Equation]
and return the solution p h The strong concavity condition in Proposition 1 is satisﬁed by the cross-entropy loss and KLdivergence for α “ 1 with the L 1 -norm, and by the squared loss and L 2 -distance for α “ 2 with the
L 2 -norm.
The bound suggests that the mixing coefﬁcient λ ˚ depends on how close the base classiﬁer is to the class probability function p .
3. D ISTILLATION - BASED A PPROACH
Proposition 1 directly motivates the use of a distillation-based approach for solving the churnconstrained optimization problem in (1).
We propose treating the base classiﬁer g as a teacher model, mixing the training labels y with scores from the teacher g p x q , and minimizing a classiﬁcation loss against the transformed labels:
[Equation]
(3) where e y P t 0 , 1 u m denotes a one-hot encoding of the label y P
r m s and φ is a strictly proper scoring function.
It is straight-forward to show that when λ “ λ ˚ , the optimal classiﬁer for the above distillation loss takes the same form in Proposition 1, i.e. h ˚ p x q “ λ ˚ p p x q ` p 1 ´ λ ˚ q g p x q .
While the optimal mixing parameter λ ˚ is unknown, we propose treating this as a hyper-parameter and tuning it to reach the desired level of churn.
In practice, we do not have direct access to the distribution D and will need to work with a sample
S “ tp x 1 , y 1 q , . .
, p x n , y n qu drawn from D .
To this end, we deﬁne the empirical risk and the empirical churn as follows:
[Equation]
where (cid:96) φ and d φ are deﬁned as in (2) for a scoring function φ .
Our proposal is to then solve the following empirical risk minimization problem over a hypothesis class H Ă t h : X Ñ ∆ m u for different values of coefﬁcient λ k chosen from a ﬁnite grid t λ 1 , . . .
λ L u Ă r 0 , 1 s :
[Equation]
To construct the ﬁnal classiﬁer, we ﬁnd a convex combination of the L classiﬁers h 1 , . . .
, h L that minimizes p R p
h q while satisfying the constraint p C p h q ď (cid:15) , and return an ensemble of the L classiﬁers.
The overall procedure is outlined in Algorithm 1, where we denote the set of convex combinations of classiﬁers h 1 , . .
, h L by co p h 1 , . . .
, h L q “ t h : x ÞÑ ř Lj “ 1 α j h j p x q | α P ∆ L u .
The post-processing step in Algorithm 1 amounts to solving a simple convex program in L variables.
This is needed for technical reasons in our theoretical results, speciﬁcally, to translate a solution to a dual-optimal solution to (1) to a primal-feasible solution.
In practice, however, we do not construct an ensemble, and instead simply return a single classiﬁer that achieves the least empirical risk while satisfying the churn constraint.
In our experiments, we use the cross-entropy loss for training, i.e. set φ y p u q “ ´ log p u y q .
T HEORETICAL G UARANTEES
We provide optimality and feasibility guarantees for the proposed algorithm and also explain why our approach is better-suited for optimizing accuracy (subject to a churn constraint) compared to the previous churn-reduction method of Fard et al.
4. O PTIMALITY AND F EASIBILITY G UARANTEES
We now show that the classiﬁer p h returned by Algorithm 1 approximately satisﬁes the churn constraint, while achieving a risk close to that of the optimal-feasible classiﬁer in H .
This result assumes that we are provided with generalization bounds for the classiﬁcation risk and churn.
Theorem 2.
Let the scoring function φ : ∆ m Ñ R m ` be convex, and } φ p z q} 8 ă B, @
z P ∆ m .
Let the set of classiﬁers H be convex, with the base classiﬁer g P H .
Suppose C and R enjoy the following generalization bounds: for any δ P p 0 , 1 q , w.p.
ě 1 ´ δ over draw of S „ D n , for any h P H ,
[Equation]
for some ∆ R p n, δ q and ∆ C p n, δ q that is decreasing in n and approaches 0 as n Ñ8 .
Let r h be an optimal-feasible classiﬁer in H , i.e. C p r h q ď (cid:15) and R p r h q ď R p h q for all classiﬁers h for which C p h q ď (cid:15) .
Let p
h
be the classiﬁer returned by Algorithm 1 with Λ “ (cid:32) max t (cid:15) (cid:15) ` 2 B , u u ˇˇ u P t 1 L , 2 L , . .
. , 1 u ( for some L P N ` .
For any δ P p 0 , 1 q , w.p. ě
´ δ over draw of S „ D n ,
Optimality : R p p
h
q ď R p r
[Equation]
In practice, we expect the churn metric to generalize better than the classiﬁcation risk, i.e. for ∆ C p n, δ q to be smaller than ∆ R p n, δ q .
This is because the classiﬁcation risk is computed on “hard” labels y P
r m s from the training sample, whereas the churn metric is computed on “soft” labels g p x q P ∆ m from the base model.
The traditional view of distillation (Hinton et al., 2015) suggests that the soft labels from a teacher model come with conﬁdence scores for each example, and thus allow the student to generalize well to unseen new examples.
A similar view is also posed by Menon et al.
(2020) , who argue that the soft labels from the teacher have “lower variance” than the hard labels from the training sample, and therefore aid in better generalization of the student.
To this end, we apply the generalization bound from (Menon et al., 2020, Proposition 2) to the student’s churn. Proposition 3 (Generalization bound for churn) . Let the scoring function φ : ∆ m Ñ R m ` be bounded. For base classiﬁer g , let U φ Ď R X denote the corresponding class of divergence functions
u p x q “ d φ p h p x q , g p x qq “ g p x q J ` φ p h p x qq ´ φ p g p x qq ˘ induced by classiﬁers h P H . Let
M Cn “ N 8 p 1 n , U φ , 2 n q denote the uniform L 8 covering number for U φ . Fix δ P p 0 , 1 q . Then with probability ě 1 ´ δ over draw of S „ D n , for any h P H : [Equation]
where V Cn p h q denotes the empirical variance of the divergence values computed on n examples t
g p x
i q J
[Equation]
; the lower the variance, the tighter is the bound. In fact, for certain base classiﬁers g , generalizing well on “churn” can have the additional beneﬁt of improving classiﬁcation performance, as shown in Proposition 7 in Appendix B.
4. A DVANTAGE OVER A NCHOR L OSS
We next compare our distillation loss in (3) with the previous anchor loss of Fard et al.
(2016), which uses the base model’s prediction only when it agrees with the original label, and uses a scaled version of the original label otherwise.
While originally proposed for churn reduction with binary labels, we provide below an analogous version of this loss for a multiclass setup:
[Equation]
where
[Equation]
otherwise
[Equation]
for hyper-parameters α,
η P r 0 , 1 s and a strictly proper scoring function φ .
Here, we have used argmax to denote ties being broken in favor of the larger class.
While this helps us simplify the exposition, our results can be easily extended to a version of the loss which includes ties.
The anchor loss does not take into account the conﬁdence with which the base model disagrees with the sampled label y .
For example, if the base model predicts near-equal probabilities for all classes, but happens to assign a slightly higher probability to a class different from y , the anchor loss would still completely ignore the base model
’s score (even though it might be the case that all the labels are indeed equally likely to occur).
In some cases, this selective use of the teacher labels can result in a biased objective and may hurt the classiﬁer’s accuracy.
To see this, consider an ideal scenario where the base model predicts the true conditional-probabilities p p x q and the student hypothesis class is universal.
In this case, minimizing the churn w.r.t.
the base model has the effect of maximizing classiﬁcation accuracy, i.e. a classiﬁer that has zero churn w.r.t.
the base model also produces the least classiﬁcation error.
However, as shown below, even in this ideal setup, minimizing the anchor loss may result in a classiﬁer different from the base model.
Proposition 4.
When g p x q “ p p x q , @ x , for any given λ P r 0 , 1 s , the minimizer for the distillation loss in (3) over all classiﬁers h is given by:
[Equation]
whereas the minimizer of the anchor loss in (5) is given by:
[Equation]
Unless α “ 0 and η “ 1 (which amounts to completely ignoring the base model) or the base model makes hard predictions on all points, i.e. p j p x q P t 0 , 1 u , @ x , the anchor loss encourages scores that differ from the base model p .
For example, when α “ η “ 1 (and the base model predicts soft probabilities), the anchor loss has the effect of down weighting the label that the base model is most conﬁdent about, and as a result, encourages lower scores on that label and higher scores on all other labels.
While one can indeed tweak the two hyper-parameters to reduce the gap between the learned classiﬁer and the base model, our proposal requires only one hyper-parameter λ , which represents an intuitive trade-off between the one-hot and teacher labels.
In fact, irrespective of the choice of λ , the classiﬁer that minimizes our distillation loss in Proposition 4 mimics the base model p exactly, and as a result, achieves both zero churn and optimal accuracy.
We shall see in the next section that even on real-world datasets, where the base classiﬁer does not necessarily make predictions close to the true class probabilities (and where the student hypothesis class is not necessarily universal and of limited capacity), our proposal performs substantially better than the anchor loss in minimizing churn at a particular accuracy.
Figure 3 provides a further ablation study, effectively interpolating between the anchor and distillation methods, and provides evidence that using the true (hard) label instead of the teacher (soft) label can steadily degrade performance.
E XPERIMENTS
We now show empirically that distillation is an effective method to train models for both accuracy and low churn.
We test our method across a large number of datasets and neural network architectures
. 5. S ETUP
Datasets and architectures :
The following are the datasets we use in our experiments, along with the associated model architectures:
• 12 OpenML datasets using fully-connected neural networks.
• 10 MNIST variants, SVHN [Table]
• CIFAR10 and CIFAR100 with ResNet-50, ResNet-101, and ResNet-152.
• IMDB dataset using transformer network
For each architecture (besides ResNet), we use 5 different sizes.
For the fully connected network, we use a simple network with one-hidden layer of 10 , 10 2 , 10 3 , 10 4 , and 10 5 units, which we call fcn- x where x is the respective size of the hidden layer.
For the convolutional neural network, we start with the LeNet5 architecture (LeCun et al., 1998) and scale the number of hidden units by a factor of x for x “ 1 , 2 , 4 , 8 , 16 , which we call ConvNet- x for the respective x .
Finally, we use the basic transformer architecture from Keras tutorial (Keras, 2020) and scale the number of hidden units by x for x “ 1 , 2 , 4 , 8 , 16 , which we call Transformer- x for the respective x .
Code for the models in
Keras can be found in the Appendix.
For each dataset, we use the standard train/test split if available, otherwise, we ﬁx a random train/test split with ratio 2:1.
Setup :
For each dataset and neural network, we randomly select from the training set 1000 initial examples, 100 validation examples, and a batch of 1000 examples, and train an initial model using
Adam optimizer with default settings on the initial set and early stopping (i.e. stop when there’s no improvement on the validation loss after 5 epochs) and default random initialization, and use that model as the base model.
Then, for each baseline, we train on the combined initial set and batch ( 2000 datapoints), again using the Adam optimizer with default settings and the same early stopping scheme and calculate the accuracy and churn against the base model on the test set.
We average across 100 runs and provide the error bands in the Appendix.
For all the datasets except the OpenML datasets, we also have results for the case of 10000 initial examples, 1000 validation examples, and a batch 1000 .
We also show results for the case of 100 initial samples, 1000 validation examples, and a batch of 1000 for all of the datasets.
Due to space, we show those results in the Appendix.
We ran our experiments on a cloud environment.
For each run, we used a NVIDIA V100 GPU, which took up to several days to ﬁnish all 100 trials.
Baselines : We test our method against the following baselines.
(1) Cold start , where we train the model from scratch with the default initializer.
(2) Warm start , where we initialize the model’s parameters to that of the base model before training.
(3) Shrink-perturb (Ash & Adams, 2019), which is a method designed to improve warm-starting by initializing the model’s weights to α ¨ θ base ` p 1 ´ α q¨ θ init before training, where θ base are the weights of the base model, θ init is a randomly initialized model, and α is a hyperparameter we tune across t 0 .
9 u .
(4) Mixup (Zhang et al., 2017) (a baseline suggested for a different notion of churn (Bahri & Jiang, 2021)), which trains an convex combinations of pairs of datapoints.
We search over its hyperparameter α P t 0 .
9 u , as deﬁned in Zhang et al.
(5) Label smoothing (Szegedy et al., 2016), which was suggested by Bahri &
Jiang (2021) for the variance notion of churn, proceeds by training on a convex combination between the original labels and the base models’ soft prediction.
We tune across the convex combination weight α P t 0 .
9 u .
(6) Co-distillation (Anil et al., 2018), which was proposed for the variance notion of churn, where we train two warm-started networks that train simultaneously on a loss that is a convex combination on the original loss and a loss on the difference between their predictions.
We tune across the convex combination weight α P t 0 .
9 u .
(7) Anchor (Fard [Table] et al., 2016), which as noted in Section 4.2, proceeds by optimizing the cross-entropy loss on a modiﬁed label: we use the label α g p x q ` p 1 ´ α q e y when the base model g agrees with the true label y , and η e y otherwise.
We tune across α P t 0 .
9 u and η P t 0 .
7 , 1 u .
For distillation, we tune the trade-off parameter λ across t 0 .
9 u .
Metric : All of the methods will produce a model that we evaluate for both accuracy and churn with respect to the base model on the test set.
We consider the hard notion of churn, which measures the average difference in hard predictions w.r.t.
the base classiﬁer on a test set.
We will see later that there is often-times a trade-off between accuracy and churn, and in an effort to produce one metric for quantitative evaluation, we propose churn at cold accuracy metric, which is deﬁned as follows.
Each baseline produces a set of models (one for each hyperparameter setting).
We take the averaged churn and accuracy across the 100 runs and choose the model with the lowest churn that is at least as accurate as the cold-start model (it’s possible that no such model exists for that method).
This way, we can identify the method that delivers the lowest churn but still performs at least as well as if we trained on the updated dataset in a vanilla manner.
We believe this metric is practically relevant as a practitioner is unlikely to accept a reduction in accuracy to reduce churn.
5. R ESULTS
The detailed results for the following experiments can be found in the Appendix.
Given space constraints, we only provide a high level summary in this section
OpenML datasets with fully-connected networks : In Table 1 we show the results for the OpenML datasets using the fcn-1000 network.
We see that distillation performs the well across the board, and for the other fully connected network sizes, distillation is the best in the majority of cases ( 84% of the time for initial batch size 1000 and 52% of time for initial batch size 100 ).
MNIST variants, SVHN, and CIFAR10 with convolutional networks :
In Table 2, we show the results for 10 MNIST variants, SVHN and CIFAR10 using convnet-4.
We see that distillation performs strongly across the board.
We found that distillation performs best in 84% of combinations between dataset and network.
When we increase the initial sample size to 10000 and keep the batch size ﬁxed at 1000 , then we found that label smoothing starts becoming competitive with distillation, where distillation is best 64% of the time, and label smoothing wins by a small margin all other times.
We only saw this phenomenon for a handful of the MNIST variants, which suggests that label smoothing may be especially effective in these situations.
When we decreased the initial sample down to 100 and kept the batch size the same, we found that distillation was best 48% of the time, with Anchor being the second best method winning 24% of the time.
For SVHN and CIFAR10, of the 10 combinations, distillation performs the best on all 10 out of the 10 .
If we increased the initial sample size to 10000 and kept the batch size ﬁxed at 1000 , then we ﬁnd that distillation still performs the best all 10 out of 10 combinations.
If we decreased the initial sample size to 100 and kept the same batch size, then distillation performs the best on 8 out of thecombinations.
CelebA with convolutional networks : Across all 200 combinations of task and network, distillation performs the best 79% of the time.
Moreover, if we increased the initial sample size to 10000 and kept the batch size ﬁxed at 1000 , distillation is even better, performing the best 91 .
5% of the time.
If we decreased the initial sample size to 100 , then distillation is best 96% of the time.
CIFAR10 and CIFAR100 with ResNet : Due to the computational costs, we only run these experiments for initial sample size 1000 .
In all cases (across ResNet-50, ResNet-101 and ResNet-152), we see that distillation outperforms the other baselines.
IMDB with transformer network :
We experimented for initial batch size 100 , 1000 , and 10000 .
We found that distillation performed the best the majority of the time, where the only notable weak performance was in some instances where no baselines were even able to reach the accuracy of the cold starting method.
In Figure 2 we show the Pareto frontiers of the various baselines as well as plotting cost of each method as we vary the trade-off between accuracy and churn.
We see that not only does distillation do well in churn, but it performs the best at any trade-off between churn and accuracy for the cases shown.
Conclusion : We have proposed knowledge distillation as a new practical solution to churn reduction, and provided both theoretical and empirical justiﬁcations for the approach.
Reproducibility Statement : All details of experimental setup are in the main text, along with descriptions of the baselines and what hyperparameters were swept across.
Code can be found in the
Appendix.
All proofs are in the Appendix.
A P ROOFS A. P ROOF OF P ROPOSITION
Proposition (Restated) .
Let p (cid:96), d q be deﬁned as in (2) for a strictly proper scoring function φ .
Suppose φ p u q is strictly convex in u .
Then there exists λ ˚ P r 0 , 1 s such that the following is an optimal-feasible classiﬁer for (1) :
[Equation]
Furthermore, if u ¨ φ p u q is α -strongly concave over u P ∆ m w.r.t.
the L q -norm, then
[Equation]
Proof.
Let h ˚ denote an optimal feasible solution for (1).
We ﬁrst note that
[Equation]
and
[Equation]
Because φ i is strictly convex in its argument, both R p h q and C p
h q are strictly convex in h .
In other words, for any α P r 0 , 1 s , and classiﬁers h 1 , h 2 , R p
α h 1 ` p 1 ´ α q h 2 q ă αR p h 1 q ` p 1 ´ α q R p h 2 q , and similarly for C .
Furthermore because C p g q “ 0 ă (cid:15) , the constraint is strictly feasible, and hence strong duality holds for (1) (as a result of Slater’s condition being satisﬁed).
Therefore (1) can be equivalently formulated as a max-min problem:
[Equation]
for which there exists a µ ˚ P R ` such that p µ ˚ ,
h ˚ q is a saddle point.
The strict convexity of R p h q and C p h q gives us that h ˚ is the unique minimizer of R p
h q ` µ ˚ C p
h q .
Setting λ ˚ “ 1 1 ` µ ˚ , we equivalently have that h ˚ is a unique minimizer of the weighted objective λ ˚ R p
h q ` p 1 ´ λ ˚ q C p
h q .
We next show that the minimizer h ˚ is of the required form.
Expanding the R and C , we have:
[Equation]
of h
[Equation]
where ¯ p p x q “ λ ˚ p p x q ` p 1 ´ λ ˚ q g p x q .
Note that it sufﬁces to minimize (6) point-wise, i.e. to choose h ˚ so that the term within the expectation ř i Pr m s ¯ p i p x q φ i p h p x qq is minimized for each x .
For a ﬁxed x , the inner term is minimized when h ˚ p x q “ ¯ p p x q .
This is because of our assumption that φ is a strictly proper scoring function, i.e. for any distribution u , the weighted loss
ř i u
i φ i p v q is uniquely minimized by v “ u .
Therefore (6) is minimized by h ˚ p
[Equation]
` p 1 ´ λ ˚ q g p x q .
To bound λ ˚ , we use a result from Williamson et al.
; Agarwal (2014) to lower bound C p h q in terms of the norm difference } h p x q ´ g p x q} q .
Deﬁne Q p u q “ inf v P ∆ m u ¨ φ p v q .
Because φ is a proper scoring function, the inﬁmum is attained at v “ u .
Therefore Q p u q “ u ¨ φ p u q , which recall is assumed to be strongly concave.
Also, note that Q p u q “ inf v P ∆ m u ¨ φ p v q is an inﬁmum of “linear” functions in u , and therefore ∇Q p u q “ φ p u q is a super-differential for Q at u .
See Proposition 7 in
Williamson et al.
(2016) for more details.
We now re-write C p h
q in terms of Q and lower bound it using the strong concavity property:
[Equation]
where the last step uses the fact that Q is α -strongly concave over u P ∆ m w.r.t.
the L q -norm.
Since the optimal scorer h ˚ satisﬁes the coverage constraint C p h ˚ q ď (cid:15) , we have from the above bound
[Equation]
Substituting for h ˚ , we have:
[Equation]
or
[Equation]
which gives us the desired bound on λ ˚ .
A. P ROOF OF T HEOREM
Theorem (Restated) .
Let the scoring function φ : ∆ m Ñ R m ` be convex, and } φ p z q} 8 ă B, @
z P ∆ m .
Let the set of classiﬁers H be convex, with the base classiﬁer g P H .
Suppose C and R enjoy the following generalization bounds: for any δ P p 0 , 1 q , w.p.
ě 1 ´ δ over draw of S „ D n , for any h P
[Equation]
n, [Equation] , for some ∆ R p n, δ q and ∆ C p n, δ q that is decreasing in n and approaches 0 as n Ñ8 .
Let r h be an optimal-feasible classiﬁer in H , i.e. C p r h q ď (cid:15) and R p r h q ď R p h q for all classiﬁers h for which C p h q ď (cid:15) .
Let p
h
be the classiﬁer returned by Algorithm 1 with Λ “ (cid:32) max t (cid:15) (cid:15) ` 2 B , u u ˇˇ u P t 1 L , 2 L , . .
. , 1 u ( for some L P N ` .
For any δ P p 0 , 1 q , w.p. ě
´ δ over draw of S „ D n ,
Optimality : R p p
[Equation]
Feasibility : C p p
[Equation]
We ﬁrst note that because } φ p z q} 8 ă B, @
z P ∆ m , both p R p
h q ă B and p C p
h q ă B .
Also, because φ i is convex, both p R p
h q and p C p
h q are convex in h .
In other words, for any α P r 0 , 1 s , and classiﬁers h 1 , h 2 , p R p
α h 1 ` p 1 ´ α q h 2 q ď α p R p h
q ` p 1 ´ α q p R p
h 2 q , and similarly for p C .
Furthermore, the objective in (4) can be decomposed into a convex combination of the empirical risk and churn:
[Equation]
Therefore minimizing p L λ p h q is equivalent to minimizing the Lagrangian function
[Equation]
over h .
Moreover, each h k minimizes r L λ k p h q .
We also note that the churn-constrained optimization problem in (1) can be posed as a Lagrangian game between a player that seeks to minimize the above Lagrangian over h and a player that seeks to maximize the Lagrangian over λ .
The next two lemmas show that Algorithm 1 can be seen as ﬁnding an approximate equilibrium of this two-player game.
Lemma 5.
Let the assumptions on φ and H in Theorem 2 hold.
Let p
h
be the classiﬁer returned by Algorithm 1 when Λ is set to Λ “
[Equation]
[Equation]
1 u ( of the range r (cid:15) (cid:15)
[Equation]
2 B , 1 s for some L P N ` .
Then there exists a bounded Lagrange multiplier ¯ λ P r (cid:15) (cid:15) ` 2 B , 1 s such that p p h , ¯ λ q forms an equilibrium of the Lagrangian min-max game:
[Equation]
Proof.
The classiﬁer p h returned by Algorithm 1 is a solution to the following constrained optimization problem over the convex-hull of the classiﬁers h 1 , . . .
, h L :
[Equation]
Consequently, there exists a ¯ λ P r 0 , 1 s such that:
[Equation]
To see this, note that the KKT conditions (along with the convexity of R and C ) give us that there exists a Lagrange multiplier ¯ µ ě 0 such that
[Equation]
When p C p p h q ď (cid:15) , ¯ µ “ 0 , and so (8) and (9) are satisﬁed for ¯ λ “ 1 .
When p C p p
h q “ (cid:15) , then (8) and (9) are satisﬁed for ¯ λ “ 1 1 ` ¯ µ .
It remains to show that that ¯ λ P r (cid:15) (cid:15) ` 2 B , 1 s .
For this, we ﬁrst show that there exists a h 1 P co p h 1 , . . .
, h L q such that p C p h 1 q ď (cid:15) { 2 .
To see why, pick h 1 to be the minimizer of the Lagrangian r
L λ p h q over all h P H for λ “ (cid:15) (cid:15) ` 2 B .
Because r L λ p h 1 q ď r L λ p g q ď λB ´ p 1 ´ λ q (cid:15) , where g is the base classiﬁer that we have assumed is in H , it follows that p C p h 1 q ď λ 1 ´ λ B ď (cid:15) { 2 .
Next, by combining (8) and (9), we have
[Equation]
Lower bounding the LHS by setting λ “ 1 and upper bounding the RHS by setting h “ h 1 , we get:
[Equation]
which gives us:
[Equation]
Hence ¯ λ ě (cid:15) (cid:15) ` 2 B , which completes the proof.
Lemma 6.
Let p
h
be the classiﬁer returned by Algorithm 1 when Λ is set to Λ “ (cid:32) max t (cid:15) (cid:15) ` 2 B , u u | u P t 1 L , . .
. , 1 u ( of the range r (cid:15) (cid:15) ` 2 B , 1 s for some L P N ` .
Fix δ P p 0 , 1 q .
Suppose R and C satisfy the generalization bounds in Theorem 2 with error bounds ∆ R p n, δ q and ∆ C p n, δ q respectively.
Then there exists a bounded Lagrange multiplier p λ P r (cid:15) (cid:15) ` 2 B , 1 s such that p p h , p µ q forms an approximate equilibrium for the Lagrangian min-max game, i.e. w.p.
ě 1 ´ δ over draw of sample S „ D n ,
[Equation]
[Equation] [Equation]
{ L
[Equation]
10) and max
[Equation] [Equation]
Proof.
We have from Lemma 5 that there exists ¯ λ P r (cid:15) (cid:15) ` 2 B , 1 s such that
[Equation]
Algorithm 1 works with a discretization Λ “
[Equation]
of the range r
[Equation]
` 2 B , 1 s .
Allowing p λ to denote the closest value to ¯ λ in this set, we have from (12):
[Equation]
where the last step follows from the fact that co p h 1 , . .
, h L q Ď H and each h k was chosen to minimize p 1 ´ λ k q p R p
h q ` λ k p p C p
h q ´ (cid:15) q for λ k P Λ .
Similarly, we have from (13),
[Equation]
What remains is to apply the generalization bounds for R and C to (14) and (15).
We ﬁrst bound the
LHS of (14).
We have with probability at least 1 ´ δ over draw of S „ D n :
[Equation]
where the last step uses the fact that 0 ď p λ ď 1 .
For the RHS
[Equation]
we have with the same probability:
[Equation]
where we again use 0 ď p λ ď 1 .
Combining (14) with (16) and (17) completes the proof for the ﬁrst part of the lemma.
Applying the generalization bounds to (15)
[Equation]
we have with the same probability:
B [Equation] L
[Equation]
which completes the proof for the second part of the lemma
[Equation]
We are now ready to prove Theorem
[Equation]
Proof of Theorem 2.
To show optimality
[Equation]
we combine (10) and (11) and get:
[Equation]
We then lower bound the LHS in (17) by setting λ “ 1 and upper bound the RHS by setting h to the optimal feasible solution r h , giving us:
[Equation]
Dividing both sides by p λ ,
[Equation]
Lower bounding p λ by
[Equation]
gives us the desired optimality result.
The feasibility result directly follows from the fact that Algorithm 1 chooses a p h that satisﬁes the empirical churn constraint p C p p h q ď (cid:15) , and from the generalization bound for C .
A. P ROOF OF P ROPOSITION
Proposition (Restated) .
When g p x q “ p p x q , @ x , for any given λ P r 0 , 1 s , the minimizer for the distillation loss in (3) over all classiﬁers h is given by:
[Equation]
whereas the minimizer of the anchor loss in (5) is given by: h ˚
j p x q “ z j ř j z j where z j “
[Equation]
αp
[Equation]
j p x q
[Equation]
´ α q
[Equation]
q
[Equation]
j “
[Equation]
p
[Equation]
x q p
[Equation]
` α max k p k p x qq p
j p x q otherwise .
Proof.
For the ﬁrst part, we expand (3) with g p x q “ p p x q , and have for any λ P r 0 , 1 s ,
[Equation]
[Equation]
For a ﬁxed x , the inner term in (19) is minimized when h ˚ p x q “ p p x q .
This is because of our assumption that φ is a strictly proper scoring function, i.e. for any distribution u , the weighted loss
ř i u
i φ i p v q is uniquely minimized by v “ u .
Therefore (19) is minimized by h ˚ p x q “ p p x q , @ x .
For the second part, we expand (5) with g p x q “ p p x q , and have:
[Equation]
where
[Equation]
otherwise
[Equation]
For a given x , let us denote j x “ argmax k
p k p x q .
We then have:
[Equation]
[Equation]
[Table]
[Equation]
ÿ
[Equation]
j
[Equation]
p
[Equation]
ÿ
[Equation] x [Equation] (20) where
[Equation]
For a ﬁxed x , the inner term in (20) is minimized when h ˚ p x q “ 1 Z p x q r p p x q , where Z p x q “ ř k r p k p x q .
This follows from the fact that for a ﬁxed x , the minimizer of the inner term r p p x q ¨ φ p h p x qq is the same as the the minimizer of the scaled term
[Equation]
p x q r p p x q ¨ φ p h p x qq , and from φ being a strictly proper scoring function.
This completes the proof.
B A DDITIONAL T HEORETICAL R ESULTS B. R ELATIONSHIP BETWEEN CHURN AND CLASSIFICATION RISK
For certain base classiﬁers g , generalizing well on “churn” can have the additional beneﬁt of improving classiﬁcation performance, as shown by the proposition below. Proposition 7. Let p (cid:96), d q be deﬁned as in (2) for a strictly proper scoring function φ .
Suppose φ p u q is strictly convex in u , Φ -Lipschitz w.r.t.
L 1 -norm for each y P r m s , and } φ p z q} 8 ă B, @
z P ∆ m .
Let λ ˚ be the optimal mixing coefﬁcient deﬁned in Proposition 1.
Let ∆ C p n, δ q be the churn generalization bound deﬁned in Theorem 2.
Let r h be an optimal feasible classiﬁer in H and p h be the classiﬁer returned by Algorithm 1.
Then for any δ P p 0 , 1 q , w.p. ě
´ δ over draw of S „ D n :
[Equation]
Proof.
Let h ˚ be the Bayes optimal classiﬁer, i.e. the optimal-feasible classiﬁer over all classiﬁers (not just those in H ).
We have:
[Equation]
¯ı
[Equation]
ˇˇˇˇˇˇ E x „ D X » – ÿ y Pr m s p
y p x q p φ y p
h ˚ p x qq ´ φ y p g
p x qqq ﬁ ﬂ ˇˇˇˇˇˇ ď E x „ D X ” p p x q ¨ ´ φ p p h p x qq ´ φ p g
p x qq ¯ı ` E x „ D X » – ÿ y Pr m s p
y p x q ˇ ˇ φ y p
h ˚ p x qq ´ φ y p g
p x qq ˇ ˇ ﬁ ﬂ ď E
[Table]
D
[Table]
” p p x q ¨ ´ φ p p h p x qq ´ φ p g p x qq ¯ı ` Φ E x „ D X » – ÿ y Pr m s p
y p x q} h ˚ p x q ´ g p x q}ﬁ ﬂ
[Equation] where the second-last step follows from Jensen’s inequality, and the last step uses the Lipschitz assumption on φ y . We further have: [Equation]
g p x q} 1 s , where the second step applies Hölder’s inequality to each x , the third step follows from the boundedness assumption on φ , and the fourth step uses the characterization h ˚ p x q “ λ ˚ p p x q`p 1 ´ λ ˚ q g p x q , for λ ˚ P r 0 , 1 s from Proposition 1.
Applying Theorem 2 to the churn C p p h q completes the proof.
This result bounds the excess classiﬁcation risk in terms of the churn generalization bound and the expected difference between the base classiﬁer g and the underlying class probability function p . When the base classiﬁer is close to p , low values of ∆ C p n, δ q result in low classiﬁcation risk. B. G ENERALIZATION BOUND FOR CLASSIFICATION RISK As a follow-up to Proposition 3, we also provide generalization bounds for the classiﬁcation risk in terms of the empirical variance of the loss values based on a result from (Menon et al., 2020,
Proposition 2).
Proposition 8 (Generalization bound for classiﬁcation risk) .
Let the scoring function φ : ∆ m Ñ R m ` be bounded.
Let V φ Ď R X denote the class of loss functions v p x, y q “ (cid:96) φ p y, h p x qq “ φ y p h p x qq induced by classiﬁers h P H .
Let M Rn “ N 8 p 1 n , V φ , 2 n q denote the uniform L 8 covering number for V φ .
Fix δ P p 0 , 1 q .
Then with probability ě 1 ´ δ over draw of S „ D n , for any h P H :
[Equation]
where V Rn p h q denotes the empirical variance of the loss computed on n examples t
φ y i p
h p x
i qqu ni “ 1 .
C D EFINITIONS OF NETWORK ARCHITECTURES USED C. F ULLY CONNECTED NETWORK
FCN-x refers to the following model with size set to "x".
In other words, it’s a simple fully connected network with def get_fcn(n_columns, num_classes=10, size=100, weight_init=
None): model =
None model = tf.keras.
Sequential([ tf.keras.layers.
Input(shape=(n_columns,)), tf.keras.layers.
Dense(size,activation=tf.nn.relu), tf.keras.layers.
Dense(num_classes,activation="softmax"), ]) model.compile( optimizer=tf.keras.optimizers.
Adam(), loss=tf.keras.losses.
CategoricalCrossentropy(), metrics=[tf.keras.metrics.categorical_accuracy]) returnmodel
C. C ONVOLUTIONAL NETWORK
Convnet-x refers to the following model with size set to "x".
Convnet-1 is based on the lenet architecture LeCun et al.
def get_convnet( input_shape=(28, 28, 3), size=1, num_classes=2, weight_init=
None): model = tf.keras.
Sequential() model.add( tf.keras.layers.
Conv2D( filters=16*size, kernel_size=(5,5), padding="same", activation="relu", input_shape=input_shape)) model.add(tf.keras.layers.
MaxPool2D(strides=2))
model.add( tf.keras.layers.
Conv2D( filters=24*size, kernel_size=(5,5), padding="valid", activation="relu")) model.add(tf.keras.layers.
MaxPool2D(strides=2)) model.add(tf.keras layers.
Flatten()) model.add(tf keras. layers.
Dense(128 * size
, activation="relu"))
model.add(
tf.
keras
.layers. Dense
(84, activation="relu"))
model.add
return model C. T RANSFORMER
Transformer-x refers to the following with size set to "x".
It is based on keras tutorial on text classiﬁcation ( https://keras.io/examples/nlp/text_classification_ with_transformer/ licensed under the Apache License, Version 2.0).
def get_transformer(maxlen, size=1, num_classes=2, weight_init=
None): model =
None class TransformerBlock(tf.keras.layers.
Layer): def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1, weight_init=
None): super(TransformerBlock, self).__init__() self.att =
tf.keras.layers.
MultiHeadAttention( num_heads=num_heads, key_dim=embed_dim)
self.ffn = tf.keras.
Sequential([ tf.keras.layers.
Dense(ff_dim, activation="relu"), tf.keras.layers.Dense(embed_dim), ]) self.layernorm
=
tf.keras.layers.
LayerNormalization(epsilon=1e-6) self.layernorm
=
tf.keras.layers.
LayerNormalization(epsilon=1e-6) def call(self, inputs, training): attn_output = self.att(inputs, inputs)
#attn_output = self.dropout1(attn_output, training=training) out
= self.layernorm1(inputs + attn_output) ffn_output = self.ffn(out1) return self.layernorm2(out1 + ffn_output) class TokenAndPositionEmbedding(tf.keras.layers.
Layer): def __init__( self, maxlen, vocab_size, embed_dim, ): super(TokenAndPositionEmbedding, self).__init__() self.token_emb =
tf.keras.layers.
Embedding( input_dim=vocab_size, output_dim=embed_dim) self.pos_emb
= tf.keras.layers.
Embedding( input_dim=maxlen, output_dim=embed_dim) def call(self, x): maxlen tf. 1] positions tf. positions
.pos return x + positions embed_dim = 32 * size # Embedding size for each token num_heads = 2 * size # Number of attention heads ff_dim = 32 * size # Hidden layer size in feed forward network inside transformer inputs =
tf.keras.layers.
Input(shape=(maxlen,))
embedding_layer
= TokenAndPositionEmbedding(maxlen, 20000, embed_dim) x = embedding_layer(inputs)
transformer_block = TransformerBlock(embed_dim, num_heads, ff_dim, weight_init) x = transformer_block(x) x
= tf.keras.layers.
GlobalAveragePooling1D()(x) outputs =
tf.keras.layers.
Dense(num_classes, activation="softmax")(x) model = tf.keras.
Model(inputs=inputs, outputs=outputs) model.compile( optimizer=tf.keras.optimizers.
Adam(), loss=tf.keras.losses.
CategoricalCrossentropy(), metrics=[tf.keras.metrics.categorical_accuracy]) returnmodel
D M ODEL TRAINING CODE
def model_trainer(get_model,
X_train, y_train,
X_test, y_test, weight_init=
None, validation_data=
None, warm=
True, mixup_alpha=-1, codistill_alpha=-1, distill_alpha=-1, anchor_alpha=-1, anchor_eps=-1): model = get_model() if weight_init is not None and warm: model.set_weights(weight_init) if FLAGS.loss == "squared": model.compile( optimizer=tf.keras.optimizers.
Adam(), loss=tf.keras.losses.
MeanSquaredError(), metrics=[tf.keras.metrics.categorical_accuracy]) callback = tf.keras.callbacks.
EarlyStopping(monitor="val_loss", patience=3) history =
None if distill_alpha >= {+
+}
original_model = get_model() original_model.set_weights(weight_init) y_pred=original_model.predict(X_train)
y_use
=distill_alpha*y_pred+(1-distill_alpha) * y_train history = model.fit( x= train, y=y use, epochs
FLAGS.n epochs, callbacks=[callback], validation_data=validation_data) elif anchor_alpha >= 0 and anchor_eps >= 0:
original_model = get_model() original_model.set_weights _init) y_pred
= original_model.predict(X_train)
y_pred_hard
= np.argmax(y_pred, axis=1) y_hard
=
np.argmax(y_train, axis=1) correct = (y_pred_hard == y_hard) correct = np.tile(correct, (y_train.shape[1], 1)) correct = np.transpose(correct) correct = correct.reshape(y_train.shape)
y_use =
np.where(correct, anchor_alpha * y_pred + (1 - anchor_alpha) * y_train, y_train * anchor_eps) history =
model.fit( x=X_train, y=y_use, epochs=FLAGS.n_epochs, callbacks=[callback], validation_data=validation_data) elif mixup_alpha >= 0:
training_generator = deep_utils.
MixupGenerator
X_train, y_train, alpha=mixup_alpha)() history =
model.fit( x=training_generator, validation_data=validation_data, steps_per_epoch=int(X_train.shape[0] / 32), epochs=FLAGS.n_epochs, callbacks=[callback]) elif codistill_alpha >= 0:
teacher_model = get_model() if weight_init is not None and warm: teacher_model.set_weights(weight_init) val_losses =
[] optimizer = tf.keras.optimizers.
Adam() global_step =alpha =
codistillation_warmup_steps
=for epoch in range(FLAGS.n_epochs):
X_train_, y_train_ = sklearn.utils.shuffle(X_train, y_train) batch_size =
for i in range(int(X_train_.shape[0] / batch_size)): if global_step >= codistillation_warmup_steps: alpha =
codistill_alpha else: alpha = 0.
with tf.
GradientTape
() as tape: batch
:] prob_student = model batch, training=
True)
prob_teacher =
teacher_model(
, training
True) loss = deep_utils.compute_loss(prob_student, prob_teacher, y_batch, alpha) trainable_weights =
model.trainable_weights + teacher_model.trainable_weights grads =
tape.gradient(loss, trainable_weights) optimizer.apply_gradients(zip(grads, trainable_weights))
global_step +=val_preds = model.predict(validation_data[0]) val_loss = np.sum( deep_utils.cross_entropy(validation_data[
].astype( , val_preds)) val_losses.append(val_loss) if len(val_losses) > 3 and min(val_losses[-3:]) >
val_losses[-4]: break else: history =
model.fit(
X_train, y_train, epochs=FLAGS.n_epochs, callbacks=[callback], validation_data=validation_data) y_pred_train = model.predict(X_train)
y_pred_test = model.predict(X_test) return y_pred_train, y_pred_test, model.get_weights()
E A DDITIONAL E XPERIMENTAL R ESULTS E. A DDITIONAL O PEN ML RESULTS E.1. I NITIAL SAMPLE 100, BATCH SIZE 1000, VALIDATION SIZE
In Tables 3 and we show the churn at cold accuracy metric across network sizes (fcn-10, fcn-100, fcn-1000, fcn-10000, fcn-100000).
Table 5 shows the standard error bars.
They are obtained by ﬁxing the dataset and model and taking the 100 accuracy and churn results from each baseline and calculating the standard error, which is the standard deviation of the mean.
We then report the average standard error across the baselines We see that distillation is the best 52% of the time.
E.1. I NITIAL SAMPLE 1000, BATCH SIZE 1000, VALIDATION SIZE
In Tables {+
+} and we show the churn at cold accuracy metric across network sizes (fcn-10, fcn-100, fcn-1000, fcn-10000, fcn-100000).
We see that distillation consistently performs strongly across datasets and sizes of networks.
Table 8 shows the standard error bars.
We see that distillation is the best 84% of the time
. E. A DDITIONAL MNIST VARIANT RESULTS E.2. I NITIAL SAMPLE SIZE 100, BATCH SIZE 1000, VALIDATION SIZE
We show full results in Table 9.
We see that distillation is the best for 24 out of the 50 combinations of dataset and network.
Error bands can be found in Table 10.
E.2. I NITIAL SAMPLE SIZE 1000, BATCH SIZE 1000, VALIDATION SIZE
We show full results in Table 11.
We see that distillation is the best for 42 out of the 50 combinations of dataset and network.
Error bands can be found in Table 12.
E.2. I NITIAL SAMPLE SIZE 10000, BATCH SIZE 1000, VALIDATION SIZE
We show full results in Table 13.
We see that in this situation, label smoothing starts becoming competitive with distillation with either of them being the best.
Distillation is the best for 32 out of the 50 combinations of dataset and network and losing marginally to label smoothing in other cases.
See Table 14 for error bands.
E. A DDITIONAL SVHN AND CIFAR RESULTS E.3. I NITIAL SAMPLE 100, BATCH SIZE 1000, VALIDATION SIZE
Results are in Table 15, where we see that distillation is best on 8 out of 10 combinations of dataset and network.
Error bands can be found in Table 16.
E.3. I NITIAL SAMPLE 1000, BATCH SIZE 1000, VALIDATION SIZE
The results can be found in Table 17.
We include the error bands here in Table 18.
Distillation is best in all combinations.
E.3. I NITIAL SAMPLE 10000, BATCH SIZE 1000, VALIDATION SIZE
Results are in Table 19, where we see that distillation is best on all combinations of dataset and network.
Error bands can be found in Table 20.
E. A DDITIONAL C ELEB A RESULTS E.4. I NITIAL S AMPLE 100, BATCH SIZE 1000, VALIDATION SIZE
Tables 21, 22, 23, and 24 show the performance of CelebA tasks when we instead use an initial sample size of 100 .
We see that across the 200 combinations of task and network, distillation is the best 192 of time, or 96% of the time.
The error bands can be found in Table 25.
E.4. I NITIAL SAMPLE 1000, BATCH SIZE 1000, VALIDATION SIZE
We show some additional CelebA results for initial sample 1000 and batch size 1000 in Tables 26, 27, and 29 which show performance for each dataset across convnet-1, convnet-2, convnet-4, convent-8, convnet-16.
This gives us 40 ¨ 5 “ 200 results, of which distillation performs the best 158 out of those settings or 79% of the time.
The error bands can be found in Table 30.
E.4. I NITIAL SAMPLE SIZE 10000, BATCH SIZE 1000, VALIDATION SIZE
Tables 31, 32, 33, and 34 show the performance of CelebA tasks when we instead use an initial sample size of 10000 .
We see that across the 200 combinations of task and network, distillation is the best 183 of time, or 91 .
5% of the time.
The error bands can be found in Table 35.
E. CIFAR10 AND CIFAR100 ON R ES N ET
Results can be found in Table 36.
We see that distillation outperforms in every case.
E. A DDITIONAL IMDB RESULTS
In Table 37, we show the results for the IMDB dataset and transformer networks for initial batch sizes of 100 , 1000 and 10000 with the batch size ﬁxed at 1000 .
The error bands can be found in Table 38.
We see that for initial sample size of 100 , distillation performs poorly for the smaller networks as the process of distillation hurts the performance with a weak teacher trained on only 100 examples, but performs well for the larger networks.
For initial sample size of 1000 and 10000 , distillation is the clear winner losing in only one instance.
We show the full Pareto frontiers and cost curves in
Figure 4.
[Table]
N/A 11.
fcn23.
N/A 21.
N/A
N/A fcn24.
N/A 24. N/A 24.
N/A
N/A magic fcn30.
fcn32.
N/A 28.
N/A 25.
N/A
N/A fcn32.
N/A 29.
N/A 28.
fcn30.
fcn27.
phonemes fcn18.
N/A N/A
N/A 18.
N/A fcn18.
N/A 17.
N/A
N/A N/A
N/A 16.
fcn18.
N/A
N/A 18.
fcn20.
fcn19.
electricity fcn38.
N/A 30.
fcn
[Table]
N/A N/A
N/A N/A N/A
N/A N/A fcn
[Table]
N/A 33.
fcn35.
N/A 35.
fcn
[Table]
N/A
N/A 32.
N/A N/A
[Table] [Table]
N/A N/A
N/A N/A
N/A N/A
N/A
[Table]
N/A 41.
[Table]
N/A 44.
N/A 27.
[Table]
N/A N/A
N/A N/A N/A
N/A N/A fcn
[Table] [Table]
[Table]
warm
[Table]
anchor
[Table]
N
[Table] [Table]
N
[Table]
N/A
N
[Table]
[Table]
[Table]
[Table] [Table]
N
[Table]
[Table]
[Table]
N
[Table]
[Table] [Table]
[Table] [Table]
N
[Table] [Table]
[Table] [Table]
[Table] [Table]
elevators
[Table]
N
[Table]
N
[Table] [Table]
N
[Table] [Table]
N
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
N
[Table]
pollen
[Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
N
[Table]
N/A
N/A
N/A
N/A 36.
fcn45.
N/A 45.
N/A fcn45.
N/A 46.
N/A 48.
phishing fcn9.
N/A 8.
N/A 8. fcn7.
N/A 7.
N/A
N/A 7. fcn8.
N/A
N/A 7.
N/A
N/A fcn9.
fcn10.
N/A 9.
N/A wilt fcn7.
N/A 6.
N/A N/A
N/A N/A
N/A fcn3.
N/A 3.
N/A 2. fcn6.
N/A 0. fcn5.
N/A 0.
fcn7.
letters fcn91.
N/A 92.
[Table] [Table]
churn 1.
elevators 0.
pollen 0.
phishing 0.
wilt 0.
letters 0.
[Table]
fcn26.
fcn18.
phonemes fcn12.
N/A
N/A 10.
fcn9.
N/A
N/A 8.
fcn10.
N/A 9.
fcn13.
N/A 12.
fcn14.
electricity fcn16.
fcn17.
fcn18.
fcn19.
fcn20.
eeg fcn47.
[Table]
[Table]
[Table]
[Table]
N
[Table]
N
[Table] N [Table] [Table]
N
[Table]
[Table]
[Table]
Dataset network cold warm s [Table] mixup ls co [Table]
anchor
[Table]
fcn
[Table]
N/A 14.
fcn [Table]
N/A 20.
fcn [Table]
N/A 18.
fcn [Table]
N/A 20.
fcn [Table]
elevators fcn
[Table] fcn
[Table] fcn [Table]
fcn [Table] fcn
[Table]
N/A 22. N/A 22.
N/A
N/A pollen fcn46.
N/A 23.
N/A 31.
N/A 33.
N/A fcn
[Table] fcn [Table]
N/A 42.
fcn45.
N/A 41.
fcn45.
N/A 43.
phishing fcn3.
N/A N/A
N/A N/A
N/A N/A
N/A fcn3.
N/A 3. fcn4.
fcn5. fcn5.
wilt fcn4.
N/A 2. fcn4.
fcn9.
N/A 7.
fcn11.
N/A 9. fcn5.
letters fcn38.
N/A 25.
N/A N/A
N/
[Table] [Table]
churn 1.
elevators 0.
pollen 0.
phishing 0.
wilt 0.
letters 0.
Dataset network cold warm
[Table]
mixup
[Table]
anchor
[Table]
mnist
[Table]
convnet
[Table]
[Table]
fashion mnist
[Table]
convnet
[Table]
convnet
[Table]
[Table] [Table]
convnet-
[Table] [Table]
[Table]
[Table]
emnist balanced convnet93.
[Table] [Table]
[Table] [Table]
N/
[Table]
[Table] [Table]
N/
[Table]
convnet-
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
N/
[Table]
convnet85.
convnet84.
convnet88.
N/A 87. N/A 87.
emnist byclass convnet73.
convnet68.
N/A convnet64.
convnet67.
convnet63.
emnist bymerge convnet76.
N/A convnet71.
N/A 67. convnet61.
N/A 58.
N/A convnet65.
convnet65.
emnist letters convnet77.
N/A 74.
N/A convnet68.
N/A convnet61.
N/A 59. convnet63.
N/A 62.
N/A 60.
N/A convnet66.
N/A 66.
N/A 65.
N/A emnist digits convnet33.
convnet23.
N/A 22. N/A 22.
N/A N/A
N/A convnet23.
N/A 20.
N/A convnet18.
N/A 19.
N/A 16. convnet21.
N/A 23.
N/A 20.
emnist mnist convnet33.
N/A 32.
N/A
N/A convnet22.
N/A 21.
N/A convnet19.
N/A
N/A 19.
N/A N/A
N/A convnet21.
N/A 20. N/A 20.
N/A convnet22.
N/A
N/A 22.
N/A 21.
N/A kmnist convnet53.
N/A convnet42.
convnet40.
N/A 37.
N/A 37. convnet39.
N/A 40.
N/A 37. convnet41.
N/A
N/A 40.
N/A 38.
N/A k49 mnist convnet93.
N/A convnet86.
convnet83.
N/A 82. convnet76. convnet79.
convnetconvnet- convnetconvnet- convnetDataset
Error
[Table]
Error
[Table]
Error
[Table]
Error
[Table]
mnist 0.
fashion mnist 0.
emnist balanced 0.
emnist byclass 0.
emnist bymerge 0.
emnist letters 0. emnist digits 0.
emnist mnist 0.
kmnist 0.
k49 mnist 0.
Dataset network cold warm s-perturb mixup ls co-dist anchor distill mnist convnet11.
N/A 10. convnet6.
N/A 6.
N/A 5. convnet6.
N/A 6.
N/A 5. convnet7.
N/A
N/A 5.
N/A
N/A 5. convnet7.
N/A 8.
N/A 6.
fashion mnist convnet22.
convnet19. convnet18.
N/A 17. convnet17.
N/A 16. convnet17.
N/A 17.
emnist balanced convnet52.
N/A convnet45.
N/A 39. convnet42.
N/A 37.
N/A 33. convnet40.
N/A 35.
N/A 32. convnet42.
N/A 39.
N/A
N/A 29.
emnist byclass convnet44. convnet38.
convnet36.
N/A 32. convnet36.
N/A 32.
N/A
N/A 25. convnet36.
emnist bymerge convnet42.
N/A 34.
N/A convnet37.
N/A 32.
N/A 30.
N/A convnet34.
convnet33.
N/A 30.
N/A N/A
N/A convnet35.
N/A 31.
N/A 28.
emnist letters convnet38.
convnet31.
N/A 28.
N/A 24. convnet29.
N/A 26.
N/A 23. convnet29.
N/A 28.
N/A 22. convnet30.
N/A 27.
N/A 23.
emnist digits convnet9.
N/A 8.
N/A 7. convnet6.
N/A 6.
N/A 5. convnet6.
N/A 6.
N/A 5. convnet6.
N/A
N/A 6.
N/A 5. convnet6.
N/A
N/A 5.
N/A
N/A 5.
emnist mnist convnet9.
N/A 8.
convnet5.
N/A 6.
N/A 4.
convnet6.
N/A 6.
N/A 5. convnet6.
N/A 6.
N/A 5. convnet7.
N/A 7.
N/A 6.
kmnist convnet23.
convnet16.
N/A 15.
N/A 13. convnet15.
N/A 14.
N/A 12. convnet16.
N/A 15.
N/A 12. convnet18.
N/A 16.
N/A 13.
k49 mnist convnet56.
convnet48.
convnet46.
N/A 39. convnet47.
N/A 41.
N/A 36. convnet49.
convnetconvnet- convnetconvnet- convnetDataset
[Table]
Error
[Table]
Error
[Table]
Error
[Table]
mnist 0.
fashion mnist 0.
emnist balanced 0.
emnist byclass 0.
emnist bymerge 0.
emnist letters 0. emnist digits 0.
emnist mnist 0.
kmnist 0.
k49 mnist 0.
Dataset network cold warm s-perturb mixup ls co-dist anchor distill mnist convnet4.
convnet4.
convnet4. convnet4.
convnet4.
fashion mnist convnet14.
convnet13.
convnet13.
convnet12. convnet13.
emnist balanced convnet25.
convnet23.
convnet24. convnet22. convnet22.
emnist byclass convnet23.
N/A 19.
N/A 13. convnet21.
N/A 19.
N/A 12. convnet21.
N/A 19. convnet21.
N/A 19.
N/A 11. convnet21.
N/A 19.
emnist bymerge convnet21.
convnet19. convnet18.
convnet19.
convnet18.
N/A 17.
emnist letters convnet17.
convnet15.
convnet15. convnet15.
convnet15.
emnist digits convnet3.
convnet3.
convnet3. convnet3.
convnet4.
emnist mnist convnet4.
convnet3. convnet3.
convnet3. convnet3.
kmnist convnet8.
convnet7.
convnet7. convnet7.
convnet8.
k49 mnist convnet27.
convnet23.
convnet22.
convnet22. convnet22.
convnetconvnet- convnetconvnet- convnetDataset
Error
Churn
Error
Churn
Error
Churn
Error
Churn
Error
Churn mnist
[Table]
fashion mnist 0.
emnist balanced 0.
emnist byclass 0.
emnist bymerge 0.
emnist letters 0. emnist digits 0.
emnist mnist 0.
kmnist 0.
k49 mnist 0.
[Table]
[Table] [Table]
N/A 39. convnet52.
convnet52.
N/A 47. convnet53.
[Table]
[Table] [Table] [Table] anchor
[Table]
N
[Table]
N
[Table]
[Table] N [Table] [Table]
[Table] [Table] [Table] [Table]
Arched Eyebrows [Table] [Table]
N [Table] [Table] [Table]
N [Table] N/ [Table]
convnet
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] N/ [Table]
convnet
[Table] [Table] [Table] [Table]
[Table]
[Table] N/A [Table]
convnet-
[Table] [Table] [Table]
[Table]
[Table] N/A [Table]
Attractive convnet-
[Table] convnet
[Table]
convnet37.
convnet40.
N/A 40. convnet44.
Bags Under Eyes convnet11.
N/A 7.
N/A 0. convnet11.
convnet12.
convnet10.
N/A 6.
N/A 0. convnet7.
N/A 0.
Bald convnet0.
N/A 0.
convnet0.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
convnet0.
N/A 0.
convnet0.
N/A 0.
Bangs convnet11.
N/A
N/A 10.
N/A
N/A convnet12.
N/A 12. convnet12. convnet12.
N/A 11. convnet13.
N/A 12.
Big Lips convnet3.
N/A 1.
N/A 0.
convnet5.
N/A 1.
N/A 2.
N/A 0.
convnet3.
N/A 2. N/A 2.
N/A 0. convnet2.
N/A 1. N/A 1.
N/A 0. convnet2.
N/A 0. N/A 0.
Big Nose convnet14.
N/A 11.
N/A 0. convnet15.
convnet11.
N/A
N/A N/A
N/A 10.
N/A 2. convnet13.
convnet11.
Black Hair convnet19.
N/A 19.
N/A
N/A convnet21. convnet22.
convnet23.
convnet21.
Blond Hair convnet11.
convnet12.
N/A 12. convnet11. N/A 11. convnet12.
N/A
N/A 10.
N/A 8. convnet13.
Dataset network cold warm s-perturb mixup ls co-dist anchor distill
Blurry convnet0.
N/A 0.
N/A
N/A 0.
convnet0.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
N/A 0.
convnet0. convnet0.
N/A 0. N/A 0.
Brown Hair convnet14.
N/A 12.
N/A 0. convnet14. convnet13.
convnet14.
N/A 12.
N/A 0. convnet13.
N/A 0.
Bushy Eyebrows convnet5.
convnet5.
N/A 4.
N/A 0.
convnet5.
N/A 3.
N/A 0.
convnet6.
N/A 3.
N/A 0.
convnet4.
N/A 0.
Chubby convnet1.
N/A 0. N/A 0.
convnet1. convnet1.
convnet1.
N/A 0.
convnet0.
N/A 0. N/A 0.
Double Chin convnet0.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
convnet1.
N/A 0.
convnet0.
N/A 0.
convnet0.
N/A 0. N/A 0.
Eyeglasses convnet4.
convnet4.
N/A 3.
convnet4.
N/A 4.
convnet4. convnet4.
N/A 3.
Goatee convnet1.
N/A 1. convnet1.
N/A 1.
N/A 0. convnet2.
convnet1.
N/A 0.
convnet1.
Gray Hair convnet2.
convnet2.
convnet2.
N/A 2. convnet2. N/A 1. convnet2.
N/A 1.
Heavy Makeup convnet33.
N/A 32. convnet33.
N/A
N/A 31.
N/A
N/A 30. convnet34.
N/A 32.
N/A 27. convnet34.
convnet37.
High Cheekbones convnet42.
N/A 43.
N/A 41.
N/A
N/A convnet43.
convnet42.
N/A
N/A 43. convnet42.
N/A 40.
N/A 37. convnet44.
Dataset network cold warm s
[Table]
mixup ls co
[Table]
anchor distill
Male convnet32.
N/A 32.
N/A 33. convnet32.
convnet31.
N/A
N/A 31. convnet34.
convnet37.
N/A 35.
Mouth Slightly Open convnet44.
N/A 44. convnet46.
N/A 42. convnet45.
N/A 45. convnet45.
N/A N/A
N/A 46.
N/A 47. convnet51.
N/A N/A
N/A N/A
N/A N/A
N/A
Mustache convnet0.
N/A 0.
convnet0.
N/A 0.
convnet0. convnet0.
convnet0.
Narrow Eyes convnet0.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
convnet0. convnet0.
N/A 0.
N/A N/A
N/A 0.
convnet0.
N/A 0. N/A 0.
No Beard convnet11.
N/A 10. convnet12. convnet13.
convnet12.
N/A 10. convnet11.
Oval Face convnet9.
N/A 7.
N/A 5.
N/A 0. convnet13.
convnet11.
N/A 7.
N/A 6.
N/A 0. convnet7.
N/A 4.
N/A 0.
convnet8.
Pale Skin convnet1.
N/A 1. convnet1.
N/A N/A
N/A 0.
convnet1.
N/A 0.
convnet1.
N/A 0. N/A 0.
convnet1.
Pointy Nose convnet13.
N/A 0. convnet14.
N/A 8.
N/A 7.
N/A 0. convnet10.
N/A 6.
N/A 7.
N/A 0. convnet10.
N/A 0.
convnet8.
N/A 0.
Receding Hairline convnet2.
N/A 2.
N/A 0.
convnet3. convnet3.
N/A 2.
N/A 0. convnet2.
N/A 2.
N/A 0.
convnet3.
Rosy Cheeks convnet1.
N/A 0. N/A 0.
convnet1. convnet1.
N/A 0.
convnet0.
N/A 0. N/A 0.
convnet0.
N/A 0.
Dataset network cold warm s-perturb mixup ls co-dist anchor distill
Sideburns convnet1.
N/A 0.
convnet1.
N/A 0.
convnet1.
N/A 1. convnet1.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
Smiling convnet42.
N/A 40.
N/A 40. convnet42.
N/A 41.
N/A
N/A convnet44.
N/A
N/A 43. N/A 43.
N/A
N/A convnet45.
convnet47.
Straight Hair convnet2.
N/A 0.
convnet3.
N/A 1.
N/A 0.
convnet3.
N/A 1. N/A 1.
N/A 0. convnet2.
N/A 1. N/A 1.
N/A 0. convnet2.
N/A 0. N/A 0.
Wavy Hair convnet27.
convnet24.
N/A 25.
N/A 22.
N/A 22. convnet26.
N/A 25.
N/A 22.
N/A
N/A convnet24.
N/A 24.
N/A 23.
N/A 23. convnet25.
Wearing Earrings convnet6.
N/A 3.
N/A 0. convnet7.
N/A 4.
N/A 0.
convnet5.
N/A 3.
N/A 0.
convnet3.
N/A 2.
N/A 0.
convnet4.
N/A 0.
Wearing Hat convnet3.
convnet3.
convnet3. convnet3.
N/A N/A
N/A 3.
N/A
N/A 2.
convnet3.
Wearing Lipstick convnet33.
convnet32.
N/A 32. convnet34.
N/A
N/A 33.
N/A
N/A 29. convnet36.
N/A
N/A 36. convnet38.
N/A 37.
Wearing Necklace convnet1.
N/A 0. N/A 0.
convnet1.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
convnet1.
N/A 0. N/A 0.
convnet0.
N/A 0.
Wearing Necktie convnet5.
convnet5.
N/A 4.
convnet5.
N/A N/A
N/A 3.
convnet5.
N/A
N/A 5.
N/A 1.
convnet5.
N/A 5.
Young convnet15.
convnet15.
convnet15.
N/A 13. convnet14. convnet13.
N/A 12.
N/A 0.
[Table]
Narrow Eyes 0.
No Beard 0.
Oval Face 0.
Pale Skin 0.
Pointy Nose 0.
Receding Hairline 0.
Rosy Cheeks 0.
Sideburns 0.
Smiling 0.
Straight Hair 0. Wavy Hair 0.
Wearing Earrings
[Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
Wearing Hat
[Table]
[Table]
[Table]
Wearing Lipstick
[Table]
[Table]
Wearing Necklace
[Table]
Wearing Necktie
[Table]
Young
[Table]
[Table]
warm
[Table]
anchor
[Table]
Shadow convnet
[Table]
[Table]
N/A [Table] [Table] [Table]
N/A 1. convnet-
[Table]
N/A
N/A 6.
[Table]
[Table]
N/A 1. convnet-
[Table] N/A [Table] [Table]
[Table]
N/A 1. convnet6.
N/A
N/A 6.
[Table]
N/A 2.
convnet5.
N/A 5.
N/A 1.
Arched Eyebrows convnet19.
N/A
N/A 16.
N/A 15.
N/A 12. convnet19.
N/A 6. convnet21.
N/A
N/A 18.
N/A 17.
N/A 12. convnet20.
N/A 18.
N/A 3. convnet18.
N/A 17.
N/A 11.
Attractive convnet22.
N/A 20.
N/A N/A
N/A N/A
N/A convnet25.
N/A 20.
N/A convnet23.
N/A 21.
N/A 22.
N/A 13. convnet24. convnet24.
N/A 21.
Bags Under Eyes convnet13.
N/A 11.
N/A 10.
N/A 2. convnet14.
N/A 12.
N/A 3. convnet13.
N/A 12.
N/A 2. convnet12.
N/A 11.
N/A 2. convnet12.
N/A 10.
N/A 1.
Bald convnet1.
convnet1.
N/A 0.
convnet1.
N/A 1.
N/A 0. N/A 0.
convnet1. convnet1.
N/A 0.
Bangs convnet6.
convnet7. convnet7.
convnet8.
convnet7.
N/A 7.
N/A 5. N/A 5.
Big Lips convnet7.
N/A 5.
N/A N/A
N/A
N/A 1. convnet7.
N/A 5.
N/A N/A
N/A
N/A 1. convnet6.
N/A 4.
N/A N/A
N/A
N/A 1. convnet6.
N/A N/A
N/A N/A
N/A N/A
N/A convnet4.
N/A 4.
N/A N/A
N/A
Big Nose convnet14.
N/A 12. N/A 12.
N/A 2. convnet15.
N/A N/A
N/A N/A
N/A N/A
N/A convnet15.
N/A 13.
N/A 12.
N/A 2. convnet14.
N/A 13.
N/A 2. convnet13.
N/A 13.
N/A 3.
Black Hair convnet15.
N/A 12.
N/A 13. convnet14.
N/A
N/A N/A
N/A 13.
N/A 10. convnet15.
N/A N/A
N/A 14.
N/A N/A
N/A convnet14.
N/A 14.
N/A
N/A 13. convnet14.
N/A 14.
Blond Hair convnet7.
convnet8. convnet8.
convnet9.
convnet9.
N/A 9.
Dataset network cold warm s-perturb mixup ls co-dist anchor distill Blurry convnet0.
N/A 0. N/A
N/A N/A
N/A
N/A convnet0. N/A 0.
N/A
N/A N/A
N/A
N/A convnet0. N/A 0. N/A N/A
N/A
N/A convnet0. N/A 0.
N/A
N/A N/A
N/A
N/A convnet0. N/A 0. N/A 0. N/A
N/A N/A
Brown Hair convnet14. N/A 13. N/A 6. convnet13.
N/A
N/A N/A N/A
N/A
N/A 6. convnet13. N/A 13. N/A 11. N/A 7. convnet13.
N/A 11. N/A 3. convnet13. N/A 12.
N/A 3.
Bushy Eyebrows convnet7.
N/A 6.
N/A 1. convnet7. N/A 6. convnet7. N/A 6. N/A 6.
N/A 1. convnet6.
N/A 6.
N/A
N/A 2.
convnet5.
N/A 5.
N/A 4.
N/A N/A
N/A
Chubby convnet2.
N/A 1.
N/A 0. convnet2.
N/A 2.
N/A 0. convnet2.
N/A 2.
N/A 0. convnet2.
N/A 1.
N/A 0.
convnet1.
N/A 1.
N/A 0.
Double Chin convnet2.
N/A 1.
N/A 0.
convnet1.
N/A 1.
N/A 0.
N/A 0. convnet2.
N/A 1.
N/A 0. convnet2.
N/A 1.
N/A 0.
convnet1.
N/A 1.
N/A 0.
N/A
N/A 0.
Eyeglasses convnet3.
N/A 2. convnet2.
N/A 2.
convnet3.
N/A 2.
convnet3.
N/A 2.
N/A convnet3.
N/A N/A
N/A 2.
N/A N/A
Goatee convnet3.
N/A N/A
N/A 1.
N/A
N/A 1. convnet3.
N/A 3.
N/A 0.
convnet3.
N/A 2.
N/A 0.
convnet3.
N/A 2. convnet2. N/A 2.
N/A 0.
Gray Hair convnet2.
N/A 2.
N/A 0. convnet2. N/A 2. convnet2.
N/A 2.
convnet3.
N/A 3.
convnet3.
Heavy Makeup convnet17.
N/A 16.
N/A 14.
N/A
N/A convnet17.
N/A 16.
N/A 14.
N/A
N/A convnet19.
N/A 18.
N/A 16.
N/A convnet20.
N/A 20.
N/A 17.
N/A
N/A convnet22.
N/A 20.
High Cheekbones convnet20.
N/A 17.
N/A N/A
N/A N/A
N/A convnet20.
N/A 18.
N/A
N/A 16.
N/A
N/A convnet23.
N/A 20.
N/A
N/A 18.
N/A
N/A convnet24.
N/A 22.
N/A N/A
N/A N/A
N/A convnet29.
N/A 27.
N/A 25.
N/A N/A
[Table]
warm
[Table]
anchor
[Table]
Male
[Table] [Table]
N/A
[Table]
[Table]
[Table] [Table]
[Table]
N/A convnet
[Table]
[Table]
N/A
[Table]
N/A
N/A N/A
N/A convnet-
[Table]
N/A 13.
[Table]
convnet16. N/A 15.
[Table] [Table]
N/A
[Table] convnet- [Table] Mouth Slightly Open convnet22.
N/A 17.
N/A
N/A 16.
N/A
N/A convnet22.
N/A 20. N/A
N/A N/A N/A N/A convnet23.
N/A N/A
N/A N/A
N/A N/A
N/A convnet26.
N/A N/A N/A
N/A N/A N/A N/A convnet31.
Mustache convnet0.
N/A 0.
N/A 0.
convnet0.
N/A 0.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
convnet0.
N/A 0. N/A 0.
N/A
N/A 0.
Narrow Eyes convnet0.
N/A 0. N/A 0.
convnet1.
N/A 0.
N/A N/A
N/A 0.
convnet0.
N/A 0. N/A 0.
N/A N/A
N/A convnet0.
N/A N/A
N/A N/A
N/A N/A
N/A convnet0.
N/A 0.
N/A N/A
N/A N/A
N/A
No Beard convnet12.
N/A 10.
N/A 9. convnet12. convnet12.
N/A 11.
N/A 10.
N/A 6. convnet13.
N/A 12. convnet13.
Oval Face convnet14.
N/A 12.
N/A N/A
N/A
N/A 2. convnet16.
N/A 14.
N/A N/A
N/A 3. convnet13.
N/A 12.
N/A 14.
N/A 2. convnet12.
N/A 11.
N/A
N/A 2. convnet10.
N/A
N/A 8.
N/A N/A
N/A 1.
Pale Skin convnet2.
N/A 2.
N/A 1.
convnet1.
convnet2.
N/A 1.
N/A 0.
convnet1. convnet1.
N/A N/A
N/A 1.
N/A N/A
Pointy Nose convnet15.
N/A 13.
N/A N/A
N/A
N/A 2. convnet16.
N/A 14.
N/A N/A
N/A
N/A convnet13.
N/A 12.
N/A N/A
N/A 3. convnet11.
N/A 9.
N/A 11.
N/A
N/A 2.
convnet9.
N/A 10.
N/A
N/A 3.
Receding Hairline convnet4.
N/A 3.
convnet4.
N/A 4.
N/A 1.
convnet4.
N/A 4.
N/A 3.
N/A 1.
convnet4.
N/A 4.
N/A 1. convnet3.
N/A 3.
N/A
N/A 0.
Rosy Cheeks convnet2.
N/A 2.
N/A 0. convnet2.
N/A 2.
N/A 0.
convnet3.
N/A 2.
N/A 0. convnet2.
N/A 1.
N/A
N/A 0.
convnet1.
N/A 1.
N/A
N/A 0.
Dataset network [Table] warm
[Table]
mixup
[Table]
anchor
[Table]
Sideburns convnet-
[Table]
N
[Table] [Table]
[Table]
[Table]
N/A
[Table]
convnet-
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
convnet
[Table]
[Table]
N
[Table]
A
[Table] [Table] [Table]
[Table]
N/A
[Table]
convnet-
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
Smiling convnet-
[Table]
N/A 13.
N/A
N/A 12. [Table]
[Table] convnet14. N/A [Table]
N/A
N/A 12.
N/A
N/A convnet-
[Table] N/A [Table]
N/A
N/A [Table] [Table] convnet-
[Table] N/A
[Table]
N/A 15.
N/A
N/A convnet23.
N/A [Table] [Table] [Table]
N/A
Straight Hair convnet5.
N/A [Table]
[Table] [Table] N/A [Table]
convnet6.
N/A 5.
N/A N/A
N/A 1. convnet6.
N/A 5.
N/A
N/A N/A
N/A 1.
convnet4.
N/A 4.
N/A
N/A N/A
N/A 1.
convnet5.
N/A 5.
N/A 4.
N/A N/A
N/A
Wavy Hair convnet19.
N/A 17.
N/A 16.
N/A 9. convnet20.
N/A 19.
N/A 18.
N/A
N/A convnet19.
N/A 17.
N/A 7. convnet19.
N/A 17. convnet19.
N/A 19.
N/A 11.
Wearing Earrings convnet12.
convnet10.
N/A 9.
N/A 8.
N/A 1. convnet10.
N/A 9.
N/A 2. convnet7.
N/A 7.
N/A
N/A 1. convnet7.
N/A 7.
N/A
N/A 1.
Wearing Hat convnet2.
convnet2.
N/A 2. convnet2.
convnet3. convnet3.
N/A 3.
N/A 2.
Wearing Lipstick convnet15.
N/A 14.
N/A N/A
N/A N/A
N/A convnet14.
convnet17.
N/A 15.
N/A 11. convnet18.
N/A 16.
N/A 15.
N/A
N/A convnet20.
Wearing Necklace convnet2.
N/A 1. N/A 1.
N/A N/A
N/A convnet2.
N/A 1.
N/A 2.
N/A N/A
N/A convnet1.
N/A 1.
N/A N/A
N/A N/A
N/A convnet2.
N/A N/A
N/A 1.
N/A N/A
N/A convnet1.
N/A 1. N/A 1.
N/A N/A
N/A
Wearing Necktie convnet4.
N/A 3.
convnet4.
N/A 3.
convnet4.
N/A 4. N/A 4.
N/A convnet4.
convnet4.
Young convnet13.
N/A
N/A N/A
N/A 10.
N/A
N/A convnet13.
N/A 12.
N/A 11.
N/A 4. convnet13.
N/A 12.
N/A N/A
N/A 5. convnet13.
N/A 12.
N/A 11.
N/A
N/A convnet12.
N/A 12.
N/A 11.
N/A 6.
[Table]
Narrow Eyes 0.
No Beard 0.
Oval Face 0.
Pale Skin 0.
Pointy Nose 0.
Receding Hairline 0.
Rosy Cheeks 0.
Sideburns 0.
Smiling 0.
Straight Hair 0. Wavy Hair 0.
Wearing Earrings 0.
Wearing Hat 0.
[Table]
Wearing Lipstick 0.
[Table]
Wearing Necklace
[Table] [Table]
Wearing Necktie
[Table]
[Table]
Young
[Table]
Dataset network cold warm s [Table] perturb mixup ls co-dist anchor distill [Table]
convnet6.
N/A 6.
N/A N/A
N/A 1. convnet6.
N/A 6.
N/A N/A
N/A
N/A 1. convnet6.
N/A 5.
N/A N/A
N/A 1. convnet6.
N/A N/A
N/A N/A
N/A
N/A 1. convnet6.
N/A 6.
N/A 5.
N/A 1.
Arched Eyebrows convnet15.
N/A 12.
N/A N/A
N/A 3. convnet15.
N/A 14.
N/A N/A
N/A
N/A 5. convnet14.
N/A 13.
N/A N/A
N/A
N/A 3. convnet15.
N/A N/A
N/A N/A
N/A
N/A 5. convnet15.
N/A 12. N/A 12.
N/A 3.
Attractive convnet16.
N/A
N/A N/A
N/A N/A
N/A 5. convnet16.
N/A 14.
N/A N/A
N/A
N/A 4. convnet16.
N/A 15.
N/A N/A
N/A
N/A 4. convnet16.
N/A 14.
N/A N/A
N/A
N/A 4. convnet16.
N/A 14.
N/A
N/A N/A
N/A 4.
Bags Under Eyes convnet10.
N/A 8.
N/A N/A
N/A
N/A 2. convnet10.
N/A 8.
N/A N/A
N/A
N/A 2.
convnet9.
N/A N/A
N/A N/A
N/A
N/A 3.
convnet9.
N/A 8.
N/A N/A
N/A
N/A 2.
convnet9.
N/A N/A
N/A N/A
N/A N/A
N/A
Bald convnet1.
N/A 1. convnet1. N/A 1. convnet1.
N/A 1. convnet1. N/A 1. convnet1.
N/A 1.
Bangs convnet5.
N/A 4.
N/A
N/A 4.
N/A 1.
convnet5.
N/A N/A
N/A
N/A 4.
N/A 1.
convnet5.
N/A N/A
N/A N/A
N/A
N/A 2.
convnet5.
N/A 5.
N/A
N/A 4.
N/A 2.
convnet5.
N/A N/A
N/A N/A
N/A
N/A 2.
Big Lips convnet8.
N/A 7.
N/A N/A
N/A
N/A 1. convnet7.
N/A 7.
N/A N/A
N/A
N/A 1. convnet7.
N/A N/A
N/A N/A
N/A N/A
N/A convnet6.
N/A N/A
N/A N/A
N/A
N/A 1.
convnet4.
N/A N/A
N/A N/A
N/A
N/A 1.
Big Nose convnet12.
N/A 10.
N/A N/A
N/A
N/A 4. convnet12.
N/A 10.
N/A N/A
N/A
N/A 2. convnet11.
N/A N/A
N/A N/A
N/A N/A
N/A convnet11.
N/A
N/A N/A
N/A N/A
N/A 2. convnet11.
N/A 10.
N/A
N/A N/A
N/A 2.
Black Hair convnet11.
N/A 10.
N/A
N/A 8.
N/A 2. convnet10. N/A 10.
N/A N/A N/A N/A 3. convnet10.
N/A 10.
N/A N/A
N/A
N/A 2. convnet10.
N/A 10.
N/A N/A
N/A
N/A 2. convnet10.
N/A 10.
N/A
N/A N/A
N/A 2.
Blond Hair convnet5.
N/A
N/A N/A
N/A N/A
N/A 2.
convnet5.
N/A 5.
N/A
N/A 4.
N/A 1.
convnet5.
N/A 5.
N/A 4.
N/A 1.
convnet5.
N/A 4.
convnet5.
N/A 5.
Dataset network cold warm s-perturb mixup ls co-dist anchor distill
Blurry convnet0.
N/A 0.
N/A N/A
N/A N/A
N/A convnet0.
N/A 0.
N/A N/A
N/A N/A
N/A convnet0.
N/A N/A
N/A N/A
N/A N/A
N/A convnet0.
N/A N/A
N/A N/A
N/A N/A
N/A convnet0.
N/A N/A
N/A N/A
N/A N/A
N/A
Brown Hair convnet11.
N/A 10.
N/A N/A
N/A
N/A 3. convnet11.
N/A 11.
N/A
N/A N/A
N/A 2. convnet10.
N/A N/A
N/A N/A
N/A
N/A 3. convnet10.
N/A 10.
N/A N/A
N/A
N/A 2. convnet10.
N/A 10.
N/A
N/A N/A
N/A 2.
Bushy Eyebrows convnet7.
N/A 6.
N/A N/A
N/A
N/A 2. convnet7.
N/A 6.
N/A N/A
N/A
N/A 1. convnet7.
N/A 7.
N/A N/A
N/A
N/A 2. convnet6.
N/A 6.
N/A N/A
N/A
N/A 1. convnet6.
N/A 6.
N/A N/A
N/A
N/A 1.
Chubby convnet3.
N/A 2. N/A 2.
N/A
N/A 0. convnet2.
N/A 2.
N/A N/A
N/A
N/A 0. convnet2.
N/A 2.
N/A N/A
N/A
N/A 0. convnet2.
N/A 2.
N/A N/A
N/A
N/A 0. convnet2.
N/A N/A
N/A N/A
N/A
N/A 0.
Double Chin convnet2.
N/A 2.
N/A 1.
N/A
N/A 0. convnet2.
N/A N/A
N/A 1.
N/A
N/A 0. convnet2.
N/A 1.
N/A N/A
N/A
N/A 0. convnet2.
N/A 1.
N/A N/A
N/A
N/A 0.
convnet1.
N/A 2.
N/A N/A
N/A
N/A 0.
Eyeglasses convnet2.
convnet2.
N/A 2.
N/A 0. convnet2.
N/A 1.
N/A 0. convnet2.
N/A 2.
N/A 1.
N/A 1. convnet2.
Goatee convnet3.
N/A 3.
N/A N/A
N/A
N/A 1. convnet3.
N/A N/A
N/A N/A
N/A
N/A 1. convnet3.
N/A N/A
N/A N/A
N/A
N/A 1. convnet3.
N/A 3.
N/A N/A
N/A
N/A 0.
convnet3.
N/A 3.
N/A N/A
N/A
N/A 0.
Gray Hair convnet2.
N/A 2. N/A 2.
N/A 0. convnet2.
N/A 2.
N/A 0. convnet2.
N/A 2. N/A 2.
N/A 0. convnet2.
N/A 2.
N/A 2. convnet2.
N/A 2.
Heavy Makeup convnet11.
N/A 9.
N/A 7.
N/A 3. convnet10.
N/A 9.
N/A N/A
N/A
N/A 3. convnet10.
N/A 9.
N/A
N/A 8.
N/A 3. convnet10.
N/A 9.
N/A
N/A 8.
N/A 3. convnet10.
N/A 9.
N/A
N/A 8.
N/A 3.
High Cheekbones convnet12.
N/A 10.
N/A
N/A 9.
N/A 3. convnet12. N/A 10. N/A N/A
N/A N/A 3. convnet12.
N/A 10.
N/A N/A
N/A
N/A 3. convnet12.
N/A 9.
N/A
N/A 8.
N/A 2. convnet13.
N/A 11.
N/A
N/A N/A
N/A 4.
[Table]
N/A 1. N/A 1.
N/A
N/A 0.
convnet1.
N/A N/A
N/A 1.
N/A N/A
N/A convnet1.
N/A 1. N/A 1.
N/A
N/A 0.
Narrow Eyes convnet4.
N/A 3.
N/A N/A
N/A
N/A 0.
convnet3.
N/A N/A
N/A N/A
N/A
N/A 0.
convnet3.
N/A 2.
N/A N/A
N/A
N/A 0. convnet2.
N/A N/A
N/A N/A
N/A
N/A 0.
convnet1.
N/A 0.
N/A N/A
N/A
N/A 0.
No Beard convnet8.
N/A 6.
N/A
N/A 6.
N/A 2.
convnet8.
N/A 7.
N/A
N/A 6.
N/A 2. convnet7.
N/A 6.
N/A
N/A 6.
N/A 2.
convnet8.
N/A 7.
N/A
N/A 6.
N/A 2.
convnet8.
N/A 7.
N/A 6.
Oval Face convnet14.
N/A 13.
N/A N/A
N/A
N/A 3. convnet14.
N/A 13.
N/A N/A
N/A
N/A 3. convnet14.
N/A N/A
N/A N/A
N/A N/A
N/A convnet14.
N/A 12.
N/A N/A
N/A
N/A 3. convnet12.
N/A 12.
N/A
N/A N/A
N/A 2.
Pale Skin convnet2.
N/A 2.
N/A 0. convnet2.
N/A 2.
N/A N/A
N/A
N/A 0. convnet2.
N/A 2. N/A 2.
N/A 0. convnet2.
N/A 2. N/A 2.
N/A 0. convnet2.
N/A 2.
N/A
N/A 2.
N/A 0.
Pointy Nose convnet15.
N/A 15.
N/A N/A
N/A
N/A 3. convnet15.
N/A 15.
N/A N/A
N/A
N/A 3. convnet15.
N/A 13.
N/A N/A
N/A
N/A 3. convnet13.
N/A N/A
N/A N/A
N/A N/A
N/A convnet11.
N/A
N/A N/A
N/A N/A
N/A 2.
Receding Hairline convnet4.
N/A 4.
N/A
N/A N/A
N/A 1.
convnet4.
N/A 4.
N/A
N/A N/A
N/A 1.
convnet4.
N/A 4.
N/A
N/A N/A
N/A 1.
convnet4.
N/A 4.
N/A
N/A N/A
N/A 1.
convnet4.
N/A N/A
N/A N/A
N/A
N/A 1.
Rosy Cheeks convnet4.
N/A 3.
N/A
N/A 1.
convnet4.
N/A 4.
N/A 1.
convnet4.
N/A 4.
N/A 3.
N/A 1. convnet3.
N/A 3.
N/A
N/A 1. convnet3.
N/A 3. N/A 3.
N/A 0.
Dataset network cold warm s-perturb mixup ls co-dist anchor distill
Sideburns convnet3.
N/A
N/A 2. N/A 2.
N/A 0.
convnet3.
N/A 2.
N/A N/A
N/A
N/A 1. convnet3.
N/A 2. N/A 2.
N/A 0. convnet2.
N/A 2. N/A 2.
N/A 0. convnet2.
N/A 2. N/A 2.
N/A 0.
Smiling convnet9.
N/A 7. N/A 7.
N/A 4.
convnet9.
N/A 8.
N/A
N/A 7.
N/A 3.
convnet9.
N/A 9.
N/A
N/A 6.
N/A 3.
convnet9.
N/A 7.
N/A 6. convnet10.
Straight Hair convnet6.
N/A 6.
N/A N/A
N/A
N/A 1. convnet6.
N/A N/A
N/A N/A
N/A
N/A 1. convnet7.
N/A N/A
N/A N/A
N/A N/A
N/A convnet6.
N/A N/A
N/A N/A
N/A
N/A 1.
convnet5.
N/A N/A
N/A N/A
N/A
N/A 1.
Wavy Hair convnet16.
N/A N/A
N/A N/A
N/A N/A
N/A convnet16.
N/A N/A
N/A N/A
N/A
N/A 5. convnet15.
N/A 14.
N/A 13.
N/A 3. convnet15.
N/A
N/A N/A
N/A N/A
N/A 3. convnet15.
N/A 13.
N/A
N/A N/A
N/A 3.
Wearing Earrings convnet12.
N/A 10.
N/A N/A
N/A 2. convnet12.
N/A 11.
N/A
N/A N/A
N/A 3. convnet12.
N/A 10.
N/A N/A
N/A 3. convnet11.
N/A 10.
N/A 9.
N/A 3. convnet11.
N/A 10.
N/A 9.
N/A 2.
Wearing Hat convnet1.
N/A 1. convnet1. N/A 1. convnet1.
N/A
N/A N/A
N/A 1.
N/A 0.
convnet1.
N/A N/A
N/A
N/A 1.
N/A 0.
convnet1.
Wearing Lipstick convnet9.
N/A 7.
N/A 6.
N/A 3.
convnet9.
N/A 7.
N/A 6.
N/A 2.
convnet8.
N/A
N/A 7.
N/A 6.
N/A 2.
convnet8.
N/A N/A
N/A
N/A 6.
N/A 2.
convnet9.
N/A 7.
N/A
N/A 6.
N/A 2.
Wearing Necklace convnet0.
N/A 0.
N/A N/A
N/A N/A
N/A convnet1.
N/A 1.
N/A N/A
N/A N/A
N/A convnet1.
N/A 1.
N/A N/A
N/A
N/A 0.
convnet1.
N/A 1.
N/A N/A
N/A
N/A 0.
convnet0.
N/A N/A
N/A N/A
N/A
N/A 0.
Wearing Necktie convnet3.
N/A 3.
N/A 2.
N/A 0.
convnet3.
N/A N/A
N/A N/A
N/A
N/A 0.
convnet3.
N/A 2.
N/A 1. convnet3.
N/A 3.
N/A 2.
N/A 1. convnet3.
N/A
N/A N/A
N/A 2.
N/A 1.
Young convnet10.
N/A 9.
N/A N/A
N/A 2. convnet10. N/A 9. N/A N/A
N/A N/A 2. convnet10.
N/A 9.
N/A N/A
N/A
N/A 2. convnet10.
N/A 9.
N/A
N/A N/A
N/A 2.
convnet9.
N/A N/A
N/A
N/A 8.
N/A 2.
[Table]
Narrow Eyes 0.
No Beard 0.
Oval Face 0.
Pale Skin 0.
Pointy Nose 0.
Receding Hairline 0.
Rosy Cheeks 0.
Sideburns 0.
Smiling 0.
Straight Hair 0.
Wavy Hair 0.
Wearing Earrings
[Table] [Table]
[Table] [Table]
Wearing Hat
[Table]
Wearing Lipstick
[Table]
Wearing Necklace
[Table]
Wearing Necktie
[Table] [Table]
[Table]
/A 43. transformer50.
N/A 47.
transformer15.
N/A 15.
N/A
N/A 13. transformer16.
N/A 13. transformer18.
N/A N/A
N/A N/A
N/A 16. transformer20.
N/A 19.
N/A 17. transformer24.
N/A 22.
N/A 20.
transformer8.
N/A N/A
N/A N/A
N/A 8.
transformer9.
N/A N/A
N/A N/A
N/A
N/A 6. transformer9.
N/A
N/A N/A
N/A N/A
N/A 6. transformer9.
N/A N/A
N/A 6.
N/A
N/A 6. transformer12.
N/A N/A
N/A N/A
N/A N/A
N/A transformer
[Table]
transformertransformer
[Table]
transformertransformer-
[Table]