Original: T HERE ARE M ANY C ONSISTENT E XPLANATIONS OF U NLABELED D ATA : W HY Y OU S HOULD A VERAGE
Revised: T HERE ARE M ANY C ONSISTENT E XPLANATIONS OF U NLABELED D ATA : W HY Y OU S HOULD A VERAGE

Original: Anonymous authors

Original: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: Presently the most successful approaches to semi-supervised learning are based on consistency regularization , whereby a model is trained to be robust to small perturbations of its inputs and parameters.
Revised: Presently the most successful approaches to semi-supervised learning are based on consistency regularization , whereby a model is trained to be robust to small perturbations of its inputs and parameters.

Revised: To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures.

Original: The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data.
Revised: The consistency loss dramatically improves generalization performance over supervisedonly training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data.

Original: We show that averaging weights can signiﬁcantly improve their generalization performance.

Original: Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modiﬁed learning rate schedule.
Revised: Motivated by these observations, we propose to train consistencybased methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modiﬁed learning rate schedule.

Original: We also propose fast-SWA , which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule.
Revised: We also propose fast-SWA , which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule.

Original: With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100 over many different settings of  training labels.
Revised: With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data.

Original: For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.
Revised: For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent advances in deep unsupervised learning, such as generative adversarial networks (GANs) (Goodfellow et al., 2014) have led to an explosion of interest in semi-supervised learning.
Revised: Recent advances in deep unsupervised learning, such as generative adversarial networks (GANs) (Goodfellow et al., 2014) have led to an explosion of interest in semi-supervised learning.

Original: Semisupervised methods make use of both unlabeled and labeled training data to improve performance over purely supervised methods.
Revised: Semisupervised methods make use of both unlabeled and labeled training data to improve performance over purely supervised methods.

Original: Semi-supervised learning is particularly valuable in applications such as medical imaging, where labeled data may be scarce and expensive (Oliver et al., 2018).
Revised: Semi-supervised learning is particularly valuable in applications such as medical imaging, where labeled data may be scarce and expensive (Oliver et al., 2018).

Original: Currently the best semi-supervised results are obtained by consistency-enforcing approaches (Bachman et al., 2014; Laine and Aila, 2017; Tarvainen and Valpola, 2017;
Revised: Currently the best semi-supervised results are obtained by consistency-enforcing approaches (Bachman et al., 2014; Laine and Aila, 2017; Tarvainen and Valpola, 2017;

Original: Miyato et al., 2017; Park et al., 2017).
Revised: Miyato et al., 2017; Park et al., 2017).

Original: These methods use unlabeled data to stabilize their predictions under input or weight perturbations.
Revised: These methods use unlabeled data to stabilize their predictions under input or weight perturbations.

Original: Consistency-enforcing methods can be used at scale with state-of-the-art architectures.
Revised: Consistency-enforcing methods can be used at scale with state-of-the-art architectures.

Original: For example, the recent Mean Teacher (Tarvainen and Valpola, 2017) model has been used with the
Revised: For example, the recent Mean Teacher (Tarvainen and Valpola, 2017) model has been used with the

Original: Shake-Shake (Gastaldi, 2017) architecture and has achieved the best semi-supervised performance on the consequential CIFAR benchmarks.
Revised: Shake-Shake (Gastaldi, 2017) architecture and has achieved the best semi-supervised performance on the consequential CIFAR benchmarks.

Original: This paper is dedicated to the understanding  consistency-based semi-supervised learning methods.
Revised: This paper is about conceptually understanding and improving consistency-based semi-supervised learning methods.

Revised: Our approach can be used as a guide for exploring how loss geometry interacts with training procedures in general.

Original: We provide several novel observations about the training objective and optimization trajectories of the popular Π (Laine and Aila, 2017) and Mean Teacher (Tarvainen and Valpola, 2017) consistency-based models.
Revised: We provide several novel observations about the training objective and optimization trajectories of the popular ⇧ (Laine and Aila, 2017) and Mean Teacher (Tarvainen and Valpola, 2017) consistency-based models.

Original: Inspired by these ﬁndings, we propose to improve SGD solutions via stochastic weight averaging (SWA) (Izmailov et al., 2018), a recent method that averages weights of the networks corresponding to different training epochs to obtain a single model with improved generalization.
Revised: Inspired by these ﬁndings, we propose to improve SGD solutions via stochastic weight averaging (SWA) (Izmailov et al., 2018), a recent method that averages weights of the networks corresponding to different training epochs to obtain a single model with improved generalization.

Original: On a thorough empirical study we show that this procedure achieves the best known semi-supervised results on consequential benchmarks.
Revised: On a thorough empirical study we show that this procedure achieves the best known semi-supervised results on consequential benchmarks.

Original: In particular, our contributions are the following:
Revised: In particular:

Original: • We show in Section 3.1 that a simpliﬁed Π model implicitly regularizes the norm of the Jacobian of the network outputs with respect to both its inputs and its weights, which in turn encourages ﬂatter solutions.
Revised: • We show in Section 3.1 that a simpliﬁed ⇧ model implicitly regularizes the norm of the Jacobian of the network outputs with respect to both its inputs and its weights, which in turn encourages ﬂatter solutions.

Original: Both the reduced Jacobian norm and ﬂatness of solutions have been related to generalization in the literature (Sokoli´c et al., 2017;
Revised: Both the reduced Jacobian norm and ﬂatness of solutions have been related to generalization in the literature (Sokoli´c et al., 2017;

Original: Novak et al., 2018; Chaudhari et al., 2016; Schmidhuber and Hochreiter, 1997; Keskar et al., 2017; Izmailov et al., 2018).
Revised: Novak et al., 2018; Chaudhari et al., 2016; Schmidhuber and Hochreiter, 1997; Keskar et al., 2017; Izmailov et al., 2018).

Original: Interpolating between the weights corresponding to different epochs of training we demonstrate that the solutions of Π and Mean Teacher models are indeed ﬂatter along these directions (Figure 1b).
Revised: Interpolating between the weights corresponding to different epochs of training we demonstrate that the solutions of ⇧ and Mean Teacher models are indeed ﬂatter along these directions (Figure 1b).

Original: • In Section 3.2, we compare the training trajectories of the Π , Mean Teacher, and supervised models and ﬁnd that the distances between the weights corresponding to different epochs are much larger for the consistency based models.
Revised: • In Section 3.2, we compare the training trajectories of the ⇧ , Mean Teacher, and supervised models and ﬁnd that the distances between the weights corresponding to different epochs are much larger for the consistency based models.

Original: The error curves of consistency models are alsomore wider (Figure 1b), which can be explained by  ﬂatness of the solutions discussed in section 3.1.
Revised: The error curves of consistency models are also wider (Figure 1b), which can be explained by the ﬂatness of the solutions discussed in section 3.1.

Original: Further we observe that the predictions of the SGD iterates can differ signiﬁcantly between different iterations of SGD.
Revised: Further we observe that the predictions of the SGD iterates can differ signiﬁcantly between different iterations of SGD.

Original: • The observation that  SGD struggles to converge to an isolated solution for consistency-based methods motivates us to apply ensembling to  the predictions of SGD iterates or average their weights.
Revised: • We observe that for consistency-based methods, SGD does not converge to a single point but continues to explore many solutions with high distances apart. Inspired by this observation, we propose to average the weights corresponding to SGD iterates, or ensemble the predictions of the models corresponding to these weights.

Original: Averaging weights of SGD iterates allows to compensate for larger steps, stabilize SGD trajectories and obtain a solution that is centered in avast ﬂat region of the loss in the weight space .
Revised: Averaging weights of SGD iterates compensates for larger steps, stabilizes SGD trajectories and obtains a solution that is centered in a ﬂat region of the loss (as a function of weights).

Original: Further, both ensembling and weight averaging allow us to make use of the improved diversityof the SGD iterates and obtain a  solution that is centered in the prediction space .
Revised: Further, we show that the SGD iterates correspond to models with diverse predictions – using weight averaging or ensembling allows us to make use of the improved diversity and obtain a better solution compared to the SGD iterates.

Original: In Section 3.3 we demonstrate that both ensembling predictions and averaging weights of the networks corresponding to different training epochs signiﬁcantly improve generalization performance and ﬁnd that the improvement is much larger for the Π and Mean Teacher models compared to supervised training.
Revised: In Section 3.3 we demonstrate that both ensembling predictions and averaging weights of the networks corresponding to different training epochs signiﬁcantly improve generalization performance and ﬁnd that the improvement is much larger for the ⇧ and Mean Teacher models compared to supervised training.

Original: We ﬁnd that averaging weights provides similar or larger beneﬁts compared to ensembling, while being more practical.
Revised: We ﬁnd that averaging weights provides similar or improved accuracy compared to ensembling, while offering the computational beneﬁts and convenience of working with a single model.

Original: Thus, we focus on weight averaging for the remaining of the paper.
Revised: Thus, we focus on weight averaging for the remainder of the paper.

Original: • Motivated by our observations in Section 3 we propose to apply Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) to the Π and Mean Teacher models.
Revised: • Motivated by our observations in Section 3 we propose to apply Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) to the ⇧ and Mean Teacher models.

Original: Based on our results in Section 3.3 we propose several modiﬁcations to SWA in Section 4 and refer to the modiﬁed procedure as fast-SWA: we use a learning rate schedule with longer cycles to increase the distance between the weights that are averaged and the diversity of the corresponding predictions; we also average weights of multiple networks within each cycle while SWA only averagesthe weights corresponding to the lowest values of the learning rate within each cycle.
Revised: Based on our results in Section 3.3 we propose several modiﬁcations to SWA in Section 4. In particular, we propose fast-SWA, which (1) uses a learning rate schedule with longer cycles to increase the distance between the weights that are averaged and the diversity of the corresponding predictions; and (2) averages weights of multiple networks within each cycle (while SWA only averages weights corresponding to the lowest values of the learning rate within each cycle).

Original: In Section 5, we show that fast-SWA improves the performance substantially faster than SWA.
Revised: In Section 5, we show that fast-SWA converges to a good solution much faster than SWA.

Original: • Applying weight averaging to the Π and Mean Teacher models we improve the best reported results on CIFAR-10 for 1 k , 2 k , 4 k and 10 k labeled examples, as well as on CIFAR-100 with 10 k labeled examples.
Revised: Applying weight averaging to the ⇧ and Mean Teacher models we improve the best reported results on CIFAR-10 for 1 k , 2 k , 4 k and 10 k labeled examples, as well as on CIFAR-100 with 10 k labeled examples.

Original: For example, we obtain 5 . 0% error on CIFAR-10 with only 4 k labels, improving the best result reported in the literature (Tarvainen and Valpola, 2017) by 1 . 3% .
Revised: For example, we obtain 5 . 0% error on CIFAR-10 with only 4 k labels, improving the best result reported in the literature (Tarvainen and Valpola, 2017) by 1 . 3% .

Original: We also apply weight averaging to a state-of-the-art domain adaptation technique (French et al., 2018) closely related to the Mean Teacher model and improve the best reported results on domain adaptation from CIFAR-10 to STL from 19 .
Revised: We also apply weight averaging to a state-of-the-art domain adaptation technique (French et al., 2018) closely related to the Mean Teacher model and improve the best reported results on domain adaptation from CIFAR-10 to STL from 19 .

Original: 9% to 16 .
Revised: 9% to 16 .

Original: 8% error.
Revised: 8% error.

Original: • Wewill release our code to replicate the experiments upon publication.
Revised: • We release our code at https://github.com/benathi/fastswa-semi-sup

Original:  B ACKGROUND 2. C ONSISTENCY B ASED M ODELS
Revised:  B ACKGROUND 2. C ONSISTENCY B ASED M ODELS

Original: We brieﬂy review semi-supervised learning with consistency-based models.
Revised: We brieﬂy review semi-supervised learning with consistency-based models.

Original: This class of models encourages predictions to stay similar under small perturbations of inputs or network parameters.
Revised: This class of models encourages predictions to stay similar under small perturbations of inputs or network parameters.

Original: For instance, two different translations of the same image should result in similar predicted probabilities.
Revised: For instance, two different translations of the same image should result in similar predicted probabilities.

Original: The consistency of a model (student) can be measured against its own predictions (e.g. Π model) or predictions of a different teacher network (e.g. Mean Teacher model).
Revised: The consistency of a model (student) can be measured against its own predictions (e.g. ⇧ model) or predictions of a different teacher network (e g. Mean Teacher model).

Original: In both cases we will say a student network measures consistency against a teacher network
Revised: In both cases we will say a student network measures consistency against a teacher network.

Original: Consistency Loss
Revised: Consistency Loss

Original: In the semi-supervised setting, we have access to labeled data D L = { ( x Li , y Li ) } N L
Revised: In the semi-supervised setting, we have access to labeled data D L = { ( x Li , y Li ) } N L

Original: i
Revised: i

Original: =1 , and unlabeled data D U
Revised: =1 , and unlabeled data D U

Original: = { x Ui } N U
Revised: = { x Ui } N U

Original: i =1 . Given two perturbed inputs x (cid:48) , x (cid:48)(cid:48) of x and the perturbed weights w (cid:48) f and w (cid:48) g , the consistency loss penalizes the difference between the student ’s predicted probablities f ( x (cid:48) ; w (cid:48) f ) and the teacher ’s g ( x (cid:48)(cid:48) ; w (cid:48) g ) .
Revised: Given two perturbed inputs x 0 , x 00 of x and the perturbed weights w 0 f and w 0 g , the consistency loss penalizes the difference between the student ’s predicted probablities f ( x 0 ; w 0 f ) and the teacher ’s g ( x 00 ; w 0 g ) .

Original: This loss is typically the Mean Squared Error or KL divergence:
Revised: This loss is typically the Mean Squared Error or KL divergence:

Original: [Equation]
Revised: [Equation]

Original: The total loss used to train the model can be written as
Revised: The total loss used to train the model can be written as

Original: [Equation]
Revised: [Equation]

Original: (cid:88) x ∈D

Original: [Equation]

Original: ∪D U (cid:96)

Original: [Equation]

Original: [Equation]

Original: , x ) (cid:124) (cid:123)(cid:122) (cid:125) L

Original: [Equation]

Original: , (2) where for classiﬁcation L CE is the cross entropy between the model predictions and supervised training labels.
Revised: where for classiﬁcation L CE is the cross entropy between the model predictions and supervised training labels.

Original: The parameter λ > 0 controls the relative importance of the consistency term in the overall loss.
Revised: The parameter  > 0 controls the relative importance of the consistency term in the overall loss.

Original: Π Model The Π model, introduced in Laine and Aila (2017) and Sajjadi et al.
Revised: ⇧ Model The ⇧ model, introduced in Laine and Aila (2017) and Sajjadi et al.

Original: (2016), uses the student model f as its own teacher.
Revised: (2016), uses the student model f as its own teacher.

Original: The data (input) perturbations include random translations, crops, ﬂips and additive Gaussian noise.
Revised: The data (input) perturbations include random translations, crops, ﬂips and additive Gaussian noise.

Original: Binary dropout (Srivastava et al., 2014) is used for weight perturbation.
Revised: Binary dropout (Srivastava et al., 2014) is used for weight perturbation.

Original: Mean Teacher Model
Revised: Mean Teacher Model

Original: The Mean Teacher model (MT) proposed in Tarvainen and Valpola (2017) uses the same data and weight perturbations as the Π model; however, the teacher weights w g are the exponential moving average (EMA) of the student weights w f : w kg =
Revised: The Mean Teacher model (MT) proposed in Tarvainen and Valpola (2017) uses the same data and weight perturbations as the ⇧ model; however, the teacher weights w g are the exponential moving average (EMA) of the student weights w f : w kg =

Original: α · w k − 1 g
Revised: ↵ · w k  1 g

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: kf .

Original: The decay rate α is usually set between 0 . 9 and 0 .
Revised: The decay rate ↵ is usually set between 0 . 9 and 0 .

Original: The Mean Teacher model has the best known results on the CIFAR-10 semi-supervised learning benchmark.
Revised: The Mean Teacher model has the best known results on the CIFAR-10 semi-supervised learning benchmark (Tarvainen and Valpola, 2017).

Original: Other Consistency-Based Models Temporal Ensembling (TE) (Laine and Aila, 2017) uses an exponential moving average of the student outputs as the teacher outputs in the consistency term for training.
Revised: Other Consistency-Based Models Temporal Ensembling (TE) (Laine and Aila, 2017) uses an exponential moving average of the student outputs as the teacher outputs in the consistency term for training.

Original: Another approach, Virtual Adversarial Training (VAT) (Miyato et al., 2017), enforces the consistency between predictions on the original data inputs and the data perturbed in an adversarial direction x (cid:48) = x + (cid:15)r adv , where r adv = arg max r : (cid:107) r (cid:107) =1 KL [ f ( x, w ) (cid:107) f
Revised: Another approach, Virtual Adversarial Training (VAT) (Miyato et al., 2017), enforces the consistency between predictions on the original data inputs and the data perturbed in an adversarial direction x 0 = x + ✏ r adv , where r adv = arg max r : k r k =1 KL [ f ( x, w ) k

Original:  ( x + ξr, w )] .
Revised: f ( x + ⇠ r, w )] .

Original:  U NDERSTANDING C ONSISTENCY -E NFORCING M ODELS
Revised:  U NDERSTANDING C ONSISTENCY -E NFORCING M ODELS

Original: In Section 3.1, we study a simpliﬁed version of the Π model theoretically and show that it penalizes the norm of the Jacobian of the outputs with respect to inputs as well as the eigenvalues of theHessian both of which have been related to generalization in the literature (Sokoli´c et al., 2017;
Revised: In Section 3.1, we study a simpliﬁed version of the ⇧ model theoretically and show that it penalizes the norm of the Jacobian of the outputs with respect to inputs, as well as the eigenvalues of the

Original:  Novak et al., 2018; Dinh et al., 2017a; Chaudhari et al., 2016).
Revised: Hessian, both of which have been related to generalization (Sokoli´c et al., 2017; Novak et al., 2018; Dinh et al., 2017a; Chaudhari et al., 2016).

Original: In Section 3.2 we empirically study the training trajectories of the Π and MT models and compare them to the training trajectories in supervised learning.
Revised: In Section 3.2 we empirically study the training trajectories of the ⇧ and MT models and compare them to the training trajectories in supervised learning.

Original: We show that even late in training consistency-based methods make large training steps leading to signiﬁcant changes in predictions on test.
Revised: We show that even late in training consistency-based methods make large training steps, leading to signiﬁcant changes in predictions on test.

Original: In section 3.3 we show that averaging weights or ensembling predictions of the models proposed by SGD at different training epochs can lead to substantial gains in accuracy and that these gains are much larger for Π and MT than for supervised training.
Revised: In Section 3.3 we show that averaging weights or ensembling predictions of the models proposed by SGD at different training epochs can lead to substantial gains in accuracy and that these gains are much larger for ⇧ and MT than for supervised training.

Original: 3. S IMPLIFIED Π M ODEL P ENALIZES L OCAL S HARPNESS
Revised: 3. S IMPLIFIED ⇧ M ODEL P ENALIZES L OCAL S HARPNESS

Original: Penalization of the input-output Jacobian norm Consider a simple version of the Π model, where we only apply small additive perturbations to the student inputs: x (cid:48) = x + (cid:15)z, z ∼ N (0 , I ) with (cid:15) (cid:28) 1 , and the teacher input is unchanged: x (cid:48)(cid:48){+00 = x. 
+}
Revised: Penalization of the input-output Jacobian norm. Consider a simple version of the ⇧ model, where we only apply small additive perturbations to the student inputs: x 0 = x + ✏ z, z ⇠ N (0 , I ) with ✏ ⌧ 1 , and the teacher input is unchanged: x {+00 = x. 
+}

Original: [-= x. 
-] Then the consistency loss (cid:96) cons (Eq. 1) becomes (cid:96) cons ( w, x, (cid:15) ) =(cid:107) f
Revised: [-= x. 
-] Then the consistency loss ` cons (Eq. 1) becomes ` cons ( w, x, ✏ ) =

Original:  ( w, x + (cid:15)z ) − f ( w, x ) (cid:107) 2 .
Revised: k f ( w, x + ✏ z )  f ( w, x ) k 2 .

Original: Consider the estimator ˆ Q = lim (cid:15)
Revised: Consider the estimator ˆ Q = lim ✏

Original: [Equation]
Revised: [Equation]

Original: 2 1 m (cid:80) mi =1 (cid:96) cons ( w, x i , (cid:15) ) , we show in Section A.5 that [Equation] [Equation] Q ]
Revised: , x i , ✏ ) .

Original: [Equation]

Original: E x [ (cid:107)

Original: [Equation]

Original: x (cid:107)

Original: [Equation] F ] and Var [ ˆ Q ] [Equation] (cid:19)
Revised: We show in Section A.5 that

Original: [Equation]
Revised: [Equation]

Original: where J x is the Jacobian of the network’s outputs with respect to its inputs evaluated at x , (cid:107) · (cid:107) F represents Frobenius norm and the expectation E x is taken over the distribution of labeled and unlabeled data.
Revised: where J x is the Jacobian of the network’s outputs with respect to its inputs evaluated at x , k · k F represents Frobenius norm, and the expectation E x is taken over the distribution of labeled and unlabeled data.

Original: That is, ˆ Q is an unbiased estimator of E x [ (cid:107) J x (cid:107) 2 F ] with variance controlled by the minibatch size m .
Revised: That is, ˆ Q is an unbiased estimator of E x [ k J x k 2 F ] with variance controlled by the minibatch size m .

Original: Therefore, the consistency loss implicitly penalizes E x [ (cid:107) J x (cid:107) 2 F ] .
Revised: Therefore, the consistency loss implicitly penalizes E x [ k J x k 2 F ] .

Original: The quantity || J x || F has been related to generalization both theoretically (Sokoli´c et al., 2017) and empirically (Novak et al., 2018).
Revised: The quantity || J x || F has been related to generalization both theoretically (Sokoli´c et al., 2017) and empirically (Novak et al., 2018).

Original: For linear models f ( x ) =
Revised: For linear models f ( x ) =

Original: Wx , penalizing ||
Revised: Wx , penalizing ||

Original: J x || F exactly corresponds to weight decay, also known as L 2 regularization, since for linear models J x = W , and (cid:107) W (cid:107) 2 F = (cid:107) vec ( W ) (cid:107) 22 .
Revised: J x || F exactly corresponds to weight decay, also known as L 2 regularization, since for linear models J x = W , and k W k 2 F = k vec ( W ) k 22 .

Original: Penalizing E x [ (cid:107) J x (cid:107) 2 F ] is also closely related to the graph based (manifold) regularization in (Zhu et al., 2003) which uses the graph Laplacian to approximate E
Revised: Penalizing E x [ k J x k 2 F ] is also closely related to the graph based (manifold) regularization in Zhu et al. (2003) which uses the graph Laplacian to approximate E

Original: x 
Revised: x [

Original: [ (cid:107)∇ M f (cid:107) 2 ] for nonlinear models, making use of the manifold structure of unlabeled data.
Revised: [Equation] ] for nonlinear models, making use of the manifold structure of unlabeled data.

Original: While isotropic perturbations investigated in this simpliﬁed  modelΠ will not in general lie along the data manifold,more targeted perturbations like image translations used to penalize gradients along the manifold and can also be understood as above. Consider perturbations sampled from thetangent space  with z ∼ P ( x ) N (0 , I ) where P ( x ) =
Revised: Isotropic perturbations investigated in this simpliﬁed ⇧ model will not in general lie along the data manifold, and it would be more pertinent to enforce consistency to perturbations sampled from the space of natural images. In fact, we can interpret consistency with respect to standard data augmentations (which are used in practice) as penalizing the manifold Jacobian norm in the same manner as above.

Original: P ( x ) 2 is the orthogonal projection matrix that projects down from R d to T x ( M ) , the tangent space of the image manifold at x . In this case, the consistency regularization penalizes the Laplacian norm on the manifold 2 E x [ (cid:107) P ( x ) J x (cid:107) 2 F ] = E x [ (cid:107) J M (cid:107) 2 F ] . We view the standard data augmentations (used in the Π and MT models) as samples from a subset of this tangent space.
Revised: See Section A.5 for more details.

Original: Penalization of the Hessian’s eigenvalues Now, instead of the input perturbation, consider the weight perturbation w (cid:48) = w + (cid:15)z .
Revised: Penalization of the Hessian’s eigenvalues. Now, instead of the input perturbation, consider the weight perturbation w 0 = w + ✏ z .

Original: Similarly, the consistency loss is an unbiased estimator for E x [ (cid:107) J w (cid:107) 2 F ] , where J w is the Jacobian of the network outputs with respect to the weights w .
Revised: Similarly, the consistency loss is an unbiased estimator for E x [ k J w k 2 F ] , where J w is the Jacobian of the network outputs with respect to the weights w .

Original: In section A.6 we show that for the MSE loss, the expected trace of the Hessian of the loss E x [ tr ( H )] can be decomposed into two terms, one of which is E x[ (cid:107) J w (cid:107) 2 F ] .
Revised: In Section A.6 we show that for the MSE loss, the expected trace of the Hessian of the loss E x [ tr ( H )] can be decomposed into two terms, one of which is E x

Original:  As minimizing the consistency loss of a simpliﬁed Π model penalizes E x [ (cid:107) J w (cid:107) 2 F ] , it also penalizes E x [ tr ( H )] .
Revised: [ k J w k 2 F ] . As minimizing the consistency loss of a simpliﬁed ⇧ model penalizes E x [ k J w k 2 F ] , it also penalizes E x [ tr ( H )] .

Original: As pointed out in Dinh et al. (2017a) and Chaudhari et al.
Revised: As pointed out in Dinh et al. (2017a) and Chaudhari et al.

Original: (2016), the eigenvalues of H encode the local information about sharpness of the loss for a given solution w .
Revised: (2016), the eigenvalues of H encode the local information about sharpness of the loss for a given solution w .

Original: Consequently, the quantity tr ( H ) which is the sum of the Hessian eigenvalues is related to the notion of sharp and ﬂat optima, which has recently gained attention as a proxy for generalization performance (see e.g. Schmidhuber and Hochreiter, 1997;
Revised: Consequently, the quantity tr ( H ) which is the sum of the Hessian eigenvalues is related to the notion of sharp and ﬂat optima, which has recently gained attention as a proxy for generalization performance (see e.g. Schmidhuber and Hochreiter, 1997;

Original: Keskar et al., 2017; Izmailov et al., 2018).
Revised: Keskar et al., 2017; Izmailov et al., 2018).

Original: Thus, based on our analysis, the consistency loss in the simpliﬁed Π model encourages ﬂatter solutions.
Revised: Thus, based on our analysis, the consistency loss in the simpliﬁed ⇧ model encourages ﬂatter solutions.

Original: 3. A NALYSIS OF S OLUTIONS ALONG SGD T RAJECTORIES
Revised: 3. A NALYSIS OF S OLUTIONS ALONG SGD T RAJECTORIES

Original: In the previous section we have seen that in a simpliﬁed Π model, the consistency loss encourages several desirable properties that have been related to  generalization.
Revised: In the previous section we have seen that in a simpliﬁed ⇧ model, the consistency loss encourages lower input-output Jacobian norm and Hessian’s eigenvalues, which are related to better generalization.

Original: In this section we analyze the properties of minimizing the consistency loss in a practical setting.
Revised: In this section we analyze the properties of minimizing the consistency loss in a practical setting.

Original: Speciﬁcally, we explore the trajectories followed by SGD for the consistency-based models and compare them to the trajectories in supervised training.
Revised: Speciﬁcally, we explore the trajectories followed by SGD for the consistency-based models and compare them to the trajectories in supervised training.

Original: [Figure]
Revised: [Figure]

Revised: ) : The evolution of the gradient norm for the consistency regularization term (Cons) and the cross-entropy term (CE) in the ⇧ , MT, and standard supervised (CE only) models during training.

Revised: and test errors along rays connecting two SGD solutions for each respective model.

Revised: and

Revised: Comparison of errors along rays connecting two SGD solutions, random rays, and adversarial rays

Original:  We train our models on CIFAR-10 using 4 k labeled data for 180 epochs.
Revised: the and supervised models. See Section A.1 for the analogous Mean Teacher model’s plot. We train our models on CIFAR-10 using 4 k labeled data for 180 epochs.

Original: The Π and Mean Teacher models use 46 k data points as unlabeled data (see Sections A.8 and A.9 for details).
Revised: The ⇧ and Mean Teacher models use 46 k data points as unlabeled data (see Sections A.8 and A.9 for details).

Original: First, in Figure 1a we visualize the evolution of norms of the gradients of the cross-entropy term (cid:107)∇ L CE (cid:107) and consistency term (cid:107)∇ L cons (cid:107) along the trajectories of the Π , MT, and  supervised models.
Revised: First, in Figure 1a we visualize the evolution of norms of the gradients of the cross-entropy term kr L CE k and consistency term kr L cons k along the trajectories of the ⇧ , MT, and standard supervised models (using CE loss only).

Original: We observe that (cid:107)∇ L Cons (cid:107) remains high until the end of training and dominates the gradient (cid:107)∇ L CE (cid:107) of the cross-entropy term for the Π and MT models.
Revised: We observe that kr L Cons k remains high until the end of training and dominates the gradient kr L CE k of the cross-entropy term for the ⇧ and MT models.

Original: Further, for both the Π and MT models, (cid:107)∇ L Cons (cid:107) is much larger than in supervised training, implying that the Π and MT models are making substantially larger steps until the end of training.
Revised: Further, for both the ⇧ and MT models, kr L Cons k is much larger than in supervised training implying that the ⇧ and MT models are making substantially larger steps until the end of training.

Original: These larger steps suggest that rather than converging to a single minimizer, SGD continues to actively explore a large set of solutionsin a ﬂat region of loss when applied to consistency-based methods.
Revised: These larger steps suggest that rather than converging to a single minimizer, SGD continues to actively explore a large set of solutions when applied to consistency-based methods.

Original: To get more insight into the observed phenomenon we analyze the behavior of train and test errors in the region of weight space around the solutions of the Π and Mean Teacher models.
Revised: For further understand this observation, we analyze the behavior of train and test errors in the region of weight space around the solutions of the ⇧ and Mean Teacher models.

Original: First, we consider the one-dimensional rays φ ( t ) =
Revised: First, we consider the onedimensional rays  ( t ) =

Original: t · w 180 + (1 − t ) w 170 , t ≥ 0 , connecting the weight vectors w 170 and w 180 corresponding to epochs 170 and 180 of training.
Revised: t · w 180 + (1  t ) w 170 , t  0 , connecting the weight vectors w 170 and wcorresponding to epochs 170 and 180 of training.

Original: We visualize the train  (measured on the labeled data)and test errors as functions of the distance from the weights win Figure 1b.
Revised: We visualize the train and test errors (measured on the labeled data) as functions of the distance from the weights w 170 in Figure 1b.

Original: We observe that the distance between the weight vectors w 170 and w 180 is much larger for the semi-supervised methods compared to supervised training, which is consistent with our observation that optimization takes larger steps  in Π and MT models.
Revised: We observe that the distance between the weight vectors w 170 and w 180 is much larger for the semi-supervised methods compared to supervised training, which is consistent with our observation that the gradient norms are larger which implies larger steps during optimization in the ⇧ and MT models.

Original: Further, we observe that the train and test error surfaces are much wider along the directions connecting w 170 and w 180 for the consistency-based methods compared to supervised training.
Revised: Further, we observe that the train and test error surfaces are much wider along the directions connecting w 170 and w 180 for the consistency-based methods compared to supervised training.

Original: One possible explanation for the increased width is the effect of the consistency loss on the Jacobian of the network and the eigenvalues of the Hessian of the loss discussed in section 3.1.
Revised: One possible explanation for the increased width is the effect of the consistency loss on the Jacobian of the network and the eigenvalues of the Hessian of the loss discussed in Section 3.1.

Original: We also observe that the test errors of interpolated weights can be lower than errors of the two SGD solutions between which we interpolate.
Revised: We also observe that the test errors of interpolated weights can be lower than errors of the two SGD solutions between which we interpolate.

Original: This error reduction is larger in the consistency models (Figure 1b).
Revised: This error reduction is larger in the consistency models (Figure 1b).

Original: We also analyze the error surfaces along random and adversarial rays starting at the SGD solution w 180 for each model.
Revised: We also analyze the error surfaces along random and adversarial rays starting at the SGD solution w 180 for each model.

Original: For the random rays we sample 5 random vectors d from the unit sphere and calculate the average train and test errors of the network with weights w t 1 + sd for s ∈ [0 , 30] .
Revised: For the random rays we sample 5 random vectors d from the unit sphere and calculate the average train and test errors of the network with weights w t 1 + sd for s 2 [0 , 30] .

Original: With adversarial rays we evaluate the error along the directions of the fastest ascent of test or train loss d adv = ∇ L CE ||∇ L CE || .
Revised: With adversarial rays we evaluate the error along the directions of the fastest ascent of test or train loss d adv = r L CE || r L CE || .

Original: Interestingly, we observe that while the solutions of the Π and MT models are much wider than supervised training solutions along the SGD-SGD directions (Figure 1b), their widths along random and adversarial rays are comparable (Figure 1c, 1d).
Revised: We observe that while the solutions of the ⇧ and MT models are much wider than supervised training solutions along the SGD-SGD directions (Figure 1b), their widths along random and adversarial rays are comparable (Figure 1c, 1d)

Original: We analyze the error along SGD-SGD rays for two reasons.
Revised: We analyze the error along SGD-SGD rays for two reasons.

Original: Firstly, in fast-SWA we are averaging solutions traversed by SGD so the rays connecting SGD iterates serve as a proxy for the space we average over.
Revised: Firstly, in fast-SWA we are averaging solutions traversed by SGD, so the rays connecting SGD iterates serve as a proxy for the space we average over.

Original: Secondly, we are interested in evaluating the width of the solutions that we explore during training which we expect will be improved by the consistency training as discussed in Section 3.1 and A.6.
Revised: Secondly, we are interested in evaluating the width of the solutions that we explore during training which we expect will be improved by the consistency training, as discussed in Section 3.1 and A.6.

Original: Wedo not expect width along random rays to be very meaningful because there are many directions in the parameter space that do not change the network outputs (Dinh et al., 2017b;
Revised: We expect width along random rays to be less meaningful because there are many directions in the parameter space that do not change the network outputs (Dinh et al., 2017b;

Original: Anonymous, 2018; Sagun et al., 2017).
Revised: GurAri et al., 2018; Sagun et al., 2017).

Original: However, by evaluating SGD-SGD rays, we can expect that these directions corresponds to meaningful changes to our model because individual SGD updates correspond to directions that change the predictions on the training set.
Revised: However, by evaluating SGD-SGD rays, we can expect that these directions corresponds to meaningful changes to our model because individual SGD updates correspond to directions that change the predictions on the training set.

Original: Furthermore, we observe that different SGD iterates produce signiﬁcantly different predictions on the test data.
Revised: Furthermore, we observe that different SGD iterates produce signiﬁcantly different predictions on the test data.

Original: Neural networks in general are known to be resilient to noise, explaining why both MT, Π and supervised models are ﬂat along random directions (Arora et al., 2018).
Revised: Neural networks in general are known to be resilient to noise, explaining why both MT, ⇧ and supervised models are ﬂat along random directions (Arora et al., 2018).

Original: At the same time neural networks are susceptible to targeted perturbations (such as adversarial attacks).
Revised: At the same time neural networks are susceptible to targeted perturbations (such as adversarial attacks).

Original: We hypothesize that we do not observe improved ﬂatness for semi-supervised methods along adversarial rays because we do not choose our input or weight perturbations adversarially, but rather they are sampled from a predeﬁned set of transformations.
Revised: We hypothesize that we do not observe improved ﬂatness for semi-supervised methods along adversarial rays because we do not choose our input or weight perturbations adversarially, but rather they are sampled from a predeﬁned set of transformations.

Original: Additionally, we analyze whether the larger optimization steps for the Π and MT models translate into higher diversity in predictions.
Revised: Additionally, we analyze whether the larger optimization steps for the ⇧ and MT models translate into higher diversity in predictions.

Original: We deﬁne diversity of a pair of models w 1 , w 2 as Diversity ( w 1 , w 2 ) 
Revised: We deﬁne diversity of a pair of models w 1 , w 2 as Diversity ( w 1 , w 2 ) =

Original: = 1 N (cid:80) N i =1 1 [ y ( w 1 ) i (cid:54) =
Revised: [Equation]

Original: y ( w 2 ) i ] , the fraction of test samples where the predicted labels between the two models differ.
Revised: ] , the fraction of test samples where the predicted labels between the two models differ.

Original: We found that for the Π and MT models, the Diversity( w 170 , w 180 ) is 7 . 1% and 6 .
Revised: We found that for the ⇧ and MT models, the Diversity( w 170 , w 180 ) is 7 . 1% and 6 .

Original: 1% of the test data points respectively, which is much higher than 3 .
Revised: 1% of the test data points respectively, which is much higher than 3 .

Original: 9% in supervised learning.
Revised: 9% in supervised learning.

Original: The increased diversity in the predictions of the networks traversed by SGD supports our conjecture that for the Π and MT models SGD struggles to converge to a single solution and continues to actively explore the set of plausible solutions until the end of training.
Revised: The increased diversity in the predictions of the networks traversed by SGD supports our conjecture that for the ⇧ and MT models SGD struggles to converge to a single solution and continues to actively explore the set of plausible solutions until the end of training.

Original: 3. E NSEMBLING AND W EIGHT A VERAGING
Revised: 3. E NSEMBLING AND W EIGHT A VERAGING

Original: In Section 3.2, we observed that the Π and MT models continue taking large steps in the weight space at the end of training.
Revised: In Section 3.2, we observed that the ⇧ and MT models continue taking large steps in the weight space at the end of training.

Original: Not only are the distances between weights larger, we observe these models to have higher diversity.
Revised: Not only are the distances between weights larger, we observe these models to have higher diversity.

Original: In this setting, using the last SGD iterate to perform prediction is not ideal since many solutions explored by SGD are equally accurate but produce different predictions.
Revised: In this setting, using the last SGD iterate to perform prediction is not ideal since many solutions explored by SGD are equally accurate but produce different predictions.

Original: Ensembling As mentioned in Section 3.2, the diversity in predictions is signiﬁcantly larger for the Π and Mean Teacher models compared to purely supervised learning.
Revised: Ensembling. In Section 3.2 we showed that the diversity in predictions is signiﬁcantly larger for the ⇧ and Mean Teacher models compared to purely supervised learning.

Original: The diversity of these iterates suggests that we can achieve greater beneﬁts from ensembling.
Revised: The diversity of these iterates suggests that we can achieve greater beneﬁts from ensembling.

Original: We use the same CNN architecture and hyper-parameters as in Section 3.2 but extend the training time by doing 5 learning rate cycles of 30 epochs after the normal training ends at epoch 180 (see A.8 and A.9 for details).
Revised: We use the same CNN architecture and hyper-parameters as in Section 3.2 but extend the training time by doing 5 learning rate cycles of 30 epochs after the normal training ends at epoch 180 (see A.8 and A.9 for details).

Original: We sample random pairs of weights w 1 , w 2 from epochs 180 , 183 , . . .
Revised: We sample random pairs of weights w 1 , w 2 from epochs 180 , 183 , . . .

Original: , 330 and measure the error reduction from ensembling these pairs of models, C ens ≡
Revised: , 330 and measure the error reduction from ensembling these pairs of models, C ens ⌘

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: In
Revised: In

Original: Figure 2c we visualize C ens , against the diversity of the corresponding pair of models.
Revised: Figure 2c we visualize C ens , against the diversity of the corresponding pair of models.

Original: We observe a strong correlation between the diversity in predictions of the constituent models and ensemble performance, and therefore C ens is substantially larger for Π and Mean Teacher models.
Revised: We observe a strong correlation between the diversity in predictions of the constituent models and ensemble performance, and therefore C ens is substantially larger for ⇧ and Mean Teacher models.

Original: As shown in (Izmailov et al., 2018), ensembling can be well approximated by weight averaging if the weights are close by.
Revised: As shown in Izmailov et al. (2018), ensembling can be well approximated by weight averaging if the weights are close by.

Original: Weight Averaging
Revised: Weight Averaging.

Original: First, we experiment on averaging random pairs of weights at the end of training and analyze the performance with respect to the weight distances.
Revised: First, we experiment on averaging random pairs of weights at the end of training and analyze the performance with respect to the weight distances.

Original: Using the the same pairs from above, we evaluate the performance of the model formed by averaging the pairs of weights,
Revised: Using the the same pairs from above, we evaluate the performance of the model formed by averaging the pairs of weights,

Original: C avg ( w 1 , w 2 ) ≡ 12 Err ( w 1 )+ 12 Err ( w 2 ) − Err (cid:0)
Revised: C avg [Equation] [Equation]

Original: 12 w 1 + 12 w 2 (cid:1) .
Revised:  .

Original: Note that C avg is a proxy for convexity: if C avg ( w 1 , w 2 ) ≥ 0 for any pair of points w 1 , w 2 , then by Jensen’s inequality the error function is convex (see  Figure 2 left).
Revised: Note that C avg is a proxy for convexity: if C avg ( w 1 , w 2 )  0 for any pair of points w 1 , w 2 , then by Jensen’s inequality the error function is convex (see the left panel of Figure 2).

Original: While the error surfaces for neural networks are known to be highly non-convex, they may be approximately convex in the region traversed by SGD late intothe training (Goodfellow et al., 2015).
Revised: While the error surfaces for neural networks are known to be highly non-convex, they may be approximately convex in the region traversed by SGD late into training (Goodfellow et al., 2015).

Original: In fact, in Figure 2b, we ﬁnd that the error surface of the SGD trajectory is approximately convex due to C avg ( w 1 , w 2 ) being mostly positive.
Revised: In fact, in Figure 2b, we ﬁnd that the error surface of the SGD trajectory is approximately convex due to C avg ( w 1 , w 2 ) being mostly positive.

Original: Here we also observe that the distances between pairs of weights are much larger for the Π and MT models than for the supervised training; and as a result, weight averaging achieves a larger gain for these models.
Revised: Here we also observe that the distances between pairs of weights are much larger for the ⇧ and MT models than for the supervised training; and as a result, weight averaging achieves a larger gain for these models.

Original: In section 3.2 we observed that for the Π and Mean Teacher models SGD traverses a large ﬂat region of the weight space late in training.
Revised: In Section 3.2 we observed that for the ⇧ and Mean Teacher models SGD traverses a large ﬂat region of the weight space late in training.

Original: Being very high-dimensional, this set has most of its volume concentrated near its boundary.
Revised: Being very high-dimensional, this set has most of its volume concentrated near its boundary.

Original: Thus, we ﬁnd SGD iterates at the periphery of this ﬂat region (see
Revised: Thus, we ﬁnd SGD iterates at the periphery of this ﬂat region (see

Original: Figure 2d).
Revised: Figure 2d).

Original: We can also explain this behavior via the argument of (Mandt et al., 2017).
Revised: We can also explain this behavior via the argument of (Mandt et al., 2017).

Original: Under certain assumptions SGD iterates can be thought of as samples from a Gaussian distribution centered at the minimum of the loss, and samples from high-dimensional Gaussians are known to be concentrated on the surface of an ellipse and never be close to the mean.
Revised: Under certain assumptions SGD iterates can be thought of as samples from a Gaussian distribution centered at the minimum of the loss, and samples from high-dimensional Gaussians are known to be concentrated on the surface of an ellipse and never be close to the mean.

Original: Averaging the SGD iterates (shown in red in Figure 2d) we can move towards the center (shown in blue) of the ﬂat region, stabilizing the SGD trajectory and improving the width of the resulting solution and consequently  generalization. 
Revised: Averaging the SGD iterates (shown in red in Figure 2d) we can move towards the center (shown in blue) of the ﬂat region, stabilizing the SGD trajectory and improving the width of the resulting solution, and consequently improving generalization. [Figure]

Original: We observe that the improvement C avg from weight averaging ( 1 . 2 ± 0 .
Revised: We observe that the improvement C avg from weight averaging ( 1 . 2 ± 0 .

Original: 2% over MT and Π pairs) is on par or larger than the beneﬁt C ens of prediction ensembling ( 0 .
Revised: 2% over MT and ⇧ pairs) is on par or larger than the beneﬁt C ens of prediction ensembling ( 0 .

Original: 9 ± 0 .
Revised: 9 ± 0 .

Revised: The smaller gain from

Revised: [Figure]

Revised: ensembling might be due to the dependency of the ensembled solutions, since they are from the same

Revised: SGD run as opposed to independent restarts as in typical ensembling settings.

Original: For the rest of the paper, we focus attention on weight averaging because of its lower costs at test time and slightly higher performance compared to ensembling.
Revised: For the rest of the paper, we focus attention on weight averaging because of its lower costs at test time and slightly higher performance compared to ensembling.

Original:  SWA AND FAST -SWA
Revised:  SWA AND FAST -SWA

Original: In Section 3 we analyzed the training trajectories of the Π , MT, and supervised models.
Revised: In Section 3 we analyzed the training trajectories of the ⇧ , MT, and supervised models.

Original: We observed that the Π and MT models continue to actively explore the set of plausible solutions, producing diverse predictions on the test set even in the late stages of training.
Revised: We observed that the ⇧ and MT models continue to actively explore the set of plausible solutions, producing diverse predictions on the test set even in the late stages of training.

Original: Further, in section 3.3 we have seen that averaging weights leads to signiﬁcant gains in performance for the Π and MT models.
Revised: Further, in section 3.3 we have seen that averaging weights leads to signiﬁcant gains in performance for the ⇧ and MT models.

Original: In particular these gains are much larger than in supervised setting.
Revised: In particular these gains are much larger than in supervised setting.

Original: Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) is a recent approach that is based on averaging weights traversed by SGD with a modiﬁed learning rate schedule.
Revised: Stochastic Weight Averaging (SWA) (Izmailov et al., 2018) is a recent approach that is based on averaging weights traversed by SGD with a modiﬁed learning rate schedule.

Original: In Section 3 we analyzed averaging pairs of weights corresponding to different epochs of training and showed that it improves the test accuracy.[-Averaging multiple weights reinforces this effect, and SWA was shown ww [Figure] 40 || w 1 w 
-]
Revised: In Section 3 we analyzed averaging pairs of weights corresponding to different epochs of training and showed that it improves the test accuracy.[-Averaging multiple weights reinforces this effect, and SWA was shown ww [Figure] 40 || w 1 w 
-]

Original: || 0.

Original: C a v g ( w 1 , w 2 )

Original: MT

Original: [-Sup (b) C avg vs. || w 1 − w 
-]
Revised: [-Sup (b) C avg vs. || w 1 − w 
-]

Original: || 0.

Original: 0.10 Diversity( w 1 , w 2 ) 0.

Original: C e n s ( w 1 , w 2 )

Original: MT

Original: Sup (

Original: [Figure]

Original: C ens

Original: [Figure]

Original: diversity [Figure] [Figure] to signiﬁcantly improve generalization performance in supervised learning.
Revised: Averaging multiple weights reinforces this effect, and SWA was shown to signiﬁcantly improve generalization performance in supervised learning.

Original: Based on our results in section 3. we can expect even larger improvements in generalization when applying SWA to the Π and MT models.
Revised: Based on our results in section 3.3, we can expect even larger improvements in generalization when applying SWA to the ⇧ and MT models.

Original: SWA typically starts from a pre-trained model, and then averages points in weight space traversed by
Revised: SWA typically starts from a pre-trained model, and then averages points in weight space traversed by

Original: SGD with a constant or cyclical learning rate.
Revised: SGD with a constant or cyclical learning rate.

Original: We illustrate the cyclical cosine learning rate schedule in Figure 3 (left) and the SGD solutions explored in Figure 3 (middle).
Revised: We illustrate the cyclical cosine learning rate schedule in Figure 3 (left) and the SGD solutions explored in Figure 3 (middle).

Original: For the ﬁrst (cid:96) ≤ (cid:96) 0 epochs the network is pre-trained using the cosine annealing schedule where the learning rate at epoch i is set equal to η ( i ) = 0 .
Revised: For the ﬁrst `  ` 0 epochs the network is pre-trained using the cosine annealing schedule where the learning rate at epoch i is set equal to ⌘ ( i ) = 0 .

Original: 5 · η 0 (1 + cos ( π · i/(cid:96) 0 )) .
Revised: 5 · ⌘ 0 (1 + cos ( ⇡ · i/ ` 0 )) .

Original: After (cid:96) epochs, we use a cyclical schedule, repeating the learning rates from epochs [ (cid:96) − c, (cid:96) ] , where c is the cycle length.
Revised: After ` epochs, we use a cyclical schedule, repeating the learning rates from epochs [ `  c, ` ] , where c is the cycle length.

Original: SWA collects the networks corresponding to the minimum values of the learning rate (shown in green in Figure 3, left) and averages their weights.
Revised: SWA collects the networks corresponding to the minimum values of the learning rate (shown in green in Figure 3, left) and averages their weights.

Original: The model with the averaged weights w SWA is then used to make predictions.
Revised: The model with the averaged weights w SWA is then used to make predictions.

Original: We propose to apply SWA to the student network both for the Π and Mean Teacher models.
Revised: We propose to apply SWA to the student network both for the ⇧ and Mean Teacher models.

Original: Note that the SWA weights do not interfere with training.
Revised: Note that the SWA weights do not interfere with training.

Original: Originally, Izmailov et al.
Revised: Originally, Izmailov et al.

Original: (2018) proposed using cyclical learning rates with small cycle length for
Revised: (2018) proposed using cyclical learning rates with small cycle length for

Original: SWA.
Revised: SWA.

Original: However, as we have seen in Section 3.3 (Figure 2, left) the beneﬁts of averaging are the most prominent when the distance between the averaged points is large.
Revised: However, as we have seen in Section 3.3 (Figure 2, left) the beneﬁts of averaging are the most prominent when the distance between the averaged points is large.

Original: Motivated by this observation, we instead use longer learning rate cycles c .
Revised: Motivated by this observation, we instead use longer learning rate cycles c .

Original: Moreover, SWA updates the average weights only once per cycle, which means that many additional training epochs are needed in order to collect enough weights for averaging.
Revised: Moreover, SWA updates the average weights only once per cycle, which means that many additional training epochs are needed in order to collect enough weights for averaging.

Original: To overcome this limitation, we propose fast-SWA , a modiﬁcation of SWA that averages networks corresponding to every k < c epochs starting from epoch (cid:96) − c .
Revised: To overcome this limitation, we propose fast-SWA , a modiﬁcation of SWA that averages networks corresponding to every k < c epochs starting from epoch `  c .

Original: We can also average multiple weights within a single epoch setting k < 1 .
Revised: We can also average multiple weights within a single epoch setting k < 1 .

Original: Notice that most of the models included in the fast-SWA average (shown in red in Figure 3, left) have higher errors than those included in the SWA average (shown in green in Figure 3, right) since they are obtained when the learning rate is high.
Revised: Notice that most of the models included in the fast-SWA average (shown in red in Figure 3, left) have higher errors than those included in the SWA average (shown in green in Figure 3, right) since they are obtained when the learning rate is high.

Original: It is our contention that including more models in the fast-SWA weight average can more than compensate for the larger errors of the individual models.
Revised: It is our contention that including more models in the fast-SWA weight average can more than compensate for the larger errors of the individual models.

Original: (see Section A.7). Indeed, our experiments in Section 5 show that fast-SWA converges substantially faster than SWA and has lower performance variance. 
Revised: Indeed, our experiments in Section 5 show that fast-SWA converges substantially faster than SWA and has lower performance variance. We analyze this result theoretically in Section A.7).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the Π and MT models (Section 4) on CIFAR-10 and CIFAR-100 with varying numbers of labeled examples.
Revised: We evaluate the ⇧ and MT models (Section 4) on CIFAR-10 and CIFAR-100 with varying numbers of labeled examples.

Original: We show that fast-SWA and SWA improve the performance of the Π and MT models, as we expect from our observations in Section 3.
Revised: We show that fast-SWA and SWA improve the performance of the ⇧ and MT models, as we expect from our observations in Section 3.

Original: In fact, in many cases fast-SWA improves on the best results reported in the semi-supervised literature.
Revised: In fact, in many cases fast-SWA improves on the best results reported in the semi-supervised literature.

Original: We also demonstrate that the preposed fast-SWA obtains high performance much faster than SWA.
Revised: We also demonstrate that the preposed fast-SWA obtains high performance much faster than SWA.

Original: We also evaluate SWA applied to a consistency-based domain adaptation model (French et al., 2018), closely related to the MT model, for adapting CIFAR-10 to STL.
Revised: We also evaluate SWA applied to a consistency-based domain adaptation model (French et al., 2018), closely related to the MT model, for adapting CIFAR-10 to STL.

Original: We improve the best reported test error rate for this task from 19 .
Revised: We improve the best reported test error rate for this task from 19 .

Original: 9% to 16 .
Revised: 9% to 16 .

Original: We discuss the experimental setup in Section 5.1.
Revised: We discuss the experimental setup in Section 5.1.

Original: We provide the results for CIFAR-10 and CIFAR100 datasets in Section 5.2 and 5.3.
Revised: We provide the results for CIFAR-10 and CIFAR100 datasets in Section 5.2 and 5.3.

Original: We summarize our results in comparison to the best previous results in Section 5.4.
Revised: We summarize our results in comparison to the best previous results in Section 5.4.

Original: We show several additional results and detailed comparisons in Appendix
Revised: We show several additional results and detailed comparisons in Appendix

Original: A.2.
Revised: A.2.

Original: We provide analysis of train and test error surfaces of fast-SWA solutions along the directions connecting fast-SWA and SGD in Section A.1.
Revised: We provide analysis of train and test error surfaces of fast-SWA solutions along the directions connecting fast-SWA and SGD in Section A.1.

Original: 5. S ETUP
Revised: 5. S ETUP

Original: We evaluate the weight averaging methods SWA and fast-SWA on different network architectures and learning rate schedules.
Revised: We evaluate the weight averaging methods SWA and fast-SWA on different network architectures and learning rate schedules.

Original: We are able to improve on the base models in all settings.
Revised: We are able to improve on the base models in all settings.

Original: In particular, we consider a 13 -layer CNN and a 12-block (26-layer) Residual Network (He et al., 2015) with Shake-Shake regularization (Gastaldi, 2017), which we refer to simply as CNN and Shake-Shake respectively (see Section A.8 for details on the architectures).
Revised: In particular, we consider a 13 -layer CNN and a 12-block (26-layer) Residual Network (He et al., 2015) with ShakeShake regularization (Gastaldi, 2017), which we refer to simply as CNN and Shake-Shake respectively (see Section A.8 for details on the architectures).

Original: For training all methods we use the stochastic gradient descent (SGD) optimizer with the cosine annealing learning rate described in Section 4.
Revised: For training all methods we use the stochastic gradient descent (SGD) optimizer with the cosine annealing learning rate described in Section 4.

Original: We use two learning rate schedules, the short schedule with (cid:96) = 180 , [-(cid:96) 
-] = 210 , c =similar to the experiments in Tarvainen and Valpola (2017) and the long schedule with (cid:96) = 1500 , [-(cid:96) 
-] = 1800 , c = 200  similar to the experiments in Gastaldi (2017).
Revised: We use two learning rate schedules, the short schedule with ` = 180 , [-(cid:96) 
-]` 0 = 210 , c = 30 , similar to the experiments in Tarvainen and Valpola (2017), and the long schedule with ` = 1500 , [-(cid:96) 
-]` 0 = 1800 , c = 200 , similar to the experiments in Gastaldi (2017).

Original: We note that the long schedule improves the performance of the base models compared to the short schedule; however, SWA can still further improve the results.
Revised: We note that the long schedule improves the performance of the base models compared to the short schedule; however, SWA can still further improve the results.

Original: See Section A.9 of the Appendix for more details on other hyperparameters.
Revised: See Section A.9 of the Appendix for more details on other hyperparameters.

Original: We repeat each CNN experiment 3 times with different random seeds to estimate the standard deviations for the results in the Appendix.
Revised: We repeat each CNN experiment 3 times with different random seeds to estimate the standard deviations for the results in the Appendix.

Original: 5. CIFAR- 
Revised: 5. CIFAR-

Original: [Figure]
Revised: [Figure]

Original:  We evaluate the proposed fast-SWA method using the Π and MT models on the CIFAR-10 dataset (Krizhevsky).
Revised: [Table] [Figure] We evaluate the proposed fast-SWA method using the ⇧ and MT models on the CIFAR-10 dataset (Krizhevsky).

Original: We use 50 k images for training with 1 k , 2 k , 4 k , 10 k and 50 k labels and report the top-1 errors on the test set ( 10 k images).
Revised: We use 50 k images for training with 1 k , 2 k , 4 k , 10 k and 50 k labels and report the top-1 errors on the test set ( 10 k images).

Original: We visualize the results for the CNN and Shake-Shake architectures in Figures 4a, 4c, and 4d.
Revised: We visualize the results for the CNN and Shake-Shake architectures in Figures 4a, 4c, and 4d.

Original: For all quantities of labeled data, fast-SWA substantially improves test accuracy in both architectures.
Revised: For all quantities of labeled data, fast-SWA substantially improves test accuracy in both architectures.

Original: Additionally, in Tables 2, 4 of the Appendix we provide a thorough comparison of different averaging strategies as well as results for VAT (Miyato et al., 2017), TE (Laine and Aila, 2016), and other baselines.
Revised: Additionally, in Tables 2, 4 of the Appendix we provide a thorough comparison of different averaging strategies as well as results for VAT (Miyato et al., 2017), TE (Laine and Aila, 2016), and other baselines.

Original: Note that we applied fast-SWA for VAT as well which is another popular approach for semi-supervised learning.
Revised: Note that we applied fast-SWA for VAT as well which is another popular approach for semi-supervised learning.

Original: We found that the improvement on VAT is not drastic – our base implementation obtains 11 .
Revised: We found that the improvement on VAT is not drastic – our base implementation obtains 11 .

Original: 26% error where fast-SWA reduces it to 10 .
Revised: 26% error where fast-SWA reduces it to 10 .

Original: 97% (see Table 2 in Section A.2).
Revised: 97% (see Table 2 in Section A.2).

Revised: It is possible that the solutions explored by VAT are not as diverse as in ⇧ and MT models due to VAT loss function.

Original: Throughout the experiments, we focus on the Π and MT models as they have been shown to scale to powerful networks such as Shake-Shake and obtained previous state-of-the-art performance.
Revised: Throughout the experiments, we focus on the ⇧ and MT models as they have been shown to scale to powerful networks such as Shake-Shake and obtained previous state-of-the-art performance.

Original: In Figure 5 (left), we visualize the test error as a function of iteration using the CNN.
Revised: In Figure 5 (left), we visualize the test error as a function of iteration using the CNN.

Original: We observe that when the cyclical learning rate starts after epoch (cid:96) = 180 , the base models drop in performance due to the sudden increase in learning rate (see Figure 3 left).
Revised: We observe that when the cyclical learning rate starts after epoch ` = 180 , the base models drop in performance due to the sudden increase in learning rate (see Figure 3 left).

Original: However, fast-SWA continues to improve while collecting the weights corresponding to high learning rates for averaging.
Revised: However, fast-SWA continues to improve while collecting the weights corresponding to high learning rates for averaging.

Original: In general, we also ﬁnd that the cyclical learning rate improves the base models beyond the usual cosine annealing schedule and increases the performance of fast-SWA as training progresses.
Revised: In general, we also ﬁnd that the cyclical learning rate improves the base models beyond the usual cosine annealing schedule and increases the performance of fast-SWA as training progresses.

Original: Compared to SWA, we also observe that fast-SWA converges substantially faster, for instance, reducing the error to 10 .
Revised: Compared to SWA, we also observe that fast-SWA converges substantially faster, for instance, reducing the error to 10 .

Original: 5% at epoch 200 while SWA attains similar error at epoch 350 for CIFAR-10 4 k labels (Figure 5 left).
Revised: 5% at epoch 200 while SWA attains similar error at epoch 350 for CIFAR-10 4 k labels (Figure 5 left).

Original: We provide additional plots in Section A.2 showing the convergence of the Π and MT models in all label settings, where we observe similar trends that fast-SWA results in faster error reduction.
Revised: We provide additional plots in Section A.2 showing the convergence of the ⇧ and MT models in all label settings, where we observe similar trends that fast-SWA results in faster error reduction.

Original: We also ﬁnd that the performance gains of fast-SWA over base models are higher for the Π model compared to the MT model, which is consistent with the convexity observation in Section 3.3 and
Revised: We also ﬁnd that the performance gains of fast-SWA over base models are higher for the ⇧ model compared to the MT model, which is consistent with the convexity observation in Section 3.3 and

Original: Figure 2.
Revised: Figure 2.

Original: In the previous evaluations (see e.g. Oliver et al., 2018; Tarvainen and Valpola, 2017), the Π model was shown to be inferior to the MT model.
Revised: In the previous evaluations (see e.g. Oliver et al., 2018; Tarvainen and Valpola, 2017), the ⇧ model was shown to be inferior to the MT model.

Original: However, with weight averaging, fast-SWA reduces the gap between Π and MT performance.
Revised: However, with weight averaging, fast-SWA reduces the gap between ⇧ and MT performance.

Original: Surprisingly, we ﬁnd that the Π model can outperform
Revised: Surprisingly, we ﬁnd that the ⇧ model can outperform

Original: MT after applying fast-SWA with moderate to large numbers of labeled points.
Revised: MT after applying fast-SWA with moderate to large numbers of labeled points.

Original: In particular, the Π +fast-SWA model outperforms MT+fast-SWA on CIFAR-10 with 4 k , 10 k , and 50 k labeled data points for the Shake-Shake architecture.[Figure]
Revised: In particular, the ⇧ +fast-SWA model outperforms MT+fast-SWA on CIFAR-10 with 4 k , 10 k , and 50 k labeled data points for the Shake-Shake architecture.

Original: 5. CIFAR-100 AND E XTRA U NLABELED D ATA
Revised: 5. CIFAR-100 AND E XTRA U NLABELED D ATA

Original: We evaluate the Π and MT models with fast-SWA on CIFAR-100.
Revised: We evaluate the ⇧ and MT models with fast-SWA on CIFAR-100.

Original: We train our models usingimages with 10 k and 50 k labels using the 13 -layer CNN.
Revised: We train our models usingimages with 10 k and 50 k labels using the 13 -layer CNN.

Original: We also analyze the effect of using the Tiny
Revised: We also analyze the effect of using the Tiny

Original: Images dataset (Torralba et al., 2008) as an additional source of unlabeled data.
Revised: Images dataset (Torralba et al., 2008) as an additional source of unlabeled data.

Original: The Tiny Images dataset consists of 80 million images, mostly unlabeled, and contains CIFARas a subset.
Revised: The Tiny Images dataset consists of 80 million images, mostly unlabeled, and contains CIFARas a subset.

Original: Following Laine and Aila (2016), we use two settings of unlabeled data, 50 k + 500 k and 50 k + 237 k ∗ where the 50 k images corresponds to CIFAR-100 images from the training set and the +500 k or +237 k ∗ images corresponds to additional 500 k or 237 k images from the Tiny Images dataset.
Revised: Following Laine and Aila (2016), we use two settings of unlabeled data, 50 k + 500 k and 50 k + 237 k ⇤ where the 50 k images corresponds to CIFAR-100 images from the training set and the +500 k or +237 k ⇤ images corresponds to additional 500 k or 237 k images from the Tiny Images dataset.

Original: For the 237 k ∗ setting, we select only the images that belong to the classes in CIFAR-100, corresponding to 237203 images.
Revised: For the 237 k ⇤ setting, we select only the images that belong to the classes in CIFAR-100, corresponding to 237203 images.

Original: For the 500 k setting, we use a random set of 500 k images whose classes can be different from CIFAR-100’s.
Revised: For the 500 k setting, we use a random set of 500 k images whose classes can be different from CIFAR-100.

Original: We visualize the results in Figure 4b, where we again observe that fast-SWA substantially improves performance for every conﬁguration of the number of labeled and unlabeled data.
Revised: We visualize the results in Figure 4b, where we again observe that fast-SWA substantially improves performance for every conﬁguration of the number of labeled and unlabeled data.

Original: In Figure 5 (middle, right) we show the errors of MT, SWA and fast-SWA as a function of iteration on CIFAR-100 for the 10 k and 50 k + 500 k label settings.
Revised: In Figure 5 (middle, right) we show the errors of MT, SWA and fast-SWA as a function of iteration on CIFAR-100 for the 10 k and 50 k + 500 k label settings.

Original: Similar to the
Revised: Similar to the

Original: CIFAR-10 experiments, we observe that fast-SWA reduces the errors substantially faster than SWA.
Revised: CIFAR-10 experiments, we observe that fast-SWA reduces the errors substantially faster than SWA.

Original: We provide detailed experimental results in Table 3 of the Appendix and include preliminary results using the Shake-Shake architecture in Table 5, Section A.2.
Revised: We provide detailed experimental results in Table 3 of the Appendix and include preliminary results using the Shake-Shake architecture in Table 5, Section A.2.

Original: 5. A DVANCING S TATE - OF - THE -A RT
Revised: 5. A DVANCING S TATE - OF - THE -A RT

Original: We have shown that fast-SWA can signiﬁcantly improve the performance of both the Π and MT models.
Revised: We have shown that fast-SWA can signiﬁcantly improve the performance of both the ⇧ and MT models.

Original: We provide a summary comparing our results with the previous best results in the literature in Table 1, using the 13 -layer CNN and the Shake-Shake architecture that had been applied previously.
Revised: We provide a summary comparing our results with the previous best results in the literature in Table 1, using the 13 -layer CNN and the Shake-Shake architecture that had been applied previously.

Original: We also provide detailed results the Appendix A.2.
Revised: We also provide detailed results the Appendix A.

Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  P RELIMINARY R ESULTS ON D OMAIN A DAPTATION
Revised: 5. P RELIMINARY R ESULTS ON D OMAIN A DAPTATION

Original: Domain adaptation problems involve learning using a source domain X s equipped with labels Y s and performing classiﬁcation on the target domain X t while having no access to the target labels at training time.
Revised: Domain adaptation problems involve learning using a source domain X s equipped with labels Y s and performing classiﬁcation on the target domain X t while having no access to the target labels at training time.

Original: A recent model by French et al.
Revised: A recent model by French et al.

Original: (2018) applies the consistency enforcing principle for domain adaptation and achieves state-of-the-art results on many datasets.
Revised: (2018) applies the consistency enforcing principle for domain adaptation and achieves state-of-the-art results on many datasets.

Original: Applying fast-SWA to this model on domain adaptation from CIFAR-10 to STL we were able to improve the best results reported in the literature from 19 .
Revised: Applying fast-SWA to this model on domain adaptation from CIFAR-10 to STL we were able to improve the best results reported in the literature from 19 .

Original: 9% to 16 .
Revised: 9% to 16 .

Original: Interestingly, we found that for this task averaging the weights in the end of every iteration in fast-SWA converges signiﬁcantly faster than averaging once per epoch and results in better performance.

Original: See Section A.10 for more details on the domain adaptation experiments.
Revised: See Section A.10 for more details on the domain adaptation experiments.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Semi-supervised learning is crucial for reducing the dependency of deep learning on large labeled datasets.
Revised: Semi-supervised learning is crucial for reducing the dependency of deep learning on large labeled datasets.

Original: Recently, there have been great advances in semi-supervised learning, with consistency regularization models achieving the best known results.
Revised: Recently, there have been great advances in semi-supervised learning, with consistency regularization models achieving the best known results.

Original: By analyzing solutions along the training trajectories for two of the most successful models in this class, the Π and Mean Teacher models, we have seen that rather than converging to a single solution SGD continues to explore a diverse set of plausible solutions late into training.
Revised: By analyzing solutions along the training trajectories for two of the most successful models in this class, the ⇧ and Mean Teacher models, we have seen that rather than converging to a single solution SGD continues to explore a diverse set of plausible solutions late into training.

Original: As a result, we can expect that averaging predictions or weights will lead to much larger gains in performance than for supervised training.
Revised: As a result, we can expect that averaging predictions or weights will lead to much larger gains in performance than for supervised training.

Original: Indeed we ﬁnd this to be the case, applying a variant of the recently proposed stochastic weight averaging (SWA) we advance the best known semi-supervised results on classiﬁcation benchmarks.
Revised: Indeed, applying a variant of the recently proposed stochastic weight averaging (SWA) we advance the best known semi-supervised results on classiﬁcation benchmarks.

Original: While not the focus of our paper, we have also shown that weight averaging has great promise in domain adaptation (French et al., 2018).
Revised: While not the focus of our paper, we have also shown that weight averaging has great promise in domain adaptation (French et al., 2018).

Original: Although the domain adaptation model we use is closely related to the Mean Teacher model, we observe qualitative differences in their behavior: the test error surface is ﬂatter, requiring much higher frequency of averaging to obtain signiﬁcant improvement.

Original: We believe that application-speciﬁc analysis of the geometric properties of the training objective and optimization trajectories will further improve results in domain adaptation. We also expect that other application areas such as reinforcement learning with sparse rewards,semi-supervised natural language processing, or generative adversarial networks (Yazıcı et al., 2018) to have large beneﬁts from averaging, which we see as exciting directions for future research.
Revised: We believe that application-speciﬁc analysis of the geometric properties of the training objective and optimization trajectories will further improve results over a wide range of application speciﬁc areas, including reinforcement learning with sparse rewards, generative adversarial networks (Yazıcı et al., 2018), or semi-supervised natural language processing.

Original: A. D ETAILED R ESULTS
Revised: A. D ETAILED R ESULTS

Original: In this section we report detailed results for the Π and Mean Teacher models and various baselines on
Revised: In this section we report detailed results for the ⇧ and Mean Teacher models and various baselines on

Original: CIFAR-10 and CIFAR-100 using the 13 -layer CNN and Shake-Shake.
Revised: CIFAR-10 and CIFAR-100 using the 13 -layer CNN and Shake-Shake.

Original: The results using the 13-layer CNN are summarized in Tables 2 and 3 for CIFAR-10 and CIFARrespectively.
Revised: The results using the 13-layer CNN are summarized in Tables 2 and 3 for CIFAR-10 and CIFARrespectively.

Original: Tables 4 and 5 summarize the results using Shake-Shake on CIFAR-10 and CIFAR100.
Revised: Tables 4 and 5 summarize the results using Shake-Shake on CIFAR-10 and CIFAR100.

Original: In the tables Π EMA is the same method as Π , where instead of SWA we apply Exponential
Revised: In the tables ⇧ EMA is the same method as ⇧ , where instead of SWA we apply Exponential

Original: Moving Averaging (EMA) for the student weights.
Revised: Moving Averaging (EMA) for the student weights.

Original: We show that simply performing EMA for the student network in the Π model without using it as a teacher (as in MT) typically results in a small improvement in the test error.
Revised: We show that simply performing EMA for the student network in the ⇧ model without using it as a teacher (as in MT) typically results in a small improvement in the test error.

Original: Figures 8 and 9 show the performance of the Π and Mean Teacher models as a function of the training epoch for CIFAR-10 and CIFAR-100 respectively for SWA and fast-SWA.
Revised: Figures 8 and 9 show the performance of the ⇧ and Mean Teacher models as a function of the training epoch for CIFAR-10 and CIFAR-100 respectively for SWA and fast-SWA.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] [Table] [Table]
Revised: MT

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table]
Revised: MT

Original: [Table] [Table] [Table]
Revised: + fast

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: SWA (1700) 6. MT + SWA (1700) 6. MT + fast-SWA (3500) 6.

Original: [Table]
Revised: MT + SWA (3500) 6. ⇧ (1500) 8. ⇧

Original: [Table]
Revised: [Table]

Revised: fast

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: SWA (1700) 7. ⇧ + SWA (1700) 7. ⇧ + fast-SWA (3500) 7. ⇧ + SWA (3500) 7.

Original: [Figure]
Revised: [Table]

Original: A. E FFECT OF L EARNING R ATE S CHEDULES
Revised: A. E FFECT OF L EARNING R ATE S CHEDULES

Original: The only hyperparameter for the fast-SWA setting is the cycle length c .
Revised: The only hyperparameter for the fast-SWA setting is the cycle length c .

Original: We demonstrate in Figure 10a that fast-SWA’s performance is not sensitive to c over a wide range of c values.
Revised: We demonstrate in Figure 10a that fast-SWA’s performance is not sensitive to c over a wide range of c values.

Original: We also demonstrate the performance for constant learning schedule.
Revised: We also demonstrate the performance for constant learning schedule.

Original: fast-SWA with cyclical learning rates generally converges faster due to higher variety in the collected weights
Revised: fast-SWA with cyclical learning rates generally converges faster due to higher variety in the collected weights.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: A. EMA VERSUS SWA AS A T EACHER
Revised: A. EMA VERSUS SWA AS A T EACHER

Original: The MT model uses an exponential moving average (EMA) of the student weights as a teacher in the consistency regularization term.
Revised: The MT model uses an exponential moving average (EMA) of the student weights as a teacher in the consistency regularization term.

Original: We consider two potential effects of using EMA as a teacher: ﬁrst, averaging weights improves performance of the teacher for the reasons discussed in Sections 3.2, 3.3; second, having a better teacher model leads to better student performance which in turn further improves the teacher.
Revised: We consider two potential effects of using EMA as a teacher: ﬁrst, averaging weights improves performance of the teacher for the reasons discussed in Sections 3.2, 3.3; second, having a better teacher model leads to better student performance which in turn further improves the teacher.

Original: In this section we try to separate these two effects.
Revised: In this section we try to separate these two effects.

Original: We apply EMA to the Π model in the same way in which we apply fast-SWA instead of using EMA as a teacher and compare the resulting performance to the Mean Teacher.
Revised: We apply EMA to the ⇧ model in the same way in which we apply fast-SWA instead of using EMA as a teacher and compare the resulting performance to the Mean Teacher.

Original: Figure 11 shows the improvement in error-rate obtained by applying EMA to the Π model in different label settings.
Revised: Figure 11 shows the improvement in error-rate obtained by applying EMA to the ⇧ model in different label settings.

Original: As we can see while
Revised: As we can see while

Original: EMA improves the results over the baseline Π model, the performance of Π -EMA is still inferior to that of the Mean Teacher method, especially when the labeled data is scarce.
Revised: EMA improves the results over the baseline ⇧ model, the performance of ⇧ -EMA is still inferior to that of the Mean Teacher method, especially when the labeled data is scarce.

Original: This observation suggests that the improvement of the Mean Teacher over the Π model can not be simply attributed to
Revised: This observation suggests that the improvement of the Mean Teacher over the ⇧ model can not be simply attributed to

Original: EMA improving the student performance and we should take the second effect discussed above into account.
Revised: EMA improving the student performance and we should take the second effect discussed above into account.

Original: Like SWA, EMA is a way to average weights of the networks, but it puts more emphasis on very recent models compared to SWA.
Revised: Like SWA, EMA is a way to average weights of the networks, but it puts more emphasis on very recent models compared to SWA.

Original: Early in training when the student model changes rapidly EMA signiﬁcantly improves performance and helps a lot when used as a teacher.
Revised: Early in training when the student model changes rapidly EMA signiﬁcantly improves performance and helps a lot when used as a teacher.

Original: However once the student model converges to the vicinity of the optimum, EMA offers little gain.
Revised: However once the student model converges to the vicinity of the optimum, EMA offers little gain.

Original: In this regime SWA is a much better way to average weights.
Revised: In this regime SWA is a much better way to average weights.

Original: We show the performance of SWA applied to Π model in Figure 11 (left).[Figure]
Revised: We show the performance of SWA applied to ⇧ model in Figure 11 (left).

Original: Since SWA performs better than EMA, we also experiment with using SWA as a teacher instead of EMA.
Revised: Since SWA performs better than EMA, we also experiment with using SWA as a teacher instead of EMA.

Original: We start with the usual MT model pretrained until epoch 150 .
Revised: We start with the usual MT model pretrained until epoch 150 .

Original: Then we switch to using
Revised: Then we switch to using

Original: SWA as a teacher at epoch 150 .
Revised: SWA as a teacher at epoch 150 .

Original: In Figure 11 (right), our results suggest that using SWA as a teacher performs on par with using EMA as a teacher.
Revised: In Figure 11 (right), our results suggest that using SWA as a teacher performs on par with using EMA as a teacher.

Original: We conjecture that once we are at a convex region of test error close to the optimum (epoch 150 ), having a better teacher doesn’t lead to substantially improved performance.
Revised: We conjecture that once we are at a convex region of test error close to the optimum (epoch 150 ), having a better teacher doesn’t lead to substantially improved performance.

Original: It is possible to start using SWA as a teacher earlier in training; however, during early epochs where the model undergoes rapid improvement EMA is more sensible than SWA as we discussed above.
Revised: It is possible to start using SWA as a teacher earlier in training; however, during early epochs where the model undergoes rapid improvement EMA is more sensible than SWA as we discussed above.

Original: A. C ONSISTENCY L OSS A PPROXIMATES J ACOBIAN N ORM
Revised: A. C ONSISTENCY L OSS A PPROXIMATES J ACOBIAN N ORM

Original: Estimator mean and variance:
Revised: Estimator mean and variance:

Original: In the simpliﬁed Π model with small additive data perturbations that are normally distributed, z ∼ N (0 , I 
Revised: In the simpliﬁed ⇧ model with small additive data perturbations that are normally distributed, z ⇠ N (0 , I ) ,

Original: [Equation]
Revised: [Equation]

Original: [Equation] Taylor expanding (cid:96) cons in (cid:15) , we obtain (cid:96) cons ( w, x, (cid:15) ) =[-(cid:15) 
-]
Revised: Taylor expanding ` cons in ✏ , we obtain ` cons ( w, x, ✏ ) =[-(cid:15) 
-]

Original:  z T J Tx J x z + O ( (cid:15) 4 ) , where J x is the
Revised: ✏ 2 z T J Tx J x z + O ( ✏ 4 ) , where J x is the

Original: Jacobian of the network outputs f with respect to the input at a particular value of x .
Revised: Jacobian of the network outputs f with respect to the input at a particular value of x .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: We can now recognize this term as a one sample stochastic trace estimator for tr ( J ( x i ) T J ( x i )) with a
Revised: We can now recognize this term as a one sample stochastic trace estimator for tr ( J ( x i ) T J ( x i )) with a

Original: Gaussian probe variable z i ; see Avron and Toledo (2011) for derivations and guarantees on stochastic trace estimators.
Revised: Gaussian probe variable z i ; see Avron and Toledo (2011) for derivations and guarantees on stochastic trace estimators.

Original: [Equation]
Revised: [Equation]

Original: Taking an expectation over the m samples of x , we get E [ ˆ Q ] =
Revised: Taking an expectation over the m samples of x , we get E [ ˆ Q ] =

Original: E x
Revised: E x

Original: [Equation] J x (cid:107) 2 F .
Revised: [ k J x k 2 F .

Original: In general if we have m samples of x and n sampled perturbations for each x , then for a symmetric matrix A with z ik iid ∼ N (0 , I ) and independent x i iid ∼ p ( x ) , the estimator
Revised: In general if we have m samples of x and n sampled perturbations for each x , then for a symmetric matrix A with z ik iid ⇠ N (0 , I ) and independent x i iid ⇠ p ( x ) , the estimator

Original: [Equation]
Revised: [Equation]

Original: has variance
Revised: has variance

Original: [Equation]
Revised: [Equation]

Original: Proof: Let q ik ≡ z Tik A ( x i )
Revised: Proof: Let q ik ⌘ z Tik A ( x i )

Original: z
Revised: z

Original: ik .
Revised: ik .

Original: It is well known that for ﬁxed x , E z [ q 11 | x 1 ] = 2 tr ( A ( x 1 ) 2 ) + tr ( A ( x 1 )) 2 , (see e.g. Avron and Toledo, 2011).
Revised: It is easy to show that for ﬁxed x , E z [ q 11 | x 1 ] = 2 tr ( A ( x 1 ) 2 ) + tr ( A ( x 1 )) 2 , (see e.g. Avron and Toledo, 2011).

Original: Note that E z 1 ,z 

Revised: Note that E z 1 ,z

Original: [ q i 1 q i 2 | x i ] = tr ( A ( x i )) 2 .
Revised: [ q i 1 q i 2 | x i ] = tr ( A ( x i )) 2 .

Original: Since (cid:8)n (cid:80) nk q ik(cid:9) m i =1 are i.i.d random variables, 
Revised: Since  1 n P nk q ik m i =1 are i.i.d random variables, Var

Original: [Equation]
Revised: [Equation]

Original: whereas this does not hold for the opposite ordering of the sum.
Revised: whereas this does not hold for the opposite ordering of the sum.

Original: E
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Plugging in A
Revised: Plugging in A

Original: =
Revised: =

Original: J T J and n = 1 , we get
Revised: J T J and n = 1 , we get

Original: [Equation]
Revised: [Equation]

Revised: Non-isotropic perturbations along data manifold Consistency regularization with natural perturbations such as image translation can also be understood as penalizing a Jacobian norm as in Section 3.1.

Original: Extension to perturbationsfrom the tangent space T x ( M ) : Suppose that we have small additive perturbations that are sampled from a normal distribution over the tangent space, z ∼ P ( x ) N (0 , I )  .
Revised: For example, consider perturbations sampled from a normal distribution on the tangent space, z ⇠ P ( x ) N (0 , I ) where P ( x ) = P ( x ) 2 is the orthogonal projection matrix that projects down from R d to T x ( M ) , the tangent space of the image manifold at x .

Revised: Then the consistency regularization penalizes the Laplacian norm of the network on the manifold (with the inherited metric from R d ).

Original: E [ z ] = 0 and E [ zz T ] = PP T (=) P 2 = P which follows if P is an orthogonal projection matrix.
Revised: E [ z ] = 0 and E [ zz T ] = PP T (=) P 2 = P which follows if P is an orthogonal projection matrix.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Revised: We view the standard data augmentations such as random translation (that are applied in the ⇧ and MT models) as approximating samples of nearby elements of the data manifold and therefore differences x 0  x approximate elements of its tangent space.

Original: A. R ELATIONSHIP B ETWEEN E x [ (cid:107) J w (cid:107) 2 F ] AND R ANDOM R AY S HARPNESS
Revised: A. R ELATIONSHIP B ETWEEN E x [ k J w k 2 F ] AND R ANDOM R AY S HARPNESS

Original: In the following analysis we review an argument for why smaller E x [ (cid:107) J w (cid:107) 2 F ] , implies broader optima.
Revised: In the following analysis we review an argument for why smaller E x [ k J w k 2 F ] , implies broader optima.

Original: To keep things simple, we focus on the MSE loss, but in principle a similar argument should apply for the Cross Entropy and the Error rate.
Revised: To keep things simple, we focus on the MSE loss, but in principle a similar argument should apply for the Cross Entropy and the Error rate.

Original: For a single data point x and one hot vector y with k classes, the hessian of (cid:96) MSE ( w ) =(cid:107) f
Revised: For a single data point x and one hot vector y with k classes, the hessian of ` MSE ( w ) =

Original:  ( x, w ) − y (cid:107) 2 can be decomposed into two terms, the Gauss-Newton matrix G =
Revised: k f ( x, w )  y k 2 can be decomposed into two terms, the Gauss-Newton matrix G =

Original: J Tw J w and a term which depends on the labels.
Revised: J Tw J w and a term which depends on the labels.

Original: [Equation]
Revised: [Equation]

Original: Thus tr ( H ) is also the sum of two terms, (cid:107) J w (cid:107) 2 F and α .
Revised: Thus tr ( H ) is also the sum of two terms, k J w k 2 F and ↵ .

Original: As the solution improves, the relative size of α goes down.
Revised: As the solution improves, the relative size of ↵ goes down.

Original: In terms of random ray sharpness, consider the expected MSE loss, or risk,
Revised: In terms of random ray sharpness, consider the expected MSE loss, or risk,

Original: R MSE ( w ) =
Revised: R MSE ( w ) =

Original: E ( x,y ) (cid:107) f
Revised: E ( x,y ) k

Original:  ( x, w ) − y (cid:107) 2 along random rays.
Revised: f ( x, w )  y k 2 along random rays.

Original: Let d be a random vector sampled from the unit sphere
Revised: Let d be a random vector sampled from the unit sphere

Original: and s is the distance along the random ray.
Revised: and s is the distance along the random ray.

Original: Evaluating the risk on a random ray, and
Revised: Evaluating the risk on a random ray, and

Original: Taylor expanding in s we have
Revised: Taylor expanding in s we have

Original: [Equation]
Revised: [Equation]

Original: Since d is from the unit sphere, E [ d ] = 0 and E [ dd T ] =
Revised: Since d is from the unit sphere, E [ d ] = 0 and E [ dd T ] =

Original: I/p where p is the dimension.
Revised: I/p where p is the dimension.

Original: Averaging over the rays, d ∼ Unif ( S p − 1 ) , we have
Revised: Averaging over the rays, d ⇠ Unif ( S p  1 ) , we have

Original: [Equation]
Revised: [Equation]

Original: All of the odd terms vanish because of the reﬂection symmetry of the unit sphere.
Revised: All of the odd terms vanish because of the reﬂection symmetry of the unit sphere.

Original: This means that locally, the sharpness of the optima (as measured by random rays) can be lowered by decreasing E x [ (cid:107) J w (cid:107) 2 F ] .
Revised: This means that locally, the sharpness of the optima (as measured by random rays) can be lowered by decreasing E x [ k J w k 2 F ] .

Original: A. I NCLUDING H IGH L EARNING R ATE I TERATES I NTO SWA
Revised: A. I NCLUDING H IGH L EARNING R ATE I TERATES I NTO SWA

Original: As discussed in Mandt et al.
Revised: As discussed in Mandt et al.

Original: (2017), under certain assumptions
Revised: (2017), under certain assumptions

Original: SGD samples from a Gaussian distribution centered at the optimum of the loss w 0 with covariance proportional to the learning rate.
Revised: SGD samples from a Gaussian distribution centered at the optimum of the loss w 0 with covariance proportional to the learning rate.

Original: Suppose then that we have n weights sampled at learning rate η 1 , w (1) i iid ∼ N ( w 0 , η 1 Σ) and m weights sampled with the higher learning rate η 2 , w (2) j iid ∼ N ( w 0 , η 2 Σ) .
Revised: Suppose then that we have n weights sampled at learning rate ⌘ 1 , w (1) i ⇠ N ( w 0 , ⌘ 1 ⌃ ) and m weights sampled with the higher learning rate ⌘ 2 , w (2) j iid ⇠ N ( w 0 , ⌘ 2 ⌃ ) .

Original: For the SWA estimator ˆ w SWA = 1 n(cid:80) i w (1) i , E
Revised: For the SWA estimator ˆ w SWA = 1 n

Original:  [ (cid:107) ˆ w SWA − w 0 (cid:107) 2 ] = tr ( Cov ( ˆ w SWA )) 
Revised: P i w (1) i , E [ k ˆ w SWA  w 0 k 2 ] = tr ( Cov ( ˆ w SWA )) =

Original: = η 1 n tr (Σ) .
Revised: ⌘ 1 n tr ( ⌃ ) .

Original: But if we include the high variance points in the average, as in fast-SWA, ˆ w fSWA = 1 n + m (cid:0) (cid:80) i w (1) i + (cid:80) j w (2) j (cid:1) , then
Revised: But if we include the high variance points in the average, as in fast-SWA, ˆ w fSWA = 1 n + m  P i w (1) i + P j w (2) j  , then

Original: E [ (cid:107) ˆ w fSWA − w 0 (cid:107) 2 ] = nη 1 + mη 2 ( n + m ) 2 tr (Σ) .
Revised: E [ k ˆ w fSWA  w 0 k 2 ] = n ⌘ 1 + m ⌘ 2 ( n + m ) 2 tr ( ⌃ ) .

Original: If nη 1 + mη 2 ( n + m ) 2 < η 1 n then including the high learning rate points decreases the MSE of the estimator for m > n (cid:0) η 2 η 1 − 2 (cid:1) .
Revised: If n ⌘ 1 + m ⌘ 2 ( n + m ) 2 < ⌘ 1 n then including the high learning rate points decreases the MSE of the estimator for m > n  ⌘ 2 ⌘ 1  .

Original: If we include enough points, we will still improve the estimate.
Revised: If we include enough points, we will still improve the estimate.

Original: A. N ETWORK A RCHITECTURES
Revised: A. N ETWORK A RCHITECTURES

Original: In the experiments we use two DNN architectures – 13 layer CNN and Shake-Shake.
Revised: In the experiments we use two DNN architectures – 13 layer CNN and Shake-Shake.

Original: The architecture of 13 -layer CNN is described in Table 6.
Revised: The architecture of 13 -layer CNN is described in Table 6.

Original: It closely follows the architecture used in (Laine and
Revised: It closely follows the architecture used in (Laine and

Original: Aila, 2017; Miyato et al., 2017; Tarvainen and Valpola, 2017).
Revised: Aila, 2017; Miyato et al., 2017; Tarvainen and Valpola, 2017).

Original: We re-implement it in PyTorch and removed the Gaussian input noise, since we found having no such noise improves generalization.
Revised: We re-implement it in PyTorch and removed the Gaussian input noise, since we found having no such noise improves generalization.

Original: For Shake-Shake we use 26-2x96d Shake-Shake regularized architecture of Gastaldi (2017) withresidual blocks.
Revised: For Shake-Shake we use 26-2x96d Shake-Shake regularized architecture of Gastaldi (2017) withresidual blocks.

Original: [Table]
Revised: [Table]

Original: A. H YPERPARAMETERS
Revised: . H YPERPARAMETERS

Original: We consider two different schedules.
Revised: We consider two different schedules.

Original: In the short schedule we set the cosine half-period [-(cid:96) 
-] = 210 and training length (cid:96) = 180 , following the schedule used in Tarvainen and Valpola (2017) in Shake-Shake experiments.
Revised: In the short schedule we set the cosine half-period [-(cid:96) 
-]` 0 = 210 and training length ` = 180 , following the schedule used in Tarvainen and Valpola (2017) in Shake-Shake experiments.

Original: For our Shake-Shake experiments we also report results with long schedule where we set (cid:96) = 1800 , [-(cid:96) 
-] = 1500 following Gastaldi (2017).
Revised: For our Shake-Shake experiments we also report results with long schedule where we set ` = 1800 , [-(cid:96) 
-]` 0 = 1500 following Gastaldi (2017).

Original: To determine the initial learning rate η 0 and the cycle length
Revised: To determine the initial learning rate ⌘ 0 and the cycle length

Original: c we used a separate validation set of size 5000 taken from the unlabeled data.
Revised: c we used a separate validation set of size 5000 taken from the unlabeled data.

Original: After determining these values, we added the validation set to the unlabeled data and trained again.
Revised: After determining these values, we added the validation set to the unlabeled data and trained again.

Original: We reuse the same values of η 0 and c for all experiments with different numbers of labeled data for both Π model and Mean Teacher for a ﬁxed architecture ( 13 -layer CNN or Shake-Shake).
Revised: We reuse the same values of ⌘ 0 and c for all experiments with different numbers of labeled data for both ⇧ model and Mean Teacher for a ﬁxed architecture ( 13 -layer CNN or Shake-Shake).

Original: For the short schedule we use cycle length c = 30 and average models once every k = 3 epochs.
Revised: For the short schedule we use cycle length c = 30 and average models once every k = 3 epochs.

Original: For long schedule we use c = 200 , k = 20 .
Revised: For long schedule we use c = 200 , k = 20 .

Original: In all experiments we use stochastic gradient descent optimizer with Nesterov momentum (Loshchilov and Hutter, 2016).
Revised: In all experiments we use stochastic gradient descent optimizer with Nesterov momentum (Loshchilov and Hutter, 2016).

Original: In fast-SWA we average every the weights of the models corresponding to every third epoch.
Revised: In fast-SWA we average every the weights of the models corresponding to every third epoch.

Original: In the Π model, we back-propagate the gradients through the student side only (as opposed to both sides in (Laine and Aila, 2016)).
Revised: In the ⇧ model, we back-propagate the gradients through the student side only (as opposed to both sides in (Laine and Aila, 2016)).

Original: For Mean Teacher we use α = 0 .
Revised: For Mean Teacher we use ↵ = 0 .

Original: 97 decay rate in the Exponential Moving Average (EMA) of the student’s weights.
Revised: 97 decay rate in the Exponential Moving Average (EMA) of the student’s weights.

Original: For all other hyper-parameters we reuse the values from Tarvainen and Valpola (2017) unless mentioned otherwise.
Revised: For all other hyper-parameters we reuse the values from Tarvainen and Valpola (2017) unless mentioned otherwise.

Original: Like in Tarvainen and Valpola (2017), we use (cid:107) · (cid:107) 2 for divergence in the consistency loss.
Revised: Like in Tarvainen and Valpola (2017), we use k · k 2 for divergence in the consistency loss.

Original: Similarly, we ramp up the consistency cost λ over the ﬁrst 5 epochs from 0 up to it’s maximum value of 100 as done in Tarvainen and Valpola (2017).
Revised: Similarly, we ramp up the consistency cost  over the ﬁrst 5 epochs from 0 up to it’s maximum value of 100 as done in Tarvainen and Valpola (2017).

Original: We use cosine annealing learning rates with no learning rate ramp up, unlike in the original MT implementation (Tarvainen and Valpola, 2017).
Revised: We use cosine annealing learning rates with no learning rate ramp up, unlike in the original MT implementation (Tarvainen and Valpola, 2017).

Original: Note that this is similar to the same hyperparameter settings as in Tarvainen and Valpola (2017) for ResNet 3 .
Revised: Note that this is similar to the same hyperparameter settings as in Tarvainen and Valpola (2017) for ResNet 2 .

Original: We note that we use the exact same hyperparameters for the Π and MT models in each experiment setting.
Revised: We note that we use the exact same hyperparameters for the ⇧ and MT models in each experiment setting.

Original: In contrast to the original implementation in Tarvainen and Valpola (2017) of CNN experiments, we use
Revised: In contrast to the original implementation in Tarvainen and Valpola (2017) of CNN experiments, we use

Original: SGD instead of Adam.
Revised: SGD instead of Adam.

Original: Understanding Experiments in Sections 3.2, 3.3 We use the 13 -layer CNN with the short learning rate schedule.
Revised: Understanding Experiments in Sections 3.2, 3.3 We use the 13 -layer CNN with the short learning rate schedule.

Original: We use a total batch size of 100 for CNN experiments with a labeled batch size offor the Π and Mean Teacher models.
Revised: We use a total batch size of 100 for CNN experiments with a labeled batch size offor the ⇧ and Mean Teacher models.

Original: We use the maximum learning rate η 0 = 0 .
Revised: We use the maximum learning rate ⌘ 0 = 0 .

Original: For Section 3.
Revised: For Section 3.

Original: we run SGD only for 180 epochs, so 0 learning rate cycles are done.
Revised: we run SGD only for 180 epochs, so 0 learning rate cycles are done.

Original: For Section 3.3 we additionally run 5 learning rate cycles and sample pairs of SGD iterates from epochs 180 - 330 corresponding to these cycles.
Revised: For Section 3.3 we additionally run 5 learning rate cycles and sample pairs of SGD iterates from epochs 180 - 330 corresponding to these cycles.

Original: CIFAR-10 CNN Experiments
Revised: CIFAR-10 CNN Experiments

Original: We use a total batch size of 100 for CNN experiments with a labeled batch size of 50 .
Revised: We use a total batch size of 100 for CNN experiments with a labeled batch size of 50 .

Original: We use the maximum learning rate η 0 = 0 .
Revised: We use the maximum learning rate ⌘ 0 = 0 .

Original: CIFAR-

Revised: CIFAR-

Original: ResNet + Shake-Shake
Revised: ResNet + Shake-Shake

Original: We use a total batch size of 128 for ResNet experiments with a labeled batch size of 31 .
Revised: We use a total batch size of 128 for ResNet experiments with a labeled batch size of 31 .

Original: We use the maximum learning rate η 0 = 0 . 05 for CIFAR-10.
Revised: We use the maximum learning rate ⌘ 0 = 0 . 05 for CIFAR-10.

Original: This applies for both the short and long schedules.
Revised: This applies for both the short and long schedules.

Original: CIFAR-100 CNN Experiments We use a total batch size of 128 with a labeled batch size offor 10 k and 50 k label settings.
Revised: CIFAR-100 CNN Experiments We use a total batch size of 128 with a labeled batch size offor 10 k and 50 k label settings.

Original: For the settings 50 k + 500 k and 50 k + 237 k ∗ , we use a labeled batch size of 64 .
Revised: For the settings 50 k + 500 k and 50 k + 237 k ⇤ , we use a labeled batch size of 64 .

Original: We also limit the number of unlabeled images used in each epoch to 100 k images.
Revised: We also limit the number of unlabeled images used in each epoch to 100 k images.

Original: We use the maximum learning rate η 0 = 0 .
Revised: We use the maximum learning rate ⌘ 0 = 0 .

Original: CIFAR-100 ResNet + Shake-Shake We use a total batch size of 128 for ResNet experiments with a labeled batch size of 31 in all label settings.
Revised: CIFAR-100 ResNet + Shake-Shake We use a total batch size of 128 for ResNet experiments with a labeled batch size of 31 in all label settings.

Original: For the settings 50 k + 500 k and 50 k + 237 k ∗ , we also limit the number of unlabeled images used in each epoch to 100 k images.
Revised: For the settings 50 k + 500 k and 50 k + 237 k ⇤ , we also limit the number of unlabeled images used in each epoch to 100 k images.

Original: We use the maximum learning rate η 0 = 0 .
Revised: We use the maximum learning rate ⌘ 0 = 0 .

Original: This applies for both the short and long schedules.
Revised: This applies for both the short and long schedules.

Original: A. D OMAIN A DAPTATION
Revised: A. D OMAIN A DAPTATION

Original: We apply fast-SWA to the best experiment setting MT+CT+TFA for CIFAR-10 to STL according to French et al.
Revised: We apply fast-SWA to the best experiment setting MT+CT+TFA for CIFAR-10 to STL according to French et al.

Original: This setting involves using conﬁdence thresholding (CT) and also an augmentation scheme with translation, ﬂipping, and afﬁne transformation (TFA).
Revised: This setting involves using conﬁdence thresholding (CT) and also an augmentation scheme with translation, ﬂipping, and afﬁne transformation (TFA).

Original: We modify the optimizer to use SGD instead of Adam (Kingma and Ba, 2015) and use cosine annealing schedule with [-(cid:96) 
-] = 600 , (cid:96) = 550 , c = 50 .
Revised: We modify the optimizer to use SGD instead of Adam (Kingma and Ba, 2015) and use cosine annealing schedule with [-(cid:96) 
-]` 0 = 600 , ` = 550 , c = 50 .

Original: We experimented with two fast-SWA methods: averaging weights once per epoch and averaging once every iteration.
Revised: We experimented with two fast-SWA methods: averaging weights once per epoch and averaging once every iteration, which is much more frequent that averaging every epoch as in the semi-supervised case.

Revised: Interestingly, we found that for this task averaging the weights in the end of every iteration in fast-SWA converges signiﬁcantly faster than averaging once per epoch and results in better performance.

Original: We report the results in Table 7.
Revised: We report the results in Table 7.

Original: We observe that averaging every iteration converges much faster ( 600 epochs instead of 3000 ) and results in better test accuracy.
Revised: We observe that averaging every iteration converges much faster ( 600 epochs instead of 3000 ) and results in better test accuracy.

Original: In our experiments with semi-supervised learning averaging more often than once per epoch didn’t improve convergence or ﬁnal results.
Revised: In our experiments with semi-supervised learning averaging more often than once per epoch didn’t improve convergence or ﬁnal results.

Original: We hypothesize that the improvement from more frequent averaging is a result of speciﬁc geometry of the loss surfaces and training trajectories in domain adaptation.
Revised: We hypothesize that the improvement from more frequent averaging is a result of speciﬁc geometry of the loss surfaces and training trajectories in domain adaptation.

Original: We leave further analysis of applying fast-SWA to domain adaptation for future work.
Revised: We leave further analysis of applying fast-SWA to domain adaptation for future work.

Original: [Table]

Original: Implementation Details We use the public code 4 of French et al.
Revised: Implementation Details We use the public code 3 of French et al.

Original: (2018) to train the model and apply fast-SWA.
Revised: (2018) to train the model and apply fast-SWA.

Original: While the original implementation uses Adam (Kingma and Ba, 2015), we use stochastic gradient descent with Nesterov momentum and cosine annealing learning rate with [-(cid:96) 
-] = 600 , (cid:96) = 550 , c = 100 and k = 100 .
Revised: While the original implementation uses Adam (Kingma and Ba, 2015), we use stochastic gradient descent with Nesterov momentum and cosine annealing learning rate with [-(cid:96) 
-]` 0 = 600 , ` = 550 , c = 100 and k = 100 .

Original: We use the maximum learning rate η 0 = 0 .
Revised: We use the maximum learning rate ⌘ 0 = 0 .

Original: 1 and momentum0 .
Revised: 1 and momentum

Original: 9 with weight decay of scale 2 × 10 − 4 .
Revised: [Table]

Original: We use the data augmentation setting MT+CF+TFA in Table 1 of French et al. and apply fast-SWA. The result reported is from epoch 4000 .
Revised: [Table]