Original: Entropic gradient descent algorithms and wide flat minima
Revised: Entropic gradient descent algorithms and wide flat minima

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: Abstract
Revised: Abstract

Original: The properties of ﬂat minima in the empirical risk landscape of neural networks have been debated for some time.
Revised: The properties of ﬂat minima in the empirical risk landscape of neural networks have been debated for some time.

Original: Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones.
Revised: Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones.

Original: In this work we ﬁrst discuss the relationship between alternative measures of ﬂatness: The local entropy , which is useful for analysis and algorithm development, and the local energy , which is easier to compute and was shown empirically in extensive tests on state-of-the-art networks to be the best predictor of generalization capabilities.
Revised: In this work we ﬁrst discuss the relationship between alternative measures of ﬂatness: The local entropy , which is useful for analysis and algorithm development, and the local energy , which is easier to compute and was shown empirically in extensive tests on state-of-the-art networks to be the best predictor of generalization capabilities.

Original: We show semi-analytically in simple controlled scenarios that these two measures correlate strongly with each other and with generalization.
Revised: We show semi-analytically in simple controlled scenarios that these two measures correlate strongly with each other and with generalization.

Original: Then, we extend the analysis to the deep learning scenario by extensive numerical validations.
Revised: Then, we extend the analysis to the deep learning scenario by extensive numerical validations.

Original: We study two algorithms, Entropy-SGD and Replicated-SGD, that explicitly include the local entropy in the optimization objective.
Revised: We study two algorithms, Entropy-SGD and Replicated-SGD, that explicitly include the local entropy in the optimization objective.

Original: We devise a training schedule by which we consistently ﬁnd ﬂatter minima (using both ﬂatness measures), and improve the generalization error for common architectures (e.g. ResNet,
Revised: We devise a training schedule by which we consistently ﬁnd ﬂatter minima (using both ﬂatness measures), and improve the generalization error for common architectures (e.g. ResNet,

Original: EﬃcientNet).
Revised: EﬃcientNet).

Original:  Introduction
Revised:  Introduction

Original: The geometrical structure of the loss landscape of neural networks has been a key topic of study for several decades (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016).
Revised: The geometrical structure of the loss landscape of neural networks has been a key topic of study for several decades (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016).

Original: One area of ongoing research is the connection between the ﬂatness of minima found by optimization algorithms like stochastic gradient descent (SGD) and the generalization performance of the network (Baldassi et al., 2020; Keskar et al., 2016).
Revised: One area of ongoing research is the connection between the ﬂatness of minima found by optimization algorithms like stochastic gradient descent (SGD) and the generalization performance of the network (Baldassi et al., 2020; Keskar et al., 2016).

Original: There are open conceptual problems in this context:
Revised: There are open conceptual problems in this context:

Original: On the one hand, there is accumulating evidence that ﬂatness is a good predictor of generalization (Jiang et al., 2019).
Revised: On the one hand, there is accumulating evidence that ﬂatness is a good predictor of generalization (Jiang et al., 2019).

Original: On the other hand, modern deep networks using ReLU activations are invariant in their outputs with respect to rescaling of weights in diﬀerent layers (Dinh et al., 2017) which makes the mathematical picture complicated 1 .
Revised: On the other hand, modern deep networks using ReLU activations are invariant in their outputs with respect to rescaling of weights in diﬀerent layers (Dinh et al., 2017) which makes the mathematical picture complicated 1 .

Original: General results are lacking.
Revised: General results are lacking.

Original: Some initial progress has been made in connecting PAC-Bayes bounds for the generalization gap with ﬂatness (Dziugaite & Roy, 2018).
Revised: Some initial progress has been made in connecting PAC-Bayes bounds for the generalization gap with ﬂatness (Dziugaite & Roy, 2018).

Original: The purpose of this work is to shed light on the connection between ﬂatness and generalization by using methods and algorithms from the statistical physics of disordered systems, and to corroborate the results with a performance study on state-of-the-art deep architectures.
Revised: The purpose of this work is to shed light on the connection between ﬂatness and generalization by using methods and algorithms from the statistical physics of disordered systems, and to corroborate the results with a performance study on state-of-the-art deep architectures.

Original: Methods from statistical physics have led to several results in the last years.
Revised: Methods from statistical physics have led to several results in the last years.

Original: Firstly, wide ﬂat minima have been shown to be a structural property of shallow networks.
Revised: Firstly, wide ﬂat minima have been shown to be a structural property of shallow networks.

Original: They exist even when training on random data and are accessible by relatively simple algorithms, even though coexisting with exponentially more numerous minima (Baldassi et al., 2015; 2016a; 2020).
Revised: They exist even when training on random data and are accessible by relatively simple algorithms, even though coexisting with exponentially more numerous minima (Baldassi et al., 2015; 2016a; 2020).

Original: We believe this to be an overlooked property of neural networks, which makes them particularly suited for learning.
Revised: We believe this to be an overlooked property of neural networks, which makes them particularly suited for learning.

Original: In analytically tractable settings, it has been shown that ﬂatness depends on the choice of the loss and activation functions, and that it correlates with generalization (Baldassi et al., 2020; 2019).
Revised: In analytically tractable settings, it has been shown that ﬂatness depends on the choice of the loss and activation functions, and that it correlates with generalization (Baldassi et al., 2020; 2019).

Original: In the above-mentioned works, the notion of ﬂatness used was the so-called local entropy (Baldassi et al., 2015; 2016a).
Revised: In the above-mentioned works, the notion of ﬂatness used was the so-called local entropy (Baldassi et al., 2015; 2016a).

Original: It measures the low-loss volume in the weight space around a minimizer, as a function of the distance (i.e. roughly speaking it measures the amount of “good” conﬁgurations around a given one).
Revised: It measures the low-loss volume in the weight space around a minimizer, as a function of the distance (i.e. roughly speaking it measures the amount of “good” conﬁgurations around a given one).

Original: This framework is not only useful for analytical calcuations, but it has also been used to introduce a variety of eﬃcient learning algorithms that focus their search on ﬂat regions (Baldassi et al., 2016a; Chaudhari et al., 2019; 2017).
Revised: This framework is not only useful for analytical calcuations, but it has also been used to introduce a variety of eﬃcient learning algorithms that focus their search on ﬂat regions (Baldassi et al., 2016a; Chaudhari et al., 2019; 2017).

Original: In this paper we call them entropic algorithms .
Revised: In this paper we call them entropic algorithms .

Original: A diﬀerent notion of ﬂatness, that we refer to as local energy in this paper, measures the average proﬁle of the training loss function around a minimizer, as a function of the distance (i.e. it measures the typical increase in the training error when moving away from the minimizer).
Revised: A diﬀerent notion of ﬂatness, that we refer to as local energy in this paper, measures the average proﬁle of the training loss function around a minimizer, as a function of the distance (i.e. it measures the typical increase in the training error when moving away from the minimizer).

Original: This quantity is intuitively appealing and rather easy to estimate via sampling, even in large systems.
Revised: This quantity is intuitively appealing and rather easy to estimate via sampling, even in large systems.

Original: In Jiang et al.
Revised: In Jiang et al.

Original: , several candidates for predicting generalization performance were tested using an extensive numerical approach on an array of diﬀerent networks and tasks, and the local energy was found to be among the best and most consistent predictors.
Revised: , several candidates for predicting generalization performance were tested using an extensive numerical approach on an array of diﬀerent networks and tasks, and the local energy was found to be among the best and most consistent predictors.

Original: The two notions, local entropy and local energy, are distinct: in a given region of a complex landscape, the local entropy measures the size of the lowest valleys, whereas the local energy measures the average height.
Revised: The two notions, local entropy and local energy, are distinct: in a given region of a complex landscape, the local entropy measures the size of the lowest valleys, whereas the local energy measures the average height.

Original: Therefore, in principle, the two quantities could vary independently.
Revised: Therefore, in principle, the two quantities could vary independently.

Original: It seems reasonable, however, to conjecture that they would be highly correlated under mild assumptions on the roughness of the landscape (which is another way to say that they are both reasonable measures to express the intuitive notion of "ﬂatness").
Revised: It seems reasonable, however, to conjecture that they would be highly correlated under mild assumptions on the roughness of the landscape (which is another way to say that they are both reasonable measures to express the intuitive notion of "ﬂatness").

Original: In this paper, we ﬁrst show that for simple systems in controlled conditions (where all relevant quantities can be estimated well by using the Belief Propagation (BP) algorithm)
Revised: In this paper, we ﬁrst show that for simple systems in controlled conditions, where all relevant quantities can be estimated well by using the Belief Propagation (BP) algorithm (Mezard &

Original:  the two notions of ﬂatness are strongly correlated: regions of high local entropy have low local energy, and vice versa.
Revised: Montanari (2009)), the two notions of ﬂatness are strongly correlated: regions of high local entropy have low local energy, and vice versa.

Original: We also conﬁrm that they are both correlated with generalization.
Revised: We also conﬁrm that they are both correlated with generalization.

Original: This justiﬁes the expectation that, even for more complex architectures and datasets, those algorithms which are driven towards high-local-entropy regions would minimize the local energy too, and thus (based on the ﬁndings in Jiang et al.
Revised: This justiﬁes the expectation that, even for more complex architectures and datasets, those algorithms which are driven towards high-local-entropy regions would minimize the local energy too, and thus (based on the ﬁndings in Jiang et al.

Original: would ﬁnd minimizers that generalize well.
Revised: would ﬁnd minimizers that generalize well.

Original: Indeed, we systematically applied two entropic algorithms, EntropySGD (eSGD) and Replicated-SGD (rSGD), to state-of-the-art deep architectures, and found that we could achieve an improved generalization performance, at the same computational cost, compared to the original papers where those architectures were introduced. 
Revised: Indeed, we systematically applied two entropic algorithms, Entropy-SGD (eSGD) and Replicated-SGD (rSGD), to state-of-the-art deep architectures, and found that we could achieve an improved generalization performance, at the same computational cost, compared to the original papers where those architectures were introduced. We believe these results to be an important addition to the current state of knowledge, since in (Baldassi et al. (2016b))

Revised: rSGD was applied only to shallow networks with binary weights trained on random patterns and the current work represents the ﬁrst study of rSGD in a realistic deep neural network setting.

Revised: Together with the ﬁrst reported consistent improvement of eSGD over SGD on image classiﬁcation, these results point to a very promising direction for further research.

Revised: While we hope to foster the application of entropic algorithms by publishing code that can be used to adapt them easily to new architectures, we also believe that the numeric results are important for theoretical research, since they are rooted in a well-deﬁned geometric interpretation of the loss landscape.

Original: We also conﬁrmed numerically that the minimizers found in this way have a lower local energy proﬁle, as expected.
Revised: We also conﬁrmed numerically that the minimizers found in this way have a lower local energy proﬁle, as expected.

Original: Remarkably, these results go beyond even those where the eSGD and rSGD algorithms were originally introduced, thanks to a general improvement in the choice for the learning protocol, that we also discuss; apart from that, we used little to no hyper-parameter tuning.
Revised: Remarkably, these results go beyond even those where the eSGD and rSGD algorithms were originally introduced, thanks to a general improvement in the choice for the learning protocol, that we also discuss; apart from that, we used little to no hyper-parameter tuning.

Original:  Related work
Revised:  Related work

Original: The idea of using the ﬂatness of a minimum of the loss function, also called the fatness of the posterior and the local area estimate of quality , for evaluating diﬀerent minimizers is several decades old (Hochreiter & Schmidhuber, 1997; Hinton & van Camp, 1993; Buntine & Weigend, 1991).
Revised: The idea of using the ﬂatness of a minimum of the loss function, also called the fatness of the posterior and the local area estimate of quality , for evaluating diﬀerent minimizers is several decades old (Hochreiter & Schmidhuber, 1997; Hinton & van Camp, 1993; Buntine & Weigend, 1991).

Original: These works connect the ﬂatness of a minimum to information theoretical concepts like the minimum description length of its minimizer: ﬂatter minima correspond to minimizers that can be encoded using fewer bits.
Revised: These works connect the ﬂatness of a minimum to information theoretical concepts like the minimum description length of its minimizer: ﬂatter minima correspond to minimizers that can be encoded using fewer bits.

Original: For neural networks, a recent empirical study (Keskar et al., 2016) shows that large-batch methods ﬁnd sharp minima while small-batch ones ﬁnd ﬂatter ones, with a positive eﬀect on generalization performance.
Revised: For neural networks, a recent empirical study (Keskar et al., 2016) shows that large-batch methods ﬁnd sharp minima while small-batch ones ﬁnd ﬂatter ones, with a positive eﬀect on generalization performance.

Original: PAC-Bayes bounds can be used for deriving generalization bounds for neural networks (Zhou et al., 2018).
Revised: PAC-Bayes bounds can be used for deriving generalization bounds for neural networks (Zhou et al., 2018).

Original: In Dziugaite & Roy (2017), a method for optimizing the PAC-Bayes bound directly is introduced and the authors note similarities between the resulting objective function and an objective function that searches for ﬂat minima.
Revised: In Dziugaite & Roy (2017), a method for optimizing the PAC-Bayes bound directly is introduced and the authors note similarities between the resulting objective function and an objective function that searches for ﬂat minima.

Original: This connection is further analyzed in Dziugaite & Roy (2018).
Revised: This connection is further analyzed in Dziugaite & Roy (2018).

Original: In Jiang et al.
Revised: In Jiang et al.

Original: (2019), the authors present a large-scale empirical study of the correlation between diﬀerent complexity measures of neural networks and their generalization perfor- mance.
Revised: (2019), the authors present a large-scale empirical study of the correlation between diﬀerent complexity measures of neural networks and their generalization performance.

Original: The authors conclude that PAC-Bayes bounds and ﬂatness measures (in particular, what we call local energy in this paper) are the most predictive measures of generalization.
Revised: The authors conclude that PAC-Bayes bounds and ﬂatness measures (in particular, what we call local energy in this paper) are the most predictive measures of generalization.

Original: The concept of local entropy has been introduced in the context of a statistical mechanics approach to machine learning for discrete neural networks in Baldassi et al.
Revised: The concept of local entropy has been introduced in the context of a statistical mechanics approach to machine learning for discrete neural networks in Baldassi et al.

Original: (2015), and subsequently extended to models with continuous weights.
Revised: (2015), and subsequently extended to models with continuous weights.

Original: We provide a detailed deﬁnition in the next section, but mention here that it measures a volume in the space of conﬁgurations, which poses computational diﬃculties.
Revised: We provide a detailed deﬁnition in the next section, but mention here that it measures a volume in the space of conﬁgurations, which poses computational diﬃculties.

Original: On relatively tractable shallow networks, the local entropy of any given conﬁguration can be computed eﬃciently using Belief Propagation (see next section), and it can be also used directly as a training objective.
Revised: On relatively tractable shallow networks, the local entropy of any given conﬁguration can be computed eﬃciently using Belief Propagation, and it can be also used directly as a training objective.

Original: In this setting, detailed analytical studies accompanied by numerical experiments have shown that the local entropy correlates with the generalization error and the eigenvalues of the Hessian (Baldassi et al., 2015; 2020).
Revised: In this setting, detailed analytical studies accompanied by numerical experiments have shown that the local entropy correlates with the generalization error and the eigenvalues of the Hessian (Baldassi et al., 2015; 2020).

Original: Another interesting ﬁnding is that the cross-entropy loss (Baldassi et al., 2020) and ReLU transfer functions (Baldassi et al., 2019), which have become the de-facto standard for neural networks, tend to bias the models towards high local entropy regions (computed based on the error loss).
Revised: Another interesting ﬁnding is that the cross-entropy loss (Baldassi et al., 2020) and ReLU transfer functions (Baldassi et al., 2019), which have become the de-facto standard for neural networks, tend to bias the models towards high local entropy regions (computed based on the error loss).

Original: Extending such techniques for general architectures is an open problem.
Revised: Extending such techniques for general architectures is an open problem.

Original: However, the local entropy objective can be approximated to derive general algorithmic schemes.
Revised: However, the local entropy objective can be approximated to derive general algorithmic schemes.

Original: Replicated stochastic gradient descent (rSGD) replaces the local entropy objective by an objective involving several replicas of the model, each one moving in the potential induced by the loss while also attracting each other.
Revised: Replicated stochastic gradient descent (rSGD) replaces the local entropy objective by an objective involving several replicas of the model, each one moving in the potential induced by the loss while also attracting each other.

Original: The method has been introduced in Baldassi et al.
Revised: The method has been introduced in Baldassi et al.

Original: (2016a), but only demonstrated on shallow networks.
Revised: (2016a), but only demonstrated on shallow networks.

Original: The rSGD algorithm is closely related to Elastic
Revised: The rSGD algorithm is closely related to Elastic

Original: Averaging SGD (EASGD), presented in Zhang et al.
Revised: Averaging SGD (EASGD), presented in Zhang et al.

Original: (2014), even though the latter was motivated purely by the idea of enabling massively parallel training and had no theoretical basis.
Revised: (2014), even though the latter was motivated purely by the idea of enabling massively parallel training and had no theoretical basis.

Original: The substantial distinguishing feature of rSGD compared to EASGD when applied to deep networks is the focusing procedure, discussed in more detail below.
Revised: The substantial distinguishing feature of rSGD compared to EASGD when applied to deep networks is the focusing procedure, discussed in more detail below.

Original: Another diﬀerence is that in rSGD there is no explicit master replica.
Revised: Another diﬀerence is that in rSGD there is no explicit master replica.

Original: Entropy-SGD (eSGD), introduced in Chaudhari et al.
Revised: Entropy-SGD (eSGD), introduced in Chaudhari et al.

Original: (2019), is a method that directly optimizes the local entropy using stochastic gradient Langevin dynamics (SGLD)
Revised: (2019), is a method that directly optimizes the local entropy using stochastic gradient Langevin dynamics (SGLD)

Original: (Welling &
Revised: (Welling &

Original: Teh, 2011).
Revised: Teh, 2011).

Original: While the goal of this method is the same as rSGD, the optimization techniques involves a double loop instead of replicas.
Revised: While the goal of this method is the same as rSGD, the optimization techniques involves a double loop instead of replicas.

Original: Parle (Chaudhari et al., 2017), combines eSGD and
Revised: Parle (Chaudhari et al., 2017), combines eSGD and

Original: EASGD (with added focusing) to obtain a distributed algorithm that shows also excellent generalization performance, consistently with the results obtained in this work.
Revised: EASGD (with added focusing) to obtain a distributed algorithm that shows also excellent generalization performance, consistently with the results obtained in this work.

Original:  Flatness measures: local entropy, local energy
Revised:  Flatness measures: local entropy, local energy

Original: The general deﬁnition of the local entropy loss L LE for a system in a given conﬁguration w (a vector of size N ) can be given in terms of any common (usually, data-dependent) loss L as:
Revised: The general deﬁnition of the local entropy loss L LE for a system in a given conﬁguration w (a vector of size N ) can be given in terms of any common (usually, data-dependent) loss L as:

Original: [Equation]
Revised: [Equation]

Original: The function d measures a distance and is commonly taken to be the squared norm of the diﬀerence of the conﬁgurations w and w (cid:48) :
Revised: The function d measures a distance and is commonly taken to be the squared norm of the diﬀerence of the conﬁgurations w and w (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: The integral is performed over all possible conﬁgurations w

Original: [Equation]
Revised: [Equation]

Original:  ; for discrete systems, it can be substituted by a sum.
Revised: The integral is performed over all possible conﬁgurations w (cid:48) ; for discrete systems, it can be substituted by a sum.

Original: The two parameters β and ˜ γ = βγ are Legendre conjugates of the loss and the distance.
Revised: The two parameters β and ˜ γ = βγ are Legendre conjugates of the loss and the distance.

Original: For large systems, N (cid:29) 1 ,their eﬀect is to jointly restrict the integral to conﬁgurations w (cid:48) below a certain loss  L ∗ ( w, β, γ ) andwithin a certain distance d ∗ ( w, β, γ ) from the reference conﬁguration w . 
Revised: For large systems, N (cid:29) 1 , the integral is dominated by conﬁgurations having a certain loss value L ∗ ( w, β, γ ) and a certain distance d ∗ ( w, β, γ ) from the reference conﬁguration w . These functional dependencies can be obtained by a saddle point approximation.

Original: In general, increasing β reduces L ∗ and increasing ˜ γ reduces d ∗ .
Revised: In general, increasing β reduces L ∗ and increasing ˜ γ reduces d ∗ .

Original: While it is convenient to use Eq.
Revised: While it is convenient to use Eq.

Original: (1) as an objective function in algorithms and for the theoretical analysis of shallow networks, it is more natural to use a normalized deﬁnition with explicit parameters when we want to measure the ﬂatness of a minimum.
Revised: (1) as an objective function in algorithms and for the theoretical analysis of shallow networks, it is more natural to use a normalized deﬁnition with explicit parameters when we want to measure the ﬂatness of a minimum.

Original: We thus also introduce the normalized local entropy Φ LE ( w, d ) , which, for a given conﬁguration w ∈ R N , measures the logarithm of the volume fraction of conﬁgurations whose training error is smaller or equal than that of the reference w in a ball of squared-radius 2 d centered in w :
Revised: We thus also introduce the normalized local entropy Φ LE ( w, d ) , which, for a given conﬁguration w ∈ R N , measures the logarithm of the volume fraction of conﬁgurations whose training error is smaller or equal than that of the reference w in a ball of squared-radius 2 d centered in w :

Original: [Equation]
Revised: [Equation]

Original: Here,  Θ ( x ) is the Heaviside step function, Θ ( x ) = 1{+
+} if x ≥ 0 and 0 otherwise.
Revised: Here, E train ( w ) is the error on the training set for a given conﬁguration w and Θ ( x ) is the Heaviside step function, Θ ( x ) = {+
+} if x ≥ 0 and 0 otherwise.

Original: This quantity is upper-bounded by zero and always zero at d = 0 .
Revised: This quantity is upper-bounded by zero and tends to zero for d → 0 (since for almost any w , except for a set with null measure, there is always a suﬃciently small neighborhood in which E train is constant).

Original: For sharp minima, it is expected to drop rapidly with d , whereas for ﬂat regions it is expected to stay close to zero within some range.
Revised: For sharp minima, it is expected to drop rapidly with d , whereas for ﬂat regions it is expected to stay close to zero within some range.

Original: A diﬀerent notion of ﬂatness is that used in Jiang et al.
Revised: A diﬀerent notion of ﬂatness is that used in Jiang et al.

Original: (2019), which we call local energy.
Revised: (2019), which we call local energy.

Original: Given a weight conﬁguration w ∈ R N , we deﬁne δE train ( w, σ ) as the average training error diﬀerence with respect to E train ( w ) when perturbing w by a (multiplicative) noise proportional to a parameter σ :
Revised: Given a weight conﬁguration w ∈ R N , we deﬁne δE train ( w, σ ) as the average training error diﬀerence with respect to E train ( w ) when perturbing w by a (multiplicative) noise proportional to a parameter σ :

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) denotes the Hadamard (element-wise) product and the expectation is over normally distributed z ∼ N (0 , I N ) .
Revised: where (cid:12) denotes the Hadamard (element-wise) product and the expectation is over normally distributed z ∼ N (0 , I N ) .

Original: In Jiang et al.
Revised: In Jiang et al.

Original: (2019), a single, arbitrarily chosen value of σ was used, whereas we compute entire proﬁles within some range [0 , σ max ] in all our tests.
Revised: (2019), a single, arbitrarily chosen value of σ was used, whereas we compute entire proﬁles within some range [0 , σ max ] in all our tests.

Original:  Entropic algorithms
Revised:  Entropic algorithms

Original: For our numerical experiments we have used two entropic algorithms, rSGD and eSGD, mentioned in the introduction.
Revised: For our numerical experiments we have used two entropic algorithms, rSGD and eSGD, mentioned in the introduction.

Original: They both approximately optimize the local entropy loss
Revised: They both approximately optimize the local entropy loss

Original: L LE as deﬁned in eq.
Revised: L LE as deﬁned in Eq.

Original: (1), for which an exact evaluation of the integral is in the general case intractable.
Revised: (1), for which an exact evaluation of the integral is in the general case intractable.

Original: The two algorithms employ diﬀerent but related approximation strategies.
Revised: The two algorithms employ diﬀerent but related approximation strategies.

Original: Entropy-SGD.
Revised: Entropy-SGD.

Original: Entropy-SGD (eSGD), introduced in Chaudhari et al.
Revised: Entropy-SGD (eSGD), introduced in Chaudhari et al.

Original: (2019), minimizes the local entropy loss  (1) by approximate evaluations of its gradient.
Revised: (2019), minimizes the local entropy loss Eq. (1) by approximate evaluations of its gradient.

Original: The gradient can be expressed as
Revised: The gradient can be expressed as

Original: [Equation]
Revised: [Equation]

Original:  where (cid:104)·(cid:105) denotes the expectation over the measure Z − 1 e − β L ( w (cid:48) )
Revised: (5) where (cid:104)·(cid:105) denotes the expectation over the measure Z − 1 e − β L ( w [Equation]

Original:  − βγ d ( w (cid:48) ,w ) , where Z is a normalization factor.
Revised: ) − βγ d ( w (cid:48) ,w ) , where Z is a normalization factor.

Original: The eSGD strategy is to approximate (cid:104) w (cid:48) (cid:105) (which implicitly depends on w ) using L steps of stochastic gradient Langevin dynamics (SGLD).
Revised: The eSGD strategy is to approximate (cid:104) w (cid:48) (cid:105) (which implicitly depends on w ) using L steps of stochastic gradient Langevin dynamics (SGLD).

Original: The resulting double-loop algorithm is presented as Algorithm 1.
Revised: The resulting double-loop algorithm is presented as Algorithm 1.

Original: The noise parameter (cid:15) in the algorithm is linked to the inverse temperature by the usual Langevin relation (cid:15) =
Revised: The noise parameter (cid:15) in the algorithm is linked to the inverse temperature by the usual Langevin relation (cid:15) =

Original: (cid:112) 2 /β .
Revised: (cid:112) 2 /β .

Original: In practice we always set it to the small value (cid:15) =
Revised: In practice we always set it to the small value (cid:15) =

Original: 10 − 4 as in Chaudhari et al.
Revised: 10 − 4 as in Chaudhari et al.

Original: For (cid:15) = 0 , eSGD approximately computes a proximal operator (Chaudhari et al., 2018).
Revised: For (cid:15) = 0 , eSGD approximately computes a proximal operator (Chaudhari et al., 2018).

Original: For (cid:15) =
Revised: For (cid:15) =

Original: α = γ = 0 , eSGD reduces to the recently introduced Lookahead optimizer (Zhang et al., 2019).
Revised: α = γ = 0 , eSGD reduces to the recently introduced Lookahead optimizer (Zhang et al., 2019).

Original: Replicated-SGD.
Revised: Replicated-SGD.

Original: Replicated-SGD (rSGD) consists in a replicated version of the usual stochastic gradient (SGD) method.
Revised: Replicated-SGD (rSGD) consists in a replicated version of the usual stochastic gradient (SGD) method.

Original: In rSGD, a number y of replicas of the same system, each with its own parameters w a where a = 1 , ..., y , are trained in parallel for K iterations.
Revised: In rSGD, a number y of replicas of the same system, each with its own parameters w a where a = 1 , ..., y , are trained in parallel for K iterations.

Original: During training, they interact with each other indirectly through an attractive term towardstheir center of mass.
Revised: During training, they interact with each other indirectly through an attractive term towards

Original: As detailed in Baldassi et al.

Original: (2016a; 2020) in the simple case of shallow networks (committee machines), the replicated system, when trained with a stochastic algorithm such as SGD, collectively explores an approximation of the local entropy landscape without the need to explicitly estimate the integral in eq.

Original: In principle, the larger y the better the approximation, but already with y = 3 the eﬀect of the replication is signiﬁcant.

Original: To summarize, rSGD replaces the local entropy (1) with the replicated loss L R :

Original: [Equation]

Original: Here, ¯ w is a center replica deﬁned as ¯ w = 1 y (cid:80) ya

Original: [-=
-]
Revised: [-=
-]

Original: w a .

Original: The algorithm is presented as

Original: Algorithm 2.

Original: Thanks to focusing (see below), any of the replicas or the center ¯ w can be used after training for prediction.

Original: This procedure is parallelizable over the replicas, so that

Original: Input : w
Revised: Input : w

Original: Hyper-parameters : L, η , γ , η (cid:48) , (cid:15) , α 1 for t = 1 , 2 , . . .
Revised: Hyper-parameters : L, η , γ , η (cid:48) , (cid:15) , α 1 for t = 1 , 2 , . . .

Original: do w (cid:48) , µ ← w for l = 1 , . . .
Revised: do w (cid:48) , µ ← w for l = 1 , . . .

Original: , L do Ξ ← sample minibatch dw (cid:48) ← ∇L ( w (cid:48) ; Ξ) + γ ( w (cid:48) − w ) w (cid:48) ← w (cid:48) − η (cid:48) dw (cid:48)
Revised: , L do Ξ ← sample minibatch dw (cid:48) ← ∇L ( w (cid:48) ; Ξ) + γ ( w (cid:48) − w ) w (cid:48) ← w (cid:48) − η (cid:48) dw (cid:48)

Original: + √ η (cid:48) (cid:15) N
Revised: + √ η (cid:48) (cid:15) N

Original: (0 , I ) µ ← αµ
Revised: (0 , I ) µ ← αµ

Original: + (1 − α ) w (cid:48)
Revised: + (1 − α ) w (cid:48)

Original: w ← w − η ( w − µ )
Revised: w ← w − η ( w − µ )

Original: Input : { w a }
Revised: Input : { w a }

Original: Hyper-parameters : y , η , γ , K 1 for t = 1 , 2 , . . .
Revised: Hyper-parameters : y , η , γ , K 1 for t = 1 , 2 , . . .

Original: do ¯ w ← 1 y (cid:80) ya =
Revised: do ¯ w ← 1 y (cid:80) ya =

Original: 1 w a for a = 1 , . . .
Revised: 1 w a for a = 1 , . . .

Original: , y do Ξ ← sample minibatch dw a ← ∇L ( w a ; Ξ) if t = 

Revised: , y do Ξ ← sample minibatch dw a ← ∇L ( w a ; Ξ) if t =

Original: mod K then dw a ← dw a + Kγ ( w a − ¯ w ) w a ← w a − η dw a wall-clock time for training is comparable to SGD, excluding the communication which happens every K parallel optimization steps.
Revised: mod K then dw a ← dw a + Kγ ( w a − ¯ w ) w a ← w a − η dw a their center of mass. As detailed in Baldassi et al. (2016a; 2020) in the simple case of shallow networks (committee machines), the replicated system, when trained with a stochastic algorithm such as SGD, collectively explores an approximation of the local entropy landscape without the need to explicitly estimate the integral in Eq. In principle, the larger y the better the approximation, but already with y = 3 the eﬀect of the replication is signiﬁcant.

Revised: To summarize, rSGD replaces the local entropy Eq.

Revised: (1) with the replicated loss L R :

Revised: [Equation]

Revised: Here, ¯ w is a center replica deﬁned as ¯ w = 1 y (cid:80) ya

Original: {+=
+}
Revised: {+=
+}

Revised: w a .

Revised: The algorithm is presented as

Revised: Algorithm 2.

Revised: Thanks to focusing (see below), any of the replicas or the center ¯ w can be used after training for prediction.

Revised: This procedure is parallelizable over the replicas, so that wall-clock time for training is comparable to SGD, excluding the communication which happens every K parallel optimization steps.

Original: In order to decouple the communication period and the coupling hyperparameter γ , we let the coupling strength take the value Kγ .
Revised: In order to decouple the communication period and the coupling hyperparameter γ , we let the coupling strength take the value Kγ .

Original: In our experiments, we did not observe degradation in generalization performance with K up to 10 .
Revised: In our experiments, we did not observe degradation in generalization performance with K up to 10 .

Original: Focusing.
Revised: Focusing.

Original: A common feature of both algorithms is that the parameter γ in the objective
Revised: A common feature of both algorithms is that the parameter γ in the objective

Original: L LE changes during the optimization process.
Revised: L LE changes during the optimization process.

Original: We start with a small γ (targeting large regions and allowing a wider exploration of the landscape) and gradually increase it.
Revised: We start with a small γ (targeting large regions and allowing a wider exploration of the landscape) and gradually increase it.

Original: We call this process focusing .
Revised: We call this process focusing .

Original: Focusing improves the dynamics by driving the system quickly to wide regions and then, once there, gradually trading oﬀ the width in order to get to the minima of the loss within those regions, see Baldassi et al.
Revised: Focusing improves the dynamics by driving the system quickly to wide regions and then, once there, gradually trading oﬀ the width in order to get to the minima of the loss within those regions, see Baldassi et al.

Original: (2016b;a).
Revised: (2016b;a).

Original: We adopt an exponential schedule for γ , where its value at epoch τ is given by γ τ = γ 

Revised: We adopt an exponential schedule for γ , where its value at epoch τ is given by γ τ = γ

Original: (1 + γ 1 ) τ .
Revised: (1 + γ 1 ) τ .

Original: For rSGD, we ﬁx γ 0 by balancing the distance and the data term in the objective before training starts, i.e. we set γ 

Revised: For rSGD, we ﬁx γ 0 by balancing the distance and the data term in the objective before training starts, i.e. we set γ

Original: = (cid:80) a L ( w a ) / (cid:80) a d( w a , ¯ w ) for rSGD.
Revised: = (cid:80) a L ( w a ) / (cid:80) a d( w a , ¯ w ) for rSGD.

Original: The parameter γ 1 is chosen such that γ increases by a factor 10 4 .
Revised: The parameter γ 1 is chosen such that γ increases by a factor 10 4 .

Original: For eSGD, we were unable to ﬁnd a criterion that worked for all experiments and manually tuned it.
Revised: For eSGD, we were unable to ﬁnd a criterion that worked for all experiments and manually tuned it.

Original: Optimizers.
Revised: Optimizers.

Original: Vanilla SGD updates in Algorithms 1 and 2 can be replaced by optimization steps of any commonly used gradient-based optimizers.
Revised: Vanilla SGD updates in Algorithms 1 and 2 can be replaced by optimization steps of any commonly used gradient-based optimizers.

Original:  Detailed comparison of flatness measures in shallow networks
Revised:  Detailed comparison of flatness measures in shallow networks

Original: In this section, we explore in detail the connection between the two ﬂatness measures and the generalization properties in a one-hidden-layer network that performs a binary classiﬁcation task, also called a committee machine.
Revised: In this section, we explore in detail the connection between the two ﬂatness measures and the generalization properties in a one-hidden-layer network that performs a binary classiﬁcation task, also called a committee machine.

Original: This model has a symmetry that allows to ﬁx all the weights in the last layer to 1 , and thus only the ﬁrst layer is trained.
Revised: This model has a symmetry that allows to ﬁx all the weights in the last layer to 1 , and thus only the ﬁrst layer is trained.

Original: It is also invariant to rescaling of the weights.
Revised: It is also invariant to rescaling of the weights.

Original: This allows to study its typical properties analytically with statistical mechanics techniques, and it was shown in Baldassi et al.
Revised: This allows to study its typical properties analytically with statistical mechanics techniques, and it was shown in Baldassi et al.

Original: (2020) that it has a rich non-convex error-loss landscape, in which rare ﬂat minima coexist with narrower ones.
Revised: (2020) that it has a rich non-convex error-loss landscape, in which rare ﬂat minima coexist with narrower ones.

Original: It is amenable to be studied semi-analytically: for individual instances, the minimizers found by diﬀerent algorithms can be compared by computing their local entropy eﬃciently with the
Revised: It is amenable to be studied semi-analytically: for individual instances, the minimizers found by diﬀerent algorithms can be compared by computing their local entropy eﬃciently with the

Original: Belief Propagation  algorithm Mezard & Montanari (2009), bypassing the need to perform the integral in eq.
Revised: Belief Propagation (BP) algorithm (see Appendix B.1), bypassing the need to perform the integral in Eq.

Original: (1) explicitly (doing the same for general architectures is an open problem).
Revised: (1) explicitly. Doing the same for general architectures is an open problem.

Original: For a network with K hidden units, the output predicted for a given input pattern x reads:
Revised: For a network with K hidden units, the output predicted for a given input pattern x reads:

Original: [Equation]
Revised: [Equation]

Original: We follow the numerical setting of Baldassi et al.
Revised: We follow the numerical setting of Baldassi et al.

Original: and train this network to perform binary classiﬁcation on two classes of the Fashion-MNIST dataset with binarized patterns, comparing the results of standard SGD with cross-entropy loss (CE) with the entropic[Figure] counterparts rSGD and eSGD.
Revised: and train this network to perform binary classiﬁcation on two classes of the Fashion-MNIST dataset with binarized patterns, comparing the results of standard SGD with cross-entropy loss (CE) with the entropic counterparts rSGD and eSGD.

Original: All these algorithms require a diﬀerentiable objective, thus we approximate sign activation functions on the hidden layer with tanh( βx ) functions, where the β parameter increases during the training.
Revised: All these algorithms require a diﬀerentiable objective, thus we approximate sign activation functions on the hidden layer with tanh( βx ) functions, where the β parameter increases during the training.

Original: The CE loss is not invariant with respect to weight rescaling: we control the norm of the weights explicitly by keeping them normalized and introducing an overall scale parameter ω that we insert explicitly in the loss:
Revised: The CE loss is not invariant with respect to weight rescaling: we control the norm of the weights explicitly by keeping them normalized and introducing an overall scale parameter ω that we insert explicitly in the loss:

Original: [Equation]
Revised: [Equation]

Original: Here, we have deﬁned f ( x, ω ) =
Revised: Here, we have deﬁned f ( x, ω ) =

Original: − x 2 + 12 ω log(2 cosh( ωx )) as in Baldassi et al.
Revised: − x 2 + 12 ω log(2 cosh( ωx )) as in Baldassi et al.

Original: The ω parameter is increased gradually in the training process in order to control the growth rate of the weight norms.
Revised: The ω parameter is increased gradually in the training process in order to control the growth rate of the weight norms.

Original: Notice that the parameter β could also be interpreted as a norm that grows over time.
Revised: Notice that the parameter β could also be interpreted as a norm that grows over time.

Original: As shown in Baldassi et al.
Revised: As shown in Baldassi et al.

Original: (2020), slowing down the norm growth rate results in better generalization performance and increased ﬂatness of the minima found at the end of the training.
Revised: (2020), slowing down the norm growth rate results in better generalization performance and increased ﬂatness of the minima found at the end of the training.

Original: To appreciate this eﬀect we used two diﬀerent parameters settings for optimizing the loss in eq.(8) with SGD, that we name “SGD slow” and “SGD fast”.
Revised: To appreciate this eﬀect we used two diﬀerent parameters settings for optimizing the loss in Eq.(8) with SGD, that we name “SGD slow” and “SGD fast”.

Original: In the fast setting both β and ω start with a large value and grow quickly, while in the slow setting they start from small values and grow more slowly, requiring more epochs to converge.
Revised: In the fast setting both β and ω start with a large value and grow quickly, while in the slow setting they start from small values and grow more slowly, requiring more epochs to converge.

Original: For rSGD, we also used two diﬀerent “fast” and “slow” settings, where the diﬀerence is in a faster or slower increase of the γ parameter that controls the distance between replicas.
Revised: For rSGD, we also used two diﬀerent “fast” and “slow” settings, where the diﬀerence is in a faster or slower increase of the γ parameter that controls the distance between replicas.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: In the left panel, we report Φ LE computed with BP around the solutions found by the diﬀerent algorithms, as a function of the distance from the solution.
Revised: In the left panel, we report Φ LE computed with BP around the solutions found by the diﬀerent algorithms, as a function of the distance from the solution.

Original: Even if the slow SGD setting improves the ﬂatness of the solution found, entropy-driven algorithms are biased towards ﬂatter minima, in the sense of the local entropy, as expected.
Revised: Even if the slow SGD setting improves the ﬂatness of the solution found, entropy-driven algorithms are biased towards ﬂatter minima, in the sense of the local entropy, as expected.

Original: In the central panel we plot the local energy proﬁles δE train for the same solutions, and we can see that the ranking of the algorithm is preserved: the two ﬂatness measures agree.
Revised: In the central panel we plot the local energy proﬁles δE train for the same solutions, and we can see that the ranking of the algorithm is preserved: the two ﬂatness measures agree.

Original: The same ranking is also clearly visible when comparing the generalization errors, in the right panel of the ﬁgure: ﬂatter minima generalize better 2 .
Revised: The same ranking is also clearly visible when comparing the generalization errors, in the right panel of the ﬁgure: ﬂatter minima generalize better 2 .

Original:  Numerical experiments on deep networks 6. Comparisons across several architectures and datasets
Revised:  Numerical experiments on deep networks 6. Comparisons across several architectures and datasets

Original: In this section we show that, by optimizing the local entropy with eSGD and rSGD, we are able to systematically improve the generalization performance compared to standard SGD.
Revised: In this section we show that, by optimizing the local entropy with eSGD and rSGD, we are able to systematically improve the generalization performance compared to standard SGD.

Original: We perform experiments on image classiﬁcation tasks, using common benchmark datasets, state-of-the-art deep architectures and the usual cross-entropy loss.
Revised: We perform experiments on image classiﬁcation tasks, using common benchmark datasets, state-of-the-art deep architectures and the usual cross-entropy loss.

Original: The detailed settings of the experiments are reported in the SM.
Revised: The detailed settings of the experiments are reported in the SM.

Original: For the experiments with eSGD and rSGD, we use the same settings and hyper-parameters (architecture, dropout, learning rate schedule,...) as for the baseline, unless otherwise stated in the SM and apart from the hyper-parameters speciﬁc to these algorithms. 
Revised: For the experiments with eSGD and rSGD, we use the same settings and hyper-parameters (architecture, dropout, learning rate schedule,...) as for the baseline, unless otherwise stated in the SM and apart from the hyper-parameters speciﬁc to these algorithms. While it would be interesting to add weight normalization (Salimans & Kingma (2016)) with frozen norm, [Figure] the baselines that we compare against uses this method.

Revised: We also note that for the local energy as deﬁned in Eq. 4, the noise is multiplicative and local energy is norm-invariant if the model itself is norm-invariant.

Original: While we do some little hyper-parameter exploration to obtain a reasonable baseline
Revised: While we do some little hyper-parameter exploration to obtain a reasonable baseline

Original: [Table]
Revised: [Figure]

Original: we do not aim to reproduce the best achievable results with these networks, since we are only interested in comparing diﬀerent algorithms in similar contexts.
Revised: we do not aim to reproduce the best achievable results with these networks, since we are only interested in comparing diﬀerent algorithms in similar contexts.

Original: For instance, we train
Revised: For instance, we train

Original: PyramidNet+ShakeDrop for 300 epochs [Table] instead of the 1800 epochs used in Cubuk et al.
Revised: PyramidNet+ShakeDrop for 300 epochs, instead of the 1800 epochs used in Cubuk et al.

Original: [Table]
Revised: [Figure]

Original: and we start from random initial conditions for EﬃcientNet instead of doing transfer learning as done in Tan & Le (2019).
Revised: and we start from random initial conditions for EﬃcientNet instead of doing transfer learning as done in Tan & Le (2019).

Original: In the case of the ResNet110 architecture instead, we use the training speciﬁcation of the original paper (He et al., 2016)
Revised: In the case of the ResNet110 architecture instead, we use the training speciﬁcation of the original paper (He et al., 2016)

Original: [Table]
Revised: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] All combinations of datasets and architectures we tested are reported in Table 1.
Revised: All combinations of datasets and architectures we tested are reported in Table 1, while representative test error curves are reported in Fig.

Original: Blanks correspond to untested combinations.
Revised: Blanks correspond to untested combinations.

Original: The ﬁrst 3 columns correspond to experiments with the same number of eﬀective epochs, that is considering that in each iteration of the outer loop in Algorithms 1 and 2 we sample L and y mini-batches respectively.
Revised: The ﬁrst 3 columns correspond to experiments with the same number of eﬀective epochs, that is considering that in each iteration of the outer loop in Algorithms 1 and 2 we sample L and y mini-batches respectively.

Original: In the last column instead, each replica consumes individually the same amount of data as the baseline.
Revised: In the last column instead, each replica consumes individually the same amount of data as the baseline.

Original: Being a distributable algorithm, rSGD enjoys the same scalability as the related EASGD and Parle (Zhang et al., 2014; Chaudhari et al., 2017).
Revised: Being a distributable algorithm, rSGD enjoys the same scalability as the related EASGD and Parle (Zhang et al., 2014; Chaudhari et al., 2017).

Original: For rSGD, we use y = 3 replicas and the scoping schedules described in Sec. 4.
Revised: For rSGD, we use y = 3 replicas and the scoping schedules described in Sec. 4.

Original: In our explorations, rSGD proved to be robust with respect to speciﬁc choices of the hyper- parameters.
Revised: In our explorations, rSGD proved to be robust with respect to speciﬁc choices of the hyperparameters.

Original: The error reported is that of the center replica ¯ w .
Revised: The error reported is that of the center replica ¯ w .

Revised: We note here, however, that the distance between the replicas at the end of training is very small and they eﬀectively

Revised: [Figure]

Revised: [Figure]

Revised: collapse to a single solution.

Revised: Since training continues after the collapse and we reach a stationary value of the loss, we are conﬁdent that the minimum found by the replicated system corresponds to a minimum of a single system.

Original: For eSGD, we set L = 5 , (cid:15) = 1 e − 4 and α = 0 .
Revised: For eSGD, we set L = 5 , (cid:15) = 1 e − 4 and α = 0 .

Original: 75 in all experiments, and we perform little tuning for the the other hyper-parameters.
Revised: 75 in all experiments, and we perform little tuning for the the other hyper-parameters.

Original: The algorithm is more sensitive to hyper-parameters than rSGD, while still being quite robust.
Revised: The algorithm is more sensitive to hyper-parameters than rSGD, while still being quite robust.

Original: Moreover, it misses an automatic γ scoping schedule.
Revised: Moreover, it misses an automatic γ scoping schedule.

Original: Results in Table 1 show that entropic algorithms generally outperform the corresponding baseline with roughly the same amount of parameter tuning and computational resources.
Revised: Results in Table 1 show that entropic algorithms generally outperform the corresponding baseline with roughly the same amount of parameter tuning and computational resources.

Original: In the next section we also show that they end up in ﬂatter minima.
Revised: In the next section we also show that they end up in ﬂatter minima.

Original: 6. Flatness vs generalization
Revised: 6. Flatness vs generalization

Original: For the deep network tests, we measured the local energy proﬁles (see eq.
Revised: For the deep network tests, we measured the local energy proﬁles (see Eq.

Original: (4)) of the conﬁgurations explored by the three algorithms.
Revised: (4)) of the conﬁgurations explored by the three algorithms.

Original: The estimates of the expectations were computed by averaging over 1000 perturbations for each value of σ .
Revised: The estimates of the expectations were computed by averaging over 1000 perturbations for each value of σ .

Original: We did not limit ourselves to the end result, but rather we traced the evolution throughout the training and stopped when the training error and loss reached stationary values.
Revised: We did not limit ourselves to the end result, but rather we traced the evolution throughout the training and stopped when the training error and loss reached stationary values.

Original: In our experiments, the ﬁnal training error is close to 0.
Revised: In our experiments, the ﬁnal training error is close to 0.

Original: Representative results are shown in ﬁg. 3, which shows that the eSGD and rSGD curves are below the SGD curve across a wide range of σ values, while also achieving better generalization.
Revised: Representative results are shown in Fig. 3, which shows that the eSGD and rSGD curves are below the SGD curve across a wide range of σ values, while also achieving better generalization.

Original: Similar results are found for diﬀerent architectures, as reported in the appendix.
Revised: Similar results are found for diﬀerent architectures, as reported in Appendix B.3.

Original: This conﬁrms the results of the shallow networks experiments: entropic algorithms tend to ﬁnd ﬂatter minima that generalize better, even when the hyper-parameters of the standard SGD algorithms had already been tuned for optimal generalization (and thus presumably to end up in generally ﬂatter regions).[Figure]
Revised: This conﬁrms the results of the shallow networks experiments: entropic algorithms tend to ﬁnd ﬂatter minima that generalize better, even when the hyper-parameters of the standard SGD algorithms had already been tuned for optimal generalization (and thus presumably to end up in generally ﬂatter regions).

Original:  Discussion and conclusions
Revised:  Discussion and conclusions

Original: We studied the connection between two notions of ﬂatness and generalization.
Revised: We studied the connection between two notions of ﬂatness and generalization.

Original: We have performed detailed studies on shallow networks and an extensive numerical study on state of the art deep architectures.
Revised: We have performed detailed studies on shallow networks and an extensive numerical study on state of the art deep architectures.

Original: Our results suggest that local entropy is a good predictor of generalization performance.
Revised: Our results suggest that local entropy is a good predictor of generalization performance.

Original: This is consistent with its relation to another ﬂatness measure, the local energy, for which this property has already been established empirically.
Revised: This is consistent with its relation to another ﬂatness measure, the local energy, for which this property has already been established empirically.

Original: Furthermore, entropic algorithms can exploit this fact and be eﬀective in improving the generalization performance on existing architectures, at ﬁxed computational cost and with little hyper-parameter tuning.
Revised: Furthermore, entropic algorithms can exploit this fact and be eﬀective in improving the generalization performance on existing architectures, at ﬁxed computational cost and with little hyper-parameter tuning.

Original: Our future eﬀorts will be devoted to studying the connection between generalization bounds and the existence of wide ﬂat regions in the landscape of the classiﬁer.
Revised: Our future eﬀorts will be devoted to studying the connection between generalization bounds and the existence of wide ﬂat regions in the landscape of the classiﬁer.

Original: A Local Entropy and Replicated Systems
Revised: A Local Entropy and Replicated Systems

Original: The analytical framework of Local Entropy was introduced in Ref.
Revised: The analytical framework of Local Entropy was introduced in Ref.

Original: Baldassi et al.
Revised: Baldassi et al.

Original: (2015), while the connection between Local Entropy and systems of real replicas (as opposed to the "fake" replicas of spin glass theory (Mézard et al., 1987)) was made in Baldassi et al.
Revised: (2015), while the connection between Local Entropy and systems of real replicas (as opposed to the "fake" replicas of spin glass theory (Mézard et al., 1987)) was made in Baldassi et al.

Original: (2016a).
Revised: (2016a).

Original: For convenience, we brieﬂy recap here the simple derivation.
Revised: For convenience, we brieﬂy recap here the simple derivation.

Original: We start from the deﬁnition of the local entropy loss given in the main text:
Revised: We start from the deﬁnition of the local entropy loss given in the main text:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We then consider the Boltzmann distribution of a system with energy function β L LE ( w ) and with an inverse temperature y , that is
Revised: We then consider the Boltzmann distribution of a system with energy function β L LE ( w ) and with an inverse temperature y , that is

Original: [Equation]
Revised: [Equation]

Original: where equivalence is up to a normalization factor.
Revised: where equivalence is up to a normalization factor.

Original: If we restrict y to integer values, we can then use the deﬁnition of L LE to construct an equivalent but enlarged system, containing y +  replicas.
Revised: If we restrict y to integer values, we can then use the deﬁnition of L LE to construct an equivalent but enlarged system, containing y + 1 replicas.

Original: Their joint distribution p ( w, { w a } a ) is readily obtained by plugging Eq.
Revised: Their joint distribution p ( w, { w a } a ) is readily obtained by plugging Eq.

Original: (9) into Eq.
Revised: (9) into Eq.

Original: We can then integrate out the original conﬁguration w and obtain the marginal distributional for the y remaining replicas
Revised: We can then integrate out the original conﬁguration w and obtain the marginal distributional for the y remaining replicas

Original: [Equation]
Revised: [Equation]

Original: where the energy function is now given by
Revised: where the energy function is now given by

Original: [Equation]
Revised: [Equation]

Original: with ¯ w = 1 y (cid:80) a w a .
Revised: with ¯ w = 1 y (cid:80) a w a .

Original: We have thus recovered the loss function for the replicated SGD (rSGD) algorithm presented in the main text.
Revised: We have thus recovered the loss function for the replicated SGD (rSGD) algorithm presented in the main text.

Original: B Flatness and local entropy B. Local entropy on the committee machine
Revised: B Flatness and local entropy B. Local entropy on the committee machine

Original: In what follows, we describe the details of the numerical experiments on the committee machine.
Revised: In what follows, we describe the details of the numerical experiments on the committee machine.

Revised: We apply diﬀerent algorithms to ﬁnd zero error conﬁgurations and then use Belief

Revised: Propagation (BP) to compute the local entropy curve for each conﬁguration obtained.

Revised: We compare them in Fig.

Revised: 1, along with the local energy computed by sampling and their test errors.

Original: We deﬁne a reduced version of the Fashion-MNIST dataset following Baldassi et al.
Revised: We deﬁne a reduced version of the Fashion-MNIST dataset following Baldassi et al.

Original: (2020): we choose the classes Dress and Coat as they are non-trivial to discriminate but also diﬀerent enough so that a small network as the one we used can generalize.
Revised: (2020): we choose the classes Dress and Coat as they are non-trivial to discriminate but also diﬀerent enough so that a small network as the one we used can generalize.

Original: The network is trained on a small subset of the available examples (500 patterns) binarized to ± 1 by using the median of each image as a threshold on the inputs; we also ﬁlter both the training and test sets to use only images in which the median is between 0.25 and 0.75.
Revised: The network is trained on a small subset of the available examples (500 patterns) binarized to ± 1 by using the median of each image as a threshold on the inputs; we also ﬁlter both the training and test sets to use only images in which the median is between 0.25 and 0.75.

Original: The network has input size N = 784 and a single hidden layer with K = 9 hidden units.
Revised: The network has input size N = 784 and a single hidden layer with K = 9 hidden units.

Original: The weights between the hidden layer and the output are ﬁxed to 1 .
Revised: The weights between the hidden layer and the output are ﬁxed to 1 .

Original: It is trained using minibatches of 100 patterns.
Revised: It is trained using mini-batches of 100 patterns.

Original: All the results are averaged over 50 independent restarts.
Revised: All the results are averaged over 50 independent restarts.

Original: For all algorithms we initialize the weights with a uniform distribution and then normalize the weights of the hidden units norm before the training starts and after each weight update.
Revised: For all algorithms we initialize the weights with a uniform distribution and then normalize the weights of the hidden units norm before the training starts and after each weight update.

Original: The β and ω parameters are updated using exponential schedules, β ( t ) =
Revised: The β and ω parameters are updated using exponential schedules, β ( t ) =

Original: β 0 (1 + β 1 ) t and ω ( t )
Revised: β 0 (1 + β 1 ) t and ω ( t )

Original: =
Revised: =

Original: ω 

Revised: ω

Original: (1 + ω 1 ) t , where t is the current epoch.
Revised: (1 + ω 1 ) t , where t is the current epoch.

Original: An analogous exponential schedule is used for the elastic interaction γ for rSGD and eSGD, as described in the main text.
Revised: An analogous exponential schedule is used for the elastic interaction γ for rSGD and eSGD, as described in the main text.

Original: In the SGD fast case, we stop as soon as a solution with zero errors is found, while for SGD slow we stop when the cross entropy loss reaches a value lower than 10 − 7 .
Revised: In the SGD fast case, we stop as soon as a solution with zero errors is found, while for SGD slow we stop when the cross entropy loss reaches a value lower than 10 − 7 .

Original: For rSGD, we stop training as soon as the distance between the replicas and their center of mass is smaller than 10 − 8 .
Revised: For rSGD, we stop training as soon as the distance between the replicas and their center of mass is smaller than 10 − 8 .

Original: For eSGD, we stop training as soon as the distance between the parameters and the mean ( µ in
Revised: For eSGD, we stop training as soon as the distance between the parameters and the mean ( µ in

Original: Algorithm 1) is smaller than 10 − 8 .
Revised: Algorithm 1) is smaller than 10 − 8 .

Original: We used the following hyper-parameters for the various algorithms:
Revised: We used the following hyper-parameters for the various algorithms:

Original: SGD {+fast : η = 2 · 10 − 4 , β 0 = 2 . 0 , β 1 = 10 − 4 , ω 0 = 5 . 0 , ω 1 = 0 . 0 ; SGD slow : η = 3 · 10 − 5 , β 0 = 0 . 5 , β 1 = 10 − 3 , ω 
+}
Revised: SGD {+fast : η = 2 · 10 − 4 , β 0 = 2 . 0 , β 1 = 10 − 4 , ω 0 = 5 . 0 , ω 1 = 0 . 0 ; SGD slow : η = 3 · 10 − 5 , β 0 = 0 . 5 , β 1 = 10 − 3 , ω 
+}

Revised: = 0 .

Original: {+5 , ω 1 = 10 − 3 ; rSGD fast : η = 10 − 4 , y = 10 , γ 
+}
Revised: {+5 , ω 1 = 10 − 3 ; rSGD fast : η = 10 − 4 , y = 10 , γ 
+}

Original: [Equation]{+= 2 · 10 − 3 , γ 1 = 2 · 10 − 3 , β 0 = 1 . 0 , β 1 = 2 · 10 − 4 , ω 
 = 0 . 5 , ω 1 = 10 − 3 ; rSGD slow : η = 10 − 3 , y = 10 , γ 
 =+}
Revised: {+= 2 · 10 − 3 , γ 1 = 2 · 10 − 3 , β 0 = 1 . 0 , β 1 = 2 · 10 − 4 , ω 
 = 0 . 5 , ω 1 = 10 − 3 ; rSGD slow : η = 10 − 3 , y = 10 , γ 
 =+}

Original: {+10 − 4 , γ 1 = 10 − 4 , β 0 = 1 . 0 , β 1 = 2 · 10 − 4 , ω 
+}
Revised: {+10 − 4 , γ 1 = 10 − 4 , β 0 = 1 . 0 , β 1 = 2 · 10 − 4 , ω 
+}

Original:  η 
Revised: = 0 . 5 , ω 1 = 10 − 3 ; eSGD : η = 10 − 3 , η (cid:48)

Original: [Equation]
Revised: = 5 · 10 − 3 , (cid:15) = 10 − 6 , L = 20 , γ 0 = 10 . 0 , γ 1 = 5 · 10 − 5 , β 0 = 1 . 0 , β 1 = 10 − 4 , ω 0 = 0 . 5 , ω 1 = 5 · 10 − 4 ;

Revised: For a given conﬁguration w obtained by one of the previous algorithms, we compute the local entropy through BP.

Revised: The message passing involved is similar to the one accurately detailed in Appendix IV of Ref.

Revised: Baldassi et al.

Original: [Equation]
Revised: (2020), the only diﬀerence here being that we apply it to a fully-connected committee machine instead of the tree-like one used in Ref. Baldassi et al. We thus give here just a brief overview of the procedure.

Revised: We frame the supervised learning problem as a constraint satisfaction problem given by the network architecture and the training examples.

Original: [Table]
Revised: In order to compute the local entropy, we add a Gaussian prior centered in w to the weights, N ( w, ∆) . The variance ∆ of the prior will be related later to a speciﬁc distance from w . We run BP iterations until convergence, then compute the Bethe free-energy f Bethe ( w, ∆) as a function of the ﬁxed point messages. Applying the procedure for diﬀerent values of ∆ and then performing a Legendre transform, we ﬁnally obtain the local entropy curve Φ LE ( w, d ) .

Original: B. Flatness curves for deep networks
Revised: B. Flatness curves for deep networks

Original: In this Section we present ﬂatness curves, δE train ( w, σ ) from Eq.
Revised: In this Section we present ﬂatness curves, δE train ( w, σ ) from Eq.

Original: (4), for some of the deep networks architecture examined in this paper.
Revised: (4), for some of the deep networks architecture examined in this paper.

Original: Results are reported in Figures 4 and 5 for diﬀerent architectures and datasets.
Revised: Results are reported in Figs 4 and 5 for diﬀerent architectures and datasets.

Original: The expectation in Eq.
Revised: The expectation in Eq.

Original: (4) is computed over the complete training set using 100 andrealizations of the Gaussian noise for each data point in Figures 4 and 5 respectively.
Revised: (4) is computed over the complete training set using 100 and 400 realizations of the Gaussian noise for each data point in Figs 4 and 5 respectively.

Original: In experiments where data augmentation was used during training, it is also used when computing the ﬂatness curve.
Revised: In experiments where data augmentation was used during training, it is also used when computing the ﬂatness curve

Original: [Figure]
Revised: [Figure]

Original: t r a i n e rr o r d i ff e r en c e
Revised: DenseNet­

Original: [Figure]
Revised: [Figure]

Original: The comparison is performed between minima found by diﬀerent algorithms, at a point where the training error is near zero and the loss has reached a stationary value.
Revised: The comparison is performed between minima found by diﬀerent algorithms, at a point where the training error is near zero and the loss has reached a stationary value.

Original: We note here that this kind of comparison is sensitive to the fact that the training errors at σ = 0 are close to each other.
Revised: We note here that this kind of comparison is sensitive to the fact that the training errors at σ = 0 are close to each other.

Original: If the comparison is made for minima that show diﬀerent train errors, the correlation between ﬂatness and test error is not observed.
Revised: If the comparison is made for minima that show diﬀerent train errors, the correlation between ﬂatness and test error is not observed.

Original: [Figure]

Original: We report a generally good agreement between the ﬂatness of the δE train curve and the generalization performance, for a large range of σ values
Revised: We report a generally good agreement between the ﬂatness of the δE train curve and the generalization performance, for a large range of σ values.

Original: B. Correlation of flatness and generalization
Revised: B. Correlation of flatness and generalization

Original: In this section we test more thoroughly the correlation of our deﬁnition of ﬂatness with the generalization error.
Revised: In this section we test more thoroughly the correlation of our deﬁnition of ﬂatness with the generalization error.

Original: The entropic algorithms that we tested resulted in both increased ﬂatness and increased generalization accuracy, but this does not imply that these quantities are correlated in other settings.
Revised: The entropic algorithms that we tested resulted in both increased ﬂatness and increased generalization accuracy, but this does not imply that these quantities are correlated in other settings.

Original: For the committee machine we tested in diﬀerent settings that the local entropy provides the same information as the local energy, and that they both correlate well with the generalization  error, even when entropic algorithms are not used.
Revised: For the committee machine we tested in diﬀerent settings that the local entropy provides the same information as the local energy, and that they both correlate well with the generalization [Figure] error, even when entropic algorithms are not used.

Original: An example can be seen in ﬁg. 6, where the same architecture has been trained with SGD fast (see B.1) and diﬀerent values of the dropout probability.
Revised: An example can be seen in Fig. [Figure] where the same architecture has been trained with SGD fast (see B.1) and diﬀerent values of the dropout probability.

Original: The minima obtained with higher dropout have larger local entropy and generalize better.
Revised: The minima obtained with higher dropout have larger local entropy and generalize better

Original: [Figure]
Revised: [Figure]

Original: In order to test this correlation independently in deep networks , we use models provided in the framework of the PGDL competition (see https://sites.google.com/view/pgdl2020/ ).
Revised: In order to test this correlation independently in deep networks [Figure] we use models provided in the framework of the PGDL competition (see https://sites.google.com/view/pgdl2020/ ).

Original: These models have diﬀerent generalization errors and have been obtained without using local entropy in their training objective.
Revised: These models have diﬀerent generalization errors and have been obtained without using local entropy in their training objective.

Original: We notice that the local entropy and the local energy are only comparable in the context of the same architecture, as the loss landscape may be very diﬀerent if the network architecture varies.
Revised: We notice that the local entropy and the local energy are only comparable in the context of the same architecture, as the loss landscape may be very diﬀerent if the network architecture varies.

Original: We therefore choose among the available models a subset with the same architecture (VGG-like with 3 convolutional blocks of width 512 and 1 fully connected layer of width 128) trained on CIFAR-10.
Revised: We therefore choose among the available models a subset with the same architecture (VGG-like with 3 convolutional blocks of width 512 and 1 fully connected layer of width 128) trained on CIFAR-10.

Original: Since the models were trained with diﬀerent hyperparameters (dropout, batch size and weight decay), they still show a wide range of generalization errors.
Revised: Since the models were trained with diﬀerent hyperparameters (dropout, batch size and weight decay), they still show a wide range of generalization errors.

Original: Since we cannot compute the local entropy for these deep networks, we restrict the analysis to the computationally cheaper local energy (see eq. 4) as a measure of ﬂatness. 
Revised: Since we cannot compute the local entropy for these deep networks, we restrict the analysis to the computationally cheaper local energy (see Eq. 4) as a measure of ﬂatness. As shown in Fig.

Original: We report a good correlation between ﬂatness as measured by the train error diﬀerence (or local energy, at a given value of the perturbation intensity σ , namely σ = 0 . 5 ) and test error, resulting in a Pearson correlation coeﬃcient r (12) = 0 .
Revised: 7 we report a good correlation between ﬂatness as measured by the train error diﬀerence (or local energy, at a given value of the perturbation intensity σ , namely σ = 0 . 5 ) and test error, resulting in a Pearson correlation coeﬃcient r (12) = 0 .

Original: 90 with p -value 1e-4.
Revised: 90 with p -value 1e-4.

Original: [Figure]
Revised: [Figure]

Original: are taken from the public data of the PGDL competition.

Original: C Deep networks experimental details
Revised: C Deep networks experimental details

Original: In this Section we describe in more detail the experiments reported in the Table 1 of the main text.
Revised: In this Section we describe in more detail the experiments reported in the Table 1 of the main text.

Original: In all experiments, the loss L is the usual cross-entropy and the parameter initialization is Kaiming normal.
Revised: In all experiments, the loss L is the usual cross-entropy and the parameter initialization is Kaiming normal.

Original: We normalize images in the train and test sets by the mean and variance over the train set.
Revised: We normalize images in the train and test sets by the mean and variance over the train set.

Original: We also apply random crops (of width w if image size is w × w , with zero-padding of size 4 for CIFAR and 8 for Tiny ImageNet) and random horizontal ﬂips.
Revised: We also apply random crops (of width w if image size is w × w , with zero-padding of size 4 for CIFAR and 8 for Tiny ImageNet) and random horizontal ﬂips.

Original: In the following we refer to the latter procedure by the name "standard preprocessing".
Revised: In the following we refer to the latter procedure by the name "standard preprocessing".

Original: All experiments are implemented using PyTorch (Paszke et al., 2019).
Revised: All experiments are implemented using PyTorch (Paszke et al., 2019).

Original: For the experiments with eSGD and rSGD, we use the same settings and hyper-parameters used for SGD (unless otherwise stated and apart from the hyperparameters speciﬁc to these two algorithms).
Revised: For the experiments with eSGD and rSGD, we use the same settings and hyper-parameters used for SGD (unless otherwise stated and apart from the hyperparameters speciﬁc to these two algorithms).

Original: For rSGD and unless otherwise stated, we set y = 3 , K = 10 and use the automatic exponential focusing schedule for γ reported in the main text.
Revised: For rSGD and unless otherwise stated, we set y = 3 , K = 10 and use the automatic exponential focusing schedule for γ reported in the main text.

Original: For eSGD, we use again an exponential focusing protocol.
Revised: For eSGD, we use again an exponential focusing protocol.

Original: In some experiments, we use a value of γ 0 automatically chosen by computing the distance between the conﬁgurations w (cid:48) and w after a loop of the SGLD dynamics (i.e. in the ﬁrst L steps with γ = 0 ) and setting γ 

Revised: In some experiments, we use a value of γ 0 automatically chosen by computing the distance between the conﬁgurations w (cid:48) and w after a loop of the SGLD dynamics (i.e. in the ﬁrst L steps with γ = 0 ) and setting γ

Original: =
Revised: =

Original: L ( w ) /d
Revised: L ( w ) /d

Original: ( w (cid:48) , w ) .
Revised: ( w (cid:48) , w ) .

Original: Unfortunately, this criterion is not robust.
Revised: Unfortunately, this criterion is not robust.

Original: Therefore, in some experiments the value of γ 0 was manually tuned.
Revised: Therefore, in some experiments the value of γ 0 was manually tuned.

Original: However, we found that eSGD is not sensitive to the precise value but to the order of magnitude.
Revised: However, we found that eSGD is not sensitive to the precise value but to the order of magnitude.

Original: We choose γ 1 such that γ is increased by a factor of 10 by the end of the training.
Revised: We choose γ 1 such that γ is increased by a factor of 10 by the end of the training.

Original: Unless otherwise stated, we set the the number of SGLD iterations to L = 5 , SGLD noise to (cid:15)
Revised: Unless otherwise stated, we set the the number of SGLD iterations to L = 5 , SGLD noise to (cid:15)

Original: = 10 − 4 and α = 0 .
Revised: = 10 − 4 and α = 0 .

Original: Moreover, we use 0.9 Nesterov momentum and weight decay in both the internal and external loop.
Revised: Moreover, we use 0.9 Nesterov momentum and weight decay in both the internal and external loop.

Original: As for the learning rate schedule, when we rescale the total number of epochs for eSGD and rSGD, we use a rescaled schedule giving a comparable ﬁnal learning rate and with consequently rescaled learning rate drop times as well.
Revised: As for the learning rate schedule, when we rescale the total number of epochs for eSGD and rSGD, we use a rescaled schedule giving a comparable ﬁnal learning rate and with consequently rescaled learning rate drop times as well.

Original: C. CIFAR-10 and CIFAR- 
Revised: C. CIFAR-10 and CIFAR-

Original: SmallConvNet The smallest architecture we use in our experiments is a LeNet-like network LeCun et al.
Revised: SmallConvNet The smallest architecture we use in our experiments is a LeNet-like network LeCun et al.

Original: [Equation]
Revised: [Equation]

Original: Each convolutional layer and the dense layer before the ﬁnal output layer are followed by
Revised: Each convolutional layer and the dense layer before the ﬁnal output layer are followed by

Original: ReLU non-linearities.
Revised: ReLU non-linearities.

Original: We train the SmallConvNet model on CIFAR-10 for 300 epochs with the following settings:
Revised: We train the SmallConvNet model on CIFAR-10 for 300 epochs with the following settings:

Original: SGD optimizer with Nesterov momentum 0.9; learning rate 0.01 that decays by a factor ofat epochs 150 and 225; batch-size 128; weight decay 1e- 4; standard preprocessing is applied; default parameter initialization (PyTorch 1.3).
Revised: SGD optimizer with Nesterov momentum 0.9; learning rate 0.01 that decays by a factor ofat epochs 150 and 225; batch-size 128; weight decay 1e-4; standard preprocessing is applied; default parameter initialization (PyTorch 1.3).

Original: For rSGD we set lr = 0 .
Revised: For rSGD we set lr = 0 .

Original: 05 and γ 

Revised: 05 and γ

Original: = 0 .
Revised: = 0 .

Original: For eSGD, we train for 60 epochs with: η = 0 .
Revised: For eSGD, we train for 60 epochs with: η = 0 .

Original: 5 that drops by a factor of 10 at epochsand 45; η (cid:48)
Revised: 5 that drops by a factor of 10 at epochsand 45; η (cid:48)

Original: = 0 .
Revised: = 0 .

Original: 02 ; γ 

Revised: 02 ; γ

Original: = 0 .
Revised: = 0 .

Original: 5 ; γ 1 = 2 · 10 − 5 .
Revised: 5 ; γ 1 = 2 · 10 − 5 .

Original: ResNet-18 In order to have a fast baseline network, we adopt a simple training procedure for ResNet-18 on CIFAR-10, without further optimizations.
Revised: ResNet-18 In order to have a fast baseline network, we adopt a simple training procedure for ResNet-18 on CIFAR-10, without further optimizations.

Original: We train the model for 160 epochs with: SGD optimizer with Nesterov momentum 0.9; initial learning rate 0.01 that decays by a factor of 10 at epoch 110; batch-size 128; weight decay 5e-4; standard preprocessing.
Revised: We train the model for 160 epochs with: SGD optimizer with Nesterov momentum 0.9; initial learning rate 0.01 that decays by a factor of 10 at epoch 110; batch-size 128; weight decay 5e-4; standard preprocessing.

Original: For rSGD we set K = 1 and learning rate 0.02.
Revised: For rSGD we set K = 1 and learning rate 0.02.

Original: For eSGD, we train for 32 epochs with initial learning rate η = 0 .
Revised: For eSGD, we train for 32 epochs with initial learning rate η = 0 .

Original: 25 that drops by a factor of 10 at epochs 16 and 25; η (cid:48)
Revised: 25 that drops by a factor of 10 at epochs 16 and 25; η (cid:48)

Original: = 0 .
Revised: = 0 .

Original: In the case in which we drop the learning rate at certain epochs, we notice that it is important not to schedule it before that the training error has reached a plateau also for eSGD and rSGD.
Revised: In the case in which we drop the learning rate at certain epochs, we notice that it is important not to schedule it before that the training error has reached a plateau also for eSGD and rSGD.

Original: ResNet-110 We train the ResNet-110 model on CIFAR-10 for 164 epochs following the original settings of He et al.
Revised: ResNet-110 We train the ResNet-110 model on CIFAR-10 for 164 epochs following the original settings of He et al.

Original: (2016): SGD optimizer with momentum 0.9; batch-size 128; weight decay 1e-4.
Revised: (2016): SGD optimizer with momentum 0.9; batch-size 128; weight decay 1e-4.

Original: We perform a learning rate warm-up starting with 0.01 and increasing it at 0.1 after 1 epoch; then it is dropped by a factor of 10 at epochs 82 and 124; standard preprocessing is applied.
Revised: We perform a learning rate warm-up starting with 0.01 and increasing it at 0.1 after 1 epoch; then it is dropped by a factor of 10 at epochs 82 and 124; standard preprocessing is applied.

Original: For both eSGD and rSGD, we ﬁnd that the learning rate warm-up is not necessary.
Revised: For both eSGD and rSGD, we ﬁnd that the learning rate warm-up is not necessary.

Original: For rSGD we set γ 

Revised: For rSGD we set γ

Original: = 5e-4.
Revised: = 5e-4.

Original: For eSGD, we train for 32 epochs with initial learning rate η = 

Revised: For eSGD, we train for 32 epochs with initial learning rate η =

Original: .that
Revised: .that

Original: drops at epochs 17 and 25, SGLD learning rate η (cid:48)
Revised: drops at epochs 17 and 25, SGLD learning rate η (cid:48)

Original: = 0 .
Revised: = 0 .

Original: and we set γ 

Revised: and we set γ

Original: = 0 .
Revised: = 0 .

Original: 1 and γ 1 = 5 · 10 − 4 .
Revised: 1 and γ 1 = 5 · 10 − 4 .

Original: PyramidNet+ShakeDrop PyramidNet+ShakeDrop (Han et al., 2016; Yamada et al., 2018), together with AutoAugment or Fast-AutoAugment, is currently the state-of-theart on CIFAR-10 and CIFAR-100 without extra training data.
Revised: PyramidNet+ShakeDrop PyramidNet+ShakeDrop (Han et al., 2016; Yamada et al., 2018), together with AutoAugment or Fast-AutoAugment, is currently the state-of-theart on CIFAR-10 and CIFAR-100 without extra training data.

Original: We train this model on
Revised: We train this model on

Original: CIFAR-10 and CIFAR-100 following the settings of Cubuk et al.
Revised: CIFAR-10 and CIFAR-100 following the settings of Cubuk et al.

Original: ; Lim et al.
Revised: ; Lim et al.

Original: PyramidNet272- α 200; SGD optimizer with Nesterov momentum 0.9; batch-size 64; weight decay 5e-5.
Revised: PyramidNet272- α 200; SGD optimizer with Nesterov momentum 0.9; batch-size 64; weight decay 5e-5.

Original: At variance with Cubuk et al.
Revised: At variance with Cubuk et al.

Original: ; Lim et al.
Revised: ; Lim et al.

Original: we train for 300 epochs and not 1800.
Revised: we train for 300 epochs and not 1800.

Original: We perform a cosine annealing of the learning rate (with a single annealing cycle) with initial learning rate 0.05.
Revised: We perform a cosine annealing of the learning rate (with a single annealing cycle) with initial learning rate 0.05.

Original: ShakeDrop is applied with the same parameters as in the original paper (Yamada et al., 2018).
Revised: ShakeDrop is applied with the same parameters as in the original paper (Yamada et al., 2018).

Original: For data augmentation we add to standard preprocessing AutoAugment with the policies found on CIFAR-10 (Cubuk et al., 2018)
Revised: For data augmentation we add to standard preprocessing AutoAugment with the policies found on CIFAR-10 (Cubuk et al., 2018)

Original: (for both CIFAR-10 and CIFAR-100) and CutOut (Devries & Taylor, 2017) with size 16.
Revised: (for both CIFAR-10 and CIFAR-100) and CutOut (Devries & Taylor, 2017) with size 16.

Original: For rSGD, we use a cosine focusing protocol for γ , deﬁned at epoch τ by γ τ = 0 .
Revised: For rSGD, we use a cosine focusing protocol for γ , deﬁned at epoch τ by γ τ = 0 .

Original: 5 γ max cos ( πτ/τ tot ) , with γ max = 0 .
Revised: 5 γ max cos ( πτ/τ tot ) , with γ max = 0 .

Original: On CIFAR-10, we decrease the interaction step
Revised: On CIFAR-10, we decrease the interaction step

Original: K from 10 to 3 towards the end of the training (at epoch 220) in order to reduce noise and allow the replicas to collapse.
Revised: K from 10 to 3 towards the end of the training (at epoch 220) in order to reduce noise and allow the replicas to collapse.

Original: EﬃcientNet-B0 EﬃcientNet-B0 is the base model for the EﬃcientNet family.
Revised: EﬃcientNet-B0 EﬃcientNet-B0 is the base model for the EﬃcientNet family.

Original: In this section we train EﬃcientNet-B0 on CIFAR-100, starting from random initial conditions.
Revised: In this section we train EﬃcientNet-B0 on CIFAR-100, starting from random initial conditions.

Original: We follow the same settings as Tan & Le (2019), with some diﬀerences: we train forepochs with RMSprop optimizer with momentum 0.9; batch-size 64; weight decay 1e-5; initial learning rate 0.01 that decays by 0.97 every 2 epochs.
Revised: We follow the same settings as Tan & Le (2019), with some diﬀerences: we train forepochs with RMSprop optimizer with momentum 0.9; batch-size 64; weight decay 1e-5; initial learning rate 0.01 that decays by 0.97 every 2 epochs.

Original: We rescale image size to 224 ×and as data augmentation we apply standard preprocessing (with zero-padding of size 32) adding AutoAugment with the policies found on CIFAR-10 (Cubuk et al., 2018).
Revised: We rescale image size to 224 ×and as data augmentation we apply standard preprocessing (with zero-padding of size 32) adding AutoAugment with the policies found on CIFAR-10 (Cubuk et al., 2018).

Original: For rSGD we set γ 0 = 5 e − 6 .
Revised: For rSGD we set γ 0 = 5 e − 6 .

Original: For eSGD we used initial learning rate η = 0 .
Revised: For eSGD we used initial learning rate η = 0 .

Original: 5 that decays by 0 .
Revised: 5 that decays by 0 .

Original: 92 every 2 epochs and η (cid:48)
Revised: 92 every 2 epochs and η (cid:48)

Original: = 0 .
Revised: = 0 .

Original: C. Tiny ImageNet
Revised: C. Tiny ImageNet

Original: ResNet-50 Entropic algorithms are eﬀective also on more complex datasets.
Revised: ResNet-50 Entropic algorithms are eﬀective also on more complex datasets.

Original: We train
Revised: We train

Original: ResNet-50 on Tiny ImageNet (data downloaded from: "Tiny ImageNet Visual Recognition
Revised: ResNet-50 on Tiny ImageNet (data downloaded from: "Tiny ImageNet Visual Recognition

Original: Challenge") for 270 epochs with: SGD optimizer with Nesterov momentum 0.9; initial learning rate 0.05 that decays by a factor of 10 at epochs 90, 180 and 240; batch-size 128; weight decay 1e-4.
Revised: Challenge") for 270 epochs with: SGD optimizer with Nesterov momentum 0.9; initial learning rate 0.05 that decays by a factor of 10 at epochs 90, 180 and 240; batch-size 128; weight decay 1e-4.

Original: Standard preprocessing is applied together with Fast-AutoAugment with the policies found on ImageNet (Lim et al., 2019).
Revised: Standard preprocessing is applied together with Fast-AutoAugment with the policies found on ImageNet (Lim et al., 2019).

Original: For eSGD we train the model for 50 epochs with η = 0 . 8 that drops by a factor of 10 at epochs 18, 36, 48 and η (cid:48)
Revised: For eSGD we train the model for 50 epochs with η = 0 . 8 that drops by a factor of 10 at epochs 18, 36, 48 and η (cid:48)

Original: = 0 .
Revised: = 0 .

Original: DenseNet-121 For DenseNet-121 on Tiny ImageNet, the setting is the same as ResNet-50, except that we train the model for 200 epochs with learning rate drops at epochs 100 and 150.
Revised: DenseNet-121 For DenseNet-121 on Tiny ImageNet, the setting is the same as ResNet-50, except that we train the model for 200 epochs with learning rate drops at epochs 100 and 150.

Original: For eSGD we train the model for 40 epochs with η = 0 .
Revised: For eSGD we train the model for 40 epochs with η = 0 .

Original: 5 that drops by a factor of 10 at epochs 25 and 30, η (cid:48)
Revised: 5 that drops by a factor of 10 at epochs 25 and 30, η (cid:48)

Original: = 0 .
Revised: = [Table]

Original: and we set γ 0 = 1 . 0 and γ 1 = 2 · 10 −