Original: Fantastic Generalization Measures and Where to Find Them
Revised: F ANTASTIC G ENERALIZATION M EASURES AND W HERE TO F IND T HEM

Original: Anonymous authors

Original: Paper under double-blind review

Original: Abstract
Revised: A BSTRACT

Original: Generalization of deep networks has  been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures.
Revised: Generalization of deep networks has lately been of great interest, resulting in a number of theoretically and empirically motivated complexity measures.

Original: However, most papers proposing such measuresonly study  a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings.
Revised: However, most papers proposing such measures study only a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings.

Original: We present the ﬁrst large scale study of generalizationbounds and measures in deep networks.
Revised: We present the ﬁrst large scale study of generalization in deep networks.

Revised: We investigate more then 40 complexity measures taken from both theoretical bounds and empirical studies.

Original: We train over two thousand convolutional networks with systematic changes in commonly used hyperparameters.
Revised: We train over 10, convolutional networks by systematically varying commonly used hyperparameters.

Original: Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.
Revised: Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.

Original:  Introduction
Revised:  I NTRODUCTION

Original: Deep neural networks have seen tremendous success in a number of applications, but why (and how well) these models generalize is still a mystery (Neyshabur et al., 2014;
Revised: Deep neural networks have seen tremendous success in a number of applications, but why (and how well) these models generalize is still a mystery (Neyshabur et al., 2014;

Original: Zhang et al., 2016; Recht et al., 2019).
Revised: Zhang et al., 2016; Recht et al., 2019).

Original: It is crucial to better understand the reason behind the generalization of modern deep learning models; such an understanding has multiple beneﬁts, including providing guarantees for safety-critical scenarios and the design of better models.
Revised: It is crucial to better understand the reason behind the generalization of modern deep learning models; such an understanding has multiple beneﬁts, including providing guarantees for safety-critical scenarios and the design of better models.

Original: A number of papers have attempted to understand the generalization phenomenon in deeplearning models from a theoretical perspective e.g. (Neyshabur et al., 2015b; Bartlett et al., 2017; Neyshabur et al., 2018a; Golowich et al., 2017; Arora et al., 2018; Nagarajan & Kolter, 2019a; Wei & Ma, 2019a; Long & Sedghi, 2019).
Revised: A number of papers have attempted to understand the generalization phenomenon in deep models from a theoretical perspective e.g. (Neyshabur et al., 2015b; Bartlett et al., 2017; Neyshabur et al., 2018a; Golowich et al., 2017; Arora et al., 2018; Nagarajan and Kolter, 2019a; Wei and Ma, 2019a; Long and Sedghi, 2019).

Original: The most direct and principled approach for studying generalization in deep learning is to prove a generalization bound which is typically an upper bound on the test error based on some quantity that can be calculated on the training set.
Revised: The most direct and principled approach for studying generalization in deep learning is to prove a generalization bound ; typically an upper bound on the test error based on some quantity that can be calculated on the training set.

Original: Unfortunately, ﬁnding tight bounds has proven to be an arduous undertaking.
Revised: Unfortunately, ﬁnding tight bounds has proven to be an arduous undertaking.

Original: While encouragingly Dziugaite & Roy (2017) showed that PAC-Bayesian bounds can be optimized to achieve a reasonably tight generalization bound, current bounds are still not tight enough to accurately capture the generalization behavior.
Revised: While encouragingly Dziugaite and Roy (2017) showed that PAC-Bayesian bounds can be optimized to achieve a reasonably tight generalization bound, current bounds are still not tight enough to accurately capture the generalization behavior.

Original: Others have proposed more direct empirical ways to characterize generalization of deep networks without attempting to deriving bounds (Keskar et al., 2016; Liang et al., 2017).
Revised: Others have proposed more direct empirical ways to characterize generalization of deep networks without attempting to deriving bounds (Keskar et al., 2016; Liang et al., 2017).

Original: However, as pointed by Dziugaite & Roy (2017), empirical correlation does not necessarily translate to a casual relationship between a measure and generalization.
Revised: However, as pointed by Dziugaite and Roy (2017), empirical correlation does not necessarily translate to a casual relationship between a measure and generalization.

Original: A core component in (theoretical or empirical) analysis of generalization is the notion of complexity measure ; a quantity that monotonically relates to some aspect of generalization.
Revised: A core component in (theoretical or empirical) analysis of generalization is the notion of complexity measure ; a quantity that monotonically relates to some aspect of generalization.

Original: More speciﬁcally, lower complexity should often imply smaller generalization gap.
Revised: More speciﬁcally, lower complexity should often imply smaller generalization gap.

Original: A complexity measure may depend on the properties of the trained model, optimizer, and possibly training data, but should not have access to a validation set.
Revised: A complexity measure may depend on the properties of the trained model, optimizer, and possibly training data, but should not have access to a validation set.

Original: Theoretically motivated complexity measures such as VC-dimension, norm of parameters, etc., are often featured as the major components of generalization bounds, where the monotonic relationship between the measures and generalization is mathematically established.
Revised: Theoretically motivated complexity measures such as VC-dimension, norm of parameters, etc., are often featured as the major components of generalization bounds, where the monotonic relationship between the measures and generalization is mathematically established.

Original: In contrast, empirically motivated complexity measures such as sharpness (Keskar et al., 2016) are justiﬁed by experimentation and observation.
Revised: In contrast, empirically motivated complexity measures such as sharpness (Keskar et al., 2016) are justiﬁed by experimentation and observation.

Original: In this work, we do not need to distinguish between theoretically vs empirically motivated measures, and simply refer to both as complexity measures.
Revised: In this work, we do not need to distinguish between theoretically vs empirically motivated measures, and simply refer to both as complexity measures.

Original: Despite the prominent role of complexity measures in studying generalization, the empirical evaluation of these measures is usually limited to a few models, often on toy problems.
Revised: Despite the prominent role of complexity measures in studying generalization, the empirical evaluation of these measures is usually limited to a few models, often on toy problems.

Original: A measure can only be considered reliable as a predictor of generalization gap if it is tested extensively on many models at a realistic problem size.
Revised: A measure can only be considered reliable as a predictor of generalization gap if it is tested extensively on many models at a realistic problem size.

Original: To this end, we carefully selected a wide range of complexity measures from the literature.
Revised: To this end, we carefully selected a wide range of complexity measures from the literature.

Original: Some of the measures are motivated by generalization bounds such as those related to VC-dimension, norm or margin based bounds, and PAC-Bayesian bounds.
Revised: Some of the measures are motivated by generalization bounds such as those related to VC-dimension, norm or margin based bounds, and PAC-Bayesian bounds.

Original: We further selected a variety of empirical measures such as sharpness (Keskar et al., 2016), Fisher-Rao norm (Liang et al., 2017) and path norms (Neyshabur et al., 2017).
Revised: We further selected a variety of empirical measures such as sharpness (Keskar et al., 2016), Fisher-Rao norm (Liang et al., 2017) and path norms (Neyshabur et al., 2017).

Original: In this study, we trained a large number of models on the CIFAR-10dataset (Krizhevsky et al., 2014).
Revised: In this study, we trained more than 10,000 models over two image classiﬁcation datasets, namely, CIFAR-10 (Krizhevsky et al., 2014) and Street View House Numbers (SVHN) Netzer et al.

Original: In order to create a wide range of generalization behaviors, we carefully varied hyperparameters that are believed to inﬂuence generalization.
Revised: In order to create a wide range of generalization behaviors, we carefully varied hyperparameters that are believed to inﬂuence generalization.

Original: We also selected multiple optimization algorithms and looked at diﬀerent stopping criteria for training convergence.
Revised: We also selected multiple optimization algorithms and looked at different stopping criteria for training convergence.

Original: Details of all our measures and hyperparameter selections are provided in Appendix D. Training under all combination of hyperparameters and optimization resulted in a large pool of models.
Revised: Details of all our measures and hyperparameter selections are provided in Appendix D. Training under all combination of hyperparameters and optimization resulted in a large pool of models.

Original: For any such model, we considered 40 complexity measures.
Revised: For any such model, we consideredcomplexity measures .

Original: The key ﬁndings that arise from our large scale study are summarized below:
Revised: The key ﬁndings that arise from our large scale study are summarized below:

Original: It is easy for some complexity measures to capture spurious correlations that do not reﬂectmore causal insights about generalization; to mitigate this problem, we propose a more rigorous approach for studying them.
Revised: It is easy for some complexity measures to capture spurious correlations that do not reﬂect causal insights about generalization; to mitigate that we propose a more rigorous method to study them.

Original: Many norm-based measures not only perform poorly, but negatively correlate with generalization speciﬁcally when the optimization procedure injects some stochasticity.
Revised: Many norm-based measures not only perform poorly, but negatively correlate with generalization speciﬁcally when the optimization procedure injects some stochasticity.

Original: In particular, the generalization bound based on the product of spectral norms of the layers (similar to that of Bartlett et al.
Revised: In particular, the generalization bound based on the product of spectral norms of the layers (similar to that of Bartlett et al.

Original: (2017)) has very strong negative correlation with generalization.
Revised: (2017)) has very strong negative correlation with generalization.

Original: Sharpness-based measures such as PAC-Bayesian bounds (McAllester, 1999)bounds and sharpness measure proposed by Keskar et al.
Revised: Sharpness-based measures like PAC-Bayesian bounds (McAllester, 1999) and the measure proposed by Keskar et al.

Original: (2016) performthe bestoverall and seemto be promising candidates for further research.
Revised: (2016) perform best and seem promising candidates for further research.

Original: Measures related to the optimization procedures such as the gradient noise and the speed of the optimization can be predictive of generalization.
Revised: Measures related to the optimization procedures such as the gradient noise and the speed of the optimization can be predictive of generalization.

Original: Our ﬁndings on the relative success of sharpness-based and optimization-based complexity measures for predicting the generalization gap can provoke further study of these measures.
Revised: Our ﬁndings on the relative success of sharpness-based and optimization-based complexity measures for predicting the generalization gap can provoke further study of these measures.

Original: 1. Related Work
Revised: 1. R ELATED W ORK

Original: The theoretically motivated measures that we consider in this work belong to a few diﬀerent families: PAC-Bayes (McAllester, 1999; Dziugaite & Roy, 2017; Neyshabur et al., 2017); VC-dimension (Vapnik & Chervonenkis, 1971); and norm-based bounds (Neyshabur et al., 2015b; Bartlett et al., 2017; Neyshabur et al., 2018a).
Revised: The theoretically motivated measures that we consider in this work belong to a few different families: PAC-Bayes (McAllester, 1999; Dziugaite and Roy, 2017; Neyshabur et al., 2017); VC-dimension (Vapnik and Chervonenkis, 1971); and norm-based bounds (Neyshabur et al., 2015b; Bartlett et al., 2017; Neyshabur et al., 2018a).

Original: The empirically motivated measures from prior literature that we consider are based on sharpness measure (Keskar et al., 2016); Fisher-Rao measure (Liang et al., 2017); distance of trained weights from initialization (Nagarajan & Kolter, 2019b) and path norm (Neyshabur et al., 2015a).
Revised: The empirically motivated measures from prior literature that we consider are based on sharpness measure (Keskar et al., 2016); Fisher-Rao measure (Liang et al., 2017); distance of trained weights from initialization (Nagarajan and Kolter, 2019b) and path norm (Neyshabur et al., 2015a).

Original: Finally, we consider some optimization based measures based on the speed of the optimization method as motivated by  (Hardt et al., 2015) and (Wilson et al., 2017a), and the magnitude of the gradient noise as motivated by the work of (Chaudhari & Soatto, 2018) and (Smith & Le, 2017).
Revised: Finally, we consider some optimization based measures based on the speed of the optimization algorithm as motivated by the work of (Hardt et al., 2015) and (Wilson et al., 2017a), and the magnitude of the gradient noise as motivated by the work of (Chaudhari and Soatto, 2018) and (Smith and Le, 2017).

Original: A few papers have explored a large scale study of generalization in deep networks.
Revised: A few papers have explored a large scale study of generalization in deep networks.

Original: Neyshabur et al.
Revised: Neyshabur et al.

Original: (2017) perform a small scale study of the generalization of PAC-Bayes, sharpness and a few diﬀerent norms, and the generalization analysis is restricted to correlation.
Revised: (2017) perform a small scale study of the generalization of PAC-Bayes, sharpness and a few different norms, and the generalization analysis is restricted to correlation.

Original: Jiang et al.
Revised: Jiang et al.

Original: (2018) studied the role of margin as a predictor of the generalization gap.
Revised: (2018) studied the role of margin as a predictor of the generalization gap.

Original: However, they used a signiﬁcantly more restricted set of models (e.g. no depth variations), the experiments were not controlled for potential undesired correlation (e.g. the models can have vastly diﬀerent training errors) and some measures contained parameters that must be learned from the set of models.
Revised: However, they used a signiﬁcantly more restricted set of models (e.g. no depth variations), the experiments were not controlled for potential undesired correlation (e.g. the models can have vastly different training error) and some measures contained parameters that must be learned from the set of models.

Original: Novak et al.
Revised: Novak et al.

Original: (2018) conducted large scale study  but they only looked at correlation of a few measures to generalization.
Revised: (2018) conducted large scale study of neural networks but they only looked at correlation of a few measures to generalization.

Original: In contrast, we study thousands of models, and perform controlled experiments to avoid undesired artiﬁcial correlations.
Revised: In contrast, we study thousands of models, and perform controlled experiments to avoid undesired artiﬁcial correlations.

Original: Our work focuses on measures that can be computed on a single model and compares a large number of measures across a much wider range of models in a carefully controlled fashion.

Original: 1. Notation

Original: We denote a probability distribution as A , set as A , tensor as A , vector as a , and scalar as a or α .

Original: Let D denote the data distributions over inputs and their labels, and let κ denote number of classes. We use (cid:44) for equality by deﬁnition. We denote by S a given dataset, consisting of m i.i.d tuples { ( X 1 , y 1 ) , . . . , ( X m , y m ) } drawn from D where X i ∈ X is the input data and y i ∈ { 1 , . . .
Revised: Some of our analysis techniques are inspired by Neal (2019) who proposed the idea of studying generalization in deep models via causal graphs, but did not provide any details or empirical results connected to that idea.

Original: , κ } the corresponding class label.

Original: We denote a feedforward neural network by f w : X → R κ , its weight parameters by w , and the the number of weights by ω (cid:44) dim( w ).

Original: No activation function is applied at the output (i.e. logits).

Original: Denote the weight tensor of the i th layer of the network by W i , so that w = vec( W 1 , . .

Original: . , W d ), where d is the depth of the network, and vec represents the vectorization operator.

Original: Furthermore, denote by f w ( X )[ j ] the j -th output of the function f w ( X ).

Original: Let R be the set of binary relations, and I : R → { 0 , 1 } be the indicator function that is 1 if its input is true and zero otherwise.

Original: Let L be the 1-0 classiﬁcation loss over the data distribution D : L ( f w ) (cid:44) E ( X ,y ) ∼ D (cid:2)

Original: I (cid:0) f w ( X ) [ y ] ≤ max j 6 = y f w ( X ) [ j ] (cid:1)(cid:3) and let ˆ L be the empirical estimate of [-1-0 loss over S : ˆ L ( f w ) (cid:44) 1 m P mi =
-]
Revised: Our work focuses on measures that can be computed on a single model and compares a large number of [-1-0 loss over S : ˆ L ( f w ) (cid:44) 1 m P mi =
-]bounds and measures across a much wider range of models in a carefully controlled fashion.

Original: I (cid:0) f w ( X )[ y i ] ≤ max j 6 = y i f w ( X )[ j ] (cid:1) .

Original: We refer to L ( f w ) − ˆ L ( f w ) as the generalization gap .

Original: For any input X , we deﬁne the sample dependent margin 1 as γ ( X ) (cid:44) (cid:0) f w ( X ) (cid:1)

Original: [-[ y ] − max i 
-]
Revised: [-[ y ] − max i 
-]

Original: = y f w ( X ) i .

Original: Moreover, we deﬁne the overall margin γ as the 10 th percentile (a robust surrogate for the minimum) of γ ( X ) over the entire training set S .

Original: More notation used for derivation is located in Appendix C.

Original:  Generalization: What is the goal and how to evaluate?
Revised:  G ENERALIZATION : W HAT IS THE GOAL AND HOW TO EVALUATE ?

Original: Generalization is arguably the most fundamental and yet mysterious aspect of machine learning.
Revised: Generalization is arguably the most fundamental and yet mysterious aspect of machine learning.

Original: The core question in generalization is what causes the pair of a model and an optimization algorithm  to generalize well beyond the training set.
Revised: The core question in generalization is what causes the triplet of a model , optimization algorithm and data properties 1 , to generalize well beyond the training set.

Original: There are many hypotheses concerning this question, but what is the right way to compare these hypotheses?
Revised: There are many hypotheses concerning this question but what is the right way to compare these hypotheses?

Revised: The core component of each hypothesis is complexity measure that monotonically relates to some aspect of generalization.

Original: Here we brieﬂy discuss some potential approaches to compare diﬀerent hypotheses:
Revised: Here we brieﬂy discuss some potential approaches to compare different complexity measures:

Original: • Tightness of Generalization Bounds: Proving generalization bounds is very useful to establish the causal relationship between a complexity measure and the generalization error.
Revised: • Tightness of Generalization Bounds. Proving generalization bounds is very useful to establish the causal relationship between a complexity measure and the generalization error.

Original: However, almost all existing bounds are vacuous on current deep  models and datasets and therefore, one cannot rely on their proof as an evidence on the causal relationship between a complexity measure and generalization currently 2 .
Revised: However, almost all existing bounds are vacuous on current deep learning tasks (combination of models and datasets), and therefore, one cannot rely on their proof as an evidence on the causal relationship between a complexity measure and generalization currently 2 .

Original: Regularizing the Complexity Measure: One may evaluate a  measure by adding it as a regularizer and directly optimizing it, but this could fail due to two reasons.
Revised: Regularizing the Complexity Measure. One may evaluate a complexity measure by adding it as a regularizer and directly optimizing it, but this could fail due to two reasons.

Original: The complexity measure could change the loss landscape in non-trivial ways and make the optimization more diﬃcult.
Revised: The complexity measure could change the loss landscape in non-trivial ways and make the optimization more difﬁcult.

Original: In such cases, if the optimization fails to optimize the measure, no conclusion can be made about the causality.
Revised: In such cases, if the optimization fails to optimize the measure, no conclusion can be made about the causality.

Original: Another, and perhaps more critical, problem is the existence of implicit regularization of the optimization algorithm.
Revised: Another, and perhaps more critical, problem is the existence of implicit regularization of the optimization algorithm.

Original: This makes it hard to run a controlled experiment since one cannot simply turn oﬀ the implicit regularization; therefore, if optimizing a measure does not improve generalization it could be simply due to the fact that it is regularizing the model in the same way as the optimization is regularizing it implicitly.
Revised: This makes it hard to run a controlled experiment since one cannot simply turn off the implicit regularization; therefore, if optimizing a measure does not improve generalization it could be simply due to the fact that it is regularizing the model in the same way as the optimization is regularizing it implicitly.

Original: • Correlation with Generalization: Evaluating measures based on correlation with generalization is interesting but potentially dangerous.
Revised: • Correlation with Generalization Evaluating measures based on correlation with generalization is very useful but it can also provide a misleading picture.

Original: To check the correlation, we should vary architectures and optimization algorithms to produce a set of models.
Revised: To check the correlation, we should vary architectures and optimization algorithms to produce a set of models.

Original: If the set is generated in an artiﬁcial way and is not representative of the typical setting, the conclusions might be deceiving and might not generalize to typical cases.
Revised: If the set is generated in an artiﬁcial way and is not representative of the typical setting, the conclusions might be deceiving and might not generalize to typical cases.

Original: One such example is training with diﬀerent portions of random labels which artiﬁcially changes the dataset.
Revised: One such example is training with different portions of random labels which artiﬁcially changes the dataset.

Original: Another pitfall is drawing conclusion from changing one or two hyper-parameters (e.g changing the width or batch-size and checking if a measure would correlate with generalization).
Revised: Another pitfall is drawing conclusion from changing one or two hyper-parameters (e.g changing the width or batch-size and checking if a measure would correlate with generalization).

Original: In these cases, the hyper-parameter could be the true cause of both change in the measure and change in the generalization but the measure itself has no causal relationship with generalization.
Revised: In these cases, the hyper-parameter could be the true cause of both change in the measure and change in the generalization, but the measure itself has no causal relationship with generalization.

Revised: Therefore, one needs to be very careful with experimental design to avoid unwanted correlations.

Revised: In this work we focus on the third approach.

Original: While acknowledging all limitations of a correlation analysis, we try to improve the procedure and capture some of the causal eﬀects as much as possible through careful design of experiments and metrics.
Revised: While acknowledging all limitations of a correlation analysis, we try to improve the procedure and capture some of the causal effects as much as possible through careful design of controlled experiments.

Original: To this end, we can try to partially capture this causal eﬀect by changing some hyper-parameters and keeping the rest ﬁxed and see if the measure can still correlate with generalization; we also adopt tools from causal inference to uncover the underlying causal structure.

Original: Further, to evaluate the eﬀectiveness of a measure, it is crucial to test it on models with a wide range of variations, and each model must be suﬃciently trained, if not to complete convergence.
Revised: Further, to evaluate the effectiveness of complexity measures as accurately as possible, we analyze them over sufﬁciently trained models with a wide range of variations in hyperparameters.

Original: For practical reasons, they mustalso reach this stage within a reasonable time budget.
Revised: For practical reasons, these models must reach convergence within a reasonable time budget.

Revised: Details of the notations used are outlined in Appendix A .

Original: 2. Training Models across Hyperparameter Space
Revised: 2. T RAINING M ODELS ACROSS H YPERPARAMETER S PACE

Original: In order to create models with diﬀerent generalization behavior, we consider various hyperparameter types, which are known or believed to inﬂuence generalization (e.g. batch size, dropout rate, etc.).
Revised: In order to create models with different generalization behavior, we consider various hyperparameter types, which are known or believed to inﬂuence generalization (e.g. batch size, dropout rate, etc.).

Original: Formally, denote each hyperparameter by θ i taking values from the set Θ i , for i = 1 , . . . , n and n denoting the total number of hyperparameter types 3 .
Revised: Formally, denote each hyperparameter by θ i taking values from the set Θ i , for i = 1 , . . . , n and n denoting the total number of hyperparameter types 3 .

Original: For each value of hyperparameters θ (cid:44) ( θ 1 , θ 2 , . . .
Revised: For each value of hyperparameters θ (cid:44) ( θ 1 , θ 2 , . . .

Original: , θ n ) ∈ Θ, where Θ (cid:44) Θ 1 × Θ 2 × · · · × Θ n , we train the architecture until the training loss (cross entropy value) reaches a given threshold (cid:15) .
Revised: , θ n ) ∈ Θ , where Θ (cid:44) Θ 1 × Θ 2 × · · · × Θ n , we train the architecture until the training loss (cross-entropy value) reaches a given threshold (cid:15) .

Original: See the Appendix A.2 for a discussion on the choice of the stopping criterion.
Revised: See the Appendix C.2 for a discussion on the choice of the stopping criterion.

Original: Doing this for each hyper-parameter conﬁguration θ ∈ Θ, we obtain a total of | Θ | models.
Revised: Doing this for each hyper-parameter conﬁguration θ ∈ Θ , we obtain a total of | Θ | models.

Original: The space Θ reﬂects our prior knowledge about a reasonable hyperparameter space, both in terms of their types and values.
Revised: The space Θ reﬂects our prior knowledge about a reasonable hyperparameter space, both in terms of their types and values.

Original: Regarding the latter, one could, for example, create Θ i by grid sampling of a reasonable number of points within a reasonable range of values for θ i .
Revised: Regarding the latter, one could, for example, create Θ i by grid sampling of a reasonable number of points within a reasonable range of values for θ i .

Original: 2. Kendall’s Rank-Correlation Coefficient
Revised: 2. E VALUATION C RITERIA 2.2. K ENDALL ’ S R ANK -C ORRELATION C OEFFICIENT

Original: One way to evaluate the quality of a complexity measure µ is through ranking .
Revised: One way to evaluate the quality of a complexity measure µ is through ranking .

Original: Given a set of models resulted by training with hyperparameters in the set Θ, their associated generalization gap { g ( θ ) | θ ∈ Θ }  and their respective values of the measure, { µ ( θ ) | θ ∈ Θ } , our goal is to analyze how consistent a measure (e.g. ‘ 2 norm of network weights) is with the empirically observed generalization.
Revised: Given a set of models resulted by training with hyperparameters in the set Θ , their associated generalization gap { g ( θ ) | θ ∈ Θ } , and their respective values of the measure { µ ( θ ) | θ ∈ Θ } , our goal is to analyze how consistent a measure (e.g. (cid:96) 2 norm of network weights) is with the empirically observed generalization.

Original: To this end, we construct a set T , where each element of the set is associated with one of the trained models.
Revised: To this end, we construct a set T , where each element of the set is associated with one of the trained models.

Original: Each element has the form of a pair: complexity measure µ versus generalization gap g .
Revised: Each element has the form of a pair: complexity measure µ versus generalization gap g .

Original: [Equation]
Revised: [Equation]

Original: An ideal complexity measure must be such that, for any pair of trained models, if µ ( θ 1 ) > µ ( θ 2 ), then so is g ( θ 1 ) > g ( θ 2 ).
Revised: An ideal complexity measure must be such that, for any pair of trained models, if µ ( θ 1 ) > µ ( θ 2 ) , then so is g ( θ 1 ) > g ( θ 2 ) .

Original: We use Kendall’s rank coeﬃcient τ  to capture to what degree such consistency holds among the elements of T .
Revised: We use Kendall’s rank coefﬁcient τ (Kendall, 1938) to capture to what degree such consistency holds among the elements of T .

Original: [Equation]
Revised: [Equation]

Original: Note that τ can vary between 1 and − 1 and attains these extreme values at perfect agreement (two rankings are the same) and perfect disagreement (one ranking is the reverse of the other) respectively.
Revised: Note that τ can vary between 1 and − 1 and attains these extreme values at perfect agreement (two rankings are the same) and perfect disagreement (one ranking is the reverse of the other) respectively.

Original: If complexity and generalization are independent, the coeﬃcient becomes zero.
Revised: If complexity and generalization are independent, the coefﬁcient becomes zero.

Original: 2. An evaluation criterion based on controlled experiments
Revised: 2.2. G RANULATED K ENDALL ’ S C OEFFICIENT

Original: While Kendall’s correlation coeﬃcient is an eﬀective tool widely used to capture relationship between 2 rankings of a set of objects, we found that certain measures can achieve high τ values in a trivial manner – i.e. the measure may strongly correlate with the generalization performance without necessarily capturing the cause of generalization.
Revised: While Kendall’s correlation coefﬁcient is an effective tool widely used to capture relationship between 2 rankings of a set of objects, we found that certain measures can achieve high τ values in a trivial manner – i.e. the measure may strongly correlate with the generalization performance without necessarily capturing the cause of generalization.

Original: We will analyze this phenomenon in greater details in subsequent sections.
Revised: We will analyze this phenomenon in greater details in subsequent sections.

Original: To mitigate the eﬀect of spurious correlations, we propose a new quantity for reﬂecting the correlation between measures and generalization based on a more controlled setting.
Revised: To mitigate the effect of spurious correlations, we propose a new quantity for reﬂecting the correlation between measures and generalization based on a more controlled setting.

Original: None of the existing complexity measures is perfect.
Revised: None of the existing complexity measures is perfect.

Original: However, they might have diﬀerent sensitivity and accuracy w.r.t.
Revised: However, they might have different sensitivity and accuracy w.r.t.

Original: diﬀerent hyperparameters.
Revised: different hyperparameters.

Original: For example, sharpness may do better than other measures when only a certain hyperparameter (say batch size) changes.
Revised: For example, sharpness may do better than other measures when only a certain hyperparameter (say batch size ) changes.

Original: To understand such details, in addition to τ ( T ), we compute τ for consistency within each hyperparameter axis
Revised: To understand such details, in addition to τ ( T ) , we compute τ for consistency within each hyperparameter axis

Original: Θ i , and then average the coeﬃcient across the remaining hyperparameter space.
Revised: Θ i , and then average the coefﬁcient across the remaining hyperparameter space.

Original: Formally, we deﬁne:
Revised: Formally, we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: The inner τ reﬂects the ranking correlation between the generalization and the complexity measure for a small group of models where the only diﬀerence among them is the variation along a single hyperparameter θ i .
Revised: The inner τ reﬂects the ranking correlation between the generalization and the complexity measure for a small group of models where the only difference among them is the variation along a single hyperparameter θ i .

Original: Then average the value across all combinations of the other hyperparameter axis.
Revised: We then average the value across all combinations of the other hyperparameter axis.

Original: Intuitively, if a measure is good at predicting the eﬀect of hyperparameter θ i over the model distribution, then its corresponding ψ i should be high.
Revised: Intuitively, if a measure is good at predicting the effect of hyperparameter θ i over the model distribution, then its corresponding ψ i should be high.

Original: Then, we compute the  ψ i ’s average across all hyperparamter axis:
Revised: Finally, we compute the average ψ i of average across all hyperparamter axes, and name it Ψ :

Original: [Equation]
Revised: [Equation]

Original: If a measure achieves a high Ψ on a given model distribution Θ, then it should achieve high individual ψ across all hyperparameters.
Revised: If a measure achieves a high Ψ on a given hyperparameter distribution Θ , then it should achieve high individual ψ across all hyperparameters.

Original: A  measure that excels at predicting changes in a single hyperparameter (high ψ i ) but fails at the other hyperparameters (low ψ j for all j 6 = i ) will not do well on Ψ. On the other hand, if the measure performs well on Ψ, it means that the measure can reliably rank the generalization for each of the hyper-parameter changes.
Revised: A complexity measure that excels at predicting changes in a single hyperparameter (high ψ i ) but fails at the other hyperparameters (low ψ j for all j (cid:54) = i ) will not do well on Ψ . On the other hand, if the measure performs well on Ψ , it means that the measure can reliably rank the generalization for each of the hyper-parameter changes.

Original: A thought experiment to illustrate why Ψ captures a better causal nature of the generalization than Kendall’s τ is as follows.
Revised: A thought experiment to illustrate why Ψ captures a better causal nature of the generalization than Kendall’s τ is as follows.

Original: Suppose there exists a measure that perfectly captures the depth of the network while producing random prediction if 2 networks have the same depth, this measure would do reasonably well in terms of τ but much worse in terms of Ψ.
Revised: Suppose there exists a measure that perfectly captures the depth of the network while producing random prediction if 2 networks have the same depth, this measure would do reasonably well in terms of τ but much worse in terms of Ψ .

Original: In the experiments we consider in the following sections, we found that such a measure would achieve overall τ = 0 .
Revised: In the experiments we consider in the following sections, we found that such a measure would achieve overall τ = 0 .

Original: 362 but Ψ = 0 .
Revised: 362 but Ψ = 0 .

Revised: 2.2. C ONDITIONAL I NDEPENDENCE T EST : T OWARDS C APTURING THE C AUSAL R ELATIONSHIPS

Original: Finally, we acknowledge this this measure is only a small step towards the diﬃcult problem of capturing the causal relationship between  complexity measures and generalization in empirical settings, and we hope this encourages future work in this direction.
Revised: Relying on correlation is intuitive but perhaps unsatisfactory. In our experiments, we change several hyper-parameters and assess the correlation between a complexity measure and generalization; however, changing the hyper-parameters can also be seen as interventions or randomized tests since they are chosen independently from any other variables, and we can use this observation to uncover causal structures. Speciﬁcally, we rely on the conditional independent test which measures the conditional mutual information between random variables. Details are outlined in Appendix B.

Original:  Generating a family of trained models
Revised:  G ENERATING A F AMILY OF T RAINED M ODELS

Original: We chose 7 common hyperparameters related to optimization and architecture design, with 3 choices for each hyperparameter, to generate 3 7 = 2187 models.
Revised: We chose 7 common hyperparameter types related to optimization and architecture design, withchoices for each hyperparameter. We generated 3 7 = 2187 models that are trained on the CIFAR10 dataset.

Revised: We analyze these 2187 models in the subsequent sections; however, additional results including repeating the experiments 5 times as well as training the models using SVHN dataset are presented 4 in Appendix Section C.5.

Revised: These additional experiments, which add up to more than 10,000 trained models, suggest that the observations we make here are robust to randomness, and, more importantly, captures general behaviors of image classiﬁcation tasks.

Original: We trained these models to convergence.
Revised: We trained these models to convergence.

Original: Convergence criterion is chosen as when cross entropy loss reaches the value 0.01.
Revised: Convergence criterion is chosen as when cross-entropy loss reaches the value 0.01.

Original: Any model that was not able to achieve this value of cross-entropy 4 was discarded from further analysis.
Revised: Any model that was not able to achieve this value of cross-entropy 5 was discarded from further analysis.

Original: The latter is diﬀerent from the DEMOGEN dataset (Jiang et al., 2018) where the models are not trained to the same cross-entropy.
Revised: The latter is different from the DEMOGEN dataset (Jiang et al., 2018) where the models are not trained to the same cross-entropy.

Original: Putting the stopping criterion on the training loss rather than the number of epochs is crucial since otherwise one can simply use cross-entropy loss value to predict generalization.
Revised: Putting the stopping criterion on the training loss rather than the number of epochs is crucial since otherwise one can simply use cross-entropy loss value to predict generalization.

Original: Please see Appendix Section A.2 for a discussion on the choice of stopping criterion.
Revised: Please see Appendix Section C.2 for a discussion on the choice of stopping criterion.

Original: To construct a pool of trained models with vastly diﬀerent generalization behaviors while being able to ﬁt the training set, we covered a wide range of hyperparameters for training.
Revised: To construct a pool of trained models with vastly different generalization behaviors while being able to ﬁt the training set, we covered a wide range of hyperparameters for training.

Original: Our base model is inspired by the Network-in-Network (Gao et al., 2011).
Revised: Our base model is inspired by the Network-in-Network (Gao et al., 2011).

Original: The hyperparameter categories we test on are: weight decay coeﬃcient ( weight decay ), width of the layer ( width ), mini-batch size ( batchsize ), learning rate ( learning rate ), dropout probability ( dropout ), depth of the architecture ( depth ) and the choice of the optimization algorithms ( opt ).
Revised: The hyperparameter categories we test on are: weight decay coefﬁcient ( weight decay ), width of the layer ( width ), mini-batch size ( batch size ), learning rate ( learning rate ), dropout probability ( dropout ), depth of the architecture ( depth ) and the choice of the optimization algorithms ( optimizer ).

Original: We select 3 choices for each hyperparameter (i.e. | Θ 1 | = 3 ∀ i ), but diﬀerent optimization algorithms may use diﬀerent hyperparameter sets such as learning rate or weight regularization.
Revised: We selectchoices for each hyperparameter (i.e. | Θ i | = 3 ).

Original: Please refer to Appendix A.3 for the details on the models, and Appendix A.1 for the reasoning behind the design choices.
Revised: Please refer to Appendix C.3 for the details on the models, and Appendix C.1 for the reasoning behind the design choices.

Original: Figure 1 showssome summarizing statistics of the models in this study.
Revised: Figure 1 shows summarizing statistics of the models in this study.

Original: On the left we show the number of models that achieve above 99% training accuracy for every individual hyperparameter choice.
Revised: On the left we show the number of models that achieve above 99% training accuracy for every individual hyperparameter choice.

Original: Since we have 3 7 = 2187 models in total, the maximum number of model for each hyperparameter type is 3 7 − 1 = 718; the majority of the models in our pool were able to reach this threshold.
Revised: Since we have 3 7 = 2187 models in total, the maximum number of models for each hyperparameter type is 3 7 − 1 = 718 ; the majority of the models in our pool were able to reach this threshold.

Original: In the middle we show the distribution of the crossentropy value over the entire training set.
Revised: In the middle we show the distribution of the cross-entropy value over the entire training set.

Original: While we want the models to be at exactly 0. cross-entropy, in practice it is computationally prohibitive to constantly evaluate the loss over the entire training set; further, to enable reasonable temporal granularity, we estimate the training loss with 100 randomly sampled minibatch.
Revised: While we want the models to be at exactly 0.01 cross-entropy, in practice it is computationally prohibitive to constantly evaluate the loss over the entire training set; further, to enable reasonable temporal granularity, we estimate the training loss with 100 randomly sampled minibatch.

Original: These computational compromises result in long-tailed distribution of training loss centered at 0.01.
Revised: These computational compromises result in long-tailed distribution of training loss centered at 0.01.

Original: As shown in Table 1, even such minuscule range of cross-entropy diﬀerence could lead to positive correlation with generalization, highlighting the importance of training loss as a stopping criterion.
Revised: As shown in Table 1, even such minuscule range of cross-entropy difference could lead to positive correlation with generalization, highlighting the importance of training loss as a stopping criterion.

Original: On the right, we show the distribution of the generalization gap.
Revised: On the right, we show the distribution of the generalization gap.

Original: We see that while all the models’ training accuracy is above 0.99, there is a wide range of generalization gap, which is ideal for evaluating complexity measures.
Revised: Notice while all the models’ training accuracy is above 0.99, there is a wide range of generalization gap, which is ideal for evaluating complexity measures.

Original:  Baselines
Revised:  P ERFORMANCE OF C OMPLEXITY M EASURES 4. B ASELINE C OMPLEXITY M EASURES

Original: The ﬁrst baseline we consider is performance of a measure against an oracle who observes the noisy generalization gap.
Revised: The ﬁrst baseline we consider is performance of a measure against an oracle who observes the noisy generalization gap.

Original: Concretely, we rank the models based on the true generalization gap with some additive noise.
Revised: Concretely, we rank the models based on the true generalization gap with some additive noise.

Original: The resulting ranking correlation indicates how close the performances of all models are.
Revised: The resulting ranking correlation indicates how close the performances of all models are.

Original: As the scale of the noise approaches 0, the oracle’s prediction tends towards perfect (i.e. 1).
Revised: As the scale of the noise approaches 0, the oracle’s prediction tends towards perfect (i.e. 1).

Original: This baseline accounts for the potential noise in the training procedure and gives an anchor for gauging the diﬃculty of each hyperparameter type.
Revised: This baseline accounts for the potential noise in the training procedure and gives an anchor for gauging the difﬁculty of each hyperparameter type.

Original: Formally, given a set of hyper-parameters Θ 0 , we deﬁneoracle (cid:15)  to be the expectation of τ or Ψ where the measure is { g ( θ ) + N (0 , (cid:15) 2 ) | θ ∈ Θ 0 } .
Revised: Formally, given an arbitrary set of hyper-parameters Θ (cid:48) , we deﬁne (cid:15) -oracle to be the expectation of τ or Ψ where the measure is { g ( θ )+ N (0 , (cid:15) 2 ) | θ ∈ Θ (cid:48) } .

Original: We report the performance of the noisy oracle in Table 1 for (cid:15) ∈ { 0 . 01 , 0 . 
Revised: We report the performance of the noisy oracle in Table 1 for (cid:15) ∈ { 0 . 02 , 0 . 05 } .

Revised: For additional choices of (cid:15) please refer to Appendix C.5.

Original: Second, to understand how our hyperparameter choices aﬀect the optimization, we give each hyperparameter type a canonical order which is believed to have correlation with generalization (e.g. larger learning rate generalizes better) and measure their τ .
Revised: Second, to understand how our hyperparameter choices affect the optimization, we give each hyperparameter type a canonical order which is believed to have correlation with generalization (e.g. larger learning rate generalizes better) and measure their τ .

Original: The exact canonical ordering can be found in Appendix A.4.
Revised: The exact canonical ordering can be found in Appendix C.4.

Original: Note that unlike other measures, each canonical ordering can only predict generalization for its own hyperparameter type, since its corresponding hyperparameter remains ﬁxed in any other hyperparameter type; consequently, each column actually represents diﬀerent measure for the canonical measure row.
Revised: Note that unlike other measures, each canonical ordering can only predict generalization for its own hyperparameter type, since its corresponding hyperparameter remains ﬁxed in any other hyperparameter type; consequently, each column actually represents different measure for the canonical measure row.

Original: Assuming that each canonical measure is uninformative of any other canonical measures, the Ψ metric for each canonical measure is 17 of its performance on it corresponding hyperparameter type. 
Revised: Assuming that each canonical measure is uninformative of any other canonical measures, the Ψ criterion for each canonical measure is 17 of its performance on the corresponding hyperparameter type. [Figure] [Table]

Original:  We next look at  of the most well-known complexity measures in machine learning is the VC-Dimension.
Revised: [Figure] We next look at one of the most well-known complexity measures in machine learning; the VCDimension.

Original: Bartlett et al.
Revised: Bartlett et al.

Original: (2019) proves bounds on the VC dimension of piece-wise linear networks with potential weight sharing.
Revised: (2019) proves bounds on the VC dimension of piece-wise linear networks with potential weight sharing.

Original: In Appendix D.1, we extend their result to include pooling layers and multi-class classiﬁcation.
Revised: In Appendix D.1, we extend their result to include pooling layers and multi-class classiﬁcation.

Original: We report two measure based on VCdimension bounds and parameter counting.
Revised: We report two complexity measures based on VC-dimension bounds and parameter counting.

Original: These measures couldonly be predicative when the architecture changes whichonly happens  in depth and width hyperparameter type.
Revised: These measures could be predictive merely when the architecture changes, which happens only in depth and width hyperparameter types.

Original: We observe that in both types VC-dimension and number of parameters are negatively correlated with generalization gap which conﬁrms the widely known empirical observation that overparametrization improves generalization in deep learning.
Revised: We observe that, with both types, VC-dimension as well as the number of parameters are negatively correlated with generalization gap which conﬁrms the widely known empirical observation that overparametrization improves generalization in deep learning .

Original: Finally, we report the measures that only look at the output of the network.
Revised: Finally, we report the measures that only look at the output of the network.

Original: In particular, we look at the cross-entropy loss, margin γ  and the entropy of the output.
Revised: In particular, we look at the cross-entropy loss, margin γ , and the entropy of the output.

Original: All three measures are closely related to each other.
Revised: These three measures are closely related to each other.

Original: In fact, the results in Table 1also reﬂects this similarity.
Revised: In fact, the outcomes in Table 1 reﬂects this similarity.

Original: The results conﬁrm the general understanding that larger margin, lower cross-entropy and higher entropy would lead to better generalization.
Revised: These results conﬁrm the general understanding that larger margin, lower cross-entropy and higher entropy would lead to better generalization .

Original: Please see Appendix D.1.1 for deﬁnitions and more discussions on these measures.
Revised: Please see Appendix D.1.1 for deﬁnitions and more discussions on these measures.

Original: [Table]  Surprising Failure of some Norm/margin-based Measures
Revised: 4. S URPRISING F AILURE OF S OME (N ORM & M ARGIN )-B ASED M EASURES

Original: In machine learning, a long standing measure for quantifying the complexity of a function, and therefore generalization, is using various norms of the given function.
Revised: In machine learning, a long standing measure for quantifying the complexity of a function, and therefore generalization, is using some norm of the given function.

Original: Some of the norms can be directly optimized to improve the generalization.
Revised: Indeed, directly optimizing some of the norms can lead to improved generalization.

Original: For example, ‘ 2 regularization on the parameters of a model can be seen as imposing an isotropic Gaussian prior over the parameters in maximum a posteriori estimation.
Revised: For example, (cid:96) 2 regularization on the parameters of a model can be seen as imposing an isotropic Gaussian prior over the parameters in maximum a posteriori estimation.

Original: We choose several representative norms or measures based on norms and compute our correlation coeﬃcient between the measures and the generalization gap of the model.
Revised: We choose several representative norms (or measures based on norms) and compute our correlation coefﬁcient between the measures and the generalization gap of the model.

Original: We study the following measures and their variants ( spectral  Frobenius  Frobenius 
Revised: We study the following measures and their variants (Table 2): spectral bound , Frobenius distance from initialization , (cid:96) 2 Frobenius norm of the parameters , Fisher-Rao metric and path norm .

Original: [Table]
Revised: [Table]

Original: Spectral bound : The most surprising observation here is that the spectral complexity is strongly negatively correlated with generalization, and negatively correlated with changes within every hyperparameter type.
Revised: Spectral bound : The most surprising observation here is that the spectral complexity is strongly negatively correlated with generalization , and negatively correlated with changes within every hyperparameter type.

Original: Most notably, it has strong negative correlation with the depth of the network, which may suggest that the largest singular values are not sufﬁcient to capture the capacity of the model.
Revised: Most notably, it has strong negative correlation with the depth of the network, which may suggest that the largest singular values are not sufﬁcient to capture the capacity of the model.

Original: To better understand the reason behind this observation, we investigate using diﬀerent components of the spectral complexity as the measure.
Revised: To better understand the reason behind this observation, we investigate using different components of the spectral complexity as the measure.

Original: An interesting observation is that the Frobenius distance to initialization is negatively correlated, but the Frobenius norm of the parameters is slightly positively correlated with generalization, which contradicts some theories suggesting solutions closer to initialization should generalize better.
Revised: An interesting observation is that the Frobenius distance to initialization is negatively correlated, but the Frobenius norm of the parameters is slightly positively correlated with generalization, which contradicts some theories suggesting solutions closer to initialization should generalize better.

Original: A tempting hypothesis is that weight decay favors solution closer to the origin, but we did an ablation study on only models with 0 weight decay and found that the distance from initialization still correlates negatively with generalization.
Revised: A tempting hypothesis is that weight decay favors solution closer to the origin, but we did an ablation study on only models with 0 weight decay and found that the distance from initialization still correlates negatively with generalization.

Original: These observations correspond to choosing diﬀerent reference matrices W 

Revised: These observations correspond to choosing different reference matrices W

Original: i for the bound: the distance corresponds to using the initialization as the reference matrices while the Frobenius norm of the parameters corresponds to using the origin as the reference.
Revised: i for the bound: the distance corresponds to using the initialization as the reference matrices while the Frobenius norm of the parameters corresponds to using the origin as the reference.

Original: Since the

Original:  Frobenius norm of the parameters shows better correlation, we use zero reference matrices in the spectral bound.
Revised: Since the Frobenius norm of the parameters shows better correlation, we use zero reference matrices in the spectral bound.

Original: This improved both τ and Ψ, albeit still negative.
Revised: This improved both τ and Ψ , albeit still negative.

Original: In addition, we extensively investigate the eﬀect of diﬀerent terms of the Spectral bound to isolate the eﬀect; however, the results do not improve.
Revised: In addition, we extensively investigate the effect of different terms of the Spectral bound to isolate the effect; however, the results do not improve.

Original: We also found that the measure sum-of-fro/margin performs particularly well.

Original: This measure is invariant to rebalancing of layers but does not correspond to any existing bound.

Original: We believe investigating this quantity is an interesting direction for future work.

Original: These experiments can be found in the Appendix D.2.
Revised: These experiments can be found in the Appendix D.2.

Original: Path norm : While path-norm is a proper norm in the function space but not in parameter space, we observe that it is positively correlated with generalization in all hyper-parameter types and achieves comparable τ (0.373) and Ψ (0.311).
Revised: Path norm : While path-norm is a proper norm in the function space but not in parameter space, we observe that it is positively correlated with generalization in all hyper-parameter types and achieves comparable τ (0.373) and Ψ (0.311).

Original: Fisher-Rao metric :
Revised: Fisher-Rao metric :

Original: The Fisher-Rao metric is a lower bound (Liang et al., 2017) on the path norm that has been recently shown to capture generalization.
Revised: The Fisher-Rao metric is a lower bound (Liang et al., 2017) on the path norm that has been recently shown to capture generalization.

Original: We observed that it overall shows worse correlation than the path norm; in particular, it is negatively correlated( τ = − 0 . with the depth of the network while contrast with path norm which properly captures the eﬀect of depth on generalization.
Revised: We observed that it overall shows worse correlation than the path norm; in particular, it is negatively correlated with the depth of the network, which contrasts with path norm that properly captures the effect of depth on generalization.

Original: A more interesting observation is that the Fisher-Rao metric achieves a positive Ψ
Revised: A more interesting observation is that the Fisher-Rao metric achieves a positive Ψ

Original: = 0 . 

Revised: = 0 .

Original: but its τ = 0 .
Revised: but its τ = 0 .is

Original: 078 is essentially at chance.
Revised: essentially at chance.

Original: This may suggest that the metric can capture a single hyper-parameter change but is not able to capture the interactions between diﬀerent hyperparameter type.
Revised: This may suggest that the metric can capture a single hyper-parameter change but is not able to capture the interactions between different hyperparameter types .

Original: Eﬀect of Randomness : Dropout and batch size (ﬁrst 2 columns of Table 2) directly introduce randomness into the training dynamic.
Revised: Effect of Randomness : dropout and batch size (ﬁrst 2 columns of Table 2) directly introduce randomness into the training dynamic.

Original: For batch size, we observed that the Frobenius displacement and spectral complexity both correlate negatively with the changes in batch size while the Frobenius norm of the parameters correlates positively with generalization.
Revised: For batch size, we observed that the Frobenius displacement and spectral complexity both correlate negatively with the changes in batch size while the Frobenius norm of the parameters correlates positively with generalization.

Original: On the other hand, when changes happen to the magnitude dropout probability, we observed that all of the proper norms are negatively correlated with the generalization changes.
Revised: On the other hand, when changes happen to the magnitude dropout probability, we observed that all of the proper norms are negatively correlated with the generalization changes.

Original: Since increasing dropout usually reduces the generalization gap, this implies that increasing the dropout probability may be at least partially responsible for the growth in these norms.
Revised: Since increasing dropout usually reduces the generalization gap, this implies that increasing the dropout probability may be at least partially responsible for the growth in these norms.

Original: This is unexpected since increasing norm in principle implies higher model capacity which is usually more prone to overﬁtting.
Revised: This is unexpected since increasing norm in principle implies higher model capacity which is usually more prone to overﬁtting.

Revised: The overall picture does not change much going from the ranking correlation to mutual information, with a notable exception where spectral complexity has the highest conditional mutual information compared to all the other measures.

Original: Success of Sharpness-based Measures
Revised: This is due to the fact that the conditional mutual information is agnostic to the direction of correlation, and in the ranking correlation, spectral complexity has the highest absolute correlation. While this view might seem contradictory to classical view as the spectral complexity is a complexity measure which should be small to guarantee good generalization, it is nonetheless informative about the generalization of the model. Further, by inspecting the conditional mutual information for each hyperparameter, we ﬁnd that the majority of spectral complexity’s predictive power is due to its ability to capture the depth of the network, as the mutual information is signiﬁcantly lower if depth is already observed. 4. S UCCESS OF S HARPNESS -B ASED M EASURES

Original: A natural category of generalization measures is centered around the concept of “ sharpness ” of the local minima, capturing the sensitivity of the empirical risk (i.e. the loss over the entire training set) to perturbations in model parameters.
Revised: A natural category of generalization measures is centered around the concept of “ sharpness ” of the local minima, capturing the sensitivity of the empirical risk (i.e. the loss over the entire training set) to perturbations in model parameters.

Original: Such notion of stability under perturbation is captured elegantly by the PAC-Bayesian framework (McAllester, 1999) which has provided promising insights for studying generalization of deep neural networks (Dziugaite & Roy, 2017; Neyshabur et al., 2017; 2018a).
Revised: Such notion of stability under perturbation is captured elegantly by the PAC-Bayesian framework (McAllester, 1999) which has provided promising insights for studying generalization of deep neural networks (Dziugaite and Roy, 2017; Neyshabur et al., 2017; 2018a).

Original: In this sections, we investigate PAC-Bayesian generalization bounds and several of its variants which rely on diﬀerent priors and diﬀerent notions of sharpness (Table 3).
Revised: In this sections, we investigate PAC-Bayesian generalization bounds and several of their variants which rely on different priors and different notions of sharpness (Table 3).

Original: In order to evaluate a PAC-Bayesian bound, one needs to come up with a prior distribution over the parameters that is chosen in advance before observing the training set.
Revised: In order to evaluate a PAC-Bayesian bound, one needs to come up with a prior distribution over the parameters that is chosen in advance before observing the training set.

Original: Then, given any posterior distribution on the parameters which could depend on the training set, a PAC-Bayesian bound (Theorem 46) states that the expected generalization error of the parameters generated from the posterior can be bounded by the KL-divergence of the prior and posterior.
Revised: Then, given any posterior distribution on the parameters which could depend on the training set, a PAC-Bayesian bound (Theorem 46) states that the expected generalization error of the parameters generated from the posterior can be bounded by the KL -divergence of the prior and posterior.

Original: The posterior distribution can be seen as adding perturbation on ﬁnal parameters.
Revised: The posterior distribution can be seen as adding perturbation on ﬁnal parameters.

Original: Dziugaite & Roy (2017) shows contrary to other generalization bounds, it is possible to calculate non-vacuous PAC-Bayesian bounds by optimizing the bound over a large set of Gaussian posteriors.
Revised: Dziugaite and Roy (2017) shows contrary to other generalization bounds, it is possible to calculate non-vacuous PAC-Bayesian bounds by optimizing the bound over a large set of Gaussian posteriors.

Original: Neyshabur et al.
Revised: Neyshabur et al.

Original: (2017) demonstrates that PAC-Bayesian bounds where prior and posterior are isotropic Gaussian distributions are good measure of generalization on small scale experiments (e.q. 47).
Revised: (2017) demonstrates that when prior and posterior are isotropic Gaussian distributions, then PAC-Bayesian bounds are good measure of generalization on small scale experiments; see Eq (47).

Original: PAC-Bayesian framework captures sharpness in the expected sense since we add randomly generated perturbations to the parameters.
Revised: PAC-Bayesian framework captures sharpness in the expected sense since we add randomly generated perturbations to the parameters.

Original: Another possible notion of sharpness is the worst-case sharpness where we search for the direction that changes the loss the most.
Revised: Another possible notion of sharpness is the worst-case sharpness where we search for the direction that changes the loss the most.

Original: This is motivated by (Keskar et al., 2016) where they observe that this notion would correlate to generalization in the case of diﬀerent batch sizes.
Revised: This is motivated by (Keskar et al., 2016) where they observe that this notion would correlate with generalization in the case of different batch sizes.

Original: We can use PAC-Bayesian framework to construct generalization bounds for this worst-case perturbations as well.
Revised: We can use PAC-Bayesian framework to construct generalization bounds for this worst-case perturbations as well.

Original: We refer to this worst case bound as the sharpness bound (eq. 50).
Revised: We refer to this worst case bound as the sharpness bound in Eq (50).

Original: The main component in both PAC-Bayes and worst-case sharpness bounds is the ratio of norm of parameters to the magnitude of the perturbation where the magnitude is chosen to be the largest number so that the training error of the perturbed model is at most 0 .
Revised: The main component in both PAC-Bayes and worst-case sharpness bounds is the ratio of norm of parameters to the magnitude of the perturbation, where the magnitude is chosen to be the largest number such that the training error of the perturbed model is at most 0 .

Original: While mathematically, the sharpness bound should always yield higher complexity than the Pac-Bayes bound, we observed that the former has higher correlation both in terms of τ and Ψ.
Revised: While mathematically, the sharpness bound should always yield higher complexity than the PAC-Bayes bound, we observed that the former has higher correlation both in terms of τ and Ψ .

Original: In addition, we measured inverse of perturbation magnitude as a measure  removing the norm in the numerator to compare it with the bound and did not observe a signiﬁcant diﬀerence.
Revised: In addition, we studied inverse of perturbation magnitude as a measure by removing the norm in the numerator to compare it with the bound. However, we did not observe a signiﬁcant difference.

Original: [Table]
Revised: [Table]

Original: Magnitude-aware Perturbation Bounds Perturbing the parameters without taking their magnitude into account can cause many of them to switch signs.
Revised: Perturbing the parameters without taking their magnitude into account can cause many of them to switch signs.

Original: Therefore, one cannot apply large perturbations to the model without changing the loss signiﬁcantly.
Revised: Therefore, one cannot apply large perturbations to the model without changing the loss signiﬁcantly.

Original: One possible modiﬁcation to improve the perturbations is to choose the perturbation magnitude based on the magnitude of the parameter.
Revised: One possible modiﬁcation to improve the perturbations is to choose the perturbation magnitude based on the magnitude of the parameter.

Original: In that case, it is guaranteed that if the magnitude of perturbation is less than that of the parameter, then the sign of the parameter does not change.
Revised: In that case, it is guaranteed that if the magnitude of perturbation is less than the magnitude of the parameter, then the sign of the parameter does not change.

Original: Following Keskar et al.
Revised: Following Keskar et al.

Original: (2016), we pick the magnitude of the perturbation with respect to the magnitude of parameters.
Revised: (2016), we pick the magnitude of the perturbation with respect to the magnitude of parameters.

Original: We formalize this notion of importance based on magnitude and derive two alternative generalization bounds for expected sharpness (eq. 55) and worst case sharpness (eq. 58) that include the magnitude of the parameters into the prior.
Revised: We formalize this notion of importance based magnitude. Speciﬁcally, we derive two alternative generalization bounds for expected sharpness in Eq (55) and worst case sharpness in Eq (58) that include the magnitude of the parameters into the prior.

Original: Formally, we design α 0 and σ 0 to be the ratio of parameter magnitude to the perturbation magnitude in sharpness and PAC-Bayes bounds respectively.
Revised: Formally, we design α (cid:48) and σ (cid:48) , respectively for sharpness and PAC-Bayes bounds, to be the ratio of parameter magnitude to the perturbation magnitude.

Original: While this change did not improve upon the original PAC-Bayesian measures, we observed that  looking at 1 /α 0 has surprising predictive power in terms of the generalization which surpasses the performance of oracle 0.02.
Revised: While this change did not improve upon the original PAC-Bayesian measures, we observed that simply looking at 1 /α (cid:48) has surprising predictive power in terms of the generalization which surpasses the performance of oracle 0.02.

Original: This measure is  close the one suggested in Keskar et al.
Revised: This measure is very close to what was originally suggested in Keskar et al.

Revised: Its effectiveness is further corroborated by the conditional mutual information based metric, where we observed that 1 /α (cid:48) has the highest mutual information with generalization among all hyperparameters and also overall.

Original: Finding σ
Revised: 4.3. F INDING σ

Original: In case of models with extremely small loss, the perturbed loss should roughly increase monotonically w.r.t the perturbation scale.
Revised: In case of models with extremely small loss, the perturbed loss should roughly increase monotonically with respect to the perturbation scale.

Original: Leveraging this observation, we design algorithms for computing the perturbation scale σ such that the ﬁrst term on the RHS is as close to a ﬁxed value as possible for all models.
Revised: Leveraging this observation, we design algorithms for computing the perturbation scale σ such that the ﬁrst term on the RHS is as close to a ﬁxed value as possible for all models.

Original: In our experiments, we choose the deviation to be 0.1 which translates to 10% training error.
Revised: In our experiments, we choose the deviation to be 0.1 which translates to 10% training error.

Original: These search algorithms are paramount to compare measures between diﬀerent models.
Revised: These search algorithms are paramount to compare measures between different models.

Original: We provide the detailed algorithms in the Appendix E. To improve upon our algorithms, one could try a computational approach similar to Dziugaite & Roy (2017) to obtain a numerically better bound which may result in stronger correlation; however, due to practical computational constraints, we could not do so for the large number of models we consider.
Revised: We provide the detailed algorithms in the Appendix E. To improve upon our algorithms, one could try a computational approach similar to Dziugaite and Roy (2017) to obtain a numerically better bound which may result in stronger correlation. However, due to practical computational constraints, we could not do so for the large number of models we consider.

Original: Potential of Optimization-based Measures
Revised: 4. P OTENTIAL OF O PTIMIZATION - BASED M EASURES

Original: Optimization is an indispensable component of deep learning.
Revised: Optimization is an indispensable component of deep learning.

Original: Numerous optimizers have been proposed for more stable training and faster convergence.
Revised: Numerous optimizers have been proposed for more stable training and faster convergence.

Original: How the optimization scheme and speed of optimization inﬂuence generalization of a model has been a topic of contention among the deep learning community (Merity et al., 2017; Hardt et al., 2015).
Revised: How the optimization scheme and speed of optimization inﬂuence generalization of a model has been a topic of contention among the deep learning community (Merity et al., 2017; Hardt et al., 2015).

Original: We studyrepresentative optimizers Momentum SGD , Adam , and RMSProp with diﬀerent initial learning rates in our experiments to thoroughly evaluate this phenomenon.
Revised: We study 3 representative optimizers Momentum SGD , Adam , and RMSProp with different initial learning rates in our experiments to thoroughly evaluate this phenomenon.

Original: We also consider other optimization related measures that are believed to correlate with generalization.
Revised: We also consider other optimization related measures that are believed to correlate with generalization.

Original: These include (Table 4):
Revised: These include (Table 4):

Original: Number of iterations required to reach cross-entropy equals 0. 
Revised: Number of iterations required to reach cross-entropy equals 0. 2.

Original: Number of iterations required going from cross-entropy equals 0.1 to cross-entropy equals 0. 3.
Revised: Number of iterations required going from cross-entropy equals 0.1 to cross-entropy equals 0. 3.

Original: Variance of the gradients after only seeing the entire dataset once (1 epoch) 4.
Revised: Variance of the gradients after only seeing the entire dataset once (1 epoch) 4.

Original: Variance of the gradients when the cross-entropy is approximately 
Revised: Variance of the gradients when the cross-entropy is approximately 0.

Original: [Table]
Revised: [Table]

Original:  The number of iterations roughly characterizes the speed of optimization, which has been argued to correlate with generalization.
Revised: Number of Iterations: The number of iterations roughly characterizes the speed of optimization, which has been argued to correlate with generalization.

Original: For the models we consider, we observed that the initial phase (step to reach cross-entropy value of 0.1) of the optimization is negatively correlated with the speed of optimization for both τ and Ψ. This would suggest that the diﬃculty of optimization during the initial phase  optimization beneﬁts the ﬁnal generalization.
Revised: For the models considered here, we observed that the initial phase (to reach cross-entropy value of 0.1) of the optimization is negatively correlated with the speed of optimization for both τ and Ψ . This would suggest that the difﬁculty of optimization during the initial phase of the optimization beneﬁts the ﬁnal generalization.

Original: On the other hand, the speed of optimization going from cross-entropy 0.1 to cross-entropy 0. does not seem to be correlated with the generalization of the ﬁnal solution.
Revised: On the other hand, the speed of optimization going from cross-entropy 0.1 to cross-entropy 0.01 does not seem to be correlated with the generalization of the ﬁnal solution.

Original: On the other hand, the speed of optimization is not an explicit capacity measure so either positive or negative correlation could potentially be informative.
Revised: Importantly, the speed of optimization is not an explicit capacity measure so either positive or negative correlation could potentially be informative .

Original: On the other hand, the variance of the gradients characterizes the smoothness of the loss surface.
Revised: Variance of Gradients:

Original: Towards the end of the training, the variance of the gradients also captures a particular type of “ﬂatness” of the local minima.
Revised: Towards the end of the training, the variance of the gradients also captures a particular type of “ﬂatness” of the local minima.

Original: This measure is surprisingly predictive of the generalization both in terms of τ and Ψ, and more importantly, is positively correlated across every type of hyperparameter.
Revised: This measure is surprisingly predictive of the generalization both in terms of τ and Ψ , and more importantly, is positively correlated across every type of hyperparameter .

Original: To the best of our knowledge, this is the ﬁrst time this phenomenon has been observed.
Revised: To the best of our knowledge, this is the ﬁrst time this phenomenon has been observed.

Original: The connection between variance of the gradient and generalization is perhaps natural since much of the recent advancement in deep learning such as residual network (He et al., 2016) or batch normalization have enabled using larger learning rates to train neural networks.
Revised: The connection between variance of the gradient and generalization is perhaps natural since much of the recent advancement in deep learning such as residual networks (He et al., 2016) or batch normalization have enabled using larger learning rates to train neural networks.

Original: Stability with higher learning rates implies smaller noises in the minibatch gradient.
Revised: Stability with higher learning rates implies smaller noises in the minibatch gradient.

Revised: With the mutual information metric, the overall observation is consistent with that of ranking correlation, but the ﬁnal gradient noise also outperforms gradient noise at 1 epoch of training conditioned on the dropout probability.

Original: We hope that our work encourages future works in other possible measures based on optimization and during training.
Revised: We hope that our work encourages future works in other possible measures based on optimization and during training.

Original:  Conclusion
Revised:  C ONCLUSION

Original: We conducted large scale experiments to test the correlation of diﬀerent measures with the generalization of deep models and propose a framework to better disentangle the cause of correlation from spurious correlation.
Revised: We conducted large scale experiments to test the correlation of different measures with the generalization of deep models and propose a framework to better disentangle the cause of correlation from spurious correlation.

Original: We conﬁrmed the eﬀectiveness of the PAC-Bayesian bounds through our experiments and corroborate it as a promising direction for cracking the generalization puzzle.
Revised: We conﬁrmed the effectiveness of the PAC-Bayesian bounds through our experiments and corroborate it as a promising direction for cracking the generalization puzzle.

Original: Further, we provide an extension to existing PAC-Bayesian bounds that consider the importance of each parameter.
Revised: Further, we provide an extension to existing PAC-Bayesian bounds that consider the importance of each parameter.

Original: We also found that several measures related to optimization are surprisingly predictive of generalization and worthy of further investigation.
Revised: We also found that several measures related to optimization are surprisingly predictive of generalization and worthy of further investigation.

Original: On the other hand, several surprising failures about the norm-based measures were uncovered.
Revised: On the other hand, several surprising failures about the norm-based measures were uncovered.

Original: In particular, we found that regularization that introduces randomness into the optimization can increase various norm of the models and spectral complexity related norm-based measures are unable to capture generalization – in fact, most of them are negatively correlated.
Revised: In particular, we found that regularization that introduces randomness into the optimization can increase various norm of the models and spectral complexity related norm-based measures are unable to capture generalization – in fact, most of them are negatively correlated.

Original: Our experiments demonstrate that the study of generalization measure can be misleading when the number of models studied is small and the metric of quantifying the relationship is not carefully chosen.
Revised: Our experiments demonstrate that the study of generalization measure can be misleading when the number of models studied is small and the metric of quantifying the relationship is not carefully chosen.

Original: We hope this work will incentivize more rigorous treatment of generalization measures in future work.
Revised: We hope this work will incentivize more rigorous treatment of generalization measures in future work.

Original: To the best of our knowledge, this work is one of the most comprehensive study of generalization to date, but there are a few short-comings.
Revised: To the best of our knowledge, this work is one of the most comprehensive study of generalization to date, but there are a few short-comings.

Original: Due to computational constraints, we were only able to study 7 most common hyperparameter types and relatively smaller architectures, which do not reﬂect the models used in production.
Revised: Due to computational constraints, we were only able to study 7 most common hyperparameter types and relatively small architectures, which do not reﬂect the models used in production.

Original: Indeed, if more hyperparameters are considered, one could expect to better capture the causal relationship.
Revised: Indeed, if more hyperparameters are considered, one could expect to better capture the causal relationship.

Original: We also only studiedclassiﬁcation models  on  convolutional networks.
Revised: We also only studied models trained on two image datasets (CIFAR-10 and SVHN), only classiﬁcation models and only convolutional networks.

Original: Future work will aim to address these limitations.
Revised: We hope that future work would address these limitations.

Original: preprint
Revised: A CKNOWLEDGEMENT

Revised: We thank our colleagues at Google: Guy Gur-Ari for many insightful discussions that helped with the experiment design, Ethan Dyer, Pierre Foret, Sergey Ioffe for their feedback, and Scott Yak for help with implementation. We are grateful for insightful discussions with Brady Neal of University of Montreal about limitation of correlation analysis. We also thank Daniel Roy of University of

Revised: Toronto for insightful comments

Revised: A N OTATION

Revised: We denote a probability distribution as A , set as A , tensor as A , vector as a , and scalar as a or α .

Revised: Let D denote the data distributions over inputs and their labels, and let κ denote number of classes.

Revised: We use (cid:44) for equality by deﬁnition.

Revised: We denote by S a given dataset, consisting of m i.i.d tuples { ( X 1 , y 1 ) , . . .

Revised: , ( X m , y m ) } drawn from D where X i ∈ X is the input data and y i ∈ { 1 , . . .

Revised: , κ } the corresponding class label.

Revised: We denote a feedforward neural network by f w : X → R κ , its weight parameters by w , and the number of weights by ω (cid:44) dim( w ) .

Revised: No activation function is applied at the output (i.e. logits).

Revised: Denote the weight tensor of the i th layer of the network by W i , so that w = vec( W 1 , . .

Revised: . , W d ) , where d is the depth of the network, and vec represents the vectorization operator.

Revised: Furthermore, denote by f w ( X )[ j ] the j -th output of the function f w ( X ) .

Revised: Let R be the set of binary relations, and I : R → { 0 , 1 } be the indicator function that is 1 if its input is true and zero otherwise.

Revised: Let L be the 1-0 classiﬁcation loss over the data distribution D :

Revised: L ( f w ) (cid:44) E ( X ,y ) ∼ D (cid:2)

Revised: I (cid:0) f w ( X )

Revised: [ y ] ≤ max j (cid:54) =

Revised: y f w ( X )[ j ] (cid:1)(cid:3) and let ˆ L be the empirical estimate of 1loss over S :

Original: {+ˆ L ( f w ) (cid:44) 1 m (cid:80) mi =
+}
Revised: {+ˆ L ( f w ) (cid:44) 1 m (cid:80) mi =
+}

Revised: I (cid:0) f w ( X )[ y i ] ≤ max j (cid:54) =

Revised: y i f w ( X )

Revised: [ j ] (cid:1) .

Revised: We refer to L ( f w ) − ˆ L ( f w ) as the generalization gap .

Revised: For any input X , we deﬁne the sample dependent margin 6 as γ ( X ) (cid:44) (cid:0) f w ( X ) (cid:1)

Revised: [ y ] − max i (cid:54) =

Revised: y f w ( X ) i .

Revised: Moreover, we deﬁne the overall margin γ as the 10 th percentile (a robust surrogate for the minimum) of γ ( X ) over the entire training set S .

Revised: Given any margin value γ ≥ 0 , we deﬁne the margin loss L γ as follows:

Revised: [Equation]

Revised: (6) and ˆ L γ is deﬁned in an analogous manner on the training set.

Revised: Further, for any vector v , we denote by (cid:107) v (cid:107) 2 the (cid:96) 2 norm of v .

Revised: For any tensor W , let (cid:107) W (cid:107) F (cid:44) (cid:107) vec( W ) (cid:107) .

Revised: We also denote (cid:107) W (cid:107) 2 as the spectral norm of the tensor W when used with a convolution operator.

Revised: For convolutional operators, we compute the true singular value with the method proposed by Sedghi et al.

Revised: (2018) through FFT.

Revised: We denote a tensor as A , vector as a , and scalar as A or a .

Revised: For any 1 ≤ j ≤ k , consider a k -th order tensor A and a j -th order tensor B where dimensions of B match the last j dimensions of A .

Revised: We then deﬁne the product operator ⊗ j :

Revised: [Equation]

Revised: (7) where i 1 , . .

Revised: , i k − j are indices.

Revised: We also assume that the input images have dimension n × n and there are κ classes.

Revised: Given the number of input channels c in , number of output channels c out , 2D square kernel with side length k , stride s , and padding p , we deﬁne the convolutional layer conv W ,s,p as follows:

Revised: [Equation]

Revised: where W ∈ R c out × c in × k × k is the convolutional parameter tensor, patch i,j,k ( Z ) is a k × k patch of

Revised: Z starting from the point ( i, j ) , and pad p is the padding operator which adds p zeros to top, bottom, left and right of X :

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: p otherwise .

Revised: We also deﬁne the max-pooling operator pool k,s,p as follows:

Revised: [Equation]

Revised: We denote by f W ,s a convolutional network

Original: arXiv
Revised: such that W i ∈ R c i × c

Revised: i − 1 × k

Revised: i × k

Revised: i is the convolution tensor and s i is the convolutional stride at layer i .

Revised: At Layer i , we assume the sequence of convolution,

Revised: ReLU and max-pooling where the max pooling has kernel k (cid:48)

Revised: i and stride s (cid:48) i .

Revised: Lack of max-pooling in some layers can be achieved by setting k (cid:48)

Original: A Experiments A. More training details
Revised: i = s (cid:48) i = 1 . We consider classiﬁcation tasks and denote the number of classes by κ . B C ONDITIONAL I NDEPENDENCE T EST : T OWARDS C APTURING THE C AUSAL R ELATIONSHIPS

Revised: Relying on correlation is intuitive but perhaps unsatisfactory.

Revised: In our experiments, we change several hyper-parameters and assess the correlation between a complexity measure and generalization.

Revised: When we observe correlation between a complexity measure and generalization, we want to differentiate the following two scenarios:

Revised: • Changing a hyper-parameter causes the complexity measure to be low and lower value of the measure causes the generalization gap to be low.

Revised: • Changing a hyper-parameter causes the complexity measure to be low and changing the same hyper-parameter also causes the generalization to be low but the lower value of the complexity measure by itself has no effect on generalization.

Revised: The above two scenarios are demonstrated in Figure 2-Middle and Figure 2-Right respectively.

Revised: In attempt to truly understand these relationships, we will rely on the tools from probabilistic causality.

Revised: Our approach is inspired by the seminal work on Inductive Causation (IC) Algorithm by Verma and Pearl (1991), which provides a framework for learning a graphical model through conditional independence test.

Revised: While the IC algorithm traditionally initiates the graph to be fully connected, we will take advantage of our knowledge about generalization and prune edges of the initialized graph to expedite the computations.

Revised: Namely, we assume that the choice of hyperparameter does not directly explain generalization, but rather it induces changes in some measure µ which can be used to explain generalization.

Revised: [Figure]

Revised: Our primary interest is to establish the existence of an edge between µ and g .

Revised: Suppose there exists a large family of complexity measures and among them there is a true complexity measure that can fully explain generalization.

Revised: Then to verify the existence of the edge between µ and g , we can perform the conditional independent test by reading the conditional mutual information between µ and g given that a set of hyperparameter types S is observed 7 .

Revised: For any function φ : Θ → R , let

Revised: V φ : Θ 1 × Θ 2 → { +1 , − 1 } be as V φ ( θ 1 , θ 2 ) (cid:44) sign( φ ( θ 1 ) − φ ( θ 2 )) .

Revised: Furthermore, let U S be a random variable that correspond to the values of hyperparameters in S .

Revised: We calculate the conditional mutual information as follows:

Revised: [Equation]

Revised: The above removes the unwanted correlation between generalization and complexity measure that is caused by hyperparameter types in set S .

Revised: Since in our case the conditional mutual information between a complexity measure and generalization is at most equal to the conditional entropy of generalization, we normalize it with the conditional entropy to arrive at a criterion ranging between 0 and 1:

Revised: [Equation]

Revised: [Equation]

Revised: According to the IC algorithm, an edge is kept between two nodes if there exists no subset S of hyperparameter types such that the two nodes are independent, i.e. ˆ I ( V µ , V g | U S ) = 0 .

Revised: In our setup, setting S to the set of all hyperparameter types is not possible as both the conditional entropy and conditional mutual information would become zero.

Revised: Moreover, due to computational reasons, we only look at |S| ≤ 2 :

Revised: [Equation]

Revised: At a high level, the larger K is for a measure µ , the more likely an edge exists between µ and g , and therefore the more likely µ can explain generalization.

Revised: For details on the set-up, please refer to

Revised: Appendix B.1 on how these quantities are estimated.

Revised: B. D EFINITION OF R ANDOM V ARIABLES

Revised: Since the measures are results of complicated interactions between the data, the model, and the training procedures, we cannot manipulate it to be any values that we want.

Revised: Instead, we use the following deﬁnition of random variables: suppose S is a subset of all the components of θ (e.g.

Revised: S = { ∅ } for |S| = 0 , |S| = { learning rate } for |S| = 1 or |S| = { learning rate, dropout } for |S| = 2 ).

Revised: Speciﬁcally we denote S ab as the collective condition { θ ( a )

Revised: [Equation]

Revised: b

Revised: [Equation]

Revised: 1 = v 2 , . .

Revised: , θ ( a ) |S| = v 2 |S|− 1 , θ ( b ) |S| = v 2 |S| } .

Revised: We can then deﬁne and empirical measure four probability Pr ( µ ( a ) > µ ( b ) , g ( a ) > g ( b ) | S ab ) , Pr ( µ ( a ) > µ ( b ) , g ( a ) <

Revised: g ( b ) | S ab ) , Pr ( µ ( a ) < µ ( b ) , g ( a ) > g ( b ) | S ab ) and

Revised: Pr ( µ ( a ) < µ ( b ) , g ( a ) <

Revised: g ( b ) | S ab ) .

Revised: µ ( a ) > µ ( b ) µ ( a ) ≤ µ ( b ) g ( a ) > g ( b ) ppg ( a ) ≤ g ( b ) pp

Revised: Together forms a 2 by 2 table that deﬁnes the joint distribution of the Bernoulli random variables

Revised: Pr ( g ( a ) > g ( b ) | S ab ) and Pr ( µ ( a ) > µ ( b ) | S ab ) .

Revised: For notation convenience, we use Pr ( µ, g | S ab ) ,

Revised: Pr ( g | S ab ) and Pr ( µ | S ab ) to denote the joint and marginal.

Revised: If there are N = 3 choices for each hyperparameter in S then there will be N |S| such tables for each hyperparameter combination.

Revised: Since each conﬁguration occurs with equal probability, for that arbitrary θ ( a ) and θ ( b ) drawn from Θ conditioned on that the components of S are observed for both models, the joint distribution can be deﬁned as Pr ( µ, g | S )

Revised: = 1 N |S| (cid:80) S ab Pr ( µ, g | S ab ) and likely the marginals can be deﬁned as

Revised: Pr ( µ | S ) = 1 N |S| (cid:80) S ab Pr ( µ | S ab ) and Pr ( g | S ) = 1 N |S| (cid:80) S ab Pr ( g | S ab ) .

Revised: With these notations established, all the relevant quantities can be computed by iterating over all pairs of models.

Revised: C E XPERIMENTS C. M ORE TRAINING DETAILS

Original: During our experiments, we found that Batch Normalization (Ioﬀe & Szegedy, 2015) is crucial to reliably reach a low cross-entropy value for all models; since normalization is a indispensable components of modern neural networks, we decide to use batch normalization in all of our models.
Revised: During our experiments, we found that Batch Normalization (Ioffe and Szegedy, 2015) is crucial to reliably reach a low cross-entropy value for all models; since normalization is a indispensable components of modern neural networks, we decide to use batch normalization in all of our models.

Original: We remove batch normalization before computing any measure by fusing the γ , β and moving statistics with the convolution operator that precedes the normalization.
Revised: We remove batch normalization before computing any measure by fusing the γ , β and moving statistics with the convolution operator that precedes the normalization.

Original: This is important as Dinh et al.
Revised: This is important as Dinh et al.

Original: (2017) showed that common generalization measures such as sharpness can be easily manipulated with re-parameterization.
Revised: (2017) showed that common generalization measures such as sharpness can be easily manipulated with reparameterization.

Original: We also discovered that the models trained with data augmentation often cannot ﬁt the data (i.e. reach cross-entropy 0.01) completely.
Revised: We also discovered that the models trained with data augmentation often cannot ﬁt the data (i.e. reach cross-entropy 0.01) completely.

Original: Since a model with data augmentation tends to consistently generalize better than the models without data augmentation, measure that reﬂects the training error (i.e. value of cross-entropy) will easily predict the ranking between two models even though it has only learned that one model uses data augmentation (see the thought experiments from the previous section).
Revised: Since a model with data augmentation tends to consistently generalize better than the models without data augmentation, measure that reﬂects the training error (i.e. value of cross-entropy) will easily predict the ranking between two models even though it has only learned that one model uses data augmentation (see the thought experiments from the previous section).

Original: While certain hyperparameter conﬁguration can reach crossentropy of 0.01 even with data augmentation, it greatly limits the space of models that we can study.
Revised: While certain hyperparameter conﬁguration can reach cross-entropy of 0.01 even with data augmentation, it greatly limits the space of models that we can study.

Original: Hence, we make the design choice to not include data augmentation in the models of this study.
Revised: Hence, we make the design choice to not include data augmentation in the models of this study.

Original: Note that from a theoretical perspective, data augmentation is also challenging to analyze since the training samples generated from the procedure are no longer identical and independently distributed.
Revised: Note that from a theoretical perspective, data augmentation is also challenging to analyze since the training samples generated from the procedure are no longer identical and independently distributed.

Original: All values for all the measures we computed over these models can be found in Table 5 in Appendix A.5.
Revised: All values for all the measures we computed over these models can be found in Table 5 in Appendix C.5.

Original: A. The choice of stopping criterion
Revised: C. T HE CHOICE OF STOPPING CRITERION

Original: The choice of stopping criterion is very essential and could completely change the evaluation and the resulting conclusions.
Revised: The choice of stopping criterion is very essential and could completely change the evaluation and the resulting conclusions.

Original: In our experiments we noticed that if we pick the stopping criterion based on number of iterations or number of epochs, then since some models optimize faster than others, they end up ﬁtting the training data more and in that case the crossentropy itself can be very predictive of generalization.
Revised: In our experiments we noticed that if we pick the stopping criterion based on number of iterations or number of epochs, then since some models optimize faster than others, they end up ﬁtting the training data more and in that case the cross-entropy itself can be very predictive of generalization.

Original: To make it harder to distinguish models based on their training performance, it makes more sense to choose the stopping criterion based on the training error or training loss.
Revised: To make it harder to distinguish models based on their training performance, it makes more sense to choose the stopping criterion based on the training error or training loss.

Original: We noticed that as expected, models with the same cross-entropy usually have very similar training error so that suggests that this choice is not very important.
Revised: We noticed that as expected, models with the same cross-entropy usually have very similar training error so that suggests that this choice is not very important.

Original: However, during the optimization the training error behavior is noisier than cross-entropy and moreover, after the training error reaches zero, it cannot distinguish models while the cross-entropy is still meaningful after ﬁtting the data.
Revised: However, during the optimization the training error behavior is noisier than cross-entropy and moreover, after the training error reaches zero, it cannot distinguish models while the cross-entropy is still meaningful after ﬁtting the data.

Original: Therefore, we decided to use cross-entropy as the stopping criterion.
Revised: Therefore, we decided to use cross-entropy as the stopping criterion.

Original: A. All Model Specification
Revised: C. A LL M ODEL S PECIFICATION

Original: As mentioned in the main text, the models we use resemble Network-in-Network (Gao et al., 2011) which is a class of more parameter eﬃcient convolution neural networks that achieve reasonably competitive performance on modern image classiﬁcation benchmarks.
Revised: As mentioned in the main text, the models we use resemble Network-in-Network (Gao et al., 2011) which is a class of more parameter efﬁcient convolution neural networks that achieve reasonably competitive performance on modern image classiﬁcation benchmarks.

Original: The model consists blocks of modules that have 1 3 × 3 convolution with stride 2 followed by 2 1 ×convolution with stride 1.
Revised: The model consists blocks of modules that have 1 3 × 3 convolution with stride 2 followed by 2 1 × 1 convolution with stride 1.

Original: We refer to this single module as a NiN-block and construct models of diﬀerent size by stacking NiN-block.
Revised: We refer to this single module as a NiN-block and construct models of different size by stacking NiN-block.

Original: For simplicity, all NiN-block have the same number of output channels c out .
Revised: For simplicity, all NiN-block have the same number of output channels c out .

Original: Dropout is applied at the end of every NiN-block.
Revised: Dropout is applied at the end of every NiN-block.

Original: At the end of the model, there is a 1 × 1 convolution reducing the channel number to the class number (i.e. 10 for CIFAR-10) followed by a global average pooling to produce the output logits.
Revised: At the end of the model, there is a 1 × 1 convolution reducing the channel number to the class number (i.e. 10 for CIFAR-10) followed by a global average pooling to produce the output logits.

Original: For width, we choose from c out from 3 options: { 2 × 96 , 4 × 96 , 8 × 96 } .
Revised: For width, we choose from c out from 3 options: { 2 × 96 , 4 × 96 , 8 × 96 } .

Original: For depth, we choose from 3 options: { 2 × NiNblock , 4 × NiNblock , 8 × NiNblock } For dropout, we choose from 3 options: { 0 . 0 , 0 . 25 , 0 . 5 } For batch size, we choose from: { 32 , 64 , 128 }
Revised: For depth, we choose from 3 options: { 2 × NiNblock , 4 × NiNblock , 8 × NiNblock } For dropout, we choose from 3 options: { 0 . 0 , 0 . 25 , 0 . 5 } For batch size, we choose from: { 32 , 64 , 128 }

Original: Since each optimizer may require diﬀerent learning rate and in some cases, diﬀerent regularization, we ﬁne-tuned the hyper-parameters for each optimizer while keeping 3 options for every hyper-parameter choices 5 .
Revised: Since each optimizer may require different learning rate and in some cases, different regularization, we ﬁne-tuned the hyper-parameters for each optimizer while keeping 3 options for every hyperparameter choices 8 .

Original: Momentum SGD : We choose momentum of 0.9 and choose the initial learning rate η from { 0 . 1 , 0 . 032 , 0 . 01 } and regularization coeﬃcient λ from { 0 . 0 ,0 . 0001 , 0 . 0005 } .
Revised: Momentum SGD : We choose momentum of 0.9 and choose the initial learning rate η from { 0 . 1 , 0 . 032 , 0 . 01 } and regularization coefﬁcient λ from { 0 . 0 ,

Original: The learning rate decay schedule is ×0 .
Revised: The learning rate decay schedule is ×

Original: 1 at iterations [60000 ,90000].
Revised: 1 at iterations [60000 ,

Original: Adam : We choose initial learning rate η from { 0 . 001 , 3 . 2 e − 4 , 1 e − 4 } , (cid:15) = 1 e − 3 and regularization coeﬃcient λ from { 0 . 0 , 0 . 0001 , 0 . 0005 } .
Revised: Adam : We choose initial learning rate η from { 0 . 001 , 3 . 2 e − 4 , 1 e − 4 } , (cid:15) = 1 e − 3 and regularization coefﬁcient λ from { 0 . 0 , 0 . 0001 , 0 . 0005 } .

Original: The learning rate decay schedule is × 0 .
Revised: The learning rate decay schedule is × 0 .

Original: 1 at iterations [60000 ,90000].
Revised: 1 at iterations [60000 ,

Original: RMSProp : We choose initial learning rate η from { 0 . 001 , 3 . 2 e − 4 , 1 e − 4 } and regularization coeﬃcient λ from { 0 . 0 , 0 . 0001 , 0 . 0003 } .
Revised: RMSProp : We choose initial learning rate η from { 0 . 001 , 3 . 2 e − 4 , 1 e − 4 } and regularization coefﬁcient λ from { 0 . 0 , 0 . 0001 , 0 . 0003 } .

Original: The learning rate decay schedule is × 0 .
Revised: The learning rate decay schedule is × 0 .

Original: 1 at iterations [60000 , 90000].
Revised: 1 at iterations [60000 , rate decay.

Original: A. Canonical Measures
Revised: C. C ANONICAL M EASURES

Original: Based on empirical observations made by the community as a whole, the canonical ordering we give to each of the hyper-parameter categories are as follows:
Revised: Based on empirical observations made by the community as a whole, the canonical ordering we give to each of the hyper-parameter categories are as follows:

Original: Batchsize: smaller batchsize leads to smaller generalization gap 2.
Revised: Batchsize: smaller batchsize leads to smaller generalization gap 2.

Original: Depth: deeper network leads to smaller generalization gap 3.
Revised: Depth: deeper network leads to smaller generalization gap 3.

Original: Width: wider network leads to smaller generalization gap 4.
Revised: Width: wider network leads to smaller generalization gap 4.

Original: Dropout: The higher the dropout ( ≤ 0 . 5) the smaller the generalization gap 5.
Revised: Dropout: The higher the dropout ( ≤ 0 . 5 ) the smaller the generalization gap 5.

Original: Weight decay: The higher the weight decay (smaller than the maximum for each optimizer) the smaller the generalization gap 6.
Revised: Weight decay: The higher the weight decay (smaller than the maximum for each optimizer) the smaller the generalization gap 6.

Original: Learning rate: The higher the learning rate (smaller than the maximum for each optimizer) the smaller the generalization gap 7.
Revised: Learning rate: The higher the learning rate (smaller than the maximum for each optimizer) the smaller the generalization gap 7.

Original: Optimizer: Generalization gap of Momentum SGD <
Revised: Optimizer: Generalization gap of Momentum SGD <

Original: Generalization gap of Adam < Generalization gap of RMSProp
Revised: Generalization gap of Adam < Generalization gap of RMSProp

Original: A. All results
Revised: C. A LL R ESULTS

Original: Below we present all of the measures we computed and their respective τ and Ψ on the 2000+ models we trained and additional plots.
Revised: Below we present all of the measures we computed and their respective τ and Ψ on more than 10, models we trained and additional plots.

Revised: Unless stated otherwise, convergence is considered when the loss reaches the value of 0 . 1 .

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] entropy

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Relying on correlation is intuitive but perhaps unsatisfactory. In our experiments, we change several hyper-parameters and assess the correlation between a complexity measure and generalization. When we observe correlation between a complexity measure and generalization, we want to diﬀerentiate the following two scenarios:
Revised: [Table]

Original: • Changing a hyper-parameter causes the complexity measure to be low and lower value of the measure causes the generalization gap to be low. • Changing a hyper-parameter causes the complexity measure to be low and changing the same hyper-parameter also causes the generalization to be low but the lower value of the complexity measure by itself has no eﬀect on generalization. The above two scenarios are demonstrated in Figure 3.
Revised: [Table]

Original: In attempt to truly understand these relationships, we will rely on the tools from probabilistic causality. Our approach is inspired by the seminal work on Inductive Causation (IC) Algorithm by Verma & Pearl (1991), which provides a framework for learning a graphical model through conditional independence test. While the IC algorithm traditionally initiates the graph to be fully connected, we will take advantage of our knowledge about generalization and prune edges of the initialized graph to expedite the computations. Namely, we assume that the choice of hyperparameter does not directly explain generalization, but rather it induces changes in some measure µ which can be used to explain generalization.
Revised: [Table]

Original: [Figure]
Revised: [Table]

Original: Our primary interest is to establish the existence of an edge between µ and g . Suppose there exists a large family of complexity measures and among them there is a true complexity measure µ ∗ that can fully explain generalization. Then to verify the existence of the edge between µ and g , we can perform the conditional independent test by reading the conditional mutual information between µ and g given that a set of hyperparameter S is observed (e.g. S contains a single hyperparameter such as the learning rate).
Revised: [Table]

Original: For any function φ : Θ → R , let V φ ( θ 1 , θ 2 ) = sign( φ ( θ 1 ) − φ ( θ 2 )). Furthermore, let U S be a random variable that correspond to the values of hyperparameters in S . We calculate the conditional mutual information as follows:
Revised: [Table]

Original: [Equation]
Revised: [Table]

Original: Since in our case the conditional mutual information between a complexity measure and generalization is at most equal to the conditional entropy of generalization, we normalize it with the conditional entropy to arrive at a criterion ranging between 0 and 1:
Revised: [Table]

Original: [Equation]
Revised: [Table]

Original: According to the IC algorithm, an edge is kept between two nodes if there exists no subset S of hyperparameters such that the two nodes are independent, i.e. ˆ I ( V µ , V g | U S )
Revised: [Table]

Original: = 0.
Revised: [Table]

Original: In our setting, letting S be the set of all hyperparameters is not possible as both the conditional entropy and conditional mutual information would become zero. Moreover, due to computational reasons, we only look at |S| ≤ 2:
Revised: [Table]

Original: [Equation]
Revised: [Table]

Original: On a high level, the larger K is for a measure µ , the more likely an edge exists between µ and g , and therefore the more likely µ can explain generalization. For details on the set-up, please refer to Appendix B.1 on how these quantities are estimated. B. Definition of Random Variables Since the measures are results of complicated interactions between the data, the model, and the training procedures, we cannot manipulate it to be any values that we want.
Revised: [Table]

Original: Instead, we use the following deﬁnition of random variables: suppose S is a subset of all the components of θ (e.g. S = { ∅ } for |S| = 0, |S| = { learning rate } for |S| = 1 or |S| = { learning rate, dropout } for |S| = 2 ). Speciﬁcally we denote S ab as the collective condition { θ ( a )= v 1 , θ ( b )=
Revised: [Table]

Original: v 2 , . . . , θ ( a ) |S| = v 2 |S|− 1 , θ ( b ) |S| = v 2 |S| } .
Revised: [Table]

Original: We can then deﬁne and empirical measure four probability Pr( µ ( a ) > µ ( b ) , g ( a ) > g ( b ) | S ab ), Pr( µ ( a ) > µ ( b ) , g ( a ) < g ( b ) | S ab ), Pr( µ ( a ) < µ ( b ) , g ( a ) > g ( b ) | S ab ) and Pr( µ ( a ) < µ ( b ) , g ( a ) < g ( b ) | S ab ).
Revised: [Table]

Original: [Equation]
Revised: [Table]

Original: > [Equation] [Figure] [Equation]
Revised: N/A

Original: [Figure] [Equation] [Figure] Together forms a 2 by 2 table that deﬁnes the joint distribution of the Bernoulli random variables Pr( g ( a ) > g ( b ) | S ab ) and Pr( µ ( a ) > µ ( b ) | S ab ).
Revised: N/A

Original: For notation convenience, we use Pr( µ, g | S ab ) , Pr( g | S ab ) and Pr( µ | S ab ) to denote the joint and marginal. If there are N =choices for each hyperparameter in S then there will be N |S| such tables for each hyperparameter combination. Since each conﬁguration occurs with equal probability, for that arbitrary θ ( a ) and θ ( b ) drawn from Θ conditioned on that the components of S are observed for both models, the joint distribution can be deﬁned as Pr( µ, g | S ) =
Revised: N/A

Original: [Equation]
Revised: [Table] [Table]

Original: N
Revised: spectralcomplexity0reflast

Original: [Equation]
Revised: [Table]

Original: P
Revised: spectralcomplexity0reflast

Original: [Equation]
Revised: [Table]

Original: Pr
Revised: spectralproduct

Original: [Equation]
Revised: [Table]

Original: µ
Revised: spectralproductom

Original: [Equation]
Revised: [Table]

Original: g | S ab ) and likely the marginals can be deﬁned as Pr( µ | S )
Revised: spectralproducedd/ [Table] spectralsum

Original: = 1 N |S| P S ab Pr( µ | S ab ) and Pr( g | S ) = N |S| P S ab Pr( g | S ab ). With these notations established, all the relevant quantities can be computed by iterating over all pairs of models.
Revised: [Table]

Original: B. Results
Revised: frobproduct

Original: Table 10 contains the results from conditional independence test which is largely consistent with that of ranking correlation.
Revised: [Table] frobproductom

Original: [Table]
Revised: [Table]

Original: Given any margin value γ ≥ 0, we deﬁne the margin loss L γ as follows:
Revised: converged to Loss = 0.1 .

Original: [Equation]
Revised: [Table] [Table]

Original: and ˆ L γ is deﬁned in an analogous manner on the training set. Further, for any vector v , we denote by k v k 2 the ‘ 2 norm of v .
Revised: frnormlogitmargin

Original: For any tensor W , let k W k F (cid:44) k vec( W ) k .
Revised: [Table] [Table] [Table]

Original: We also denote k W k 2 as the spectral norm of the tensor W when used with a convolution operator. For convolutional operators, we compute the true singular value with the method proposed by Sedghi et al.
Revised: cross- entropy 0.

Original: (2018) through FFT.
Revised: [Table]

Original: We denote a tensor as A , vector as a , and scalar as A or a . For any 1 ≤ j ≤ k , consider a k -th order tensor A and a j -th order tensor B where dimensions of B match the last j dimensions of A . We then deﬁne the product operator ⊗ j :
Revised: [Table]

Original: [Equation]
Revised: [Table]

Original: where i 1 , .
Revised: 1/sigmapacbayes 0.

Original: , i k − j are indices. We also assume that the input images have dimension n × n and there are κ classes.
Revised: [Table] [Table] /sigmasharpness

Original: Given the number of input channels c in , number of output channels c out , 2D square kernel with side length k , stride s , and padding p , we deﬁne the convolutional layer conv W ,s,p as follows:
Revised: [Table] [Table] num-step

Original: [Equation]
Revised: [Table]

Revised: loss

Original: [Equation]
Revised: [Table]

Original: where W ∈ R c out × c in × k × k is the convolutional parameter tensor, patch i,j,k ( Z ) is a k × k patch of Z starting from the point ( i, j ), and pad p is the padding operator which adds p zeros to top, bottom, left and right of X :
Revised: stepto

Original: [Equation]
Revised: [Table]

Original: We also deﬁne the max-pooling operator pool k,s,p as follows:
Revised: num-step-to-

Original: [Equation]
Revised: [Table]

Original: We denote by f W ,s a convolutional network such that W i ∈
Revised: -loss

Original: R c i × c
Revised: [Table]

Original: i − 1 × k
Revised: /alphasharpnessmag

Original: i × k
Revised: [Table]

Original: [-i is the convolution tensor and s i is the convolutional stride at layer i . At Layer i , we assume the sequence of convolution, ReLU and max-pooling where the max pooling has kernel k 
 i and stride s 0 i . Lack of max-pooling in some layers can be achieved by setting k 0 i = s 
-]
Revised: [-i is the convolution tensor and s i is the convolutional stride at layer i . At Layer i , we assume the sequence of convolution, ReLU and max-pooling where the max pooling has kernel k 
 i and stride s 0 i . Lack of max-pooling in some layers can be achieved by setting k 0 i = s 
-]/alphapacbayesmag

Original: i = 1.
Revised: [Table]

Original: We consider classiﬁcation tasks and denote the number of classes by κ .

Original: D Complexity Measures
Revised: D C OMPLEXITY M EASURES

Original: In this section, we look at diﬀerent complexity measures.
Revised: In this section, we look at different complexity measures.

Original: When a measure µ is based on a generalization bound, we chose it so that the following is true with probability 0 .
Revised: When a measure µ is based on a generalization bound, we chose it so that the following is true with probability 0 .

Original: 99 (we choose the failure probability δ to be 0.01):
Revised: 99 (we choose the failure probability δ to be 0.01):

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We also deﬁnecolorredconsider? measures which do not provably bound the generalization error and evaluate those.
Revised: We also consider measures which do not provably bound the generalization error and evaluate those.

Original: Note that in almost all cases, the canonical ordering given based on some “common” assumptions are positively correlated with the generalization in terms of both τ and Ψ; however, for optimizer, the correlation τ is close to 0.
Revised: Note that in almost all cases, the canonical ordering given based on some “common" assumptions are positively correlated with the generalization in terms of both τ and Ψ ; however, for optimizer, the correlation τ is close to 0.

Original: This implies that the choice of optimizer is only essentially uncorrelated with the generalization gap in the range of models we consider.
Revised: This implies that the choice of optimizer is only essentially uncorrelated with the generalization gap in the range of models we consider.

Original: This ordering helps validate many techniques used by the practioners.
Revised: This ordering helps validate many techniques used by the practioners.

Original: D. VC-Dimension Based Measures
Revised: D. VC-D IMENSION B ASED M EASURES

Original: We start by restating the theorem in (Bartlett et al., 2019) which provides an upper bound on the VC-dimension of any piece-wise linear network.
Revised: We start by restating the theorem in (Bartlett et al., 2019) which provides an upper bound on the VC-dimension of any piece-wise linear network.

Original: Theorem 1 (Bartlett et al.
Revised: Theorem 1 (Bartlett et al.

Original: Let F be the class of feed-forward networks with a ﬁxed computation graph of depth d and ReLU activations.
Revised: Let F be the class of feed-forward networks with a ﬁxed computation graph of depth d and ReLU activations.

Original: Let a i and q i be the number of activations and parameters in layer i .
Revised: Let a i and q i be the number of activations and parameters in layer i .

Original: Then VC-dimension of F can be bounded as follows: 
Revised: Then VC-dimension of F can be bounded as follows: [Equation] ia [Equation]

Revised: (cid:88)

Revised: j

Original: [Equation]
Revised: [Equation]

Original:  Theorem 2 Given a convolutional network f , for any δ > 0 , with probability 1 − δ over the the training set:
Revised:     Theorem 2 Given a convolutional network f , for any δ > 0 , with probability 1 − δ over the the training set:

Original: [Equation]
Revised: [Equation]

Original: Proof We simplify the bound in Theorem 1 using a d 0 to refer to the depth instead of d :
Revised: Proof We simplify the bound in Theorem 1 using a d (cid:48) to refer to the depth instead of d :

Original: [Equation]
Revised: [Equation]

Original:   log  8 e d 0 X i =

Original: ia

Original: i 

Original:  ≤ d 0 + 2 log  8 e d 0 X i = ia i 

Original:  d 0 X i =

Original: ( d 0 − i + 1) q i ≤ 3 d 0 log  8 e d 0 X i =

Original: ia i 

Original:  d 0 X i =

Original: q i In order to extend the above bound to a convolutional network, we need to present a pooling layer with ReLU activations.
Revised: In order to extend the above bound to a convolutional network, we need to present a pooling layer with ReLU activations.

Original: First note that maximum of two inputs can be calculated using two layers with ReLU and linear activations as max( x 1 , x 2 )
Revised: First note that maximum of two inputs can be calculated using two layers with ReLU and linear activations as max( x 1 , x 2 )

Original: = x 1 + ReLU ( x 2 − x 1 ).
Revised: = x 1 + ReLU ( x 2 − x 1 ) .

Original: Now, since max-pooling at layer i has kernel sizes k [-
-]
Revised: Now, since max-pooling at layer i has kernel sizes k [-
-](cid:48)

Original: i , we need d 4 log 2 ( k 0 i ) e layers to present that but given that the kernel size of the max-pooling layer is at most size of the image, we have
Revised: i , we need (cid:100) 4 log 2 ( k (cid:48) i ) (cid:101) layers to present that but given that the kernel size of the max-pooling layer is at most size of the image, we have

Original: [Equation]
Revised: [Table]

Original: Therefore, we have d 0 ≤ 9 d log 2 ( n ).
Revised: Therefore, we have d (cid:48) ≤ 9 d log 2 ( n ) .

Original: The number of activations in any of these layers is at most n 2 c i since there are at most n 2 pairs of neighbor pixels in an n × n image with c i channels.
Revised: The number of activations in any of these layers is at most n 2 c i since there are at most n 2 pairs of neighbor pixels in an n × n image with c i channels.

Original: We ignore strides when calculating the upper bound since it only reduces number of activations at a few layers and does not change the bound signiﬁcantly.
Revised: We ignore strides when calculating the upper bound since it only reduces number of activations at a few layers and does not change the bound signiﬁcantly.

Original: Using these bounds on d 0 , a i and q i the equivalent network, we can bound the VC dimension as follows:
Revised: Using these bounds on d (cid:48) , a i and q i the equivalent network, we can bound the VC dimension as follows:

Original: [Equation]
Revised: [Equation]

Original: For binary classiﬁers, generalization error can be in terms of Rademacher complexity (Mohri et al., 2012) which in turn can be bounded by 72 p VC /m
Revised: For binary classiﬁers, generalization error can be in terms of Rademacher complexity (Mohri et al., 2012) which in turn can be bounded by 72 (cid:112) VC /m

Original: (Kontorovich, 2016).
Revised: (Kontorovich, 2016).

Original: Therefore, we can get the following 6 generalization bound:
Revised: Therefore, we can get the following 9 generalization bound:

Original: [Equation]
Revised: [Equation]

Original: For multi-class classiﬁcation, the generalization error can be similarly bounded by Graph dimension which is an extension of VC-dimension.
Revised: For multi-class classiﬁcation, the generalization error can be similarly bounded by Graph dimension which is an extension of VC-dimension.

Original: A simple approach get a bound on Graph dimension is to consider all pairs of classes as binary classiﬁcation problem which bounds the graph dimension by κ 2 V C ( F ).
Revised: A simple approach get a bound on Graph dimension is to consider all pairs of classes as binary classiﬁcation problem which bounds the graph dimension by κ 2 V C ( F ) .

Original: There, putting everything together, we get the following generalization bound:
Revised: There, putting everything together, we get the following generalization bound:

Original: [Equation]
Revised: [Equation]

Original: Inspired by Theorem 2, we deﬁne the following V C -based measure for generalization:
Revised: Inspired by Theorem 2, we deﬁne the following V C -based measure for generalization:

Original: [Equation]
Revised: [Equation]

Original: Since some of the dependencies in the above measure are probably proof artifacts, we also deﬁne another measure that is nothing but the number of parameters of the model:
Revised: Since some of the dependencies in the above measure are probably proof artifacts, we also deﬁne another measure that is nothing but the number of parameters of the model:

Original: [Equation]
Revised: [Equation]

Original: D.1. Measures on the output of the network
Revised: D.1. M EASURES ON THE OUTPUT OF THE NETWORK

Original: While measures that can be calculated only based on the output of the network cannot reveal complexity of the network, they can still be very informative for predicting generalization.
Revised: While measures that can be calculated only based on the output of the network cannot reveal complexity of the network, they can still be very informative for predicting generalization.

Original: Therefore, we deﬁne a few measures that can be calculated solely based on the output of the network.
Revised: Therefore, we deﬁne a few measures that can be calculated solely based on the output of the network.

Original: We start by looking at the cross-entropy over the output.
Revised: We start by looking at the cross-entropy over the output.

Original: Even though we used a crossentropy based stopping criterion, the cross-entropy of the ﬁnal models is not exactly the same as the stopping criterion and it could be informative.
Revised: Even though we used a cross-entropy based stopping criterion, the cross-entropy of the ﬁnal models is not exactly the same as the stopping criterion and it could be informative.

Original: Hence we deﬁne the following measure:
Revised: Hence we deﬁne the following measure:

Original: [Equation]
Revised: [Equation]

Original: where ‘ is the cross entropy loss.
Revised: where (cid:96) is the cross-entropy loss.

Original: Another useful and intuitive notion that appears in generalization bounds is margin.
Revised: Another useful and intuitive notion that appears in generalization bounds is margin.

Original: In all measures that involve margin γ , we set the margin γ to be the 10-th percentile of the margin values on the training set and therefore ensuring ˆ L γ ≤ 0 . 1.
Revised: In all measures that involve margin γ , we set the margin γ to be the 10 -th percentile of the margin values on the training set and therefore ensuring ˆ L γ ≤ 0 . 1 .

Original: Even though margin alone is not a sensible generalization measure and can be artiﬁcially increased by scaling up the magnitude of the weights, it could still reveal information about training dynamics and therefore be informative.
Revised: Even though margin alone is not a sensible generalization measure and can be artiﬁcially increased by scaling up the magnitude of the weights, it could still reveal information about training dynamics and therefore be informative.

Original: We report the following measure based on the margin:
Revised: We report the following measure based on the margin:

Original: [Equation]
Revised: [Equation]

Original: Finally, entropy of the output is another interesting measure and it has been shown that regularizing it can improve generalization in deep learning (Pereyra et al., 2017).
Revised: Finally, entropy of the output is another interesting measure and it has been shown that regularizing it can improve generalization in deep learning (Pereyra et al., 2017).

Original: With a ﬁxed cross-entropy, increasing the entropy corresponds to distribute the uncertainty of the predictions equally among the wrong labels which is connected to label smoothing and increasing the margin.
Revised: With a ﬁxed cross-entropy, increasing the entropy corresponds to distribute the uncertainty of the predictions equally among the wrong labels which is connected to label smoothing and increasing the margin.

Original: We deﬁne the following measure which is the negative entropy of the output of the network:
Revised: We deﬁne the following measure which is the negative entropy of the output of the network:

Original: [Equation]
Revised: [Equation]

Original: where p i [ j ] is the predicted probability of the class j for the input data X i .
Revised: where p i [ j ] is the predicted probability of the class j for the input data X i .

Original: D. Norm/Margin Based Measures
Revised: D. (N ORM & M ARGIN )-B ASED M EASURES

Original: Several generalization bounds have been proved for neural networks using margin and norm notions.
Revised: Several generalization bounds have been proved for neural networks using margin and norm notions.

Original: In this section, we go over several such measures.
Revised: In this section, we go over several such measures.

Original: For fully connected networks, 
Revised: For fully connected networks, Bartlett and

Original: Bartlett & Mendelson (2002) have shown a bound based on product of ‘ 1 , ∞ norm of the layer weights times a 2 d factor where ‘ 1 , ∞ is the maximum over hidden units of the ‘ 2 norm of the incoming weights to the hidden unit.
Revised: Mendelson (2002) have shown a bound based on product of (cid:96) 1 , ∞ norm of the layer weights times a 2 d factor where (cid:96) 1 , ∞ is the maximum over hidden units of the (cid:96) 2 norm of the incoming weights to the hidden unit.

Original: Neyshabur et al.
Revised: Neyshabur et al.

Original: (2015b) proved a bound based on product of Frobenius norms of the layer weights times a 2 d factor and Golowich et al.
Revised: (2015b) proved a bound based on product of Frobenius norms of the layer weights times a 2 d factor and Golowich et al.

Original: (2017) was able to improve the factor to √ d .
Revised: (2017) was able to improve the factor to √ d .

Original: Bartlett et al.
Revised: Bartlett et al.

Original: (2017) proved a bound based on product of spectral norm of the layer weights times sum over layers of ratio of Frobenius norm to spectral norm of the layer weights and Neyshabur et al.
Revised: (2017) proved a bound based on product of spectral norm of the layer weights times sum over layers of ratio of Frobenius norm to spectral norm of the layer weights and Neyshabur et al.

Original: (2018a) showed a similar bound can be achieved in a simpler way using PAC-bayesian framework.
Revised: (2018a) showed a similar bound can be achieved in a simpler way using PAC-bayesian framework.

Original: Spectral Norm Unfortunately, none of the above founds are directly applicable to convolutional networks.
Revised: Spectral Norm Unfortunately, none of the above founds are directly applicable to convolutional networks.

Original: Pitas et al.
Revised: Pitas et al.

Original: (2017) built on Neyshabur et al.
Revised: (2017) built on Neyshabur et al.

Original: (2018a)
Revised: (2018a)

Original: and extended the bound on the spectral norm to convolutional networks.
Revised: and extended the bound on the spectral norm to convolutional networks.

Original: The bound is very similar to the one for fully connected networks by Bartlett et al.
Revised: The bound is very similar to the one for fully connected networks by Bartlett et al.

Original: We next restate their generalization bound for convolutional networks including the constants.
Revised: We next restate their generalization bound for convolutional networks including the constants.

Original: Theorem 3 (Pitas et al.
Revised: Theorem 3 (Pitas et al.

Original: Let B an upper bound on the ‘ 2 norm of any point in the input domain.
Revised: Let B an upper bound on the (cid:96) 2 norm of any point in the input domain.

Original: For any B, γ, δ > 0 , the following bound holds with probability 1 − δ over the training set:
Revised: For any B, γ, δ > 0 , the following bound holds with probability 1 − δ over the training set:

Original: [Equation]
Revised: [Equation]

Original: Inspired by the above theorem, we deﬁne the following spectral measure:
Revised: Inspired by the above theorem, we deﬁne the following spectral measure:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The generalization bound in Theorem 3 depends on reference tensors W 

Revised: The generalization bound in Theorem 3 depends on reference tensors W

Original: i .
Revised: i .

Original: We chose the initial tensor as the reference in the above measure but another reasonable choice is the origin which gives the following measures:
Revised: We chose the initial tensor as the reference in the above measure but another reasonable choice is the origin which gives the following measures:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Since some of the terms in the generalization bounds might be proof artifacts
Revised: Since some of the terms in the generalization bounds might be proof artifacts

Original: [Equation]
Revised: [Equation]

Original: we also measure the main terms in the generalization bound:
Revised: we also measure the main terms in the generalization bound:

Original: [Equation]
Revised: [Equation]

Original: We further look at the main two terms in the bound separately to be able to diﬀerentiate their contributions.
Revised: We further look at the main two terms in the bound separately to be able to differentiate their contributions

Original: [Equation]
Revised: [Equation]

Original: Finally [Equation] since product of spectral norms almost certainly increases with depth
Revised: Finally, since product of spectral norms almost certainly increases with depth, we look at the following measure which is equal to the sum over squared spectral norms after rebalancing the layers to have the same spectral norms:

Original: [Equation]
Revised: [Equation]

Original: we look at the following measure which is equal to the sum over squared spectral norms after rebalancing the layers to have the same spectral norms:

Original: [Equation]
Revised: [Equation]

Original: Frobenius Norm
Revised: Frobenius Norm

Original: The generalization bound given in Neyshabur et al.
Revised: The generalization bound given in Neyshabur et al.

Original: (2015b) is not directly applicable to convolutional networks.
Revised: (2015b) is not directly applicable to convolutional networks.

Original: However, Since for each layer i , we have k W[-i k 2 ≤ k 
 i-]
Revised: However, Since for each layer i , we have (cid:107) W[-i k 2 ≤ k 
 i-]

Original:  k  W i k F and therefore by Theorem 3, we can get an upper bound on the test error based on product of Frobenius norms.
Revised: i (cid:107) 2 ≤ k 2 i (cid:107) W i (cid:107) F and therefore by Theorem 3, we can get an upper bound on the test error based on product of Frobenius norms.

Original: Therefore, we deﬁne the following measure based on the product of Frobenius norms:
Revised: Therefore, we deﬁne the following measure based on the product of Frobenius norms:

Original: [Equation]
Revised: [Equation]

Original: We also look at the following measure with correspond to sum of squared Frobenius norms of the layers after rebalancing them to have the same norm:
Revised: We also look at the following measure with correspond to sum of squared Frobenius norms of the layers after rebalancing them to have the same norm:

Original: [Equation]
Revised: [Equation]

Revised: F (cid:33)

Original:  Finally, given recent evidence on the importance of distance to initialization (Dziugaite & Roy, 2017; Nagarajan & Kolter, 2019b; Neyshabur et al., 2018b), we calculate the following measures:
Revised: [Equation] /d (39) Finally, given recent evidence on the importance of distance to initialization (Dziugaite and Roy, 2017; Nagarajan and Kolter, 2019b; Neyshabur et al., 2018b), we calculate the following measures:

Original: [Equation]
Revised: [Equation]

Original: In case when the reference matrix W 

Revised: In case when the reference matrix W

Original: i
Revised: i

Original: = 

Revised: =

Original: for all weights, eq. 40 the Frobenius norm of the parameters which also correspond to distance from the origin:
Revised: for all weights, Eq (40) the Frobenius norm of the parameters which also correspond to distance from the origin:

Original: [Equation]
Revised: [Equation]

Original: Path-norm Path-norm was introduced in Neyshabur et al.
Revised: Path-norm Path-norm was introduced in Neyshabur et al.

Original: (2015b)
Revised: (2015b)

Original: as an scale invariant complexity measure for generalization
Revised: as an scale invariant complexity measure for generalization

Original: and is shown to be a useful geometry for optimization Neyshabur et al.
Revised: and is shown to be a useful geometry for optimization Neyshabur et al.

Original: (2015a).
Revised: (2015a).

Original: To calculate path-norm, we square the parameters of the network, do a forward pass on an all-ones input and then take square root of sum of the network outputs.
Revised: To calculate path-norm, we square the parameters of the network, do a forward pass on an all-ones input and then take square root of sum of the network outputs.

Original: We deﬁne the following measures based on the path-norm:
Revised: We deﬁne the following measures based on the path-norm:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where w 2 = w ◦ w is the element-wise square operation on the parameters.
Revised: where w 2 = w ◦ w is the element-wise square operation on the parameters.

Original: Fisher-Rao Norm Fisher-Rao metric was introduced in Liang et al.
Revised: Fisher-Rao Norm Fisher-Rao metric was introduced in Liang et al.

Original: (2017) as a complexity measure for neural networks.
Revised: (2017) as a complexity measure for neural networks.

Original: Liang et al.
Revised: Liang et al.

Original: (2017) showed that Fisher-Rao norm is a lower bound on the path-norm and it correlates in some cases.
Revised: (2017) showed that Fisher-Rao norm is a lower bound on the path-norm and it correlates in some cases.

Original: We deﬁne a measure based on the Fisher-Rao matric of the network:
Revised: We deﬁne a measure based on the Fisher-Rao matric of the network:

Original: [Equation]
Revised: [Equation]

Original: where ‘ is the cross-entropy loss.
Revised: where (cid:96) is the cross-entropy loss.

Original: D. Flatness-based Measures
Revised: D. F LATNESS - BASED M EASURES

Original: PAC-Bayesian framework (McAllester, 1999) allows us to study ﬂatness of a solution and connect it to generalization.
Revised: PAC-Bayesian framework (McAllester, 1999) allows us to study ﬂatness of a solution and connect it to generalization.

Original: Given a prior P is is chosen before observing the training set and a posterior Q which is a distribution on the solutions of the learning algorithm (and hence depends on the training set), we can bound the expected generalization error of solutions generated from Q with high probability based on the KL divergence of P and Q .
Revised: Given a prior P is is chosen before observing the training set and a posterior Q which is a distribution on the solutions of the learning algorithm (and hence depends on the training set), we can bound the expected generalization error of solutions generated from Q with high probability based on the KL divergence of P and Q .

Original: The next theorem states a simpliﬁed version of PAC-Bayesian bounds.
Revised: The next theorem states a simpliﬁed version of PAC-Bayesian bounds.

Original: Theorem 4 For any δ > 0 , distribution D , prior P , with probability 1 − δ over the training set, for any posterior Q the following bound holds:
Revised: Theorem 4 For any δ > 0 , distribution D , prior P , with probability 1 − δ over the training set, for any posterior Q the following bound holds:

Original: [Equation]
Revised: [Equation]

Original: If P and Q are Gaussian distributions with P = N ( µ P , Σ P ) amd Q = N ( µ Q , Σ Q ), then the KL-term can be written as follows:
Revised: If P and Q are Gaussian distributions with P = N ( µ P , Σ P ) amd Q = N ( µ Q , Σ Q ) , then the KL -term can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: Setting Q = N ( w , σ 2 I ) and P = N ( w 0 , σ 2 I ) similar to Neyshabur et al.
Revised: Setting Q = N ( w , σ 2 I ) and P = N ( w 0 , σ 2 I ) similar to Neyshabur et al.

Original: (2017), the KL term will be simply k w − w 0 k 2 2 2 σ.
Revised: (2017), the KL term will be simply (cid:107) w [Equation]

Original: However, since σ belongs to prior, if we search to ﬁnd a value for σ , we need to adjust the bound to reﬂect that.
Revised: However, since σ belongs to prior, if we search to ﬁnd a value for σ , we need to adjust the bound to reﬂect that.

Original: Since we search over less thanpredeﬁned values of σ in our experiments, we can use the union bound which changes the logarithmic term to log(20000 m/δ ) and we get the following bound:
Revised: Since we search over less than 20000 predeﬁned values of σ in our experiments, we can use the union bound which changes the logarithmic term to log(20000 m/δ ) and we get the following bound:

Original: [Equation]
Revised: [Equation]

Original: Based on the above bound, we deﬁne the following measures using the origin and initialization as reference tensors:
Revised: Based on the above bound, we deﬁne the following measures using the origin and initialization as reference tensors:

Original: [Equation]
Revised: [Equation]

Original: where σ is chosen to be the largest number such that E u ∼ N( u,σ 2 I )
Revised: where σ is chosen to be the largest number such that E u ∼ N

Original: h ˆ L ( f w + u ) i ≤ 0 .
Revised: [Equation]

Original: The above framework captures ﬂatness in the expected sense since we add Gaussian perturbations to the parameters.
Revised: The above framework captures ﬂatness in the expected sense since we add Gaussian perturbations to the parameters.

Original: Another notion of ﬂatness is the worst-case ﬂatness where we search for the direction that changes the loss the most.
Revised: Another notion of ﬂatness is the worst-case ﬂatness where we search for the direction that changes the loss the most.

Original: This is motivated by (Keskar et al., 2016) where they observe that this notion would correlate to generalization in the case of diﬀerent batch sizes.
Revised: This is motivated by (Keskar et al., 2016) where they observe that this notion would correlate to generalization in the case of different batch sizes.

Original: We can use PAC-Bayesian framework to give generalization bounds for worst-case perturbations as well.
Revised: We can use PAC-Bayesian framework to give generalization bounds for worst-case perturbations as well.

Original: The magnitude of a Gaussian variable with with variance σ 2 is at most σ p 2 log(2 /δ )
Revised: The magnitude of a Gaussian variable with with variance σ 2 is at most σ (cid:112) 2 log(2 /δ )

Original: with probability 1 − δ/ 2.
Revised: with probability 1 − δ/ 2 .

Original: Applying a union bound on all parameters, we get that with probability 1 − δ/ 2 the magnitude of the Gaussian noise is at most α = σ p 2 log(2 ω/δ ) where ω is the number of parameters of the model.
Revised: Applying a union bound on all parameters, we get that with probability 1 − δ/the magnitude of the Gaussian noise is at most α = σ (cid:112) 2 log(2 ω/δ ) where ω is the number of parameters of the model.

Original: Therefore, we can get the following generalization bound:
Revised: Therefore, we can get the following generalization bound:

Original: [Equation]
Revised: [Equation]

Original: Inspired by the above bound, we deﬁne the following measures:
Revised: Inspired by the above bound, we deﬁne the following measures:

Original: [Equation]
Revised: [Equation]

Original: (52) where α is chosen to be the largest number such that max | u i |≤ α ˆ L ( f w + u ) ≤ 0 .
Revised: where α is chosen to be the largest number such that max | u i [Equation]

Original: To understand the importance of the ﬂatness parameters σ and α , we also deﬁne the following measures:
Revised: To understand the importance of the ﬂatness parameters σ and α , we also deﬁne the following measures:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where α and σ are computed as explained above.
Revised: where α and σ are computed as explained above.

Original: Magnitude-aware Perturbation Bounds
Revised: Magnitude-aware Perturbation Bounds

Original: The magnitude of perturbation in (Keskar et al., 2016) was chosen so that for each parameter the ratio of magnitude of perturbation to the magnitude of the parameter is bounded by a constant α 0 7 .
Revised: The magnitude of perturbation in (Keskar et al., 2016) was chosen so that for each parameter the ratio of magnitude of perturbation to the magnitude of the parameter is bounded by a constant α (cid:48) 10 .

Original: Following a similar approach, we can choose the posterior for parameter i in PAC-Bayesian framework to be N ( w i , σ 0 2 | w i | 2 + (cid:15) 2 ).
Revised: Following a similar approach, we can choose the posterior for parameter i in PAC-Bayesian framework to be N ( w i , σ (cid:48) 2 | w i | 2 + (cid:15) 2 ) .

Original: Now, substituting this in the Equation equation D.3 and solving for the prior N ( w 0 , σ 2 P ) that minimizes the KL term by setting the gradient with respect to σ P 2 to zero, KL can be written as follows:[Equation] Therefore
Revised: Now, substituting this in the Equation equation D.3 and solving for the prior N ( w 0 , σ 2 P ) that minimizes the KL term by setting the gradient with respect to σ P 2 to zero, KL can be written as follows:

Original: [Equation]
Revised: [Equation]

Original:  the generalization bound can be written as follows
Revised: Therefore, the generalization bound can be written as follows

Original: [Equation]
Revised: [Equation]

Original: where u i ∼ N (0 , σ 0 2 | w i | + (cid:15) 2 ), (cid:15) = 1 e − 3 and σ 0 is chosen to be the largest number 
Revised: where u i ∼ N (0 , σ (cid:48) 2 | w i | + (cid:15) 2 ) , (cid:15) = 1 e − 3 and σ (cid:48) is chosen to be the largest number such that

Original: such that E u h ˆ L ( f w + u ) i ≤ 0 .
Revised: [Equation] ≤ 0 .

Original: We deﬁne the following measures based on the generalization bound:
Revised: We deﬁne the following measures based on the generalization bound:

Original: [Equation]
Revised: [Equation]

Original: We also follow similar arguments are before to get a similar bound on the worst-case sharpness:
Revised: We also follow similar arguments are before to get a similar bound on the worst-case sharpness:

Revised: E u [ L ( f w + u )] ≤

Original: [Equation]
Revised: [Equation]

Original: We look at the following measures based on the above bound:
Revised: We look at the following measures based on the above bound:

Original: [Equation]
Revised: [Equation]

Original:  sharpness [Equation] mag
Revised: Finally, we look at measures that are only based the sharpness values computed above:

Original: [Equation]
Revised: [Equation]

Original: Finally, we look at measures that are only based the sharpness values computed above:

Original: [Equation]
Revised: [Equation]

Original: where α and σ are computed as explained above.
Revised: where α and σ are computed as explained above.

Original: D. Optimization-based Measures
Revised: D. O PTIMIZATION - BASED M EASURES

Original: There are mixed results about how the optimization speed is relevant to generalization.
Revised: There are mixed results about how the optimization speed is relevant to generalization.

Original: On one hand we know that adding Batch Normalization or using shortcuts in residual architectures help both optimization and generalization and Hardt et al.
Revised: On one hand we know that adding Batch Normalization or using shortcuts in residual architectures help both optimization and generalization and Hardt et al.

Original: (2015) suggests that faster optimization results in better generalization.
Revised: (2015) suggests that faster optimization results in better generalization.

Original: On the other hand, there are empirical results showing that adaptive optimization methods that are faster, usually generalize worse (Wilson et al., 2017b).
Revised: On the other hand, there are empirical results showing that adaptive optimization methods that are faster, usually generalize worse (Wilson et al., 2017b).

Original: Here, we put these hypothesis into test by looking at the number of steps to achieve cross entropy 0.1 and the number of steps needed to go from cross-entropy 0.1 to 0.01:[Equation] initialization [Equation]
Revised: Here, we put these hypothesis into test by looking at the number of steps to achieve cross-entropy 0.1 and the number of steps needed to go from cross-entropy 0.1 to 0.01:

Original: [Equation]

Original: 1 cross-entropy (

Original: [Equation]
Revised: [Equation]

Original: The above measures tell us if the speed of optimization at early or late stages can be informative about generalization.
Revised: The above measures tell us if the speed of optimization at early or late stages can be informative about generalization.

Original: We also deﬁne measures that look at the SGD gradient noise after the ﬁrst epoch and at the end of training at cross-entropy 0.01 to test the gradient noise can be predictive of generalization:
Revised: We also deﬁne measures that look at the SGD gradient noise after the ﬁrst epoch and at the end of training at cross-entropy 0.01 to test the gradient noise can be predictive of generalization:

Original: [Equation]
Revised: [Equation]

Original: where w 1 is the weight vector after the ﬁrst epoch.
Revised: where w 1 is the weight vector after the ﬁrst epoch.

Original: E Algorithms
Revised: E A LGORITHMS

Original: We ﬁrst lay out some common notations used in the pseudocode:
Revised: We ﬁrst lay out some common notations used in the pseudocode:

Original: f : the architecture that takes parameter θ and input x and map to f ( x ; θ ) which is the predicted label of x 2.
Revised: f : the architecture that takes parameter θ and input x and map to f ( x ; θ ) which is the predicted label of x 2.

Original: θ : parameters 3.
Revised: parameters 3.

Original: M : Some kind of iteration; M 1 : binary search depth; M 2 : Monte Carlo Estimation steps; M 3 : Iteration for estimating the loss 4.
Revised: M : Some kind of iteration; M 1 : binary search ; M 2 : Monte Carlo Estimation steps; M 3 : Iteration for estimating the loss 4.

Original: D =
Revised: D =

Original: { ( x i , y i ) } ni =

Revised: { ( x i , y i ) } ni =

Original: the dataset the model is trained on; B as a uniformly sampled minibatch from the dataset.
Revised: the dataset the model is trained on; B as a uniformly sampled minibatch from the dataset.

Original: Both search algorithm relies on the assumption that the loss increases monotonically with the perturbation magnitude σ around the ﬁnal weight.
Revised: Both search algorithm relies on the assumption that the loss increases monotonically with the perturbation magnitude σ around the ﬁnal weight.

Original: This assumption is quite mild and in reality holds across almost all the models in this study.
Revised: This assumption is quite mild and in reality holds across almost all the models in this study.

Original: Algorithm
Revised: i = f ( B i ; θ )) 6: end for 7: return Accuracy/M

Original: Note that for ﬁnding the sharpness σ , we use the cross entropy as the diﬀerentiable surrogate object instead of the 1-0 loss which is in general not diﬀerentiable.
Revised: Note that for ﬁnding the sharpness σ , we use the cross-entropy as the differentiable surrogate object instead of the 1-0 loss which is in general not differentiable.

Original: Using gradient ascent brings another additional challenge that is for a converged model, the local gradient signal is usually weak, making gradient ascent extremely ineﬃcient.
Revised: Using gradient ascent brings another additional challenge that is for a converged model, the local gradient signal is usually weak, making gradient ascent extremely inefﬁcient.

Original: To speed up thie process, we add a uniform noise with range being [ − σ new /N w , σ new /N w ] to lift the weight oﬀ the ﬂat minima where N w is the number of parameters.
Revised: To speed up thie process, we add a uniform noise with range being [ − σ new /N w , σ new /N w ] to lift the weight off the ﬂat minima where N w is the number of parameters.

Original: This empirical greatly accelerates the search.
Revised: This empirical greatly accelerates the search.

Revised: Further, for magnitude aware version of the bounds, the overall algorithm stays the same with the exception that now covariance matrices at line 7 of Algorithm 2 become as diagonal matrix containing w 2 i on the diagonal; similarly, for line 12 of Algorithm 3, the weight clipping of each w i is conditioned on σ new | w i | , i.e. clipped to [ − σ new | w i | , σ new | w i | ] .

Revised: Here w i denotes the i th parameter of ﬂattened w .

Original: 1: Inputs: f , θ 0 , model accuracy ‘ , target accuracy deviation d , Upper bound σ max , Lower bound σ min , M 1 , M 2 , M2: Initialize 3: for episode i = 1 to M 1 do 4: σ new =
Revised: 1: Inputs: f , θ 0 , model accuracy (cid:96) , target accuracy deviation d , Upper bound σ max , Lower bound σ min , M 1 , M 2 , M2: Initialize 3: for episode i = 1 to M 1 do 4: σ new =

Original: ( σ max + σ min ) /5:
Revised: ( σ max + σ min ) /5:

Original: ˆ ‘ =6: for step j = 0 to M 2 do 7: θ ← θ 

Revised: ˆ (cid:96) = 6: for step j = 0 to M 2 do 7: θ ← θ

Original: + N (0 , σ 2 new I ) 8: ˆ ‘ = ˆ ‘ + EstimateAccuracy( f, θ new , D , M 3 ) 9: end for 10: ˆ ‘ =ˆ ‘/M11:
Revised: + N (0 , σ 2 new I ) 8: ˆ (cid:96) = ˆ (cid:96) + EstimateAccuracy( f, θ new , D , M 3 ) 9: end for 10: ˆ (cid:96) =

Original: ˆ  d = | ‘ − ˆ ‘ | 12: if ˆ d < (cid:15) d or σ max − σ min < (cid:15) σ then 13: return σ new 14: end if 15: if ˆ d > d then 16: σ max = σ new 17: else 18: σ min = σ new 19: end if 20: end for
Revised: ˆ (cid:96)/M11: ˆ d = | (cid:96) − ˆ (cid:96) | 12: if ˆ d < (cid:15) d or σ max − σ min < (cid:15) σ then 13: return σ new 14: end if 15: if ˆ d > d then 16: σ max = σ new 17: else 18: σ min = σ new 19: end if 20: end for

Original:  1: Inputs: f , θ 0 , loss function L , model accuracy ‘ , target accuracy deviation d , Upper bound σ max , Lower bound σ min , M 1 , M 2 , M 3 , gradient steps M2: Initialize 3: for episode i = 1 to M 1 do 4: σ new =
Revised: Algorithm 3 Find σ for Sharpness Bound 1: Inputs: f , θ 0 , loss function L , model accuracy (cid:96) , target accuracy deviation d , Upper bound σ max , Lower bound σ min , M 1 , M 2 , M 3 , gradient steps M2: Initialize 3: for episode i = 1 to M 1 do 4: σ new =

Original: ( σ max + σ min ) /5:
Revised: ( σ max + σ min ) /5:

Original: ˆ ‘ = ∞ 6: for step j = 0 to M 2 do 7: θ = θ 

Revised: ˆ (cid:96) = ∞ 6: for step j = 0 to M 2 do 7: θ = θ

Original: + U ( σ new / 2) 8: for step k = 0 to M 4 do 9:
Revised: + U ( σ new / 2) 8: for step k = 0 to M 4 do 9:

Original: B ∼ sample( D ) 10: θ = θ + η ∇ θ ‘ ( f, B , θ ) 11: if || θ || >
Revised: B ∼ sample ( D ) 10: θ = θ + η ∇ θ (cid:96) ( f, B , θ ) 11: if || θ || >

Original: σ new then 12: θ = σ new · θ || θ || 13: end if 14: end for 15: ˆ ‘ = min(ˆ ‘, EstimateAccuracy( f, θ new , D , M 3 )) 16: end for 17: ˆ d = | ‘ − ˆ ‘ | 18:
Revised: σ new then 12: θ = σ new · θ || θ || 13: end if 14: end for 15: ˆ (cid:96) = min(ˆ (cid:96), EstimateAccuracy( f, θ new , D , M 3 )) 16: end for 17: ˆ d = | (cid:96) − ˆ (cid:96)

Original:  if ˆ d < (cid:15) d or σ max − σ min < (cid:15) σ then 19: return σ new 20: end if 21: if ˆ d > d then 22: σ max = σ new 23: else 24: σ min = σ new 25: end if 26: end for
Revised: | 18: if ˆ d < (cid:15) d or σ max − σ min < (cid:15) σ then 19: return σ new 20: end if 21: if ˆ d > d then 22: σ max = σ new 23: else 24: σ min = σ new 25: end if 26: end for

Original: Further, for magnitude aware version of the bounds, the overall algorithm stays the same with the exception that now covariance matrices at line 7 of Algorithm 2 become as diagonal matrix containing w 2 i on the diagonal; similarly, for line 12 of Algorithm 3, the weight