Original: A M ODERN T AKE ON THE B IAS -V ARIANCE T RADEOFF IN N EURAL N ETWORKS
Revised: A M ODERN T AKE ON THE B IAS -V ARIANCE T RADEOFF IN N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We revisit the bias-variance tradeoff for neural networks in light of modern empirical ﬁndings.
Revised: We revisit the bias-variance tradeoff for neural networks in light of modern empirical ﬁndings.

Original: The traditional bias-variance tradeoff in machine learning suggests that as model complexity grows, variance increases.
Revised: The traditional bias-variance tradeoff in machine learning suggests that as model complexity grows, variance increases.

Original: Classical bounds in statistical learning theory point to the number of parameters in a model as a measure of model complexity, which means the tradeoff would indicate that variance increases with the size of neural networks.
Revised: Classical bounds in statistical learning theory point to the number of parameters in a model as a measure of model complexity, which means the tradeoff would indicate that variance increases with the size of neural networks.

Original: However, we empirically ﬁnd that variance due to training set sampling is roughly constant (with both width and depth) in practice.
Revised: However, we empirically ﬁnd that variance due to training set sampling is roughly constant (with both width and depth) in practice.

Original: Variance caused by the non-convexity of the loss landscape is different.
Revised: Variance caused by the non-convexity of the loss landscape is different.

Original: We ﬁnd that it decreases with width and increases with depth, in our setting.
Revised: We ﬁnd that it decreases with width and increases with depth, in our setting.

Original: We provide theoretical analysis, in a simpliﬁed setting inspired by linear models, that is consistent with our empirical ﬁndings for width.
Revised: We provide theoretical analysis, in a simpliﬁed setting inspired by linear models, that is consistent with our empirical ﬁndings for width.

Original: We view bias-variance as a useful lens to study generalization through and encourage further theoretical explanation from this perspective.
Revised: We view bias-variance as a useful lens to study generalization through and encourage further theoretical explanation from this perspective.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The traditional view in machine learning is that increasingly complex models achieve lower bias at the expense of higher variance.
Revised: The traditional view in machine learning is that increasingly complex models achieve lower bias at the expense of higher variance.

Original: This balance between underﬁtting (high bias) and overﬁtting (high variance) is commonly known as the bias-variance tradeoff (Figure 1).
Revised: This balance between underﬁtting (high bias) and overﬁtting (high variance) is commonly known as the bias-variance tradeoff (Figure 1).

Original: In their landmark work that initially highlighted this bias-variance dilemma, Geman et al.
Revised: In their landmark work that initially highlighted this bias-variance dilemma in machine learning, Geman et al.

Original: (1992) suggest that larger neural networks suffer from higher variance.
Revised: (1992) suggest that larger neural networks suffer from higher variance.

Original: Because bias and variance contribute to test set performance (through the bias-variance decomposition), this provided strong intuition for how we think about generalization capabilities of large models.
Revised: Because bias and variance contribute to test set performance (through the bias-variance decomposition), this provided strong intuition for how we think about generalization capabilities of large models.

Original: Learning theory supports this intuition, as most classical and current bounds on generalization error grow with the size of the networks (Brutzkus et al., 2018).
Revised: Learning theory supports this intuition, as most classical and current bounds on generalization error grow with the size of the networks (Brutzkus et al., 2018).

Original: However, there is a growing amount of evidence of larger networks generalizing better than their smaller counterparts in practice (Neyshabur et al., 2014;
Revised: However, there is a growing amount of evidence of larger networks generalizing better than their smaller counterparts in practice (Neyshabur et al., 2014;

Original: Novak et al., 2018; Zhang et al., 2017;
Revised: Novak et al., 2018; Zhang et al., 2017;

Original: Canziani et al., 2016).
Revised: Canziani et al., 2016).

Original: This apparent mismatch between theory and practice is due to the use of worst-case analysis that depends only on the model class, completely agnostic to data distribution and without taking optimization into account 1 .
Revised: This apparent mismatch between theory and practice is due to the use of worst-case analysis that depends only on the model class, completely agnostic to data distribution and without taking optimization into account.

Original:  A modern empirical study of bias-variance can take all of this information into account.
Revised: 1 A modern empirical study of bias-variance can take all of this information into account.

Original: We revisit the bias-variance tradeoff in the modern setting, focusing on how variance changes with increasing size of neural networks that are trained with optimizers whose step sizes are tuned with a validation set.
Revised: We revisit the bias-variance tradeoff in the modern setting, focusing on how variance changes with increasing size of neural networks that are trained with optimizers whose step sizes are tuned with a validation set.

Original: In contrast to the traditional view of the bias-variance tradeoff (Geman et al., 1992), we ﬁnd evidence that the overall variance decreases with network width (Figure 1).
Revised: In contrast to the traditional view of the bias-variance tradeoff (Geman et al., 1992), we ﬁnd evidence that the overall variance decreases with network width (Figure 1).

Original: This can be seen as the “bias-variance analog” of the described recent evidence of larger networks generalizing better.
Revised: This can be seen as the “bias-variance analog” of the described recent evidence of larger networks generalizing better.

Original: More in line with the tradeoff, we ﬁnd that variance grows slowly with depth, using current best practices.
Revised: More in line with the tradeoff, we ﬁnd that variance grows slowly with depth, using current best practices.

Original: To better understand these coarse trends, we develop a new, more ﬁne-grain way to study variance.
Revised: To better understand these coarse trends, we develop a new, more ﬁne-grain way to study variance.

Original: We separate variance due to initialization (caused by non-convexity of the optimization landscape) from variance due to sampling of the training set.
Revised: We separate variance due to initialization (caused by non-convexity of the optimization landscape) from variance due to sampling of the training set.

Original: Surprisingly, we ﬁnd that variance due to training set sampling is roughly constant with both width and depth.
Revised: Surprisingly, we ﬁnd that variance due to training set sampling is roughly constant with both width and depth (Figure 2).

Original: Variance due to initialization  decreases with width and increases with depth, in our setting.
Revised: Variance due to initialization [Figure] [Figure] decreases with width and increases with depth, in our setting (Figure 2).

Original: To support our empirical ﬁndings, we
Revised: To support our empirical ﬁndings

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  provide a simple theoretical analysis of these sources of variance by taking inspiration from overparameterized linear models.
Revised: we provide a simple theoretical analysis of these sources of variance by taking inspiration from over-parameterized linear models.

Original: We see further theoretical treatment of variance as a fruitful direction for better understanding complexity and generalization abilities of neural networks
Revised: We see further theoretical treatment of variance as a fruitful direction for better understanding complexity and generalization abilities of neural networks

Original: [Figure]
Revised: [Figure]

Original: M AIN CONTRIBUTIONS
Revised: M AIN CONTRIBUTIONS

Original: We revisit the bias-variance analysis in the modern setting for neural networks and point out that it is not necessarily a tradeoff as overall variance decreases with width (similar to bias, which is already known to decrease), yielding better generalization .
Revised: We revisit the bias-variance analysis in the modern setting for neural networks and point out that it is not necessarily a tradeoff as overall variance decreases with width (similar to bias), yielding better generalization .

Original: We perform a more ﬁne-grain study of variance in neural networks by decomposing it into variance due to initialization and variance due to sampling.
Revised: We perform a more ﬁne-grain study of variance in neural networks by decomposing it into variance due to initialization and variance due to sampling.

Original: Variance due to sampling is roughly constant with both width and depth.
Revised: Variance due to sampling is roughly constant with both width and depth.

Original: Variance due to initialization decreases with width, while it increases with depth, in the settings we consider.
Revised: Variance due to initialization decreases with width, while it increases with depth, in the settings we consider.

Original: In a simpliﬁed setting, inspired by linear models, we provide theoretical analysis in support of our empirical ﬁndings for network width.
Revised: In a simpliﬁed setting, inspired by linear models, we provide theoretical analysis in support of our empirical ﬁndings for network width.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: Section 2 establishes necessary preliminaries.
Revised: Section 2 establishes necessary preliminaries.

Original: In Sections 3 and  4, we study the impact of network width and network depth (respectively) on variance.
Revised: In Section 3 and Section 4, we study the impact of network width and network depth (respectively) on variance.

Original: In Section 5, we present our simple theoretical variance analysis.
Revised: In Section 5, we present our simple theoretical variance analysis.

Original:  P RELIMINARIES 2. S ET - UP
Revised:  P RELIMINARIES 2. S ET - UP

Original: We consider the typical supervised learning task of predicting an output y ∈ Y from an input x ∈ X , where the pairs ( x, y ) are drawn from some unknown joint distribution, D .
Revised: We consider the typical supervised learning task of predicting an output y ∈ Y from an input x ∈ X , where the pairs ( x, y ) are drawn from some unknown joint distribution, D .

Original: The learning problem consists of inferring a function h S :
Revised: The learning problem consists of inferring a function h S :

Original: X → Y from a ﬁnite training dataset S of m i.i.d.
Revised: X → Y from a ﬁnite training dataset S of m i.i.d.

Original: samples from
Revised: samples from

Original: D .
Revised: D .

Original: The quality of a predictor h can quantiﬁed by the expected error,
Revised: The quality of a predictor h can quantiﬁed by the expected error,

Original: [Equation]
Revised: [Equation]

Original:  for some loss function (cid:96) :
Revised: ) for some loss function (cid:96) :

Original: Y × Y → R .
Revised: Y × Y → R .

Original: In this paper, predictors h θ are parametrized by the weights θ ∈ R N of deep neural networks.
Revised: In this paper, predictors h θ are parametrized by the weights θ ∈ R N of deep neural networks.

Revised: We consider a frequentist risk analysis of the learning algorithm, that is, the average performance over possible training sets (denoted by the random variable S ) of size m .

Revised: This is the same quantity Geman et al.

Revised: (1992) consider.

Original: Because of randomness in initialization and non-convexity of the loss surface, we assume the learned weights are drawn from a distribution p ( θ | S ) conditioned on  the  training dataset S ; marginalizing over all m samples of S gives a distribution p ( θ) =
Revised: While S is the only random quantity focused on in traditional bias-variance decomposition, we also focus on randomness coming from optimization. We denote the random variable for optimization randomness (e.g. initialization) by I . Formally, given a ﬁxed training set S and ﬁxed optimization randomness I , the learning algorithm A produces θ =

Original:  E S p ( θ | S ) on the  learned weights.
Revised: A ( S, I ) . Randomness in initialization translates to randomness in A ( S, · ) because of non-convexity of the loss surface. Given a ﬁxed training set, we encode the randomness due to I in a conditional distribution p ( θ | S ) ; marginalizing over the training set S of size m gives a marginal distribution p ( θ ) = E S p ( θ | S ) on the weights learned by A from m samples.

Original: In this context, the frequentist risk is obtained by averaging the expected error over the learned weights:
Revised: In this context, the frequentist risk for the learning algorithm using training sets of size m becomes:

Original: [Equation]
Revised: [Equation]

Original: 2. B IAS -V ARIANCE DECOMPOSITION
Revised: 2. B IAS -V ARIANCE DECOMPOSITION

Original: We brieﬂy recall the standard bias-variance decomposition of the frequentist risk in the case of squared-losses.
Revised: We brieﬂy recall the standard bias-variance decomposition of the frequentist risk in the case of squared-losses.

Original: We work in the context of classiﬁcation, where each class k ∈ { 1 · · · K } is represented by a one-hot vector in R K .
Revised: We work in the context of classiﬁcation, where each class k ∈ { 1 · · · K } is represented by a one-hot vector in R K .

Original: The predictor outputs a score or probability vector in R k .
Revised: The predictor outputs a score or probability vector in R k .

Original: In this context, the risk in Eqn.
Revised: In this context, the risk in Eqn.

Original: 2 decomposes into three sources of error (Geman et al., 1992):
Revised: 2 decomposes into three sources of error (Geman et al., 1992):

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is an intrinsic error term independent of the predictor; the second is a bias term
Revised: The ﬁrst term is an intrinsic error term independent of the predictor; the second is a bias term

Original: [Equation]
Revised: [Equation]

Original: where ¯ y ( x ) denotes the expectation E
Revised: where ¯ y ( x ) denotes the expectation E

Original: [ y | x ] of y given x .
Revised: [ y | x ] of y given x .

Original: The third term is the expected variance of the output predictions:
Revised: The third term is the expected variance of the output predictions:

Original: [Equation]
Revised: [Equation]

Revised: where the expectation over θ can be done as in Eqn.

Original: Finally, in the set up of Section 2.1, the sources of variance are the choice of training set S and the choice of initialization, encoded into the conditional p ( ·| S ) .
Revised: Finally, in the set-up of Section 2.1, the sources of variance are the choice of training set S and the choice of initialization I (encoded into the conditional p ( ·| S ) ).

Original: By the law of total variance, we then have the further decomposition:
Revised: By the law of total variance, we then have the further decomposition:

Original: [Equation]
Revised: [Equation]

Original: We call the ﬁrst term variance due to initialization and the second term variance due to sampling throughout the paper.
Revised: We call the ﬁrst term variance due to initialization and the second term variance due to sampling throughout the paper.

Original: Note thattrue risks computed with classiﬁcation losses (e.g cross-entropy or 0-1 loss) do not have such a clean bias-variance decomposition (Domingos, 2000; James, 2003).
Revised: Note that risks computed with classiﬁcation losses (e.g cross-entropy or 0loss) do not have such a clean bias-variance decomposition (Domingos, 2000; James, 2003).

Original: However, it is natural to expect that bias and variance are useful indicators of the performance of the models.
Revised: However, it is natural to expect that bias and variance are useful indicators of the performance of the models.

Original: In fact, the classiﬁcation risk can be bounded as 4 times the regression risk (Appendix D.2). 
Revised: In fact, the classiﬁcation risk can be bounded as 4 times the regression risk (Appendix D.2). [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Original:  V ARIANCE AND WIDTH
Revised:  V ARIANCE AND WIDTH

Original: In this section, we study how variance of single hidden layer networks varies with width, like a modern analog of Geman et al. (1992).
Revised: In this section, we study how variance of single hidden layer networks varies with width, like a modern analog of Geman et al. (1992).

Original: We study fully connected single hidden layer networks up to the largest size that ﬁts in memory, in order to search for an eventual increase in variance.
Revised: We study fully connected single hidden layer networks up to the largest size that ﬁts in memory, in order to search for an eventual increase in variance.

Original: To make our study as general as possible, we consider networks without any regularization bells and whistles such as weight decay, dropout, or data augmentation, which Zhang et al.
Revised: To make our study as general as possible, we consider networks without any regularization bells and whistles such as weight decay, dropout, or data augmentation, which Zhang et al.

Original: (2017) found to not be necessary for good generalization.
Revised: (2017) found to not be necessary for good generalization.

Original: As is commonly done in practice, these networks are trained with optimizers (e.g. SGD) whose step sizes are tuned using a validation set 2 . [Figure] [Figure] [Figure]
Revised: As is commonly done in practice, these networks are trained with optimizers (e.g. SGD) whose step sizes are tuned using a validation set.

Original: [Figure]

Original: 3. C OMMON EXPERIMENTAL DETAILS
Revised: 3. C OMMON EXPERIMENTAL DETAILS

Original: Experiments are run on different datasets: full MNIST, small MNIST, and a sinusoid regression task.
Revised: Experiments are run on different datasets: full MNIST, small MNIST, and a sinusoid regression task.

Original: Averages over data samples are performed by taking the training set S and creating 50 bootstrap (Efron, 1979) replicate training sets S (cid:48) by sampling with replacement from S .
Revised: Averages over data samples are performed by taking the training set S and creating 50 bootstrap (Efron, 1979) replicate training sets S (cid:48) by sampling with replacement from S .

Original: We train 50 different neural networks for each hidden layer size using these different training sets.
Revised: We train 50 different neural networks for each hidden layer size using these different training sets.

Original: Then, we estimate E bias and E variance as in Section 2.2, where the population expectation E x is estimated with an average over the test set inputs (Kohavi and Wolpert, 1996; Domingos, 2000).
Revised: Then, we estimate E bias and E variance as in Section 2.2, where the population expectation E x is estimated with an average over the test set inputs.

Original:  To estimate the two terms from the law of total variance (Equation 5), we use 10 random seeds for the outer expectation and 10 for the inner expectation, resulting in a total of 100 seeds.
Revised: 4 To estimate the two terms from the law of total variance (Equation 5), we use 10 random seeds for the outer expectation and 10 for the inner expectation, resulting in a total of 100 seeds.

Original: Furthermore, we compute 99% conﬁdence intervals for our bias and variance estimates using the bootstrap (Efron, 1979).
Revised: Furthermore, we compute 99% conﬁdence intervals for our bias and variance estimates using the bootstrap (Efron, 1979).

Original: The networks are trained using SGD with momentum and generally run for long after 100% training set accuracy is reached (e.g. 500 epochs for full data MNIST and 10000 epochs for small data
Revised: The networks are trained using SGD with momentum and generally run for long after 100% training set accuracy is reached (e.g. 500 epochs for full data MNIST and 10000 epochs for small data

Original: MNIST).
Revised: MNIST).

Original: The step size hyperparameter is ﬁxed to 0.1 for the full data experiment and is chosen via a validation set for the small data experiment.
Revised: The step size hyperparameter is ﬁxed to 0.1 for the full data experiment and is chosen via a validation set for the small data experiment.

Original: The momentum hyperparameter is always set to 0.9.
Revised: The momentum hyperparameter is always set to 0.9.

Original: 3. D ECREASING VARIANCE IN FULL DATA SETTING
Revised: 3. D ECREASING VARIANCE IN FULL DATA SETTING

Original: We ﬁnd a clear decreasing trend in variance with width of the network in the full data setting (Figure 1).
Revised: We ﬁnd a clear decreasing trend in variance with width of the network in the full data setting (Figure 1).

Original: The trend is the same with or without early stopping, so early stopping is not necessary to see decreasing variance, similar to how it was not necessary to see better test set performance with width in Neyshabur et al.
Revised: The trend is the same with or without early stopping, so early stopping is not necessary to see decreasing variance, similar to how it was not necessary to see better test set performance with width in Neyshabur et al.

Original: 3. T ESTING THE LIMITS : DECREASING VARIANCE IN THE SMALL DATA SETTING
Revised: 3. T ESTING THE LIMITS : DECREASING VARIANCE IN THE SMALL DATA SETTING

Original: Decreasing the size of the dataset can only increase variance.
Revised: Decreasing the size of the dataset can only increase variance.

Original: To study the robustness of the above observation, we decrease the size of the training set to just 100 examples.
Revised: To study the robustness of the above observation, we decrease the size of the training set to just 100 examples.

Original: In this small data setting, somewhat surprisingly, we still observe the same trend of decreasing variance with width (Figure 3a).
Revised: In this small data setting, somewhat surprisingly, we still observe the same trend of decreasing variance with width (Figure 3a).

Original: The test error behaves similarly (Figure 3b).
Revised: The test error behaves similarly (Figure 3b).

Original: The step size is tuned using a validation set (Appendix B.1).
Revised: The step size is tuned using a validation set (Appendix B.1).

Original: The training for tuning is stopped after 1000 epochs, whereas the training for the ﬁnal models is stopped after 10000 epochs.
Revised: The training for tuning is stopped after 1000 epochs, whereas the training for the ﬁnal models is stopped after 10000 epochs.

Original: The corresponding experiment where step size is the same 0.01 for all network sizes is in Appendix B.2.
Revised: The corresponding experiment where step size is the same 0.01 for all network sizes is in Appendix B.2.

Original: With the same step size for all networks, we do not see decreasing variance.
Revised: With the same step size for all networks, we do not see decreasing variance.

Original: Note that we are not claiming that variance decreases with width regardless of step size.
Revised: Note that we are not claiming that variance decreases with width regardless of step size.

Original: Rather, we are claiming variance decreases with width when the step size is tuned using a validation set, as is done in practice.
Revised: Rather, we are claiming variance decreases with width when the step size is tuned using a validation set, as is done in practice.

Original: By tuning the step size, we are making the experimental design choice of keeping optimality of step size constant across networks (more discussion on this in Appendix B.2).
Revised: By tuning the step size, we are making the experimental design choice of keeping optimality of step size constant across networks (more discussion on this in Appendix B.2).

Original: This sensitivity to step size in the small data setting is evidence that we are testing the limits of our hypothesis.
Revised: This sensitivity to step size in the small data setting is evidence that we are testing the limits of our hypothesis.

Original: A larger amount of data makes the networks more robust to the choice of step size (Figure 1).
Revised: A larger amount of data makes the networks more robust to the choice of step size (Figure 1).

Original: However, it is likely the case that if we were able to compute with much larger networks, we would eventually observe a U shape in the full data setting as well.
Revised: However, it is likely the case that if we were able to compute with much larger networks, we would eventually observe increasing variance in the full data setting as well.

Original: By looking at the small data setting, we are able to test our hypothesis when the ratio of size of network to dataset size is quite large, and we still ﬁnd this decreasing trend in variance (Figure 3a).
Revised: By looking at the small data setting, we are able to test our hypothesis when the ratio of size of network to dataset size is quite large, and we still ﬁnd this decreasing trend in variance (Figure 3a).

Original: To see how dependent this phenomenon is on SGD, we also run these experiments using batch gradient descent and PyTorch’s version of LBFGS.
Revised: To see how dependent this phenomenon is on SGD, we also run these experiments using batch gradient descent and PyTorch’s version of LBFGS.

Original: Interestingly, we ﬁnd a decreasing variance trend with those optimizers as well.
Revised: Interestingly, we ﬁnd a decreasing variance trend with those optimizers as well.

Original: These experiments are included in Appendix B.3.
Revised: These experiments are included in Appendix B.3.

Original: This means that this decreasing variance phenomenon is not explained by the concept that “SGD implicitly regularizes.”
Revised: This means that this decreasing variance phenomenon is not explained by the concept that “SGD implicitly regularizes.”

Original: 3. D ECOUPLING VARIANCE DUE TO SAMPLING FROM VARIANCE DUE TO INITIALIZATION
Revised: 3. D ECOUPLING VARIANCE DUE TO SAMPLING FROM VARIANCE DUE TO INITIALIZATION

Original: In order to better understand this variance phenomenon in neural networks, we separate the variance due to sampling from the variance due to initialization, according to the law of total variance (Equation 5).
Revised: In order to better understand this variance phenomenon in neural networks, we separate the variance due to sampling from the variance due to initialization, according to the law of total variance (Equation 5).

Original: Contrary to what traditional bias-variance tradeoff intuition would suggest, we ﬁnd variance due to sampling is roughly independent of width (Figure 2).
Revised: Contrary to what traditional bias-variance tradeoff intuition would suggest, we ﬁnd variance due to sampling is roughly independent of width (Figure 2).

Original: Furthermore, we ﬁnd that variance due to initialization decreases with width, causing the joint variance to decrease with width (Figure 2).
Revised: Furthermore, we ﬁnd that variance due to initialization decreases with width, causing the joint variance to decrease with width (Figure 2).

Original: A body of recent work has provided evidence that over-parameterization (in width) helps optimization (Du  and Lee, 2018;
Revised: A body of recent work has provided evidence that over-parameterization (in width) helps gradient descent optimize to global minima in neural networks (Du et al., 2019; Du and Lee, 2018;

Original: Soltanolkotabi et al., 2017; Zhang et al., 2018; Ma et al., 2018; Livni et al., 2014).
Revised: Soltanolkotabi et al., 2017; Livni et al., 2014; Zhang et al., 2018).

Revised: Always reaching a global minimum implies low variance due to initialization on the training set .

Original: Our observations show that the over-parameterization (in width) effect on optimization seems to extend to generalization, on the data sets we consider.
Revised: Our observation of decreasing variance on the test set shows that the over-parameterization (in width) effect on optimization seems to extend to generalization, on the data sets we consider.

Original: 3. V ISUALIZATION WITH REGRESSION ON SINUSOID
Revised: 3. V ISUALIZATION WITH REGRESSION ON SINUSOID

Original: We trained different width neural networks on a noisy sinusoidal distribution with 80 independent training examples.
Revised: We trained different width neural networks on a noisy sinusoidal distribution with 80 independent training examples.

Original: This sinusoid regression setting also exhibits the familiar bias-variance trends (Figure 3c) and trends of the two components of the variance (Figure 5c).
Revised: This sinusoid regression setting also exhibits the familiar bias-variance trends (Figure 3c) and trends of the two components of the variance (Figure 5c).

Original: Because this setting is low-dimensional, we can visualize the learned functions.
Revised: Because this setting is low-dimensional, we can visualize the learned functions.

Original: The classic caricature of high capacity models is that they ﬁt the training data in a very erratic way (example in Figure 11 of Appendix B.4).
Revised: The classic caricature of high capacity models is that they ﬁt the training data in a very erratic way (example in Figure 11 of Appendix B.4).

Original: We ﬁnd that wider networks learn sinusoidal functions that are much more similar than the functions learned by their narrower counterparts (Figure 4).
Revised: We ﬁnd that wider networks learn sinusoidal functions that are much more similar than the functions learned by their narrower counterparts (Figure 4).

Original: We have analogous plots for all of the other widths and ones that visualize the variance similar to how it is commonly visualized for Gaussian processes in Appendix B.4. 
Revised: We have analogous plots for all of the other widths and ones that visualize the variance similar to how it is commonly visualized for Gaussian processes in Appendix B.4. [Figure] [Figure] [Figure]

Original:  V ARIANCE AND DEPTH
Revised:  V ARIANCE AND DEPTH

Original: In this section, we study the effect of depth on bias and variance by ﬁxing width and varying depth.
Revised: In this section, we study the effect of depth on bias and variance by ﬁxing width and varying depth.

Original: Historically, there have been pathological problems that cause deeper networks to experience higher test error than their shallower counterparts (He et al., 2016).
Revised: Historically, there have been pathological problems that cause deeper networks to experience higher test error than their shallower counterparts (Glorot and Bengio, 2010; He et al., 2016; Balduzzi et al., 2017).

Original: This indicates that there are some important confounding factors to control for when varying depth.
Revised: This indicates that there are some important confounding factors to control for when varying depth.

Original: The best control that we found is to use an initialization that achieves dynamical isometry , the condition that all of the singular values of the input-output Jacobian are 1 at initialization (Saxe et al., 2014; Pennington et al., 2017), as it allows networks to achieve test error that is nearly independent of depth (Figure 5b).
Revised: The best control that we found is to use an initialization that achieves dynamical isometry , the condition that all of the singular values of the input-output Jacobian are 1 at initialization (Saxe et al., 2014; Pennington et al., 2017), as it allows networks to achieve test error that is nearly independent of depth (Figure 5b).

Original: See Appendix C.1 for more discussion on this.
Revised: See Appendix C.1 for more discussion on this.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: 4. T OTAL VARIANCE EXPERIMENTS
Revised: 4. T OTAL VARIANCE EXPERIMENTS

Original: We train fully connected networks up to 200 layers deep and observe slowly increasing variance with depth (Figure 5a).
Revised: We train fully connected networks up to 200 layers deep and observe slowly increasing variance with depth (Figure 5a).

Original: The experimental protocol is similar to what it was in Section 3, with a few differences: All networks have width 100 and achieve 0 training error.
Revised: The experimental protocol is similar to what it was in Section 3, with a few differences: All networks have width 100 and achieve 0 training error.

Original: We train them to the same loss value of 5e-5 to control for differences in training loss.
Revised: We train them to the same loss value of 5e-5 to control for differences in training loss.

Original: This value was chosen carefully by observing when training error had been 0 for a long time.
Revised: This value was chosen carefully by observing when training error had been 0 for a long time.

Original: The 3 kinds of different networks we train are vanilla fully connected, fully connected with skip connections, and fully connected with dynamical isometry initialization.
Revised: The 3 kinds of different networks we train are vanilla fully connected, fully connected with skip connections, and fully connected with dynamical isometry initialization.

Original: We only show the experiment with dynamical isometry in the main paper, but the other 2 are in Appendix C.2 and  C.3.
Revised: We only show the experiment with dynamical isometry in the main paper, but the other two are in Appendix C.2 and Appendix C.3.

Original: We settle on fully connected networks without skip connections, initialized using the initialization
Revised: We settle on fully connected networks without skip connections, initialized using the initialization

Original: Pennington et al.
Revised: Pennington et al.

Original: (2017) recommend to achieve dynamical isometry.
Revised: (2017) recommend to achieve dynamical isometry.

Original: This is the best experimental protocol of the 3 we tried because it appears to largely mitigate the pathological problems that cause deeper networks to have higher test error.
Revised: This is the best experimental protocol of the three we tried because it appears to largely mitigate the pathological problems that cause deeper networks to have higher test error.

Original: We compare the test accuracy of skip connections to that of dynamical isometry in Figure 5b 3 to see that while the test accuracy of skip connections varies by over 1% from depth 25 to 100, the corresponding error bars for the dynamical isometry test errors overlap (although test error does increase by about 0.1% from depth 25 to 200).
Revised: We compare the test accuracy of skip connections to that of dynamical isometry in Figure 5b 5 to see that while the test accuracy of skip connections varies by over 1% from depth 25 to 100, the corresponding error bars for the dynamical isometry test errors overlap (although test error does increase by about 0.1% from depth 25 to 200).

Original: This near lack of dependence of test error on depth is why we view this experiment as having controlled for confounding factors sufﬁciently well.
Revised: This near lack of dependence of test error on depth is why we view this experiment as having controlled for confounding factors sufﬁciently well.

Original: Additionally, this is the only protocol of the 3 we tested where bias monotonically decreases with depth (Figure 5a, Appendix C).
Revised: Additionally, this is the only protocol of the three we tested where bias monotonically decreases with depth (Figure 5a, Appendix C).

Original: Just as new advancements, such as skip connections and dynamical isometry, have greatly helped with test set performance, there could still be future advancements that change these results.
Revised: Just as new advancements, such as skip connections and dynamical isometry, have greatly helped with test set performance, there could still be future advancements that change these results.

Original: For example, it seems plausible that we will eventually have model families whose test error decreases (with depth) until it plateaus and, similarly, variance that increases and plateaus.
Revised: For example, it seems plausible that we will eventually have model families whose test error decreases (with depth) until it plateaus and, similarly, variance that increases and plateaus.

Original: 4. D ECOUPLING VARIANCE DUE TO SAMPLING FROM VARIANCE DUE TO INITIALIZATION
Revised: 4. D ECOUPLING VARIANCE DUE TO SAMPLING FROM VARIANCE DUE TO INITIALIZATION

Original: To get a more ﬁne-grain look at the effect of depth on variance, we estimate the terms of the law of total variance in Figure 2, just as we did for width.
Revised: To get a more ﬁne-grain look at the effect of depth on variance, we estimate the terms of the law of total variance in Figure 2, just as we did for width.

Original: Surprisingly, variance due to sampling is roughly constant again.
Revised: Surprisingly, variance due to sampling is roughly constant again.

Original: Variance due to initialization increases with depth.
Revised: Variance due to initialization increases with depth.

Original: We view the increase in variance due to initialization that we observe as consistent with the conventional wisdom that Arora et al.
Revised: We view the increase in variance due to initialization that we observe as consistent with the conventional wisdom that Arora et al.

Original: (2018) summarizes: “Conventional wisdom in deep learning states that increasing depth improves expressiveness but complicates optimization.”
Revised: (2018) summarizes: “Conventional wisdom in deep learning states that increasing depth improves expressiveness but complicates optimization.”

Original: While Arora et al.
Revised: While Arora et al.

Original: (2018) focus on speed of training, the variance we measure in Figure 2 is about the diversity of different minima.
Revised: (2018) focus on speed of training, the variance we measure in Figure 2 is about the diversity of different minima.

Original: Increasing depth seems to lead to different initial starting points optimizing to increasingly different functions, as evaluated on the test set .
Revised: Increasing depth seems to lead to different initial starting points optimizing to increasingly different functions, as evaluated on the test set .

Original: Li et al.
Revised: Li et al.

Original: (2017, Figure 7) provide visualizations that suggest that increasing depth leads to increasingly “chaotic” loss landscapes, which would indicate increasing variance on the training set.
Revised: (2017, Figure 7) provide visualizations that suggest that increasing depth leads to increasingly “chaotic” loss landscapes, which would indicate increasing variance on the training set.

Original:  D ISCUSSION AND THEORETICAL INSIGHTS FOR INCREASING WIDTH
Revised:  D ISCUSSION AND THEORETICAL INSIGHTS FOR INCREASING WIDTH

Original: Our empirical results demonstrate that in the practical setting, variance due to initialization decreases with network width while variance due to sampling remains constant.
Revised: Our empirical results demonstrate that in the practical setting, variance due to initialization decreases with network width while variance due to sampling remains constant.

Original: In subsection 5.1, we review classical results from linear models and remark that these trends can be seen in over-parameterized linear models.
Revised: In Section 5.1, we review classical results from linear models and remark that these trends can be seen in over-parameterized linear models.

Original: In subsection 5.2 we take inspiration from linear models to provide analogous arguments for this phenomenon in increasingly wide neural networks, under strong assumptions.
Revised: In Section 5.2 we take inspiration from linear models to provide analogous arguments for this phenomenon in increasingly wide neural networks, under strong assumptions.

Original: In subsection 5.3, we discuss the mismatch between the trend of variance due to initialization with width vs. the corresponding trend with depth and  why the assumptions in subsection 5.2 might be increasingly inaccurate with deeper and deeper networks.
Revised: In Section 5.3, we note the mismatch between width and depth (the trend of variance due to initialization with width is opposite the corresponding trend with depth), and we discuss why the assumptions in Section 5. might be increasingly inaccurate with deeper and deeper networks.

Original: 5. I NSIGHTS FROM LINEAR MODELS
Revised: 5. I NSIGHTS FROM LINEAR MODELS

Original: The goal here is to gain insights from simple linear models.
Revised: The goal here is to gain insights from simple linear models.

Original: We discuss the standard setting which assumes a noisy linear mapping y
Revised: We discuss the standard setting which assumes a noisy linear mapping y

Original: =
Revised: =

Original: θ T x + (cid:15) between input feature vectors x ∈ R N and real outputs,
Revised: θ T x + (cid:15) between input feature vectors x ∈ R N and real outputs,

Original: where E ( (cid:15) )
Revised: where E ( (cid:15) )

Original: =

Revised: =

Original: and Var ( (cid:15) )
Revised: and Var ( (cid:15) )

Original: =
Revised: =

Original: σ 2 (cid:15) .
Revised: σ 2 (cid:15) .

Original: Note that x is not necessarily raw data, but can be thought of as the embedding of the raw data in R N , using feature functions; this allows for the “over-parameterized” setting in linear models where N > m , regardless of the dimensionality of the raw data.
Revised: Note that x is not necessarily raw data, but can be thought of as the embedding of the raw data in R N , using feature functions; this allows for the “over-parameterized” setting in linear models where N > m , regardless of the dimensionality of the raw data.

Original: We consider linear ﬁts ˆ y
Revised: We consider linear ﬁts ˆ y

Original: =
Revised: =

Original: ˆ θ T x obtained using mean-square error gradient-descent with random initialization.
Revised: ˆ θ T x obtained using mean-square error gradient-descent with random initialization.

Original: We revisit the standard variance analysis for linear regression (Hastie et al., 2009, Sec. 7.3), where one can give the explicit form of the gradient descent solution.
Revised: We revisit the standard variance analysis for linear regression (Hastie et al., 2009, Section 7.3), where one can give the explicit form of the gradient descent solution.

Original: For a training set S of size m , let X S denote the m × N data matrix whose i th row is the training point x
Revised: For a training set S of size m , let X S denote the m × N data matrix whose i th row is the training point x

Original: Ti .
Revised: Ti .

Original: We also introduce the input correlation matrices:
Revised: We also introduce the input correlation matrices:

Original: [Equation]
Revised: [Equation]

Original: The case where N ≤ m is standard: if X S has maximal rank, Σ S is invertible; the solution is independent of the initialization and given by
Revised: The case where N ≤ m is standard: if X S has maximal rank, Σ S is invertible; the solution is independent of the initialization and given by

Original: [Equation]
Revised: [Equation]

Original: In the “ﬁxed design” scenario, where we consider ﬁxed training points x i , the expected prediction variance with respect to noise is then,
Revised: In the “ﬁxed design” scenario, where we consider ﬁxed training points x i , the expected prediction variance with respect to noise is then

Original: [Equation]
Revised: [Equation]

Original: In this case, the variance grows with the number of parameters.
Revised: In this case, the variance grows with the number of parameters.

Original: For example, by replacing Σ with its unbiased estimator m − 1 Σ S , we recover the standard value ( N/m ) σ 2 (cid:15) (Hastie et al., 2009).
Revised: For example, by replacing Σ with its unbiased estimator m − 1 Σ S , we recover the standard value ( N/m ) σ 2 (cid:15) (Hastie et al., 2009).

Original: The “over-parametrized” case where N > m is more interesting: even if X S has maximal rank, Σ S is not invertible.
Revised: The “over-parametrized” case where N > m is more interesting: even if X S has maximal rank, Σ S is not invertible.

Original: The kernel of Σ S is the subspace U ⊥ S orthogonal to the span U S of the training points x i .
Revised: The kernel of Σ S is the subspace U ⊥ S orthogonal to the span U S of the training points x i .

Original: Gradient descent updates belong to U S , independent of U ⊥ S .
Revised: Gradient descent updates belong to U S , independent of U ⊥ S .

Original: Initialized at θ 0 , it gives the solution,
Revised: Initialized at θ 0 , it gives the solution

Original: [Equation]
Revised: [Equation]

Original: where P S and P S ⊥ are the projections onto U S and U ⊥ S , and superscript + denotes the pseudoinverse.
Revised: where P S and P S ⊥ are the projections onto U S and U ⊥ S , and superscript + denotes the pseudoinverse.

Original: The ﬁrst term, orthogonal to the data, does not get updated during training and only depends on the initialization.
Revised: The ﬁrst term, orthogonal to the data, does not get updated during training and only depends on the initialization.

Original: The two others form the minimum norm solution, which lies in U S .
Revised: The two others form the minimum norm solution, which lies in U S .

Original: The form of the solution (9) has several consequences: (a) Initialization contributes to the variance.
Revised: The form of the solution (Equation 9) has several consequences: (a) Initialization contributes to the variance.

Original: Thus, for the input x and using a standard initializationθ 0 ∼ N (0 , 1 N I ) , we obtain
Revised: Thus, for the input x and using a standard initializationθ 0 ∼ N (0 , 1 N I ) , we obtain

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which is non zero whenever x have components orthogonal to the training data.
Revised: which is non zero whenever x has components orthogonal to the training data.

Original: Note however that the variance due to initialization actually decreases with the number of parameters.
Revised: Note, however, that the variance due to initialization actually decreases with the number of parameters.

Original: (b) The expected variance due to noise is,
Revised: (b) The expected variance due to noise is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: In this case, the variance scales as the dimension  the data , as opposed to the number of parameters.
Revised: In this case, the variance scales as the dimension of the data , as opposed to the number of parameters.

Original: Thus, replacing Σ by its unbiased estimator m − 1 Σ S , we ﬁnd the value ( r/m ) σ 2 (cid:15) where r = rank(Σ S ) =
Revised: Thus, replacing Σ by its unbiased estimator m − 1 Σ S , we ﬁnd the value ( r/m ) σ 2 (cid:15) where r = rank(Σ S ) =

Original: dim U S .
Revised: dim U S .

Original: We argue in the next section that, under speciﬁc assumptions that we discuss, these insights may be relevant for the non-linear case.
Revised: We argue in the next section that, under speciﬁc assumptions that we discuss, these insights may be relevant for the non-linear case.

Original: 5. B ACK TO NEURAL NETWORKS
Revised: 5. A MORE GENERAL RESULT

Original: We will illustrate our arguments in the following simpliﬁed setting.
Revised: We will illustrate our arguments in the following simpliﬁed setting.

Original: Setting.
Revised: Setting.

Original: Let N be the dimension of the parameter space.
Revised: Let N be the dimension of the parameter space.

Original: The prediction for a ﬁxed example x , given by a trained network parameterized by θ depends on:
Revised: The prediction for a ﬁxed example x , given by a trained network parameterized by θ depends on:

Original: (i) a subspace of the parameter space, M ∈ R N with relatively small dimension, d ( N ) , which depends only on the learning task.
Revised: (i) a subspace of the parameter space, M ∈ R N with relatively small dimension, d ( N ) , which depends only on the learning task.

Original: (ii) parameter components corresponding to directions orthogonal to M .
Revised: (ii) parameter components corresponding to directions orthogonal to M .

Original: The orthogonal M ⊥ of M has dimension, N − d ( N ) , and is essentially irrelevant to the learning task.
Revised: The orthogonal M ⊥ of M has dimension, N − d ( N ) , and is essentially irrelevant to the learning task.

Original: We can write the parameter vector as a sum of these two components θ = θ M + θ M ⊥ .
Revised: We can write the parameter vector as a sum of these two components θ = θ M + θ M ⊥ .

Original: We will further make the following assumptions.
Revised: We will further make the following assumptions.

Original: (a) The optimization of the loss function is invariant with respect to θ M⊥ .
Revised: (a) The optimization of the loss function is invariant with respect to θ M⊥ .

Original: (b) Along M , optimization  yields solutions independently of the initialization θ 0 .
Revised: (b) Regardless of initialization, the optimization method consistently yields a solution with the same θ M component, (i.e. the same vector when projected onto M ).

Original: These are strong assumptions.
Revised: These are strong assumptions, but there is some support for them in the literature.

Original: Li et al.
Revised: Li et al.

Original: (2018) empirically showed the existence of a critical number d ( N ) =
Revised: (2018) empirically showed the existence of a critical number d ( N ) =

Original: d of relevant parameters for a given learning task, independent of the size of the model.
Revised: d of relevant parameters for a given learning task, independent of the size of the model.

Revised: Sagun et al.

Revised: (2017) showed that the spectrum of the Hessian for over-parametrized networks splits into ( i ) a bulk centered near zero and ( ii ) a small number of large eigenvalues, which suggests 7 that learning occurs mainly in a small number of directions.

Original: The existence of a subspace M ⊥ in which no learning occurs was also conjectured byAdvani and
Revised: The existence of a subspace M ⊥ in which no learning occurs was also conjectured by

Original:  Saxe (2017) andempirically shown to hold true 5 for deep linear networks.
Revised: Advani and Saxe (2017) and shown to hold in deep linear networks under a simplifying assumption that decouples the dynamics of the weights in different layers.

Original: 5.2. V ARIANCE DUE TO INITIALIZATION
Revised: 5.2. V ARIANCE DUE TO INITIALIZATION

Original: Given the above assumptions, the following result shows that the variance from initialization vanishes as we increase N .
Revised: Given the above assumptions, the following result shows that the variance from initialization vanishes as we increase N .

Original: The full proof, which builds on concentration results for Gaussians (based on Levy’s lemma (Ledoux, 2001)), is given in Appendix D.
Revised: The full proof, which builds on concentration results for Gaussians (based on Levy’s lemma (Ledoux, 2001)), is given in Appendix D.

Original: Theorem 1 (Decay of variance due to initialization) .
Revised: Theorem 1 (Decay of variance due to initialization) .

Original: Consider the setting of Section 5.2 Let θ denote the parameters at the end of the learning process.
Revised: Consider the setting of Section 5.2 Let θ denote the parameters at the end of the learning process.

Original: Then, for a ﬁxed data set and parameters initialized as θ 0 ∼ N (0 , 1 N I ) , the variance of the prediction satisﬁes the inequality,
Revised: Then, for a ﬁxed data set and parameters initialized as θ 0 ∼ N (0 , 1 N I ) , the variance of the prediction satisﬁes the inequality,

Original: [Equation]
Revised: [Equation]

Original: N (12) where L is the Lipschitz constant of the prediction with respect to θ , and for some universal constant
Revised: (12) where L is the Lipschitz constant of the prediction with respect to θ , and for some universal constant

Original: C > O .
Revised: C > O .

Original: This result guarantees that the variance decreases to zero as N increases, provided the Lipschitz constant L grows more slowly than the square root of dimension, L = o ( √ N ) .
Revised: This result guarantees that the variance decreases to zero as N increases, provided the Lipschitz constant L grows more slowly than the square root of dimension, L = o ( √ N ) .

Original: 5.2. V ARIANCE DUE TO SAMPLING
Revised: 5.2. V ARIANCE DUE TO SAMPLING

Original: Under the above assumptions, the parameters at the end of learning take the form θ = θ ∗M + θ 0 M ⊥ .
Revised: Under the above assumptions, the parameters at the end of learning take the form θ = θ ∗M + θ 0 M ⊥ .

Original: For ﬁxed initialization, the only source of variance of the prediction is the randomness of θ ∗M on the learning manifold.
Revised: For ﬁxed initialization, the only source of variance of the prediction is the randomness of θ ∗M on the learning manifold.

Original: The variance depends on the parameter dimensionality only through dim M = d ( N ) , and hence remains constant if d ( N ) does (Li et al., 2018).
Revised: The variance depends on the parameter dimensionality only through dim M = d ( N ) , and hence remains constant if d ( N ) does (Li et al., 2018).

Original: 5. D ISCUSSION ON ASSUMPTIONS IN INCREASINGLY DEEP NETWORKS
Revised: 5. D ISCUSSION ON ASSUMPTIONS IN INCREASINGLY DEEP NETWORKS

Original: The mismatch between the outcome of our theoretical analysis and the observed trend of variance due to initialization with depth suggests that our assumptions are increasingly strong with depth.
Revised: The mismatch between the outcome of our theoretical analysis and the observed trend of variance due to initialization with depth suggests that our assumptions are increasingly inaccurate with depth.

Original: For example, even for linear networks, assumption (a) seems decreasingly plausible as depth increases (Advani and Saxe, 2017).
Revised: For some intuition about why this may be the case, consider the dependence of gradients with respect to subsets of hidden units, as these gradients are related to assumption (a): the invariance of the optimization process to θ M⊥ .

Original: 5 for small enough initial weights.
Revised: Gradients of hidden units in the same layer (related to width) do not directly depend on each other; rather, only optimization induces dependencies between them via the loss function. In sharp contrast, hidden units in different layers (related to depth) are functions of their preceding layers, and similarly, the gradients with respect to earlier layers are functionally dependent on the gradients with respect to later layers. This hints at more complex optimization interactions between parameters when increasing depth.

Original:  C ONCLUSION AND FUTURE WORK
Revised:  C ONCLUSION AND FUTURE WORK

Original: By revisiting the bias-variance decomposition and using a ﬁner-grain method to empirically study variance, we ﬁnd interesting phenomena.
Revised: By revisiting the bias-variance decomposition and using a ﬁner-grain method to empirically study variance, we ﬁnd interesting phenomena.

Original: First, the bias-variance tradeoff is misleading for network width (one way to increase size) as the measure of model complexity.
Revised: First, the bias-variance tradeoff is misleading for network width (one way to increase size) as the measure of model complexity.

Original: Second, variance due to sampling does not appear to be dependent on width or depth.
Revised: Second, variance due to sampling does not appear to be dependent on width or depth.

Original: Third, variance due to initialization is roughly consistent with the optimization literature, as we observe the test set analog of the current conventional wisdom for both width and depth.
Revised: Third, variance due to initialization is roughly consistent with the optimization literature, as we observe the test set analog of the current conventional wisdom for both width and depth.

Original: Finally, by taking inspiration from linear models, we perform a theoretical analysis of the variance that is consistent with our empirical observations for increasing width.
Revised: Finally, by taking inspiration from linear models, we perform a theoretical analysis of the variance that is consistent with our empirical observations for increasing width.

Original: We view future work that uses the bias-variance lens as promising.
Revised: We view future work that uses the bias-variance lens as promising.

Original: For example, a probabilistic notion of effective capacity of a model is natural when studying generalization through this lens (Appendix A).
Revised: For example, a probabilistic notion of effective capacity of a model is natural when studying generalization through this lens (Appendix A).

Original: We did not study how bias and variance change over the course of training; that would make an interesting direction for future work.
Revised: We did not study how bias and variance change over the course of training; that would make an interesting direction for future work.

Revised: Additionally, it may be fruitful to apply the biasvariance lens to other network architectures, such as convolutional networks and recurrent networks.

Original: We argue it is worth running variance vs. depth experiments using future best practices to train deep models, as the results could be different.
Revised: We argue it is worth running variance vs. depth experiments using future best practices to train deep models, as the results could be different.

Original: More theoretical work is also needed to achieve a full understanding of the behaviour of variance in deep models.
Revised: More theoretical work is also needed to achieve a full understanding of the behaviour of variance in deep models.

Revised: Variance is analytically different from generalization error in that the deﬁnition of variance does not involve the labels at all.

Original: We view the bias-variance lens as a useful tool for studying generalization in deep learning and hope to encourage more work in this direction.
Revised: We view the bias-variance lens as a useful tool for studying generalization in deep learning and hope to encourage more work in this direction.

Original: Appendices A PPENDIX A P ROBABILISTIC NOTION OF EFFECTIVE CAPACITY
Revised: Appendices A PPENDIX A P ROBABILISTIC NOTION OF EFFECTIVE CAPACITY

Original: The problem with classical complexity measures is that they do not take into account optimization and have no notion of what will actually be learned.
Revised: The problem with classical complexity measures is that they do not take into account optimization and have no notion of what will actually be learned.

Original: Arpit et al.
Revised: Arpit et al.

Original: (2017) deﬁne a notion of an effective hypothesis class to take into account what functions are possible to be learned by the learning algorithm.
Revised: (2017, Section 1) deﬁne a notion of an effective hypothesis class to take into account what functions are possible to be learned by the learning algorithm.

Original: However, this still has the problem of not taking into account what hypotheses are likely to be learned.
Revised: However, this still has the problem of not taking into account what hypotheses are likely to be learned.

Original: To take into account the probabilistic nature of learning, we deﬁne the (cid:15) - hypothesis class for a data distribution D and learning algorithm A , that contains the hypothesis which are at least (cid:15) -likely for some (cid:15) > 0 : 
Revised: To take into account the probabilistic nature of learning, we deﬁne the (cid:15) - hypothesis class for a data distribution D and learning algorithm A , that contains the hypotheses which are at least (cid:15) -likely for some (cid:15) > 0 : [Equation] D (

Original: [Equation]
Revised: [Equation]

Revised: = {

Revised: [Equation]

Revised: [Equation]

Original:  (13) where S is a training set drawn from D m , h ( A , S ) is a random variable drawn from the distribution over learned functions induced by D and the randomness in A ; p is the corresponding density.
Revised: [Equation] (cid:15) [Equation] , (13) where S is a training set drawn from D m , h ( A , S ) is a random variable drawn from the distribution over learned functions induced by D and the randomness in A ; p is the corresponding density.

Original: Thinking about a model’s (cid:15) -hypothesis class can lead to drastically different intuitions for the complexity of a model and its variance (Figure 6).
Revised: Thinking about a model’s (cid:15) -hypothesis class can lead to drastically different intuitions for the complexity of a model and its variance (Figure 6).

Original: This is at the core of the intuition for why the traditional view of bias-variance as a tradeoff does not hold in all cases.
Revised: This is at the core of the intuition for why the traditional view of bias-variance as a tradeoff does not hold in all cases.

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX B W IDTH AND VARIANCE : ADDITIONAL EMPIRICAL RESULTS AND DISCUSSION B. T UNED LEARNING RATES FOR SGD
Revised: A PPENDIX B W IDTH AND VARIANCE : ADDITIONAL EMPIRICAL RESULTS AND DISCUSSION B. T UNED LEARNING RATES FOR SGD

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: B. F IXED LEARNING RATE RESULTS FOR SMALL DATA MNIST
Revised: B. F IXED LEARNING RATE RESULTS FOR SMALL DATA MNIST

Original: [Figure]
Revised: [Figure]

Original: Note that the U curve shown in Figure 8 when we do not tune the step size is explained by the fact that the constant step chosen is a “good” step size for some networks and “bad” for others.
Revised: Note that the U curve shown in Figure 8 when we do not tune the step size is explained by the fact that the constant step chosen is a “good” step size for some networks and “bad” for others.

Original: Results from Keskar et al.
Revised: Results from Keskar et al.

Original: (2017) and Smith et al.
Revised: (2017) and Smith et al.

Original: (2018) show that a step size that corresponds well to the noise structure in SGD is important for achieving good test set accuracy.
Revised: (2018) show that a step size that corresponds well to the noise structure in SGD is important for achieving good test set accuracy.

Original: Because our networks are different sizes, their stochastic optimization process will have a different landscape and noise structure.
Revised: Because our networks are different sizes, their stochastic optimization process will have a different landscape and noise structure.

Original: By tuning the step size, we are making the experimental design choice to keep optimality of step size constant across networks, rather than keeping step size constant across networks.
Revised: By tuning the step size, we are making the experimental design choice to keep optimality of step size constant across networks, rather than keeping step size constant across networks.

Original: To us, choosing this control makes much more sense than choosing to control for step size.
Revised: To us, choosing this control makes much more sense than choosing to control for step size.

Original: B. O THER OPTIMIZERS FOR WIDTH EXPERIMENT ON SMALL DATA MNIST
Revised: B. O THER OPTIMIZERS FOR WIDTH EXPERIMENT ON SMALL DATA MNIST

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: B. S INUSOID REGRESSION EXPERIMENTS
Revised: B. S INUSOID REGRESSION EXPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: sinusoid
Revised: sinusoid

Original: Below, we ﬁnd that this does not happen with increasingly wide neural networks (Figure 13).
Revised: Below, we ﬁnd that this does not happen with increasingly wide neural networks (Figure 13 and Figure 14)

Original: [Figure]
Revised: [Figure]

Original: A PPENDIX C D EPTH AND VARIANCE : ADDITIONAL EMPIRICAL RESULTS AND DISCUSSION C. D ISCUSSION ON NEED FOR CAREFUL EXPERIMENTAL DESIGN
Revised: A PPENDIX C D EPTH AND VARIANCE : ADDITIONAL EMPIRICAL RESULTS AND DISCUSSION C. D ISCUSSION ON NEED FOR CAREFUL EXPERIMENTAL DESIGN

Original: Depth is an important component of deep learning.
Revised: Depth is an important component of deep learning.

Original: We study its effect on bias and variance by ﬁxing width and varying depth.
Revised: We study its effect on bias and variance by ﬁxing width and varying depth.

Original: However, there are pathological problems associated with training very deep networks such as vanishing/exploding gradient (Hochreiter, 1991; Bengio et al., 1994; Glorot and Bengio, 2010), signal not being able to propagate through through the network (Schoenholz et al., 2017), and gradients resembling white noise (Balduzzi et al., 2017).
Revised: However, there are pathological problems associated with training very deep networks such as vanishing/exploding gradient (Hochreiter, 1991; Bengio et al., 1994; Glorot and Bengio, 2010), signal not being able to propagate through through the network (Schoenholz et al., 2017), and gradients resembling white noise (Balduzzi et al., 2017).

Original: He et al.
Revised: He et al.

Original: (2016) pointed out that very deep networks experience high test set error and argued it was due to high training set loss.
Revised: (2016) pointed out that very deep networks experience high test set error and argued it was due to high training set loss.

Original: However, while skip connections (He et al., 2016), better initialization (Glorot and Bengio, 2010), and batch normalization (Ioffe and Szegedy, 2015) have largely served to facilitate low training loss in very deep networks, the problem of high test set error still remains.
Revised: However, while skip connections (He et al., 2016), better initialization (Glorot and Bengio, 2010), and batch normalization (Ioffe and Szegedy, 2015) have largely served to facilitate low training loss in very deep networks, the problem of high test set error still remains.

Original: The current best practices for achieving low test error in very deep networks arose out of trying to solve the above problems in training.
Revised: The current best practices for achieving low test error in very deep networks arose out of trying to solve the above problems in training.

Original: An initial step was to ensure the mean squared singular value of the input-output Jacobian, at initialization, is close to 1 (Glorot and Bengio, 2010).
Revised: An initial step was to ensure the mean squared singular value of the input-output Jacobian, at initialization, is close to 1 (Glorot and Bengio, 2010).

Original: More recently, there has been work on a stronger condition known as dynamical isometry , where all singular values remain close to 1 (Saxe et al., 2014; Pennington et al., 2017).
Revised: More recently, there has been work on a stronger condition known as dynamical isometry , where all singular values remain close to 1 (Saxe et al., 2014; Pennington et al., 2017).

Original: Pennington et al.
Revised: Pennington et al.

Original: (2017) also empirically found that dynamical isometry helped achieve low test set error.
Revised: (2017) also empirically found that dynamical isometry helped achieve low test set error.

Original: Furthermore, Xiao et al.
Revised: Furthermore, Xiao et al.

Original: (2018, Figure 1) found evidence that test set performance did not degrade with depth when they lifted dynamical isometry to CNNs.
Revised: (2018, Figure 1) found evidence that test set performance did not degrade with depth when they lifted dynamical isometry to CNNs.

Original: This why we settled on dynamical isometry as the best known practice to control for as many confounding factors as possible.
Revised: This why we settled on dynamical isometry as the best known practice to control for as many confounding factors as possible.

Original: We ﬁrst ran experiments with vanilla full connected networks (Figure 16).
Revised: We ﬁrst ran experiments with vanilla full connected networks (Figure 16).

Original: These have clear training issues where networks of depth more than 20 take very long to train to the target training loss of 5e-
Revised: These have clear training issues where networks of depth more than 20 take very long to train to the target training loss of 5e-5.

Original: The bias curve is not even monotonically decreasing.
Revised: The bias curve is not even monotonically decreasing.

Original: Clearly, there are important confounding factors not controlled for in this simple setting.
Revised: Clearly, there are important confounding factors not controlled for in this simple setting.

Original: Still, note that variance increases roughly linearly with depth.
Revised: Still, note that variance increases roughly linearly with depth.

Original: We then study fully connected networks with skip connections between every 2 layers (Figure 17).
Revised: We then study fully connected networks with skip connections between every 2 layers (Figure 17).

Original: While this allows us to train deeper networks than without skip connections, many of the same issues persist (e
Revised: While this allows us to train deeper networks than without skip connections, many of the same issues persist (e

Original: [Figure]
Revised: [Figure]

Original: g. bias still not monotonically decreasing).
Revised: g. bias still not monotonically decreasing).

Original: The bias, variance, and test error curves are all checkmark-shaped.
Revised: The bias, variance, and test error curves are all checkmark-shaped.

Original: C. V ANILLA FULLY CONNECTED DEPTH EXPERIMENTS
Revised: C. V ANILLA FULLY CONNECTED DEPTH EXPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: C. S KIP CONNECTIONS DEPTH EXPERIMENTS
Revised: C. S KIP CONNECTIONS DEPTH EXPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: C. D YNAMICAL ISOMETRY DEPTH EXPERIMENTS
Revised: C. D YNAMICAL ISOMETRY DEPTH EXPERIMENTS

Original: The ﬁgures in this section are included in the main paper but they are included here for comparison to the above and for completeness.
Revised: The ﬁgures in this section are included in the main paper but they are included here for comparison to the above and for completeness.

Original: [Figure]
Revised: [Figure]

Original: have training error 

Revised: have training error

Original: and are trained to the same training loss of 5e-5.
Revised: and are trained to the same training loss of 5e-5.

Original: A PPENDIX D S OME P ROOFS D. P ROOF OF T HEOREM
Revised: A PPENDIX D S OME P ROOFS D. P ROOF OF T HEOREM

Original: First we state some known concentration results (Ledoux, 2001) that we will use in the proof.
Revised: First we state some known concentration results (Ledoux, 2001) that we will use in the proof.

Original: Lemma 1 (Levy) .
Revised: Lemma 1 (Levy) .

Original: Let h : S nR → R be a function on the n -dimensional Euclidean sphere of radius
Revised: Let h : S nR → R be a function on the n -dimensional Euclidean sphere of radius

Original: R , with Lipschitz constant L ; and θ ∈ S nR chosen uniformly at random for the normalized measure.
Revised: R , with Lipschitz constant L ; and θ ∈ S nR chosen uniformly at random for the normalized measure.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: for some universal constant C > 0 .
Revised: for some universal constant C > 0 .

Original: Uniform measures on high dimensional spheres approximate Gaussian distributions (Ledoux, 2001).
Revised: Uniform measures on high dimensional spheres approximate Gaussian distributions (Ledoux, 2001).

Original: Using this, Levy’s lemma yields an analogous concentration inequality for functions of Gaussian variables:
Revised: Using this, Levy’s lemma yields an analogous concentration inequality for functions of Gaussian variables:

Original: Lemma 2 (Gaussian concentration) .
Revised: Lemma 2 (Gaussian concentration) .

Original: Let h : R n → R be a function on the Euclidean space R n , with Lipschitz constant L ; and θ ∼ N (0 , σ I n ) sampled from an isotropic n -dimensional Gaussian.
Revised: Let h : R n → R be a function on the Euclidean space R n , with Lipschitz constant L ; and θ ∼ N (0 , σ I n ) sampled from an isotropic n -dimensional Gaussian.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: for some universal constant C > 0 .
Revised: for some universal constant C > 0 .

Original: Note that in the Gaussian case, the bound is dimension free.
Revised: Note that in the Gaussian case, the bound is dimension free.

Original: In turn, concentration inequalities give variance bounds for functions of random variables.
Revised: In turn, concentration inequalities give variance bounds for functions of random variables.

Original: Corollary 1.
Revised: Corollary 1.

Original: Let h be a function satisfying the conditions of Theorem 2, and Var ( h ) =
Revised: Let h be a function satisfying the conditions of Theorem 2, and Var ( h ) =

Original: E [( h − E
Revised: E [( h − E

Original: [ h ]) 2 ] .
Revised: [ h ]) 2 ] .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let g = h − E
Revised: Let g = h − E

Original: [ h ] .
Revised: [ h ] .

Original: Then Var ( h ) =
Revised: Then Var ( h ) =

Original: Var ( g ) and
Revised: Var ( g ) and

Original: [Equation]
Revised: [Equation]

Original: Now swapping expectation and integral (by Fubini theorem), and by using the identity E 1 | g | >t = P ( | g | > t ) , we obtain
Revised: Now swapping expectation and integral (by Fubini theorem), and by using the identity E 1 | g | >t = P ( | g | > t ) , we obtain

Original: [Equation]
Revised: [Equation]

Original: We are now ready to prove Theorem 1 .
Revised: We are now ready to prove Theorem 1 .

Original: We ﬁrst recall our assumptions.
Revised: We ﬁrst recall our assumptions.

Original: Assumption 1.
Revised: Assumption 1.

Original: The optimization of the loss function is invariant with respect to θ M⊥ .
Revised: The optimization of the loss function is invariant with respect to θ M⊥ .

Original: With the exception of random initialization, the learning method does not alter θ M⊥ .

Original: Assumption 2.
Revised: Assumption 2.

Original: Along M , optimization yields solutions independently of the initialization θ 0 .
Revised: Along M , optimization yields solutions independently of the initialization θ 0 .

Original: We add the following assumptions.
Revised: We add the following assumptions.

Original: Assumption 3.
Revised: Assumption 3.

Original: The prediction h θ ( x ) is L -Lipschitz with respect to θ M⊥ .
Revised: The prediction h θ ( x ) is L -Lipschitz with respect to θ M⊥ .

Original: Assumption 4.
Revised: Assumption 4.

Original: The network parameters are initialized as
Revised: The network parameters are initialized as

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst prove that the Gaussian concentration theorem translates into concentration of predictions in the setting of Section 5.2.1.
Revised: We ﬁrst prove that the Gaussian concentration theorem translates into concentration of predictions in the setting of Section 5.2.1.

Original: Theorem 2 (Concentration of predictions) .
Revised: Theorem 2 (Concentration of predictions) .

Original: Consider the setting of Section 5.2 and Assumptionsand 4.
Revised: Consider the setting of Section 5.2 and Assumptionsand 4.

Original: Let θ denote the parameters at the end of the learning process.
Revised: Let θ denote the parameters at the end of the learning process.

Original: Then, for a ﬁxed data set, S we get concentration of the prediction, under initialization randomness,
Revised: Then, for a ﬁxed data set, S we get concentration of the prediction, under initialization randomness,

Original: [Equation]
Revised: [Equation]

Original: for some universal constant C > 0 .
Revised: for some universal constant C > 0 .

Original: Proof.
Revised: Proof.

Original: In our setting, the parameters at the end of learning can be expressed as
Revised: In our setting, the parameters at the end of learning can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where θ ∗M is independent of the initialization θ 0 .
Revised: where θ ∗M is independent of the initialization θ 0 .

Original: To simplify notation, we will assume that, at least locally around θ ∗M , M is spanned by the ﬁrst d ( N ) standard basis vectors, and M ⊥ by the remaining N − d ( N ) .
Revised: To simplify notation, we will assume that, at least locally around θ ∗M , M is spanned by the ﬁrst d ( N ) standard basis vectors, and M ⊥ by the remaining N − d ( N ) .

Original: This will allow us, from now on, to use the same variable names for θ M and θ M ⊥ to denote their lower-dimensional representations of dimension d ( N ) and N − d ( N ) respectively.
Revised: This will allow us, from now on, to use the same variable names for θ M and θ M ⊥ to denote their lower-dimensional representations of dimension d ( N ) and N − d ( N ) respectively.

Original: More generally, we can assume that there is a mapping from θ M and θ M ⊥ to those lower-dimensional representations.
Revised: More generally, we can assume that there is a mapping from θ M and θ M ⊥ to those lower-dimensional representations.

Original: From Assumptions 1 and 4 we get
Revised: From Assumptions 1 and 4 we get

Original: [Equation]
Revised: [Equation]

Original: Let g ( θ M ⊥ )
Revised: Let g ( θ M ⊥ )

Original: (cid:44) h θ ∗M
Revised: (cid:44) h θ ∗M

Original: + θ M⊥ ( x ) .
Revised: + θ M⊥ ( x ) .

Original: By Assumption 3, g ( · ) is L -Lipschitz.
Revised: By Assumption 3, g ( · ) is L -Lipschitz.

Original: Then, by the Gaussian concentration theorem we get,
Revised: Then, by the Gaussian concentration theorem we get,

Original: [Equation]
Revised: [Equation]

Original: The result of Theorem 1 immediately follows from Theorem 2 and Corollary 1, with σ 2 = 1 /N :
Revised: The result of Theorem 1 immediately follows from Theorem 2 and Corollary 1, with σ 2 = 1 /N :

Original: [Equation]
Revised: [Equation]

Original: Provided the Lipschitz constant L of the prediction grows more slowly than the square of dimension,
Revised: Provided the Lipschitz constant L of the prediction grows more slowly than the square of dimension,

Original: L = o ( √ N ) , we conclude that the variance vanishes to zero as N grows.
Revised: L = o ( √ N ) , we conclude that the variance vanishes to zero as N grows.

Original: D. B OUND ON CLASSIFICATION ERROR IN TERMS OF REGRESSION ERROR
Revised: D. B OUND ON CLASSIFICATION ERROR IN TERMS OF REGRESSION ERROR

Original: In this section we give a bound on classiﬁcation risk  in terms of the regression risk.
Revised: In this section we give a bound on classiﬁcation risk R classif in terms of the regression risk R reg .

Original:  Our classiﬁer deﬁnes a map h : X → R k , which outputs probability vectors h ( x ) ∈ R k , with (cid:80) ky =1 h ( x ) y = 1 .
Revised: Notation. Our classiﬁer deﬁnes a map h : X → R k , which outputs probability vectors h ( x ) ∈ R k , with (cid:80) ky =1 h ( x ) y = 1 .

Original: The classiﬁcation loss is deﬁned by
Revised: The classiﬁcation loss is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where I ( a ) = 1 if predicate a is true and 0 otherwise.
Revised: ) where I ( a ) = 1 if predicate a is true and 0 otherwise.

Original: We will exploit the bound L ( h ) ≤ (cid:101) L ( h ) where [Equation]
Revised: Given trained predictors h S indexed by training dataset S , the classiﬁcation and regression risks are given by,

Original: [Equation]
Revised: [Equation]

Original: Given trained predictors h S indexed by the training dataset S , we denote by R = E S L ( h S ) and (cid:101) R = E S (cid:101)
Revised: where Y denotes the one-hot vector representation of the class y . Proposition 1.

Original: L ( h S ) the frequentist risks associated to these losses. We will also consider the frequentist risk associated to the mean square error,
Revised: The classiﬁcation risk is bounded by four times the regression risk, R classif ≤ 4 R reg . Proof. First note that, if h ( x ) ∈ R k is a probability vector, then

Original: [Equation]
Revised: [Equation]

Original: where Y denotes the one-hot vector representation of the class y . We will bound the classiﬁcation risk in terms of the mean square error risk, by showing the inequality 
Revised: By taking the expectation over x, y , we obtain the inequality L ( h ) ≤ (cid:101)

Original: R ≤ 4 R . We ﬁrst note,
Revised: L ( h ) where

Original: [Equation]
Revised: [Equation]

Original: Next, we show,
Revised: We then have,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Markov’s inequality applied to the frequentist risk. Since
Revised: where the last inequality follows from Markov’s inequality.

Original: R ≤ (cid:101) R , we conclude that R ≤ 4 R .

Original: A PPENDIX E C OMMON INTUITIONS FROM IMPACTFUL WORKS
Revised: A PPENDIX E C OMMON INTUITIONS FROM IMPACTFUL WORKS

Original: “Neural Networks and the Bias/Variance Dilemma” from (Geman et al., 1992): “How big a network should we employ?
Revised: “Neural Networks and the Bias/Variance Dilemma” from (Geman et al., 1992): “How big a network should we employ?

Original: A small network, with say one hidden unit, is likely to be biased, since the repertoire of available functions spanned by f ( x ; w ) over allowable weights will in this case be quite limited.
Revised: A small network, with say one hidden unit, is likely to be biased, since the repertoire of available functions spanned by f ( x ; w ) over allowable weights will in this case be quite limited.

Original: If the true regression is poorly approximated within this class, there will necessarily be a substantial bias.
Revised: If the true regression is poorly approximated within this class, there will necessarily be a substantial bias.

Original: On the other hand, if we overparameterize, via a large number of hidden units and associated weights, then the bias will be reduced (indeed, with enough weights and hidden units, the network will interpolate the data), but there is then the danger of a signiﬁcant variance contribution to the mean-squared error.
Revised: On the other hand, if we overparameterize, via a large number of hidden units and associated weights, then the bias will be reduced (indeed, with enough weights and hidden units, the network will interpolate the data), but there is then the danger of a signiﬁcant variance contribution to the mean-squared error.

Original: (This may actually be mitigated by incomplete convergence of the minimization algorithm, as we shall see in Section 3.5.5.)”
Revised: (This may actually be mitigated by incomplete convergence of the minimization algorithm, as we shall see in Section 3.5.5.)”

Original: “An Overview of Statistical Learning Theory” from (Vapnik, 1999): “To avoid over ﬁtting (to get a small conﬁdence interval) one has to construct networks with small VC-dimension.”
Revised: “An Overview of Statistical Learning Theory” from (Vapnik, 1999): “To avoid over ﬁtting (to get a small conﬁdence interval) one has to construct networks with small VC-dimension.”

Original: “Stability and Generalization” from Bousquet and Elisseeff (2002): “It has long been known that when trying to estimate an unknown function from data, one needs to ﬁnd a tradeoff between bias and variance.
Revised: “Stability and Generalization” from Bousquet and Elisseeff (2002): “It has long been known that when trying to estimate an unknown function from data, one needs to ﬁnd a tradeoff between bias and variance.

Original: Indeed, on one hand, it is natural to use the largest model in order to be able to approximate any function, while on the other hand, if the model is too large, then the estimation of the best function in the model will be harder given a restricted amount of data.”
Revised: Indeed, on one hand, it is natural to use the largest model in order to be able to approximate any function, while on the other hand, if the model is too large, then the estimation of the best function in the model will be harder given a restricted amount of data."

Original: Footnote: “We deliberately do not provide a precise deﬁnition of bias and variance and resort to common intuition about these notions.”
Revised: Footnote: “We deliberately do not provide a precise deﬁnition of bias and variance and resort to common intuition about these notions."

Original: Pattern Recognition and Machine Learning from Bishop (2006): “Our goal is to minimize the expected loss, which we have decomposed into the sum of a (squared) bias, a variance, and a constant noise term.
Revised: Pattern Recognition and Machine Learning from Bishop (2006): “Our goal is to minimize the expected loss, which we have decomposed into the sum of a (squared) bias, a variance, and a constant noise term.

Original: As we shall see, there is a trade-off between bias and variance, with very ﬂexible models having low bias and high variance, and relatively rigid models having high bias and low variance.”
Revised: As we shall see, there is a trade-off between bias and variance, with very ﬂexible models having low bias and high variance, and relatively rigid models having high bias and low variance.”

Original: “Understanding the Bias-Variance Tradeoff” from Fortmann-Roe (2012): “At its root, dealing with bias and variance is really about dealing with over- and under-ﬁtting.
Revised: “Understanding the Bias-Variance Tradeoff” from Fortmann-Roe (2012): “At its root, dealing with bias and variance is really about dealing with over- and under-ﬁtting.

Original: Bias is reduced and variance is increased in relation to model complexity.
Revised: Bias is reduced and variance is increased in relation to model complexity.

Original: As more and more parameters are added to a model, the complexity of the model rises and variance becomes our primary concern while bias steadily falls.
Revised: As more and more parameters are added to a model, the complexity of the model rises and variance becomes our primary concern while bias steadily falls.

Original: For example, as more polynomial terms are added to a linear regression, the greater the resulting model’s complexity will be.”
Revised: For example, as more polynomial terms are added to a linear regression, the greater the resulting model’s complexity will be.”