Original: P LATEAU IN M ONOTONIC L INEAR I NTERPOLATION — A "B IASED " V IEW OF L OSS L ANDSCAPE FOR D EEP N ETWORKS
Revised: P LATEAU IN M ONOTONIC L INEAR I NTERPOLATION — A "B IASED " V IEW OF L OSS L ANDSCAPE FOR D EEP N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Monotonic linear interpolation (MLI) — on the line connecting a random initialization with the minimizer it converges to, the loss and accuracy are monotonic — is a phenomenon that is commonly observed in the training of neural networks.
Revised: Monotonic linear interpolation (MLI) — on the line connecting a random initialization with the minimizer it converges to, the loss and accuracy are monotonic — is a phenomenon that is commonly observed in the training of neural networks.

Original: Such a phenomenon may seem to suggest that optimization of neural networks is easy.
Revised: Such a phenomenon may seem to suggest that optimization of neural networks is easy.

Original: In this paper, we show that the MLI property is not necessarily related to the hardness of optimization problems, and empirical observations on MLI for deep neural networks depend heavily on the biases.
Revised: In this paper, we show that the MLI property is not necessarily related to the hardness of optimization problems, and empirical observations on MLI for deep neural networks depend heavily on the biases.

Original: In particular, we show that interpolating both weights and biases linearly leads to very different influences on the final output, and when different classes have different last-layer biases on a deep network, there will be a long plateau in both the loss and accuracy interpolation (which existing theory of MLI cannot explain).
Revised: In particular, we show that interpolating both weights and biases linearly leads to very different influences on the final output, and when different classes have different last-layer biases on a deep network, there will be a long plateau in both the loss and accuracy interpolation (which existing theory of MLI cannot explain).

Original: We also show how the last-layer biases for different classes can be different even on a perfectly balanced dataset using a simple model.
Revised: We also show how the last-layer biases for different classes can be different even on a perfectly balanced dataset using a simple model.

Original: Empirically we demonstrate that similar intuitions hold on practical networks and realistic datasets.
Revised: Empirically we demonstrate that similar intuitions hold on practical networks and realistic datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks can often be optimized using simple gradient-based methods, despite the objectives being highly nonconvex.
Revised: Deep neural networks can often be optimized using simple gradient-based methods, despite the objectives being highly nonconvex.

Original: Intuitively, this suggests that the loss landscape must have nice properties that allow efficient optimization.
Revised: Intuitively, this suggests that the loss landscape must have nice properties that allow efficient optimization.

Original: To understand the properties of loss landscape,
Revised: To understand the properties of loss landscape,

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2014) studied the linear interpolation between a random initialization and the local minimum found after training.
Revised: (2014) studied the linear interpolation between a random initialization and the local minimum found after training.

Original: They observed that the loss interpolation curve is monotonic and approximately convex (see the MNIST curve in Figure 1) and concluded that these tasks are easy to optimize.
Revised: They observed that the loss interpolation curve is monotonic and approximately convex (see the MNIST curve in Figure 1) and concluded that these tasks are easy to optimize.

Original: However, other recent empirical observations, such as Frankle (2020) observed that for deep neural networks on more complicated datasets, both the loss and the error curves have a long plateau along the interpolation path, i.e., the loss and error remain high until close to the optimum (see the CIFAR-10 curve in Figure 1).
Revised: However, other recent empirical observations, such as Frankle (2020) observed that for deep neural networks on more complicated datasets, both the loss and the error curves have a long plateau along the interpolation path, i.e., the loss and error remain high until close to the optimum (see the CIFAR-10 curve in Figure 1).

Original: Does the long plateau along the linear interpolation suggest these tasks are harder to optimize?
Revised: Does the long plateau along the linear interpolation suggest these tasks are harder to optimize?

Original: Not necessarily, since the hardness of optimization problems does not need to be related to the shape of interpolation curves (see examples in Appendix A).
Revised: Not necessarily, since the hardness of optimization problems does not need to be related to the shape of interpolation curves (see examples in Appendix A).

Original: In this paper we give the first theory that explains the plateau in both loss and error interpolations.
Revised: In this paper we give the first theory that explains the plateau in both loss and error interpolations.

Original: We attribute the plateau to simple reasons as the bias terms, the network initialization scale and the network depth, which may not necessarily be related to the difficulty of optimization.
Revised: We attribute the plateau to simple reasons as the bias terms, the network initialization scale and the network depth, which may not necessarily be related to the difficulty of optimization.

Original: Note that there are many different theories for the optimization of overparametrized neural networks,
Revised: Note that there are many different theories for the optimization of overparametrized neural networks,

Original: in particular the neural tangent kernel (NTK) analysis (Jacot et al., 2018;
Revised: in particular the neural tangent kernel (NTK) analysis (Jacot et al., 2018;

Original: Du et al., 2018; Allen-Zhu et al., 2019; Arora et al., 2019) and mean-field analysis (Chizat & Bach, 2018; Mei et al., 2018).
Revised: Du et al., 2018; Allen-Zhu et al., 2019; Arora et al., 2019) and mean-field analysis (Chizat & Bach, 2018; Mei et al., 2018).

Original: However they don’t explain the plateau in both loss and error interpolations.
Revised: However they don’t explain the plateau in both loss and error interpolations.

Original: For NTK regime, the network output is nearly linear in the parameters and the loss interpolation curve is monotonically decreasing and convex — no plateau in the loss interpolation.
Revised: For NTK regime, the network output is nearly linear in the parameters and the loss interpolation curve is monotonically decreasing and convex — no plateau in the loss interpolation.

Original: Mean-field regime often uses a smaller initialization on a homogeneous neural network (as considered in Chizat & Bach (2018); Mei et al.
Revised: Mean-field regime often uses a smaller initialization on a homogeneous neural network (as considered in Chizat & Bach (2018); Mei et al.

Original: In this case, the interpolated network output is basically a scaled version of the network output at the minimum and has same label predictions — no plateau in the error interpolation curve.
Revised: In this case, the interpolated network output is basically a scaled version of the network output at the minimum and has same label predictions — no plateau in the error interpolation curve.

Original: [Figure]
Revised: [Figure]

Original: 1. O UR RESULTS
Revised: 1. O UR RESULTS

Original: Our theoretical results consist of two parts.
Revised: Our theoretical results consist of two parts.

Original: In the first part (see Section 3), we give a plausible explanation for the plateau in both the loss and error curves.
Revised: In the first part (see Section 3), we give a plausible explanation for the plateau in both the loss and error curves.

Original: Claim 1 (informal) .
Revised: Claim 1 (informal) .

Original: If a deep network has a relatively small initialization, and its last-layer biases are significantly different for different classes, then both the loss and error curves will have a plateau.
Revised: If a deep network has a relatively small initialization, and its last-layer biases are significantly different for different classes, then both the loss and error curves will have a plateau.

Original: The length of the plateau is longer for a deeper network.
Revised: The length of the plateau is longer for a deeper network.

Original: We formalize this claim in Theorem 1.
Revised: We formalize this claim in Theorem 1.

Original: For intuition, consider an r -layer neural network that only has bias on the last layer, and consider Xavier initialization (Glorot & Bengio, 2010) which typically gives small output and zero bias.
Revised: For intuition, consider an r -layer neural network that only has bias on the last layer, and consider Xavier initialization (Glorot & Bengio, 2010) which typically gives small output and zero bias.

Original: If we consider the α -interpolation point (with coefficient α for the minimum and (1 − α ) for the initialization), then the weight “signal” from the minimum scales as α r (as it is the product of r layers) while the bias scales as α .
Revised: If we consider the α -interpolation point (with coefficient α for the minimum and (1 − α ) for the initialization), then the weight “signal” from the minimum scales as α r (as it is the product of r layers) while the bias scales as α .

Original: As illustrated in Figure 2 (right), when r is large and there is a difference in biases, the bias will dominate, which creates a plateau in error.
Revised: As illustrated in Figure 2 (right), when r is large and there is a difference in biases, the bias will dominate, which creates a plateau in error.

Original: For the loss, one can also show that the weight signal is near 0 for small α , so the network output is dominated by the biases and the loss cannot beat the random guessing at initialization.
Revised: For the loss, one can also show that the weight signal is near 0 for small α , so the network output is dominated by the biases and the loss cannot beat the random guessing at initialization.

Original: Note that this explanation for the plateau does not have any implication on the hardness of optimization.
Revised: Note that this explanation for the plateau does not have any implication on the hardness of optimization.

Original: However, why would the last-layer biases be different for different classes, especially in cases when the biases are initialized as zeros and all classes are balanced?
Revised: However, why would the last-layer biases be different for different classes, especially in cases when the biases are initialized as zeros and all classes are balanced?

Original: In the second part (see Section 4), we focus on a simple model that we call r -homogeneous-weight network.
Revised: In the second part (see Section 4), we focus on a simple model that we call r -homogeneous-weight network.

Original: This is a two-layer network whose i -th output is ⟨ W i, : , x ⟩ r + b i , where x ∈ R d is the network input, W i, : ∈ R d is the weight vector and b i ∈ R is the bias (see Figure 2 (left)).
Revised: This is a two-layer network whose i -th output is ⟨ W i, : , x ⟩ r + b i , where x ∈ R d is the network input, W i, : ∈ R d is the weight vector and b i ∈ R is the bias (see Figure 2 (left)).

Original: Our simple model simulates a depth- r ReLU/linear network with bias on the output layer, in the sense that the signal is r -homogeneous while the bias is 1 -homogeneous in the parameters.
Revised: Our simple model simulates a depth- r ReLU/linear network with bias on the output layer, in the sense that the signal is r -homogeneous while the bias is 1 -homogeneous in the parameters.

Original: Under this model we can show that:
Revised: Under this model we can show that:

Original: Claim 2 (informal) .
Revised: Claim 2 (informal) .

Original: For the r -homogeneous-weight network on a simple balanced dataset, the class that is learned last has the largest bias.
Revised: For the r -homogeneous-weight network on a simple balanced dataset, the class that is learned last has the largest bias.

Original: Here, a class is learned when all the samples in this class get classified correctly with good confidence.
Revised: Here, a class is learned when all the samples in this class get classified correctly with good confidence.

Original: We basically show that once a class gets learned, the bias associated with this class starts decreasing and eventually the class that is learned last has the largest bias.
Revised: We basically show that once a class gets learned, the bias associated with this class starts decreasing and eventually the class that is learned last has the largest bias.

Original: We formalize this claim in Theorem 2.
Revised: We formalize this claim in Theorem 2.

Original: In Section 5, we verify these ideas empirically on fully-connected networks for MNIST (Deng, 2012),
Revised: In Section 5, we verify these ideas empirically on fully-connected networks for MNIST (Deng, 2012),

Original: Fashion-MNIST (Xiao et al., 2017) and on VGG-16 (Simonyan & Zisserman, 2014) for CIFAR-10,
Revised: Fashion-MNIST (Xiao et al., 2017) and on VGG-16 (Simonyan & Zisserman, 2014) for CIFAR-10,

Original: CIFAR-100 (Krizhevsky et al., 2009).
Revised: CIFAR-100 (Krizhevsky et al., 2009).

Original: We first show that if we train a neural network without using any bias, then the error curve has much shorter plateau or no plateau at all.
Revised: We first show that if we train a neural network without using any bias, then the error curve has much shorter plateau or no plateau at all.

Original: Even for networks that are trained normally with biases, we design a homogeneous interpolation scheme for biases to make sure that both biases and weights are r -homogeneous.
Revised: Even for networks that are trained normally with biases, we design a homogeneous interpolation scheme for biases to make sure that both biases and weights are r -homogeneous.

Original: Such an interpolation indeed significantly shortens the plateau for the error.
Revised: Such an interpolation indeed significantly shortens the plateau for the error.

Original: We also show that decreasing the initialization scale or increasing the network depth can produce a longer plateau in both the error and loss curves.
Revised: We also show that decreasing the initialization scale or increasing the network depth can produce a longer plateau in both the error and loss curves.

Original: Finally, we show that the bias is correlated with the ordering in which the classes are being learned for small datasets, which suggests that even though the model we consider in the convergence analysis is simple, it captures some of the behavior in practice.
Revised: Finally, we show that the bias is correlated with the ordering in which the classes are being learned for small datasets, which suggests that even though the model we consider in the convergence analysis is simple, it captures some of the behavior in practice.

Original: 𝑧 !
Revised: 𝑧 !

Original: 𝑓 " 𝑥 +𝑏 " 𝛼 #
Revised: 𝑓 " 𝑥 +𝑏 " 𝛼 #

Original: Bias 𝛼 𝑏 $% − 𝑏 "%
Revised: Bias 𝛼 𝑏 $% − 𝑏 "%

Original: Weights 𝑊 ",:( ,𝑥 !
Revised: Weights 𝑊 ",:( ,𝑥 !

Original: 𝛼 ) 𝑊 ",: 𝛼 * 𝛼 + 𝛼
Revised: 𝛼 ) 𝑊 ",: 𝛼 * 𝛼 + 𝛼

Original: 1. R ELATED WORKS
Revised: 1. R ELATED WORKS

Original: There are two major lines of work studying interpolation between different points for neural networks, one on monotonic linear interpolation that interpolates the initial network and the learned network, and the other on mode connectivity that connects two learned networks.
Revised: There are two major lines of work studying interpolation between different points for neural networks, one on monotonic linear interpolation that interpolates the initial network and the learned network, and the other on mode connectivity that connects two learned networks.

Original: Monotonic linear interpolation.
Revised: Monotonic linear interpolation.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2014) first studied the linear interpolation between the network at initialization and the network after training on MNIST.
Revised: (2014) first studied the linear interpolation between the network at initialization and the network after training on MNIST.

Original: Frankle (2020) extended the experiments to modern networks on CIFAR-10 and ImageNet and found that though the loss/error is still monotonically non-increasing along the path, it remains high until close to the optimum.
Revised: Frankle (2020) extended the experiments to modern networks on CIFAR-10 and ImageNet and found that though the loss/error is still monotonically non-increasing along the path, it remains high until close to the optimum.

Original: Lucas et al.
Revised: Lucas et al.

Original: (2021) showed that MLI holds when the network output curve along the interpolation path is close to linear (measured by Gaussian length).
Revised: (2021) showed that MLI holds when the network output curve along the interpolation path is close to linear (measured by Gaussian length).

Original: However, the Gaussian length can only be formally controlled in the NTK regime.
Revised: However, the Gaussian length can only be formally controlled in the NTK regime.

Original: Mode connectivity.
Revised: Mode connectivity.

Original: Mode connectivity considers the interpolation between two learned networks (modes) found by SGD.
Revised: Mode connectivity considers the interpolation between two learned networks (modes) found by SGD.

Original: In general, a linear interpolation between two different local minima crosses regions of high loss (Goodfellow et al., 2014).
Revised: In general, a linear interpolation between two different local minima crosses regions of high loss (Goodfellow et al., 2014).

Original: Surprisingly, Draxler et al.
Revised: Surprisingly, Draxler et al.

Original: (2018) and Garipov et al.
Revised: (2018) and Garipov et al.

Original: (2018) observed that local minima found by SGD from different initializations can be connected via a piece-wise linear path of low loss.
Revised: (2018) observed that local minima found by SGD from different initializations can be connected via a piece-wise linear path of low loss.

Original: Frankle et al.
Revised: Frankle et al.

Original: (2020) and Fort et al.
Revised: (2020) and Fort et al.

Original: (2020) observed that local minima trained from the same initialization can also be connected using a linear path.
Revised: (2020) observed that local minima trained from the same initialization can also be connected using a linear path.

Original: Freeman &
Revised: Freeman &

Original: Bruna (2016); Venturi et al.
Revised: Bruna (2016); Venturi et al.

Original: ; Nguyen (2019; 2021); Kuditipudi et al.
Revised: ; Nguyen (2019; 2021); Kuditipudi et al.

Original: ; Shevchenko &
Revised: ; Shevchenko &

Original: Mondelli (2020); Nguyen et al.
Revised: Mondelli (2020); Nguyen et al.

Original: (2021) gave several theoretical explanations for this phenomenon.
Revised: (2021) gave several theoretical explanations for this phenomenon.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We first formally define the linear interpolation between the network at initialization and the network after training.
Revised: We first formally define the linear interpolation between the network at initialization and the network after training.

Original: Then we describe the notations that we will use in the paper.
Revised: Then we describe the notations that we will use in the paper.

Original: Linear interpolation: Consider a network with parameters θ ∈ R p .
Revised: Linear interpolation: Consider a network with parameters θ ∈ R p .

Original: Suppose the network is initialized with parameters θ (0) and it converges to θ ( T ) .
Revised: Suppose the network is initialized with parameters θ (0) and it converges to θ ( T ) .

Original: A linear interpolation is constructed by setting the parameters θ [ α ] = (1 − α ) θ (0) + αθ ( T ) for α ∈ [0 , 1] .
Revised: A linear interpolation is constructed by setting the parameters θ [ α ] = (1 − α ) θ (0) + αθ ( T ) for α ∈ [0 , 1] .

Original: The loss interpolation curve is defined as γ loss ( α ) :
Revised: The loss interpolation curve is defined as γ loss ( α ) :

Original: [0 , 1] → R such that γ loss ( α ) is the training loss of the network at θ [ α ] .
Revised: [0 , 1] → R such that γ loss ( α ) is the training loss of the network at θ [ α ] .

Original: Similarly, the error interpolation curve is defined as γ error ( α ) :
Revised: Similarly, the error interpolation curve is defined as γ error ( α ) :

Original: [0 , 1] → [0 , 1] with γ error ( α ) as the training error of the network at θ [ α ] .
Revised: [0 , 1] → [0 , 1] with γ error ( α ) as the training error of the network at θ [ α ] .

Original: Here, the training error is simply the ratio of training samples that get classified incorrectly by the network.
Revised: Here, the training error is simply the ratio of training samples that get classified incorrectly by the network.

Original: Notations: We use [ k ] to denote the set { 1 , 2 , · · · , k } .
Revised: Notations: We use [ k ] to denote the set { 1 , 2 , · · · , k } .

Original: We use N (0 , δ 2 ) to denote the Gaussian distribution with mean zero and variance δ 2 .
Revised: We use N (0 , δ 2 ) to denote the Gaussian distribution with mean zero and variance δ 2 .

Original: We use ∥·∥ to denote the ℓ 2 norm for a vector or the spectral norm for a matrix.
Revised: We use ∥·∥ to denote the ℓ 2 norm for a vector or the spectral norm for a matrix.

Original: For any non-zero vector v, we use ¯ v to denote v/ ∥ v ∥ .
Revised: For any non-zero vector v, we use ¯ v to denote v/ ∥ v ∥ .

Original: We use
Revised: We use

Original: O ( · ) , Θ( · ) , Ω( · ) to hide the dependency on constant factors and use (cid:101) O ( · ) , (cid:101) Θ( · ) , (cid:101) Ω( · ) to hide the dependency on poly-logarithmic factors.
Revised: O ( · ) , Θ( · ) , Ω( · ) to hide the dependency on constant factors and use (cid:101) O ( · ) , (cid:101) Θ( · ) , (cid:101) Ω( · ) to hide the dependency on poly-logarithmic factors.

Original: For any time t , we use θ ( t ) , f ( t ) to denote the parameters and the network at time t .
Revised: For any time t , we use θ ( t ) , f ( t ) to denote the parameters and the network at time t .

Original: For any α ∈ [0 , 1] , we use θ
Revised: For any α ∈ [0 , 1] , we use θ

Original: [ α ] , f [ α ] to denote the α interpolation point, which means θ [ α ] :
Revised: [ α ] , f [ α ] to denote the α interpolation point, which means θ [ α ] :

Original: = (1 − α ) θ
Revised: = (1 − α ) θ

Original: [0] + αθ [ T ] and f [ α ] is the network with parameters θ [ α ] .
Revised: [0] + αθ [ T ] and f [ α ] is the network with parameters θ [ α ] .

Original:  P LATEAU FOR LOSS AND ERROR INTERPOLATIONS
Revised:  P LATEAU FOR LOSS AND ERROR INTERPOLATIONS

Original: We prove that the long plateau exists in the loss and error curves when the initialization is small and the network is deep on fully-connected networks.
Revised: We prove that the long plateau exists in the loss and error curves when the initialization is small and the network is deep on fully-connected networks.

Original: The detailed proof can be found in Appendix B.3.
Revised: The detailed proof can be found in Appendix B.3.

Original: We consider an r -layer fully-connected neural network with r at least three.
Revised: We consider an r -layer fully-connected neural network with r at least three.

Original: Given input x ∈ R n 0 , the network output is
Revised: Given input x ∈ R n 0 , the network output is

Original: [Equation]
Revised: [Equation]

Original: where V i ∈ R n r × n r − 1 for each layer i ∈ [ r ] and b ∈ R n r .
Revised: where V i ∈ R n r × n r − 1 for each layer i ∈ [ r ] and b ∈ R n r .

Original: Here the activation function σ ( · ) can be either identity function or ReLU function.
Revised: Here the activation function σ ( · ) can be either identity function or ReLU function.

Original: The output layer width equals to the number of classes, i.e., n r = k.
Revised: The output layer width equals to the number of classes, i.e., n r = k.

Original: We use L ( { V i } , b ) to denote the sum of cross entropy loss over all samples.
Revised: We use L ( { V i } , b ) to denote the sum of cross entropy loss over all samples.

Original: For the biases, we initialize them as zeros and assume after training there exists a gap between the largest bias and the second largest, which also holds empirically (see Figure 8).
Revised: For the biases, we initialize them as zeros and assume after training there exists a gap between the largest bias and the second largest, which also holds empirically (see Figure 8).

Original: Note this bias gap is essential for the plateau in the error interpolation.
Revised: Note this bias gap is essential for the plateau in the error interpolation.

Original: If all the biases are equal in the trained network, the logits for different classes only differ by the weights signal and the interpolated network has same label predictions as the trained network.
Revised: If all the biases are equal in the trained network, the logits for different classes only differ by the weights signal and the interpolated network has same label predictions as the trained network.

Original: Assumption 1 (Bias Gap) .
Revised: Assumption 1 (Bias Gap) .

Original: Choosing i ∗ ∈ arg max i ∈ [ k ] b ( T ) i , we have b ( T ) i ∗ − max i ̸
Revised: Choosing i ∗ ∈ arg max i ∈ [ k ] b ( T ) i , we have b ( T ) i ∗ − max i ̸

Original: = i ∗ b ( T ) i > 0 .
Revised: = i ∗ b ( T ) i > 0 .

Original: Without loss of generality, we assume that b ( T ) k > max i ∈
Revised: Without loss of generality, we assume that b ( T ) k > max i ∈

Original: [ k − 1] b ( T ) i .
Revised: [ k − 1] b ( T ) i .

Original: We denote ∆ min :
Revised: We denote ∆ min :

Original: = b ( T ) k − max i ∈
Revised: = b ( T ) k − max i ∈

Original: [ k − 1] b ( T ) i and ∆ max :=
Revised: [ k − 1] b ( T ) i and ∆ max :=

Original: b ( T ) k − min i ∈
Revised: b ( T ) k − min i ∈

Original: [ k − 1] b ( T ) i .
Revised: [ k − 1] b ( T ) i .

Original: Then, we show both the loss and error interpolation curves have a long plateau in Theorem 1.
Revised: Then, we show both the loss and error interpolation curves have a long plateau in Theorem 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose the network is defined as in Equation (1) and suppose the weights satisfy (cid:13)(cid:13)(cid:13) V
Revised: Suppose the network is defined as in Equation (1) and suppose the weights satisfy (cid:13)(cid:13)(cid:13) V

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: 0) i (cid:13)(cid:13)(cid:13) ≤
Revised: i (cid:13)(cid:13)(cid:13) ≤

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: , (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original:  (cid:13)(cid:13)(cid:13) ≤ V max for all layers i ∈
Revised: i (cid:13)(cid:13)(cid:13) ≤ V max for all layers i ∈

Original: [ r ] .
Revised: [ r ] .

Original: On a k -class balanced dataset whose inputs have ℓ 2 norm at most 1 , if Assumption 1 holds, for any ϵ > 0 , as long as δ < min (cid:16) ϵ 1 /r r , 1 r 2 , (cid:0) 12 e (cid:1) 2 r − 2 (cid:17) , there exist α 1 =[Equation]
Revised: On a k -class balanced dataset whose inputs have ℓ 2 norm at most 1 , if Assumption 1 holds, for any ϵ > 0 , as long as δ < min (cid:16) ϵ 1 /r r , 1 r 2 , (cid:0) 12 e (cid:1) 2 r − 2 (cid:17) , there exist α 1 =

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: for all α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k ; 2.
Revised: for all α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k ; 2.

Original: for all α ∈ [0 , α 3 ] , we have log k − 2 eϵ ≤ 1 N L (cid:16)(cid:110) V
Revised: for all α ∈ [0 , α 3 ] , we have log k − 2 eϵ ≤ 1 N L (cid:16)(cid:110) V

Original: [ α ] i (cid:111) , b
Revised: [ α ] i (cid:111) , b

Original: [ α ] (cid:17) ≤ log k + α ∆ max + 2 eϵ,
Revised: [ α ] (cid:17) ≤ log k + α ∆ max + 2 eϵ,

Original: where N is the number of training examples.
Revised: where N is the number of training examples.

Original: The above theorem shows that for all α ∈ [ α 1 , α 2 ] , the error remains at 1 − 1 /k that is the same as random guessing.
Revised: The above theorem shows that for all α ∈ [ α 1 , α 2 ] , the error remains at 1 − 1 /k that is the same as random guessing.

Original: We skip the very short initial region [0 , δ ∆ min ] since the bias is very small and the error can be unpredictable due to the randomness in initial weights.
Revised: We skip the very short initial region [0 , δ ∆ min ] since the bias is very small and the error can be unpredictable due to the randomness in initial weights.

Original: When initialization scale δ is small, this error plateau region is roughly [0 , ( ∆ min 2 V r max ) 1 r − 1 ] .
Revised: When initialization scale δ is small, this error plateau region is roughly [0 , ( ∆ min 2 V r max ) 1 r − 1 ] .

Original: Empirically, ∆ min 2 V r max is smaller than 1 and does not change much when depth increases.
Revised: Empirically, ∆ min 2 V r max is smaller than 1 and does not change much when depth increases.

Original: So the plateau becomes longer in a deeper network.
Revised: So the plateau becomes longer in a deeper network.

Original: Intuitively, the plateau in error curve is there because for a small initialization, the output is close to α r V ( T ) r
Revised: Intuitively, the plateau in error curve is there because for a small initialization, the output is close to α r V ( T ) r

Original: [Equation]
Revised: [Equation]

Original: x ) · · · (cid:17) + αb ( T ) .
Revised: x ) · · · (cid:17) + αb ( T ) .

Original: When α is not large enough α r is much smaller than α , so for every class i ̸
Revised: When α is not large enough α r is much smaller than α , so for every class i ̸

Original: = k, the first term (signal part) cannot overcome the bias gap α ( b ( T ) k − b ( T ) i ) .
Revised: = k, the first term (signal part) cannot overcome the bias gap α ( b ( T ) k − b ( T ) i ) .

Original: This implies that all samples are predicted as class k and the error is 1 − 1 /k.
Revised: This implies that all samples are predicted as class k and the error is 1 − 1 /k.

Original: We also show that the average loss cannot be lower than log k − 2 eϵ when α ≤ ϵ 1 /r V max .
Revised: We also show that the average loss cannot be lower than log k − 2 eϵ when α ≤ ϵ 1 /r V max .

Original: Note a small random initialization can achieve a loss of approximately log k .
Revised: Note a small random initialization can achieve a loss of approximately log k .

Original: Usually the bias gap ∆ max in practice is not very large, so the loss curve remains nearly flat during this interpolation region.
Revised: Usually the bias gap ∆ max in practice is not very large, so the loss curve remains nearly flat during this interpolation region.

Original: Again, the loss plateau is becoming longer when depth r increases.
Revised: Again, the loss plateau is becoming longer when depth r increases.

Original: This is because the weights signal remains nearfor a larger range of α .
Revised: This is because the weights signal remains nearfor a larger range of α .

Original:  T RAINING DYNAMICS FOR CREATING A BIAS GAP
Revised:  T RAINING DYNAMICS FOR CREATING A BIAS GAP

Original: In this section, we explain how the gradient flow dynamics generates a bias gap on a balanced dataset by analyzing a simple model.
Revised: In this section, we explain how the gradient flow dynamics generates a bias gap on a balanced dataset by analyzing a simple model.

Original: Below, we first define the network model, training dataset and optimization procedure for our analysis.
Revised: Below, we first define the network model, training dataset and optimization procedure for our analysis.

Original: r -homogeneous-weight network: We consider a two-layer and k -output neural network with activation function σ ( z ) :
Revised: r -homogeneous-weight network: We consider a two-layer and k -output neural network with activation function σ ( z ) :

Original: =
Revised: =

Original: z r , where r is a positive constant that is at least three.
Revised: z r , where r is a positive constant that is at least three.

Original: As illustrated in
Revised: As illustrated in

Original: Figure 2 (left), under input x ∈ R d , the i -th output f i ( x ) is ⟨ W i, : , x ⟩ r + b i , where the weight vector
Revised: Figure 2 (left), under input x ∈ R d , the i -th output f i ( x ) is ⟨ W i, : , x ⟩ r + b i , where the weight vector

Original: W i, : ∈ R d is the i -th row of weight matrix W ∈ R k × d and b i ∈ R is the i -th entry of vector b ∈ R k .
Revised: W i, : ∈ R d is the i -th row of weight matrix W ∈ R k × d and b i ∈ R is the i -th entry of vector b ∈ R k .

Original: In output f i ( x ) , we call ⟨ W i, : , x ⟩ r the signal since it is input-dependent and call
Revised: In output f i ( x ) , we call ⟨ W i, : , x ⟩ r the signal since it is input-dependent and call

Original: b i the bias.
Revised: b i the bias.

Original: Dataset: We consider a k -class balanced dataset, with k as a constant.
Revised: Dataset: We consider a k -class balanced dataset, with k as a constant.

Original: We denote the whole dataset as S and denote the subset for each class i ∈
Revised: We denote the whole dataset as S and denote the subset for each class i ∈

Original: [ k ] as S i .
Revised: [ k ] as S i .

Original: Each subset S i has exactly N/k samples and each sample x ∈ R d is independently sampled as v i + ξ, where the noise ξ ∼ N (0 , σ 2 d I ) .
Revised: Each subset S i has exactly N/k samples and each sample x ∈ R d is independently sampled as v i + ξ, where the noise ξ ∼ N (0 , σ 2 d I ) .

Original: To differentiate the noise terms among different samples, we denote the noise associated with sample x as ξ x .
Revised: To differentiate the noise terms among different samples, we denote the noise associated with sample x as ξ x .

Original: We assume all v i ’s are orthonormal; without loss of generality, we assume v i
Revised: We assume all v i ’s are orthonormal; without loss of generality, we assume v i

Original: =
Revised: =

Original: e i for each class i. Here, we assume the orthogonal features to facilitate the convergence analysis beyond the
Revised: e i for each class i. Here, we assume the orthogonal features to facilitate the convergence analysis beyond the

Original: NTK regime, following previous works (Allen-Zhu & Li, 2020;
Revised: NTK regime, following previous works (Allen-Zhu & Li, 2020;

Original: Ge et al., 2021).
Revised: Ge et al., 2021).

Original: Optimization: We initialize each entry in weight matrix W by independently sampling from
Revised: Optimization: We initialize each entry in weight matrix W by independently sampling from

Original: Gaussian distribution N (0 , δ 2 ) and then taking the absolute value 1 .
Revised: Gaussian distribution N (0 , δ 2 ) and then taking the absolute value 1 .

Original: Our analysis can be trivially generalized to standard Gaussian initialization (without taking absolute value) when r is an even integer.
Revised: Our analysis can be trivially generalized to standard Gaussian initialization (without taking absolute value) when r is an even integer.

Original: We initialize all bias terms as zeros.
Revised: We initialize all bias terms as zeros.

Original: We use cross-entropy loss L ( W, b )
Revised: We use cross-entropy loss L ( W, b )

Original: = (cid:80) i ∈
Revised: = (cid:80) i ∈

Original: [ k ] (cid:80) x ∈S i − log (cid:16)
Revised: [ k ] (cid:80) x ∈S i − log (cid:16)

Original: [Equation]
Revised: [Equation]

Original:  and run gradient flow on kN L ( W, b ) for time T. Our analysis can also be extended to gradient descent with a small step size.
Revised: (cid:17) , and run gradient flow on kN L ( W, b ) for time T. Our analysis can also be extended to gradient descent with a small step size.

Original: Next we show that running gradient flow from a small initialization can converge to a model with zero error and constant bias gap.
Revised: Next we show that running gradient flow from a small initialization can converge to a model with zero error and constant bias gap.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose the neural network, dataset and optimization procedure are as defined in
Revised: Suppose the neural network, dataset and optimization procedure are as defined in

Original: Section 4.
Revised: Section 4.

Original: Suppose initialization scale δ ≤ Θ(1) , noise level σ ≤ (cid:101) Θ(1) , dimension d ≥ (cid:101) Θ(1 /δ 2 r − 2 ) and number of samples N ≥ (cid:101) Θ(

Revised: Suppose initialization scale δ ≤ Θ(1) , noise level σ ≤ (cid:101) Θ(1) , dimension d ≥ (cid:101) Θ(1 /δ 2 r − 2 ) and number of samples N ≥ (cid:101) Θ(

Original: /δ
Revised: /δ

Original: r − 1 ) , with probability at least 0 .
Revised: r − 1 ) , with probability at least 0 .

Original: 99 in the initialization, there exists time T =
Revised: 99 in the initialization, there exists time T =

Original: Θ(log(1 /δ )
Revised: Θ(log(1 /δ )

Original: /δ
Revised: /δ

Original: r − 2 ) such that we have
Revised: r − 2 ) such that we have

Original: zero error: for all different i, j ∈ [ k ] and for all x ∈ S i , f ( T )
Revised: zero error: for all different i, j ∈ [ k ] and for all x ∈ S i , f ( T )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: bias gap: b ( T ) i ∗ − max i ̸
Revised: bias gap: b ( T ) i ∗ − max i ̸

Original: = i ∗ b ( T ) i ≥ Ω(1) with i ∗ = arg max i ∈
Revised: = i ∗ b ( T ) i ≥ Ω(1) with i ∗ = arg max i ∈

Original: [ k ] b ( T ) i .
Revised: [ k ] b ( T ) i .

Original: Due to space limit, we only give a proof sketch here and leave the detailed proof in Appendix C.
Revised: Due to space limit, we only give a proof sketch here and leave the detailed proof in Appendix C.

Original: Since our dataset is perfectly balanced, it might seem surprising that gradient flow learns diverse biases.
Revised: Since our dataset is perfectly balanced, it might seem surprising that gradient flow learns diverse biases.

Original: We can compute the time derivative on the bias, ˙ b i = 1 − kN (cid:80) x ∈S u i ( x ) , where u i ( x ) is the softmax output for class i, that is exp( f i ( x ))
Revised: We can compute the time derivative on the bias, ˙ b i = 1 − kN (cid:80) x ∈S u i ( x ) , where u i ( x ) is the softmax output for class i, that is exp( f i ( x ))

Original: (cid:80) i
Revised: (cid:80) i

Original: ′∈ [ k ] exp( f i ′ ( x )) .
Revised: ′∈ [ k ] exp( f i ′ ( x )) .

Original: At the beginning, all logits are small, we have u i ( x ) ≈ 1 /k and ˙
Revised: At the beginning, all logits are small, we have u i ( x ) ≈ 1 /k and ˙

Original: b i ≈ 0 .
Revised: b i ≈ 0 .

Original: If all the samples are learned at the same time, we have u i ( x ) ≈ 1 , u i ( x ′ )
Revised: If all the samples are learned at the same time, we have u i ( x ) ≈ 1 , u i ( x ′ )

Original: ≈ 0 for x ∈ S
Revised: ≈ 0 for x ∈ S

Original: i , x ′ ∈ S \ S i , which again leads to ˙
Revised: i , x ′ ∈ S \ S i , which again leads to ˙

Original: b i ≈ 0 .
Revised: b i ≈ 0 .

Original: On the other hand, we can consider what happens if all samples in one class (e.g., class i ) are learned before any sample in any other class (e.g., class j ) is learned 2 .
Revised: On the other hand, we can consider what happens if all samples in one class (e.g., class i ) are learned before any sample in any other class (e.g., class j ) is learned 2 .

Original: In this case we have
Revised: In this case we have

Original: [Equation]
Revised: [Equation]

Original: where for any learned sample x ∈ S i , we have u
Revised: where for any learned sample x ∈ S i , we have u

Original: i ( x ) ≈ 1 , u j ( x ) ≈ 0 ; for any not yet learned sample x ∈ S \ S
Revised: i ( x ) ≈ 1 , u j ( x ) ≈ 0 ; for any not yet learned sample x ∈ S \ S

Original: i , we have u
Revised: i , we have u

Original: i
Revised: i

Original: ( x ) , u j ( x ) ≈ 1 /k.
Revised: ( x ) , u j ( x ) ≈ 1 /k.

Original: The above calculation shows that b i starts to decrease and all the other bias terms increase.
Revised: The above calculation shows that b i starts to decrease and all the other bias terms increase.

Original: Generalizing this intuition, we show that b i ′ starts to decrease whenever class i ′ is learned, and the class that is learned last will have the largest bias.
Revised: Generalizing this intuition, we show that b i ′ starts to decrease whenever class i ′ is learned, and the class that is learned last will have the largest bias.

Original: [Figure]
Revised: [Figure]

Original: As the weights are initialized randomly, by standard anti-concentration, one can argue that there is a gap between W (0) i,i ’s. Without loss of generality, we assume W (0)1 , 1 > W (0)2 , 2 > · · · > W (0) k,k .
Revised: As the weights are initialized randomly, by standard anti-concentration, one can argue that there is a gap between W (0) i,i ’s. Without loss of generality, we assume W (0)1 , 1 > W (0)2 , 2 > · · · > W (0) k,k .

Original: The initial difference in the weights will lead to different classes being learned at different time.
Revised: The initial difference in the weights will lead to different classes being learned at different time.

Original: We show that by doing induction on the following hypothesis through training:
Revised: We show that by doing induction on the following hypothesis through training:

Original: Proposition 1 (Induction Hypothesis) .
Revised: Proposition 1 (Induction Hypothesis) .

Original: In the same setting of Theorem 2, with probability at least 0.
Revised: In the same setting of Theorem 2, with probability at least 0.

Original: in initialization, there exist time points 

Revised: in initialization, there exist time points

Original: =: s 1 < t 1 < s 2 < t 2 < · · · < s k − 1 < t k − 1 < s k :
Revised: =: s 1 < t 1 < s 2 < t 2 < · · · < s k − 1 < t k − 1 < s k :

Original: = T with t i − s i
Revised: = T with t i − s i

Original: = Θ(log(1 /δ )
Revised: = Θ(log(1 /δ )

Original: /δ
Revised: /δ

Original: r − 2 ) and s i +1 − t i = Θ(1) for i ∈ [ k − 1] such that for any t ∈ [ s i , s i +1 ] ,
Revised: r − 2 ) and s i +1 − t i = Θ(1) for i ∈ [ k − 1] such that for any t ∈ [ s i , s i +1 ] ,

Original: (classes not yet learned) for any class j, j ′ ≥
Revised: (classes not yet learned) for any class j, j ′ ≥

Original: i +1 , we have (1) b ( t ) j ≥ max i ′ ∈
Revised: i +1 , we have (1) b ( t ) j ≥ max i ′ ∈

Original: [ k ] b ( t ) i ′ − O
Revised: [ k ] b ( t ) i ′ − O

Original: ( δ r ) , (2) (cid:12)(cid:12)(cid:12) b ( t ) j − b ( t ) j ′ (cid:12)(cid:12)(cid:12) ≤ O ( δ r ) and (3) W ( t ) j,j ≤ O ( δ ) ; 2.
Revised: ( δ r ) , (2) (cid:12)(cid:12)(cid:12) b ( t ) j − b ( t ) j ′ (cid:12)(cid:12)(cid:12) ≤ O ( δ r ) and (3) W ( t ) j,j ≤ O ( δ ) ; 2.

Original: (classes already learned) for any class j ≤ i − 1 , we have (1) b ( t ) j ≤ max i ′ ∈
Revised: (classes already learned) for any class j ≤ i − 1 , we have (1) b ( t ) j ≤ max i ′ ∈

Original: [ k ] b ( t ) i ′ − Ω(1) , (2)
Revised: [ k ] b ( t ) i ′ − Ω(1) , (2)

Original: f
Revised: f

Original: ( t ) j ( x ) ≥ f ( t ) i ′ ( x ) + Ω(1) for i ′
Revised: ( t ) j ( x ) ≥ f ( t ) i ′ ( x ) + Ω(1) for i ′

Original: ̸
Revised: ̸

Original: = j, x ∈ S j and (3) W ( t ) j,j ≥ Ω(1) ; 3.
Revised: = j, x ∈ S j and (3) W ( t ) j,j ≥ Ω(1) ; 3.

Original: (parameters movement) (1) for any j ∈ [ k ] , Θ( δ ) =
Revised: (parameters movement) (1) for any j ∈ [ k ] , Θ( δ ) =

Original: W (0) j,j < W ( t ) j,j , (2) for any distinct j, j ′ ∈
Revised: W (0) j,j < W ( t ) j,j , (2) for any distinct j, j ′ ∈

Original: [ k ] , 0 < W ( t ) j,j ′ ≤ O ( δ ) and (3) for any j, j ′ ∈
Revised: [ k ] , 0 < W ( t ) j,j ′ ≤ O ( δ ) and (3) for any j, j ′ ∈

Original: [ k ] and any x ∈ S j ′ , (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) 
Revised: [ k ] and any x ∈ S j ′ , (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) [Table] min (cid:16) [Table]

Revised: [Table]

Revised: ) , W ( t ) j,

Original: [Table]
Revised: [Table]

Original:  This proposition shows that gradient flow learns k classes one by one, from class 1 to class k .
Revised: ′ (cid:17) . This proposition shows that gradient flow learns k classes one by one, from class 1 to class k .

Original: More precisely, each class i is learned during time [ s i , s i +1 ] .
Revised: More precisely, each class i is learned during time [ s i , s i +1 ] .

Original: All the not yet learned classes
Revised: All the not yet learned classes

Original: j ≥
Revised: j ≥

Original: i +have close to maximum biases and their weights W ( t ) j,j ’s are small.
Revised: i +have close to maximum biases and their weights W ( t ) j,j ’s are small.

Original: All the already learned classes j ≤ i − 1 have small biases and large weights W ( t ) j,j ’s. For the parameters movement, we know that all the diagonal entries W ( t ) j,j ’s are larger than the initialization and all the off-diagonal entries W ( t ) j,j ′ ’s are only O ( δ ) .
Revised: All the already learned classes j ≤ i − 1 have small biases and large weights W ( t ) j,j ’s. For the parameters movement, we know that all the diagonal entries W ( t ) j,j ’s are larger than the initialization and all the off-diagonal entries W ( t ) j,j ′ ’s are only O ( δ ) .

Original: The correlation between the weights and noise terms also remains small.
Revised: The correlation between the weights and noise terms also remains small.

Original: When learning class i during time [ s i , s i +1 ] , the weight W ( t ) i,i slowly grows to a small constant in [ s i , t i ] and then quickly grows large in [ t i ,
Revised: When learning class i during time [ s i , s i +1 ] , the weight W ( t ) i,i slowly grows to a small constant in [ s i , t i ] and then quickly grows large in [ t i ,

Original: s i +1 .
Revised: s i +1 .

Original: As a result, all x ∈ S i become classified correctly.
Revised: As a result, all x ∈ S i become classified correctly.

Original: During the same time, b ( t ) i decreases and becomes smaller than the largest bias by at least a constant.
Revised: During the same time, b ( t ) i decreases and becomes smaller than the largest bias by at least a constant.

Original: At the end time T = s k , although W ( T ) k,k remains small, all x ∈ S k are also classified correctly because b ( T ) k is the largest bias.
Revised: At the end time T = s k , although W ( T ) k,k remains small, all x ∈ S k are also classified correctly because b ( T ) k is the largest bias.

Original: See an illustration of this learning process in Figure 3.
Revised: See an illustration of this learning process in Figure 3.

Original: Although we consider a simple neural network and data distribution, the analysis for the training dynamics is still non-trivial.
Revised: Although we consider a simple neural network and data distribution, the analysis for the training dynamics is still non-trivial.

Original: There are three major challenges in our proof: (1) How to ensure that class i + 1 is learned much later than class i ?
Revised: There are three major challenges in our proof: (1) How to ensure that class i + 1 is learned much later than class i ?

Original: For any class j that has not been learned, how to maintain that its bias is close to the maximum?
Revised: For any class j that has not been learned, how to maintain that its bias is close to the maximum?

Original: For any learned class j, how to maintain the large bias gap from the top bias?
Revised: For any learned class j, how to maintain the large bias gap from the top bias?

Original: Next, we give the proof ideas for these questions.
Revised: Next, we give the proof ideas for these questions.

Original: Since all the off-diagonal entries and correlations with noise terms in W ( t ) are negligible, in our proof we can essentially focus on the movement of W ( t ) i,i ’s and b ( t ) i ’s.
Revised: Since all the off-diagonal entries and correlations with noise terms in W ( t ) are negligible, in our proof we can essentially focus on the movement of W ( t ) i,i ’s and b ( t ) i ’s.

Original: Lower bounding s
Revised: Lower bounding s

Original: i +

Revised: i +

Original: − s i .
Revised: − s i .

Original: During time [ s i , t i ] , the dynamics of W ( t ) i,i is similar as in the tensor power method (Allen-Zhu & Li, 2020;
Revised: During time [ s i , t i ] , the dynamics of W ( t ) i,i is similar as in the tensor power method (Allen-Zhu & Li, 2020;

Original: Ge et al., 2021).
Revised: Ge et al., 2021).

Original: The initial gap between W (0) i,i and W (0) j,j ensures that when W ( t i ) i,i rises to a small constant, W ( t i ) j,j is still O ( δ ) for all j ≥ i + 1 .
Revised: The initial gap between W (0) i,i and W (0) j,j ensures that when W ( t i ) i,i rises to a small constant, W ( t i ) j,j is still O ( δ ) for all j ≥ i + 1 .

Original: Then after constant time s i +1 − t
Revised: Then after constant time s i +1 − t

Original: i , W s i +1 j,j is still O
Revised: i , W s i +1 j,j is still O

Original: ( δ ) since the increasing rate of W ( t ) j,j is merely O ( δ r − 1 ) .
Revised: ( δ ) since the increasing rate of W ( t ) j,j is merely O ( δ r − 1 ) .

Original: Bias for classes that are not learned.
Revised: Bias for classes that are not learned.

Original: For j ≥ i + 1 , we maintain that b ( t ) j ≥ max i ′ ∈ [ k ] b ( t ) i ′ − O
Revised: For j ≥ i + 1 , we maintain that b ( t ) j ≥ max i ′ ∈ [ k ] b ( t ) i ′ − O

Original: ( δ r ) .
Revised: ( δ r ) .

Original: First, we use the below lemma to show biases for any two classes j, j ′ ≥ i + 1 remain close.
Revised: First, we use the below lemma to show biases for any two classes j, j ′ ≥ i + 1 remain close.

Original: Lemma 1 (Coupling Biases) .
Revised: Lemma 1 (Coupling Biases) .

Original: Assuming W j ′ ,j ′ , W j,j ≤ O ( δ ) and b j ′ , b j ≥ max i ′ ∈
Revised: Assuming W j ′ ,j ′ , W j,j ≤ O ( δ ) and b j ′ , b j ≥ max i ′ ∈

Original: [ k ] b i ′ − O
Revised: [ k ] b i ′ − O

Original: ( δ r ) , we have ˙
Revised: ( δ r ) , we have ˙

Original: b j ′ − ˙
Revised: b j ′ − ˙

Original: b j > 0 if b j ′ − b j ≤ − µδ r , and ˙ b j ′ − ˙
Revised: b j > 0 if b j ′ − b j ≤ − µδ r , and ˙ b j ′ − ˙

Original: b
Revised: b

Original: j < 

Revised: j <

Original: if b j ′ − b j ≥ + µδ r for some positive constant µ.
Revised: if b j ′ − b j ≥ + µδ r for some positive constant µ.

Original: Second we show that any already learned or being learned class j ′ ≤ i cannot have bias much larger than any class j ≥ i + 1 not yet learned.
Revised: Second we show that any already learned or being learned class j ′ ≤ i cannot have bias much larger than any class j ≥ i + 1 not yet learned.

Original: Lemma 2 (Bias Gap Control I) .
Revised: Lemma 2 (Bias Gap Control I) .

Original: For any different j ′ , j ∈ [ k ] , if W j ′ ,j ′ ≥ W j,j , W j,j ≤ O ( δ ) and b j ′ − b j ≥ O
Revised: For any different j ′ , j ∈ [ k ] , if W j ′ ,j ′ ≥ W j,j , W j,j ≤ O ( δ ) and b j ′ − b j ≥ O

Original: ( δ r ) , b j ≥ max i ′ ∈
Revised: ( δ r ) , b j ≥ max i ′ ∈

Original: [ k ] b i ′ − O
Revised: [ k ] b i ′ − O

Original: ( δ r ) , we have ˙
Revised: ( δ r ) , we have ˙

Original: b j ′ − ˙
Revised: b j ′ − ˙

Original: b j < 0 .
Revised: b j < 0 .

Original: Bias for learned classes.
Revised: Bias for learned classes.

Original: At time s j +1 , we can prove that 1 − u ( s j +1 )
Revised: At time s j +1 , we can prove that 1 − u ( s j +1 )

Original: j ( x ) ≤ C 1 for all x ∈ S j and b ( s j +1 )
Revised: j ( x ) ≤ C 1 for all x ∈ S j and b ( s j +1 )

Original: j − b ( s j +1 ) k ≤ − C 2 .
Revised: j − b ( s j +1 ) k ≤ − C 2 .

Original: According to the below lemma, we can ensure that b ( t ) j − b ( t ) k ≤ − C 2 for any t ≥ s j +1 .
Revised: According to the below lemma, we can ensure that b ( t ) j − b ( t ) k ≤ − C 2 for any t ≥ s j +1 .

Original: Lemma 3 (Bias Gap Control II) .
Revised: Lemma 3 (Bias Gap Control II) .

Original: There exist small positive constants C 1 , C 2 such that for any j ∈ [ k − 1] and any x ∈ S j , if 1 − u j ( x ) ≤ C 1 , W k,k ≤ O ( δ ) and b j − b k ≥ − C 2 , we have ˙
Revised: There exist small positive constants C 1 , C 2 such that for any j ∈ [ k − 1] and any x ∈ S j , if 1 − u j ( x ) ≤ C 1 , W k,k ≤ O ( δ ) and b j − b k ≥ − C 2 , we have ˙

Original: b j − ˙
Revised: b j − ˙

Original: b k < − Ω(1) .
Revised: b k < − Ω(1) .

Original: 4. P LATEAU AND MONOTONICITY FOR r - HOMOGENEOUS - WEIGHT NETWORK
Revised: 4. P LATEAU AND MONOTONICITY FOR r - HOMOGENEOUS - WEIGHT NETWORK

Original: Now assuming the network at initialization and after training satisfies the properties described in
Revised: Now assuming the network at initialization and after training satisfies the properties described in

Original: Theorem 2 and Proposition 1, we can prove a tighter bound on the plateau region and also show the monotonicity in error and loss curve.
Revised: Theorem 2 and Proposition 1, we can prove a tighter bound on the plateau region and also show the monotonicity in error and loss curve.

Original: See the complete proofs in Appendix B.1 and Appendix B.2.
Revised: See the complete proofs in Appendix B.1 and Appendix B.2.

Original: Same as in Assumption 1, we use ∆ i to denote the bias gap b ( T ) k − b ( T ) i for i ∈
Revised: Same as in Assumption 1, we use ∆ i to denote the bias gap b ( T ) k − b ( T ) i for i ∈

Original: [ k − 1] and denote ∆ min := min i ∈ [ k − 1] ∆ i and ∆ max = max i ∈
Revised: [ k − 1] and denote ∆ min := min i ∈ [ k − 1] ∆ i and ∆ max = max i ∈

Original: [ k − 1] ∆ i .
Revised: [ k − 1] ∆ i .

Original: For the weights, we denote W min =
Revised: For the weights, we denote W min =

Original: min i ∈ [ k − 1] W
Revised: min i ∈ [ k − 1] W

Original: ( T ) i,i and W max = max i ∈
Revised: ( T ) i,i and W max = max i ∈

Original: [ k ] W ( T ) i,i .
Revised: [ k ] W ( T ) i,i .

Original: We denote R min =
Revised: We denote R min =

Original: min i ∈ [ k − 1] ∆ i /
Revised: min i ∈ [ k − 1] ∆ i /

Original: [ W ( T ) i,i ] r ,
Revised: [ W ( T ) i,i ] r ,

Original: R max = max i ∈ [ k − 1] ∆ i / [ W ( T ) i,i ] r .
Revised: R max = max i ∈ [ k − 1] ∆ i / [ W ( T ) i,i ] r .

Original: Below, we show the plateau and monotonicity of loss and error interpolations in Theorem 3.
Revised: Below, we show the plateau and monotonicity of loss and error interpolations in Theorem 3.

Original: Theorem 3.
Revised: Theorem 3.

Original: Suppose the neural network, dataset and optimization procedure are as defined in Section 4.
Revised: Suppose the neural network, dataset and optimization procedure are as defined in Section 4.

Original: Suppose the network at initialization and after training satisfies the properties described in Theorem 2 and Proposition 1.
Revised: Suppose the network at initialization and after training satisfies the properties described in Theorem 2 and Proposition 1.

Original: For any ϵ ∈ (0 , 1) , suppose δ ≤ min(Θ( ϵ 1 /r ) , Θ( R 1 r − 1 min ∆ 1 /r min ) , Θ(( W min W max )
Revised: For any ϵ ∈ (0 , 1) , suppose δ ≤ min(Θ( ϵ 1 /r ) , Θ( R 1 r − 1 min ∆ 1 /r min ) , Θ(( W min W max )

Original: 2 r r − 2 )) .
Revised: 2 r r − 2 )) .

Original: There exist α 1 = δ ∆ min , α 2 = ( 1 1+ O ( √ δ ) )
Revised: There exist α 1 = δ ∆ min , α 2 = ( 1 1+ O ( √ δ ) )

Original: rr − 1 R 1 r − 1 min , 
Revised: rr − 1 R 1 r − 1 min , α 3 =

Original: α 3 = ϵ 1 /r W max and α 4 =(1 + O ( δ ))
Revised: ϵ 1 /r W max and α 4 =

Original:  1 r − 1 (cid:0) R max r (cid:1) 1 r −
Revised: (1 + O ( δ )) 1 r − 1 (cid:0) R max r (cid:1) 1 r −

Original: such that
Revised: such that

Original: for all α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k ; for all α ∈ [ α 1 , 1] , the error is non-increasing; 2.
Revised: for all α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k ; for all α ∈ [ α 1 , 1] , the error is non-increasing; 2.

Original: for all α ∈ [0 , α 3 ] , we have log k − eϵ ≤ 1 N L ( W [ α ] , b [ α ] ) ≤ log k + α ∆ max + eϵ ; for all α ∈ [ α 4 , 1] , the loss is strictly monotonically decreasing.
Revised: for all α ∈ [0 , α 3 ] , we have log k − eϵ ≤ 1 N L ( W [ α ] , b [ α ] ) ≤ log k + α ∆ max + eϵ ; for all α ∈ [ α 4 , 1] , the loss is strictly monotonically decreasing.

Original: The analysis for the plateau is very similar as in Theorem 1 since a r -homogeneous-weight network is similar to a depth- r fully connected neural network with only last-layer biases in the sense that the weights are r -homogeneous while the (last-layer) bias is 1 -homogeneous.
Revised: The analysis for the plateau is very similar as in Theorem 1 since a r -homogeneous-weight network is similar to a depth- r fully connected neural network with only last-layer biases in the sense that the weights are r -homogeneous while the (last-layer) bias is 1 -homogeneous.

Original: For the error plateau, we prove a tighter bound on the right boundary α 2 than in Theorem 1.
Revised: For the error plateau, we prove a tighter bound on the right boundary α 2 than in Theorem 1.

Original: We also show the error is nonincreasing for α ∈ [ δ/ ∆ min , 1] by arguing that once a sample is correctly classified at interpolated point α ′ ≥ δ/ ∆ min , it will remain so for any α ≥ α ′ .
Revised: We also show the error is nonincreasing for α ∈ [ δ/ ∆ min , 1] by arguing that once a sample is correctly classified at interpolated point α ′ ≥ δ/ ∆ min , it will remain so for any α ≥ α ′ .

Original: Similar as in Theorem 1, we can show that the loss is no smaller than log k − eϵ when α ≤ ϵ 1 /r W max .
Revised: Similar as in Theorem 1, we can show that the loss is no smaller than log k − eϵ when α ≤ ϵ 1 /r W max .

Original: To show the monotonicity of loss after α 4 , we show that f [ α ] i ( x ) − f
Revised: To show the monotonicity of loss after α 4 , we show that f [ α ] i ( x ) − f

Original: [ α ] j ( x ) is increasing in α for i ̸
Revised: [ α ] j ( x ) is increasing in α for i ̸

Original: = j and x ∈ S
Revised: = j and x ∈ S

Original: i .
Revised: i .

Original: In summary, for α ∈ [ α 1 , α 2 ] , the signal is smaller than the bias gap and the error remains at 1 − 1 /k .
Revised: In summary, for α ∈ [ α 1 , α 2 ] , the signal is smaller than the bias gap and the error remains at 1 − 1 /k .

Original: Before α 3 , the signal is very small and the loss remains large; after α 4 , the signal starts to overcome the bias gap and the loss is decreasing.
Revised: Before α 3 , the signal is very small and the loss remains large; after α 4 , the signal starts to overcome the bias gap and the loss is decreasing.

Original: See an illustration in Figure 2 (right).
Revised: See an illustration in Figure 2 (right).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we empirically show that intuitions from our simple theoretical model can also be applied to more realistic datasets and architectures.
Revised: In this section we empirically show that intuitions from our simple theoretical model can also be applied to more realistic datasets and architectures.

Original: First, we show that bias plays an important role in creating the plateau in the error interpolation, as predicted by Theorem 1.
Revised: First, we show that bias plays an important role in creating the plateau in the error interpolation, as predicted by Theorem 1.

Original: We then demonstrate the influence of initialization size and network depth (also see Theorem 1).
Revised: We then demonstrate the influence of initialization size and network depth (also see Theorem 1).

Original: Finally, we show that similar to Proposition 1 the class that is learned last often has larger bias.
Revised: Finally, we show that similar to Proposition 1 the class that is learned last often has larger bias.

Original: Due to space constraint, we only show the results on MNIST and CIFAR-100 in this section, while similar results also hold on
Revised: Due to space constraint, we only show the results on MNIST and CIFAR-100 in this section, while similar results also hold on

Original: Fashion-MNIST and CIFAR-10 (see Appendix D).
Revised: Fashion-MNIST and CIFAR-10 (see Appendix D).

Original: Unless specified otherwise, we use a depth-10 and width-1024 fully-connected ReLU neural network (FCN10) for MNIST and use VGG-16 (without batch normalization) for CIFAR-100.
Revised: Unless specified otherwise, we use a depth-10 and width-1024 fully-connected ReLU neural network (FCN10) for MNIST and use VGG-16 (without batch normalization) for CIFAR-100.

Original: We use
Revised: We use

Original: Kaiming initialization (He et al., 2015) for the weights and set all bias terms as zeros.
Revised: Kaiming initialization (He et al., 2015) for the weights and set all bias terms as zeros.

Original: For FCN on MNIST, we use a small initialization by scaling the weights of each layer by (0 . 001) 1 / 10 so the output is scaled by 0 .
Revised: For FCN on MNIST, we use a small initialization by scaling the weights of each layer by (0 . 001) 1 / 10 so the output is scaled by 0 .

Original: We train each network using SGD for 100 epochs.
Revised: We train each network using SGD for 100 epochs.

Original: See more experiment settings in Appendix D.
Revised: See more experiment settings in Appendix D.

Original: We linearly interpolate using 50 evenly spaced points between the network at initialization and the network at the end of training.
Revised: We linearly interpolate using 50 evenly spaced points between the network at initialization and the network at the end of training.

Original: We evaluate error and loss on the train set.
Revised: We evaluate error and loss on the train set.

Original: For each setting, we repeat the experiments three times from different random seeds and plot the mean and deviation.
Revised: For each setting, we repeat the experiments three times from different random seeds and plot the mean and deviation.

Original: [Figure]
Revised: [Figure]

Original: Role of bias in creating plateau.
Revised: Role of bias in creating plateau.

Original: We demonstrate the importance of bias using two experiments.
Revised: We demonstrate the importance of bias using two experiments.

Original: In the first experiment
Revised: In the first experiment

Original: [Figure]
Revised: [Figure]

Original: we compare the loss/error interpolation curves between networks equipped with bias for all the layers ( all bias ), with bias only for the output layer ( last bias )
Revised: we compare the loss/error interpolation curves between networks equipped with bias for all the layers ( all bias ), with bias only for the output layer ( last bias )

Original: [Figure]
Revised: [Figure]

Original: and with no bias at all ( no bias ).
Revised: and with no bias at all ( no bias ).

Original: Figure 4 shows that networks with all bias and last bias have a much longer error plateau than networks without bias.
Revised: Figure 4 shows that networks with all bias and last bias have a much longer error plateau than networks without bias.

Original: Three bias settings have similar loss interpolation curves
Revised: Three bias settings have similar loss interpolation curves

Original: [Figure]
Revised: [Figure]

Original: By our theory, the bias dominates the signal at the beginning of the interpolation because the bias term scales as α while the signal scales as α r .
Revised: By our theory, the bias dominates the signal at the beginning of the interpolation because the bias term scales as α while the signal scales as α r .

Original: In the second experiment, to correct this discrepancy, we interpolate the bias at the h -th layer (input is at the 0 -th layer) as b [ α ] h = (1 − α )
Revised: In the second experiment, to correct this discrepancy, we interpolate the bias at the h -th layer (input is at the 0 -th layer) as b [ α ] h = (1 − α )

Original: h b (0) h
Revised: h b (0) h

Original: + α h b ( T ) h
Revised: + α h b ( T ) h

Original: = α h b ( T ) h .
Revised: = α h b ( T ) h .

Original: We call this the homogeneous interpolation as now terms involving bias and weights all have α r coefficients.
Revised: We call this the homogeneous interpolation as now terms involving bias and weights all have α r coefficients.

Original: We compare this with the normal interpolation that linearly interpolates the bias terms.
Revised: We compare this with the normal interpolation that linearly interpolates the bias terms.

Original: Figure 5 shows that for networks with all bias or last bias, using homogeneous interpolation can significantly reduce the plateau in the error interpolation, but does not affect the loss interpolation.
Revised: Figure 5 shows that for networks with all bias or last bias, using homogeneous interpolation can significantly reduce the plateau in the error interpolation, but does not affect the loss interpolation.

Original: Role of initialization scale and network depth.
Revised: Role of initialization scale and network depth.

Original: Our theory suggests that with a smaller initialization, the signal magnitude at the initial interpolation is smaller, which can create longer plateau in both loss interpolation and error interpolation.
Revised: Our theory suggests that with a smaller initialization, the signal magnitude at the initial interpolation is smaller, which can create longer plateau in both loss interpolation and error interpolation.

Original: We compare networks under initialization scales 1 , 0 . 1 , 0 . 01 and 0 .
Revised: We compare networks under initialization scales 1 , 0 . 1 , 0 . 01 and 0 .

Original: 001 , where scale 1 corresponds to the standard Kaiming initialization.
Revised: 001 , where scale 1 corresponds to the standard Kaiming initialization.

Original: For other
Revised: For other

Original: [Figure]
Revised: [Figure]

Original: initialization β, we rescale each layer by the same factor so the output is rescaled by β.
Revised: initialization β, we rescale each layer by the same factor so the output is rescaled by β.

Original: According to
Revised: According to

Original: Figure 6, smaller initialization does create longer plateau in loss and error interpolation
Revised: Figure 6, smaller initialization does create longer plateau in loss and error interpolation

Original: [Figure]
Revised: [Figure]

Original: With a deeper network, the signal grows slower at the initial interpolation phase, which can potentially create a longer plateau in both loss interpolation and error interpolation.
Revised: With a deeper network, the signal grows slower at the initial interpolation phase, which can potentially create a longer plateau in both loss interpolation and error interpolation.

Original: We compare FCN4,
Revised: We compare FCN4,

Original: FCN6, FCN8, FCN10 on MNIST and compare VGG11, VGG13, VGG16, VGG19 on CIFAR-100.
Revised: FCN6, FCN8, FCN10 on MNIST and compare VGG11, VGG13, VGG16, VGG19 on CIFAR-100.

Original: According to Figure
Revised: According to Figure

Original: [Figure]
Revised: [Figure]

Original: deeper networks do have longer plateau in loss and error interpolation
Revised: deeper networks do have longer plateau in loss and error interpolation

Original: [Figure]
Revised: [Figure]

Original: Bias learning dynamics.
Revised: Bias learning dynamics.

Original: Our dynamics analysis in Section 4 shows that gradient descent can learn diverse biases on a balanced dataset by learning different classes at different time points.
Revised: Our dynamics analysis in Section 4 shows that gradient descent can learn diverse biases on a balanced dataset by learning different classes at different time points.

Original: In particular, the last learned class should have the highest bias term.
Revised: In particular, the last learned class should have the highest bias term.

Original: We verify this theory by studying FCN with only output bias on balanced 2-class or 3-class MNIST.
Revised: We verify this theory by studying FCN with only output bias on balanced 2-class or 3-class MNIST.

Original: To separate the learning of different classes, we compute the per-class loss by only considering the examples in that particular class.
Revised: To separate the learning of different classes, we compute the per-class loss by only considering the examples in that particular class.

Original: According to Figure 8, in the 2-class MNIST, number 1 is learned last and its bias is larger, which fits our theory.
Revised: According to Figure 8, in the 2-class MNIST, number 1 is learned last and its bias is larger, which fits our theory.

Original: Also in the 3-class MNIST, class 2 is learned first, class 3 the second and class 1 the last; for the learned bias, class 2 bias is smallest, class 3 bias in the middle and class 1 bias the highest.
Revised: Also in the 3-class MNIST, class 2 is learned first, class 3 the second and class 1 the last; for the learned bias, class 2 bias is smallest, class 3 bias in the middle and class 1 bias the highest.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our theory suggests that the plateau in loss/error interpolation curves may be attributed to simple reasons, and it’s unclear if these reasons are related to the difficulty/easiness of optimization.
Revised: Our theory suggests that the plateau in loss/error interpolation curves may be attributed to simple reasons, and it’s unclear if these reasons are related to the difficulty/easiness of optimization.

Original: In our experiments although the training succeeds in all the settings, the loss and error interpolation curves can be easily manipulated by changing the initialization size, network depth and bias terms.
Revised: In our experiments although the training succeeds in all the settings, the loss and error interpolation curves can be easily manipulated by changing the initialization size, network depth and bias terms.

Original: Therefore, we believe one needs to look at structures more complicated than linear interpolation to understand why optimization succeeds for deep neural networks.
Revised: Therefore, we believe one needs to look at structures more complicated than linear interpolation to understand why optimization succeeds for deep neural networks.

Original: Though our theory requires a small initialization, we also observe plateau in CIFAR-100 with standard initialization, which suggests that the useful signal is still a high order term in α.
Revised: Though our theory requires a small initialization, we also observe plateau in CIFAR-100 with standard initialization, which suggests that the useful signal is still a high order term in α.

Original: We also observe that sometimes the ordering of the biases does not exactly follow the ordering of the learning.
Revised: We also observe that sometimes the ordering of the biases does not exactly follow the ordering of the learning.

Original: We believe this is partially due to the correlation between different-class features and offer a preliminary explanation in Appendix D.5.
Revised: We believe this is partially due to the correlation between different-class features and offer a preliminary explanation in Appendix D.5.

Original: We leave the thorough study of these problems in the future work.
Revised: We leave the thorough study of these problems in the future work.

Revised: A CKNOWLEDGEMENT

Revised: This work is supported by NSF Award DMS-2031849, CCF-1845171 (CAREER), CCF- (Tripods) and a Sloan Research Fellowship.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: For our theoretical results, we listed all the assumptions and stated the theorems in the main text and we left the complete proof for all the claims in the Appendix.
Revised: For our theoretical results we listed all the assumptions and stated the theorems in the main text and we left the complete proof for all the claims in the Appendix.

Original: For our experimental results, we defined the detailed experiment settings in the Appendix and also uploaded the source code as supplementary material.
Revised: For our experimental results, we defined the detailed experiment settings in the Appendix and also uploaded the source code as supplementary material.

Original: A E XAMPLES FOR THE DISCONNECTION BETWEEN LINEAR INTERPOLATION SHAPE AND OPTIMIZATION DIFFICULTY
Revised: A E XAMPLES FOR THE DISCONNECTION BETWEEN LINEAR INTERPOLATION SHAPE AND OPTIMIZATION DIFFICULTY

Original: We give two examples that illustrate the disconnection between the linear interpolation shape and the optimization difficulty.
Revised: We give two examples that illustrate the disconnection between the linear interpolation shape and the optimization difficulty.

Original: In Section A.1, we show a function that is NP-hard to optimize, but has a convex and monotonically decreasing loss interpolation.
Revised: In Section A.1, we show a function that is NP-hard to optimize, but has a convex and monotonically decreasing loss interpolation.

Original: Then in Section A.2, we give a function that is easy to optimize, but has a non-monotonic loss interpolation.
Revised: Then in Section A.2, we give a function that is easy to optimize, but has a non-monotonic loss interpolation.

Original: A. H ARD FUNCTION WITH CONVEX LOSS INTERPOLATION
Revised: A. H ARD FUNCTION WITH CONVEX LOSS INTERPOLATION

Original: For any symmetric third-order tensor T ∈ R d × d × d , our goal is to minimize
Revised: For any symmetric third-order tensor T ∈ R d × d × d , our goal is to minimize

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R d and z ∈ R .
Revised: where x ∈ R d and z ∈ R .

Original: It’s known that finding the spectral norm of a symmetric third-order tensor (that is, max v ∈ R d , ∥ v ∥ =1 T ( v, v, v ) ) is NP-hard (Hillar & Lim, 2013).
Revised: It’s known that finding the spectral norm of a symmetric third-order tensor (that is, max v ∈ R d , ∥ v ∥ =1 T ( v, v, v ) ) is NP-hard (Hillar & Lim, 2013).

Original: We prove that minimizing f ( x, z ) is also NP-hard by reducing the tensor spectral norm problem to it.
Revised: We prove that minimizing f ( x, z ) is also NP-hard by reducing the tensor spectral norm problem to it.

Original: Proposition 2.
Revised: Proposition 2.

Original: Minimizing f
Revised: Minimizing f

Original: ( x, z ) as defined in Eqn.
Revised: ( x, z ) as defined in Eqn.

Original: 2 is NP-hard.
Revised: 2 is NP-hard.

Original: Proof.
Revised: Proof.

Original: For any non-zero tensor T, let ( x ∗ , z ∗ ) be one minimizer of f ( x, z ) , it’s easy to verify that
Revised: For any non-zero tensor T, let ( x ∗ , z ∗ ) be one minimizer of f ( x, z ) , it’s easy to verify that

Original: T ( x ∗ , x ∗ , x ∗ ) > 0 .
Revised: T ( x ∗ , x ∗ , x ∗ ) > 0 .

Original: We show that ¯ x ∗ := x ∗ / ∥ x ∗ ∥ must be a solution to max v ∈ R d , ∥ v ∥ =1 T ( v, v, v ) .
Revised: We show that ¯ x ∗ := x ∗ / ∥ x ∗ ∥ must be a solution to max v ∈ R d , ∥ v ∥ =1 T ( v, v, v ) .

Original: For the sake of contradiction, assume there exists v ∗ with unit norm such that T ( v ∗ , v ∗ , v ∗ ) >
Revised: For the sake of contradiction, assume there exists v ∗ with unit norm such that T ( v ∗ , v ∗ , v ∗ ) >

Original: T (¯ x ∗ , ¯ x ∗ , ¯ x ∗ ) .
Revised: T (¯ x ∗ , ¯ x ∗ , ¯ x ∗ ) .

Original: It’s easy to verify that f ( ∥ x ∗ ∥ v ∗ , z ∗ ) <
Revised: It’s easy to verify that f ( ∥ x ∗ ∥ v ∗ , z ∗ ) <

Original: f
Revised: f

Original: ( x ∗ , z ∗ ) , which however contradicts the optimality of ( x ∗ , z ∗ ) .
Revised: ( x ∗ , z ∗ ) , which however contradicts the optimality of ( x ∗ , z ∗ ) .

Original: Next, we prove that start from certain initialization, the loss along the linear interpolation path is convex and monotonically decreasing.
Revised: Next, we prove that start from certain initialization, the loss along the linear interpolation path is convex and monotonically decreasing.

Original: Note that assuming the unit Frobenius norm of T does not hurt the NP-hardness of the problem.
Revised: Note that assuming the unit Frobenius norm of T does not hurt the NP-hardness of the problem.

Original: And our initialization is oblivious of the tensor T.
Revised: And our initialization is oblivious of the tensor T.

Original: Proposition 3.
Revised: Proposition 3.

Original: Assume ∥ T ∥ F = 1 .
Revised: Assume ∥ T ∥ F = 1 .

Original: Suppose we start from initialization ( x 0 , z 0 ) with x 0 =and | z 0 | > 3 √ 24 .
Revised: Suppose we start from initialization ( x 0 , z 0 ) with x 0 =and | z 0 | > 3 √ 24 .

Original: Let ( x ∗ , z ∗ ) be a minimizer of f ( x, z ) as defined in Eqn.
Revised: Let ( x ∗ , z ∗ ) be a minimizer of f ( x, z ) as defined in Eqn.

Original: We know the loss interpolation curve γ ( α ) :
Revised: We know the loss interpolation curve γ ( α ) :

Original: = f
Revised: = f

Original: ((1 − α ) x 0 + αx ∗ , (1 − α )
Revised: ((1 − α ) x 0 + αx ∗ , (1 − α )

Original: z 0 + αz ∗ ) is convex and monotonically decreasing for α ∈ [0 , 1] .
Revised: z 0 + αz ∗ ) is convex and monotonically decreasing for α ∈ [0 , 1] .

Original: Proof.
Revised: Proof.

Original: We first prove that at any minimizer ( x ∗ , z ∗ ) , we must have z ∗ = 0 .
Revised: We first prove that at any minimizer ( x ∗ , z ∗ ) , we must have z ∗ = 0 .

Original: Otherwise, we can set z as zero to further decrease the loss.
Revised: Otherwise, we can set z as zero to further decrease the loss.

Original: Starting from an initialization ( x (0) , z (0) ) with x (0) = 0 , we know at each interpolation point x
Revised: Starting from an initialization ( x (0) , z (0) ) with x (0) = 0 , we know at each interpolation point x

Original: [ α ] = αx ∗ , z
Revised: [ α ] = αx ∗ , z

Original: [ α ] = (1 − α )
Revised: [ α ] = (1 − α )

Original: z (0) .
Revised: z (0) .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: To prove the convexity of γ ( α ) for α ∈ [0 , 1] , we only need to prove γ ′′ ( α ) > 0 for α ∈ [0 , 1] .
Revised: To prove the convexity of γ ( α ) for α ∈ [0 , 1] , we only need to prove γ ′′ ( α ) > 0 for α ∈ [0 , 1] .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Since the formula for γ ′′ ( α ) involves both T (¯ x ∗ , ¯ x ∗ , ¯ x ∗ ) and ∥ x ∗ ∥ , we first figure out the relation between these two quantities.
Revised: Since the formula for γ ′′ ( α ) involves both T (¯ x ∗ , ¯ x ∗ , ¯ x ∗ ) and ∥ x ∗ ∥ , we first figure out the relation between these two quantities.

Original: Suppose T (¯ x ∗ , ¯ x ∗ , ¯ x ∗ ) = p > 0 , it’s not hard to find ∥ x ∗ ∥ must be equal to 3 p 4 .
Revised: Suppose T (¯ x ∗ , ¯ x ∗ , ¯ x ∗ ) = p > 0 , it’s not hard to find ∥ x ∗ ∥ must be equal to 3 p 4 .

Original: This is because − ∥ x ∗ ∥ 3 p + ∥ x ∗ ∥ 4 is minimized when ∥ x ∗ ∥ = 3 p 4 .
Revised: This is because − ∥ x ∗ ∥ 3 p + ∥ x ∗ ∥ 4 is minimized when ∥ x ∗ ∥ = 3 p 4 .

Original: Next, we prove γ ′′ ( α ) > 0 for α ∈ (2 / 3 , 1] and α ∈ [0 , 2 / 3] separetely.
Revised: Next, we prove γ ′′ ( α ) > 0 for α ∈ (2 / 3 , 1] and α ∈ [0 , 2 / 3] separetely.

Original: When α ∈ (2 / 3 , 1] , we have
Revised: When α ∈ (2 / 3 , 1] , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we know γ ′′ ( α ) > 0 .
Revised: Therefore, we know γ ′′ ( α ) > 0 .

Original: When α ∈ [0 , 2 / 3] , we know
Revised: When α ∈ [0 , 2 / 3] , we know

Original: [Equation]
Revised: [Equation]

Original: Since ∥ T ∥ F = 1 , we know T (¯ x ∗ , ¯ x ∗ , ¯ x ∗ ) ≤ 1 and ∥ x ∗ ∥ ≤ 3 / 4 .
Revised: Since ∥ T ∥ F = 1 , we know T (¯ x ∗ , ¯ x ∗ , ¯ x ∗ ) ≤ 1 and ∥ x ∗ ∥ ≤ 3 / 4 .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Then, we know that if (cid:12)(cid:12) z (0) (cid:12)(cid:12) > 3 √ 24 , we have γ ′′ ( α ) > 0 .
Revised: Then, we know that if (cid:12)(cid:12) z (0) (cid:12)(cid:12) > 3 √ 24 , we have γ ′′ ( α ) > 0 .

Original: A. E ASY FUNCTION WITH NON - MONOTONIC LOSS INTERPOLATION
Revised: A. E ASY FUNCTION WITH NON - MONOTONIC LOSS INTERPOLATION

Original: In this section, we give an easy-to-optimize function that however has a non-monotonic loss interpolation curve.
Revised: In this section, we give an easy-to-optimize function that however has a non-monotonic loss interpolation curve.

Original: We consider the following loss function
Revised: We consider the following loss function

Original: [Equation]
Revised: [Equation]

Original: (cid:1)

Original: [Equation]

Original: (cid:0)
Revised: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: (cid:1)(cid:17) otherwise ,
Revised: otherwise ,

Original: [Equation]
Revised: [Equation]

Original: where x, y ∈ R .
Revised: where x, y ∈ R .

Original: We can also re-parameterize f ( x, y ) using angle θ ∈ [0 , 2 π ) and length r ∈ [0 , ∞ ) as h ( θ, r ) = (cid:16) 1 − sin( θ ) 3 (cid:17) (cid:0) r 4 − 2 r 2 (cid:1) .
Revised: We can also re-parameterize f ( x, y ) using angle θ ∈ [0 , 2 π ) and length r ∈ [0 , ∞ ) as h ( θ, r ) = (cid:16) 1 − sin( θ ) 3 (cid:17) (cid:0) r 4 − 2 r 2 (cid:1) .

Original: Next, we prove that starting from any non-zero point, gradient flow converges to the global minimizer.
Revised: Next, we prove that starting from any non-zero point, gradient flow converges to the global minimizer.

Original: Proposition 4.
Revised: Proposition 4.

Original: Starting from any non-zero initialization, gradient flow on f ( x, y ) as defined in Eqn.converges to the global minimizer (0 , − 1) .
Revised: Starting from any non-zero initialization, gradient flow on f ( x, y ) as defined in Eqn.converges to the global minimizer (0 , − 1) .

Original: Proof.
Revised: Proof.

Original: We know the unique minimizer of f ( x, y ) is (0 , − 1) by considering its equivalent form h ( θ, r ) .
Revised: We know the unique minimizer of f ( x, y ) is (0 , − 1) by considering its equivalent form h ( θ, r ) .

Original: For h ( θ, r ) = (cid:16) 1 − sin( θ ) 3 (cid:17) (cid:0) r 4 − 2 r 2 (cid:1) , we know (cid:0) r 4 − 2 r 2 (cid:1) is minimized at r = 1 and (cid:16) 1 − sin( θ ) 3 (cid:17) is maximized at θ = 3 π 2 .
Revised: For h ( θ, r ) = (cid:16) 1 − sin( θ ) 3 (cid:17) (cid:0) r 4 − 2 r 2 (cid:1) , we know (cid:0) r 4 − 2 r 2 (cid:1) is minimized at r = 1 and (cid:16) 1 − sin( θ ) 3 (cid:17) is maximized at θ = 3 π 2 .

Original: Besides the minimizer (0 , − 1) , the other stationary point is at (0 , 0) .
Revised: Besides the minimizer (0 , − 1) , the other stationary point is at (0 , 0) .

Original: For any point ( x, y ) different from (0 , − 1) and (0 , 0) , if x 2 + y 2 ̸ = 1 , the gradient along the radial direction is non-zero; if y √ x 2 + y 2 ̸ = − 1 , the gradient along the tangent direction is non-zero.
Revised: For any point ( x, y ) different from (0 , − 1) and (0 , 0) , if x 2 + y 2 ̸ = 1 , the gradient along the radial direction is non-zero; if y √ x 2 + y 2 ̸ = − 1 , the gradient along the tangent direction is non-zero.

Original: It’s also easy to verify that starting from a non-zero point, gradient flow does not converge to (0 , 0) , so it must converge to (0 , − 1)
Revised: It’s also easy to verify that starting from a non-zero point, gradient flow does not converge to (0 , 0) , so it must converge to (0 , − 1)

Original: It’s also very easy to prove that gradient descent with appropriate step size converges to an ϵneighborhood of the global minimizer within poly (1 /ϵ ) number of iterations.
Revised: It’s also very easy to prove that gradient descent with appropriate step size converges to an ϵneighborhood of the global minimizer within poly (1 /ϵ ) number of iterations.

Original: This is because the gradient is at least poly ( ϵ ) for any non-zero point outside of the ϵ -neighborhood of the global minimizer.
Revised: This is because the gradient is at least poly ( ϵ ) for any non-zero point outside of the ϵ -neighborhood of the global minimizer.

Original: Starting from an initialization ( x, y ) with x 2 + y 2 = Θ(1) , the smoothness along the training is also bounded by a constant.
Revised: Starting from an initialization ( x, y ) with x 2 + y 2 = Θ(1) , the smoothness along the training is also bounded by a constant.

Original: Next, we prove that starting from certain initialization 3 , the loss interpolation between the initialization and the global minimizer is non-monotonic.
Revised: Next, we prove that starting from certain initialization 3 , the loss interpolation between the initialization and the global minimizer is non-monotonic.

Original: We prove this by identifying two points along the interpolation path such that the point closer to minimizer has a higher loss compared with the point further to theminimizer
Revised: We prove this by identifying two points along the interpolation path such that the point closer to minimizer has a higher loss compared with the point further to the

Original: Proposition 5.
Revised: Proposition 5.

Original: Suppose we start from an initialization ( x 0 , y 0 ) =
Revised: Suppose we start from an initialization ( x 0 , y 0 ) =

Original: ( r sin( β ) , r cos( β )) with r ≥ 1 and β ∈ [ − π/ 3 , π/ 3] .
Revised: ( r sin( β ) , r cos( β )) with r ≥ 1 and β ∈ [ − π/ 3 , π/ 3] .

Original: Consider the loss interpolation curve γ ( α ) =
Revised: Consider the loss interpolation curve γ ( α ) =

Original: f ((1 − α ) x 0 + αx ∗ , (1 − α )
Revised: f ((1 − α ) x 0 + αx ∗ , (1 − α )

Original: y 

Revised: y

Original: + αy ∗ ) with ( x ∗ , y ∗ )
Revised: + αy ∗ ) with ( x ∗ , y ∗ )

Original: = (0 , − 1) and f ( · , · ) defined in Eqn.
Revised: = (0 , − 1) and f ( · , · ) defined in Eqn.

Original: We know there exist 0 ≤ α 1 < α 2 ≤ 1 such that
Revised: We know there exist 0 ≤ α 1 < α 2 ≤ 1 such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We prove for any β ∈ [ − π/ 3 , π/ 3] and any r ≥ 1 , the loss interpolation between ( r sin( β ) , r cos( β )) to (0 , − 1) is non-monotonic.
Revised: We prove for any β ∈ [ − π/ 3 , π/ 3] and any r ≥ 1 , the loss interpolation between ( r sin( β ) , r cos( β )) to (0 , − 1) is non-monotonic.

Original: In particular, we show there are two points along the linear interpolation satisfying
Revised: In particular, we show there are two points along the linear interpolation satisfying

Original: [Equation]
Revised: [Equation]

Original: where (sin ( β/ 2) cos ( β/ 2) , − sin ( β/ 2) sin ( β/ 2)) is the middle point between (sin( β ) , cos( β )) and (0 , − 1) .
Revised: where (sin ( β/ 2) cos ( β/ 2) , − sin ( β/ 2) sin ( β/ 2)) is the middle point between (sin( β ) , cos( β )) and (0 , − 1) .

Original: Next, we separately upper bound f (sin( β ) , cos( β )) and lower bound f (sin ( β/ 2) cos ( β/ 2) , − sin ( β/ 2) sin ( β/ 2)) .
Revised: Next, we separately upper bound f (sin( β ) , cos( β )) and lower bound f (sin ( β/ 2) cos ( β/ 2) , − sin ( β/ 2) sin ( β/ 2)) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have f (sin ( β/ 2) cos ( β/ 2) , − sin ( β/ 2) sin ( β/ 2))
Revised: Therefore, we have f (sin ( β/ 2) cos ( β/ 2) , − sin ( β/ 2) sin ( β/ 2))

Original: − f (sin( β ) , cos( β )) ≥ 532 .
Revised: − f (sin( β ) , cos( β )) ≥ 532 .

Original: B P ROOF FOR PLATEAU AND MONOTONICITY
Revised: B P ROOF FOR PLATEAU AND MONOTONICITY

Original: We first consider the r -homogeneous-weight model.
Revised: We first consider the r -homogeneous-weight model.

Original: We prove the plateau and monotonicity properties for the error interpolation (Theorem 4) in Section B.1.
Revised: We prove the plateau and monotonicity properties for the error interpolation (Theorem 4) in Section B.1.

Original: We then prove the plateau and monotonicity properties for the loss interpolation (Theorem 5) in Section B.2.
Revised: We then prove the plateau and monotonicity properties for the loss interpolation (Theorem 5) in Section B.2.

Original: Theorem 3 is a simple combination of Theorem 4 and Theorem 5.
Revised: Theorem 3 is a simple combination of Theorem 4 and Theorem 5.

Original: Finally, we give the plateau analysis for the fully-connected neural networks (Theorem 1) in Section B.3.
Revised: Finally, we give the plateau analysis for the fully-connected neural networks (Theorem 1) in Section B.3.

Original: B. E RROR INTERPOLATION FOR r - HOMOGENEOUS - WEIGHT MODEL
Revised: B. E RROR INTERPOLATION FOR r - HOMOGENEOUS - WEIGHT MODEL

Original: Theorem 4 (Error Interpolation) .
Revised: Theorem 4 (Error Interpolation) .

Original: Suppose the network at initialization and after training satisfy the properties described in Theorem 2 and Induction Hypothesis 1.
Revised: Suppose the network at initialization and after training satisfy the properties described in Theorem 2 and Induction Hypothesis 1.

Original: Suppose δ ≤ 
Revised: Suppose δ ≤ min( O (1) , [Equation] [Equation] min

Original: [Equation]
Revised: [Equation]

Revised: min ) , O ((

Revised: [Equation]

Revised: max ) 2 r r

Original:  There exist 
Revised: [Equation] There exist α 1 = [Equation] and

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: for all α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k ; 2.
Revised: for all α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k ; 2.

Original: for all α ∈ [ α 1 , 1] , the error is non-increasing.
Revised: for all α ∈ [ α 1 , 1] , the error is non-increasing.

Original: Proof of Theorem 4.
Revised: Proof of Theorem 4.

Original: This theorem directly follows from Lemma 4 and Lemma 5.
Revised: This theorem directly follows from Lemma 4 and Lemma 5.

Original: □
Revised: □

Original: Next, we separately prove the initial plateau in Lemma 4 and the monotonicity in Lemma 5.
Revised: Next, we separately prove the initial plateau in Lemma 4 and the monotonicity in Lemma 5.

Original: Lemma 4 (Error Plateau) .
Revised: Lemma 4 (Error Plateau) .

Original: In the same setting as in Theorem 4, there exists α 1 = δ ∆ min and α 2 = (cid:16)
Revised: In the same setting as in Theorem 4, there exists α 1 = δ ∆ min and α 2 = (cid:16)

Original: [Equation]
Revised: [Equation]

Original: , such that for any interpolation point with α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k.
Revised: , such that for any interpolation point with α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k.

Original: Moreover, we have f
Revised: Moreover, we have f

Original: [ α ] i ( e j ) <
Revised: [ α ] i ( e j ) <

Original: f [ α ] k ( e j ) for all j ∈ [ k ] and all i ̸ = k .
Revised: f [ α ] k ( e j ) for all j ∈ [ k ] and all i ̸ = k .

Original: In the proof of Lemma 4, we show that for interpolation point α ∈
Revised: In the proof of Lemma 4, we show that for interpolation point α ∈

Original: [ α 1 , α 2 ] , the bias term dominates and all samples are classified as class k that has the largest bias.
Revised: [ α 1 , α 2 ] , the bias term dominates and all samples are classified as class k that has the largest bias.

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: We only need to show that for all α ∈ [ α 1 , α 2 ] , we have
Revised: We only need to show that for all α ∈ [ α 1 , α 2 ] , we have

Original: [Equation]
Revised: [Equation]

Original: for all x ∈ S and all i ̸ = k, which immediately implies the error is 1 − 1 /k.
Revised: for all x ∈ S and all i ̸ = k, which immediately implies the error is 1 − 1 /k.

Original: Without loss of generality, assume x ∈ S j where j may equal i or k.
Revised: Without loss of generality, assume x ∈ S j where j may equal i or k.

Original: For α ∈ (cid:104) α 1 , √ δ W min (cid:17) .
Revised: For α ∈ (cid:104) α 1 , √ δ W min (cid:17) .

Original: If α 1 = δ ∆ min ≥ √ δ
Revised: If α 1 = δ ∆ min ≥ √ δ

Original: W min , we only need to consider the case when α ∈ (cid:104) √
Revised: W min , we only need to consider the case when α ∈ (cid:104) √

Original: δ
Revised: δ

Original: W min , α 2 (cid:105) .
Revised: W min , α 2 (cid:105) .

Original: So here we assume
Revised: So here we assume

Original: [Equation]
Revised: [Equation]

Original: W min .
Revised: W min .

Original: We can lower bound f
Revised: We can lower bound f

Original: [ α ] k ( x ) − f
Revised: [ α ] k ( x ) − f

Original: [ α ] i ( x ) as f [ α ] k ( x)
Revised: [ α ] i ( x ) as f [ α ] [Equation] ( x

Original: [Equation]
Revised: [Equation]

Original: [ α ] i ( x )
Revised: [ α ] i ( x )

Original: [Equation]
Revised: [Equation]

Revised: W [ α ] k, :

Original: [Equation]
Revised: [Equation]

Revised: x (cid:69)(cid:105)

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: x
Revised: (cid:104)(cid:68) [Equation] [Equation] ] i, :

Original: [Equation]
Revised: [Equation]

Original: i
Revised: x (cid:69)(cid:105)

Original: [Equation]
Revised: [Equation]

Revised: b [ α ] i

Original: [Equation]
Revised: [Equation]

Original: x
Revised: W [ [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: [ α ] i
Revised: [ α ] i

Original: [Equation]
Revised: [Equation]

Original:  ≥
Revised: r [Equation] ] i ≥

Original: [Equation]
Revised: [Equation]

Original: , where the second equality uses (cid:12)(cid:12)(cid:12)(cid:68) W
Revised: , where the second equality uses (cid:12)(cid:12)(cid:12)(cid:68) W

Original: [ α ] i, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( δ ) and the inequality uses (cid:68) W [ α ] k, : , x (cid:69) ≥ 0 .
Revised: [ α ] i, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( δ ) and the inequality uses (cid:68) W [ α ] k, : , x (cid:69) ≥ 0 .

Original: To prove f [ α ] k ( x ) − f
Revised: To prove f [ α ] k ( x ) − f

Original: [ α ] i ( x ) > 0 for α ∈ (cid:104)
Revised: [ α ] i ( x ) > 0 for α ∈ (cid:104)

Original: δ ∆ min , √ δ W min (cid:105) , we only need to prove δ ∆ min ∆ i − (cid:104) W (0) i,j + √ δ W min W ( T ) i,j + O ( δ ) (cid:105) r > 0 .
Revised: δ ∆ min , √ δ W min (cid:105) , we only need to prove δ ∆ min ∆ i − (cid:104) W (0) i,j + √ δ W min W ( T ) i,j + O ( δ ) (cid:105) r > 0 .

Original: Since ∆ i ≥ ∆ min , we know δ ∆ min ∆
Revised: Since ∆ i ≥ ∆ min , we know δ ∆ min ∆

Original: i ≥ δ.
Revised: i ≥ δ.

Original: Due to full accuracy, we know (cid:68) W ( T ) i, : , x (cid:69) ≥ ∆ 1 /ri for x ∈ S i , which then implies W ( T ) i,
Revised: Due to full accuracy, we know (cid:68) W ( T ) i, : , x (cid:69) ≥ ∆ 1 /ri for x ∈ S i , which then implies W ( T ) i,

Original: i ≥ Ω (cid:16) ∆ 1 /ri (cid:17) because ∆ i ≥ Ω(1) and (cid:68) W ( T ) i, : , ξ x (cid:69) ≤ O ( δ ) ≤ O (1) .
Revised: i ≥ Ω (cid:16) ∆ 1 /ri (cid:17) because ∆ i ≥ Ω(1) and (cid:68) W ( T ) i, : , ξ x (cid:69) ≤ O ( δ ) ≤ O (1) .

Original: Since W ( T ) i,i ≥ Ω (cid:16) ∆ 1 /ri (cid:17) and W ( T ) i,j ≤ O ( δ ) for i ̸
Revised: Since W ( T ) i,i ≥ Ω (cid:16) ∆ 1 /ri (cid:17) and W ( T ) i,j ≤ O ( δ ) for i ̸

Original: = j, so we have W ( T ) i,j ≤ W ( T ) i,i ≤ W max as long as δ ≤ O (∆ 1 /r min ) .
Revised: = j, so we have W ( T ) i,j ≤ W ( T ) i,i ≤ W max as long as δ ≤ O (∆ 1 /r min ) .

Original: So we can upper bound (cid:104) W [Equation] i, [Equation]
Revised: So we can upper bound (cid:104) W (0) i,j + √ δ W min W ( T ) i,j + O

Original: √

Original: [Equation]
Revised: [Equation]

Original: min

Original: [Equation]

Original: ( T )

Original: i

Original: [Equation]

Original: j + O

Original: [Equation]

Original: δ ) (cid:105) r as follows,
Revised: δ ) (cid:105) r as follows,

Original: [Equation]
Revised: [Equation]

Original: where the second inequality assumes δ ≤ O (cid:16) W 2max W 2min (cid:17) .
Revised: where the second inequality assumes δ ≤ O (cid:16) W 2max W 2min (cid:17) .

Original: Therefore, to prove δ ∆ min ∆ i − (cid:104) W (0) i,j + √ δ W min W ( T ) i,j + O ( δ ) (cid:105) r > 0 we only need δ − e (cid:32) √ δW max
Revised: Therefore, to prove δ ∆ min ∆ i − (cid:104) W (0) i,j + √ δ W min W ( T ) i,j + O ( δ ) (cid:105) r > 0 we only need δ − e (cid:32) √ δW max

Original: W min (cid:33) r > 0 , which holds as long as δ < (cid:104) 1 e (cid:16) W min W max (cid:17) r (cid:105) 2 r − 2 .
Revised: W min (cid:33) r > 0 , which holds as long as δ < (cid:104) 1 e (cid:16) W min W max (cid:17) r (cid:105) 2 r − 2 .

Original: For α ∈ (cid:104) √
Revised: For α ∈ (cid:104) √

Original: δ
Revised: δ

Original: W min , α 2 (cid:105) .
Revised: W min , α 2 (cid:105) .

Original: Similar as above, we only need to show that α ∆ i − (cid:104) W (0) i,j + αW ( T ) i,j + O ( δ ) (cid:105) r > 0 for i ̸ = k and j ∈ [ k ] .
Revised: Similar as above, we only need to show that α ∆ i − (cid:104) W (0) i,j + αW ( T ) i,j + O ( δ ) (cid:105) r > 0 for i ̸ = k and j ∈ [ k ] .

Original: Since W (0) i,j ≤ O ( δ ) and α ≥ √ δ/W min , we have W (0) i,j ≤ O ( √ δαW min ) .
Revised: Since W (0) i,j ≤ O ( δ ) and α ≥ √ δ/W min , we have W (0) i,j ≤ O ( √ δαW min ) .

Original: Therefore, we have W (0) i,j + αW ( T ) i,j + O ( δ ) ≤ 
Revised: Therefore, we have W (0) i,j + αW ( T ) i,j + O ( δ ) ≤ (cid:16)

Original: [Equation]
Revised: [Equation]

Original: √
Revised: + O

Original: [Equation]
Revised: [Equation]

Revised: √ δ

Original: [Equation]
Revised: [Equation]

Revised: ) i,i .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality assumes α ≤ α
Revised: where the last inequality assumes α ≤ α

Original: [Equation]
Revised: [Equation]

Original: R min = min i ∈ [ k − 1] ∆ i / [ W ( T ) i,i ] r .
Revised: R min = min i ∈ [ k − 1] ∆ i / [ W ( T ) i,i ] r .

Original: □
Revised: □

Original: Next, we show that the error is non-increasing for α ∈ [ α 1 , 1] by proving that once a sample is classified correctly it will remain so.
Revised: Next, we show that the error is non-increasing for α ∈ [ α 1 , 1] by proving that once a sample is classified correctly it will remain so.

Original: Lemma 5 (Error Monotonicity) .
Revised: Lemma 5 (Error Monotonicity) .

Original: In the same setting as in Theorem 4, there exists δ 1 = δ ∆ min such that the error is non-increasing for α ∈
Revised: In the same setting as in Theorem 4, there exists δ 1 = δ ∆ min such that the error is non-increasing for α ∈

Original: [ α 1 , 1] .
Revised: [ α 1 , 1] .

Original: Proof of Lemma 5.
Revised: Proof of Lemma 5.

Original: We first show that sample e k is correctly classified for the whole range [ α 1 , 1] .
Revised: We first show that sample e k is correctly classified for the whole range [ α 1 , 1] .

Original: Second, we show for any other sample once it become classified right it will remain so.
Revised: Second, we show for any other sample once it become classified right it will remain so.

Original: Combining these two cases, we prove the monotonicity of the error rate.
Revised: Combining these two cases, we prove the monotonicity of the error rate.

Original: Class k. We first show that every x ∈ S k is classified correctly for any α ∈ [ α 1 , 1] .
Revised: Class k. We first show that every x ∈ S k is classified correctly for any α ∈ [ α 1 , 1] .

Original: According to
Revised: According to

Original: Lemma 4, we know that
Revised: Lemma 4, we know that

Original: [Equation]
Revised: [Equation]

Original: for any i ̸ = k.
Revised: for any i ̸ = k.

Original: We only need to prove that f [ α ] k ( x ) − f
Revised: We only need to prove that f [ α ] k ( x ) − f

Original: [ α ] i ( x ) is increasing for α ∈ [ α 1 , 1] .
Revised: [ α ] i ( x ) is increasing for α ∈ [ α 1 , 1] .

Original: Expanding f
Revised: Expanding f

Original: [ α ] k ( x ) − f
Revised: [ α ] k ( x ) − f

Original: [ α ] i ( x ) , we have f
Revised: [ α ] i ( x ) , we have f

Original: [ α ] k ( x ) − f
Revised: [ α ] k ( x ) − f

Original: [ α ] i ( x ) = (cid:104) (1 − α )
Revised: [ α ] i ( x ) = (cid:104) (1 − α )

Original: (cid:68) W (0) k, : , x (cid:69) + α (cid:16)(cid:68) W ( T ) k, : , x (cid:69)(cid:17)(cid:105) r − (cid:104) (1 − α ) (cid:68) W (0) i, : , x (cid:69) + α (cid:68) W ( T ) i, : , x (cid:69)(cid:105) r + α (cid:16) b ( T ) k − b ( T ) i (cid:17) , which is increasing since (cid:12)(cid:12)(cid:12)(cid:68) W ( T ) k, : , x (cid:69)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) W (0) k, : , x (cid:69)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) W ( T ) i, : , x (cid:69)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) W (0) i, : , x (cid:69)(cid:12)(cid:12)(cid:12) ≤
Revised: (cid:68) W (0) k, : , x (cid:69) + α (cid:16)(cid:68) W ( T ) k, : , x (cid:69)(cid:17)(cid:105) r − (cid:104) (1 − α ) (cid:68) W (0) i, : , x (cid:69) + α (cid:68) W ( T ) i, : , x (cid:69)(cid:105) r + α (cid:16) b ( T ) k − b ( T ) i (cid:17) , which is increasing since (cid:12)(cid:12)(cid:12)(cid:68) W ( T ) k, : , x (cid:69)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) W (0) k, : , x (cid:69)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) W ( T ) i, : , x (cid:69)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) W (0) i, : , x (cid:69)(cid:12)(cid:12)(cid:12) ≤

Original: O ( δ ) and b ( T ) k − b ( T )
Revised: O ( δ ) and b ( T ) k − b ( T )

Original: i > Ω(1) .
Revised: i > Ω(1) .

Original: Other classes.
Revised: Other classes.

Original: For any class i ̸
Revised: For any class i ̸

Original: = k, from Lemma 4, we know that it is classified incorrectly for α ∈
Revised: = k, from Lemma 4, we know that it is classified incorrectly for α ∈

Original: [ α 1 , α 2 ] .
Revised: [ α 1 , α 2 ] .

Original: We prove that once it become classified correctly at some α ′ ∈ ( α 2 , 1] , it remains so for α ∈ [ α ′ , 1] .
Revised: We prove that once it become classified correctly at some α ′ ∈ ( α 2 , 1] , it remains so for α ∈ [ α ′ , 1] .

Original: We show that at α, for any x ∈ S i , if f [ α ] i ( x ) >
Revised: We show that at α, for any x ∈ S i , if f [ α ] i ( x ) >

Original: f
Revised: f

Original: [ α ] j ( x ) for all j ̸ = i , we have ∂∂α (cid:16) f
Revised: [ α ] j ( x ) for all j ̸ = i , we have ∂∂α (cid:16) f

Original: [ α ] i ( x ) − f
Revised: [ α ] i ( x ) − f

Original: [ α ] j ( x ) (cid:17) > 0 .
Revised: [ α ] j ( x ) (cid:17) > 0 .

Original: Expanding f
Revised: Expanding f

Original: [ α ] i ( x ) − f
Revised: [ α ] i ( x ) − f

Original: [ α ] j ( x ) , we have
Revised: [ α ] j ( x ) , we have

Original: [Equation]
Revised: [Equation]

Original: Since f [ α ] i ( x ) − f
Revised: Since f [ α ] i ( x ) − f

Original: [ α ] j ( x ) > 0 , we have (cid:104)(cid:68) W [ α ]i, : , x (cid:69)(cid:105) r > α (cid:16) b ( T ) j − b ( T ) i (cid:17) , where we use (cid:68) W [ α ] j, : , x (cid:69) ≥ 0 .
Revised: [ α ] j ( x ) > 0 , we have (cid:104)(cid:68) W [ α ] i [Equation] : , x [Equation]

Revised: r > α

Revised: [Equation]

Revised: j − b

Revised: [Equation]

Revised: T

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: , where we use (cid:68) W [ α ] j, : , x (cid:69) ≥ 0 .

Original: Computing ∂∂α (cid:16) f
Revised: Computing ∂∂α (cid:16) f

Original: [ α ] i ( x ) − f
Revised: [ α ] i ( x ) − f

Original: [ α ] j ( x ) (cid:17) , we have ∂ ∂α (cid:16) [Equation]
Revised: [ α ] j ( x ) (cid:17) , we have ∂ ∂α (cid:16) f

Original: [ α ] i [Equation] x ) − f [ α ] j
Revised: [ α ] i ( x ) − f [ α ] j

Original: [Equation]
Revised: [Equation]

Original: x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: ∂ ∂α (cid:16)(cid:104)
Revised: ∂ ∂α (cid:16)(cid:104)

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: − α
Revised: − α

Original: [Equation]
Revised: [Equation]

Original: W

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: where the inequality uses (cid:12)(cid:12)(cid:12)(cid:68) W (0) j, : , x (cid:69)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) W ( T ) j, : , x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( δ ) .
Revised: where the inequality uses (cid:12)(cid:12)(cid:12)(cid:68) W (0) j, : , x (cid:69)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) W ( T ) j, : , x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( δ ) .

Original: If b ( T ) j − b ( T ) i ≤ 0 , we only need to prove r
Revised: If b ( T ) j − b ( T ) i ≤ 0 , we only need to prove r

Original: [Equation]
Revised: [Equation]

Original: (cid:69)(cid:17)(cid:105) r
Revised: (cid:69)(cid:17)(cid:105) r

Original: [Equation]
Revised: [Equation]

Original: (cid:16)(cid:68)
Revised: (cid:16)(cid:68)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:69) − (cid:68) W (0) i, : , x (cid:69)(cid:17) − O
Revised: (cid:69) [Equation] (cid:68) W (0) i, : , x (cid:69)(cid:17) − O

Original: ( δ r ) > 0 , which holds since (cid:16)(cid:68) W ( T ) i, : , x (cid:69) − (cid:68) W (0)
Revised: ( δ r ) > 0 , which holds since (cid:16)(cid:68) W ( T ) i, : , x (cid:69) − (cid:68) W (0)

Original: i, : , x (cid:69)(cid:17) , (cid:68) W [ α ] i, : , x (cid:69) ≥ Ω(1) .
Revised: i, : , x (cid:69)(cid:17) , (cid:68) W [ α ] i, : , x (cid:69) ≥ Ω(1) .

Original: If b ( T ) j − b ( T ) i > 0 , we have ∂
Revised: If b ( T ) j − b ( T ) i > 0 , we have ∂

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: >
Revised: >

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses (cid:104)(cid:68) W
Revised: where the last inequality uses (cid:104)(cid:68) W

Original: [ α ] i, : , x (cid:69)(cid:105) r > α (cid:16) b
Revised: [ α ] i, : , x (cid:69)(cid:105) [Equation] > α (cid:16) b

Original: [Equation]
Revised: [Equation]

Original: − b ( T ) i (cid:17) .
Revised: − b ( T ) i (cid:17) .

Original: Therefore, to prove ∂∂α (cid:16) f 
Revised: Therefore, to prove ∂∂α [Equation] f [Equation]

Original: [ α ] i ( e i ) − f
Revised: i [Equation] e i ) − f

Original: [ α ] j ( e i ) (cid:17) > 0 , we only need to prove
Revised: [ α ] j ( e i ) (cid:17) > 0 , we only need to prove

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The first inequality requires (cid:68) W (0) i, : , x (cid:69) ≤ α (cid:16)(cid:68) W ( T ) i, : , x (cid:69) − (cid:68) W (0) i, : , x (cid:69)(cid:17) and the second inequality uses r ≥ 3 , (1 − O ( δ r )) ≥ 2 / 3 .
Revised: The first inequality requires (cid:68) W (0) i, : , x (cid:69) ≤ α (cid:16)(cid:68) W ( T ) i, : , x (cid:69) − (cid:68) W (0) i, : , x (cid:69)(cid:17) and the second inequality uses r ≥ 3 , (1 − O ( δ r )) ≥ 2 / 3 .

Original: To prove (cid:68) W (0) i, : , x (cid:69) ≤ α (cid:16)(cid:68) W ( T ) i, : , x (cid:69) − (cid:68) W (0) i, : , x (cid:69)(cid:17) , it’s equivalent to show (cid:68) W (0) i, : , x (cid:69) ≤ α 1+ α (cid:68) W ( T ) i, : , x (cid:69) .
Revised: To prove (cid:68) W (0) i, : , x (cid:69) ≤ α (cid:16)(cid:68) W ( T ) i, : , x (cid:69) − (cid:68) W (0) i, : , x (cid:69)(cid:17) , it’s equivalent to show (cid:68) W (0) i, : , x (cid:69) ≤ α 1+ α (cid:68) W ( T ) i, : , x (cid:69) .

Original: Since α ≥ α 2 = (cid:16) 1 1+ O ( √ δ ) (cid:17) rr − 1 R 1 r − 1 min , we can lower bound α 1+ α as follows,
Revised: Since α ≥ α 2 = (cid:16) 1 1+ O ( √ δ ) (cid:17) rr − 1 R 1 r − 1 min , we can lower bound α 1+ α as follows,

Original: [Equation]
Revised: [Equation]

Original: where the first inequality uses α ≤ 1 and the second inequality uses 1 + O ( √ δ ) ≤ 2 , r ≥ 2 .
Revised: where the first inequality uses α ≤ 1 and the second inequality uses 1 + O ( √ δ ) ≤ 2 , r ≥ 2 .

Original: So we have α 1+ α (cid:68) W ( T ) i, : , x (cid:69) ≥ 18 R 1 r − 1 min ∆ 1 /r min .
Revised: So we have α 1+ α (cid:68) W ( T ) i, : , x (cid:69) ≥ 18 R 1 r − 1 min ∆ 1 /r min .

Original: Therefore, we only need δ ≤ O (cid:18) R 1 r − 1 min ∆ 1 /r min (cid:19) to ensure that
Revised: Therefore, we only need δ ≤ O (cid:18) R 1 r − 1 min ∆ 1 /r min (cid:19) to ensure that

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: B. L OSS INTERPOLATION FOR r - HOMOGENEOUS - WEIGHT MODEL
Revised: B. L OSS INTERPOLATION FOR r - HOMOGENEOUS - WEIGHT MODEL

Original: In this section, we give a proof of Theorem 5.
Revised: In this section, we give a proof of Theorem 5.

Original: Theorem 5 (Loss Interpolation) .
Revised: Theorem 5 (Loss Interpolation) .

Original: Suppose the network at initialization and after training satisfy the properties described in Theorem 2 and Induction Hypothesis 1.
Revised: Suppose the network at initialization and after training satisfy the properties described in Theorem 2 and Induction Hypothesis 1.

Original: For any ϵ ∈ (0 , 1) , suppose δ ≤ O ( ϵ 1 /r ) , there exist α 3 = ϵ 1 /r W max and α 4 = (1 + O ( δ )) 1 r − 1 (cid:0) R max r (cid:1) 1 r − 

Revised: For any ϵ ∈ (0 , 1) , suppose δ ≤ O ( ϵ 1 /r ) , there exist α 3 = ϵ 1 /r W max and α 4 = (1 + O ( δ )) 1 r − 1 (cid:0) R max r (cid:1) 1 r −

Original: such that
Revised: such that

Original: for all α ∈ [0 , α 3 ] , we have log k − eϵ ≤ 1 N L ( W [ α ] , b [ α ] ) ≤ log k + α ∆ max + eϵ ; 2. for all α ∈
Revised: for all α ∈ [0 , α 3 ] , we have log k − eϵ ≤ 1 N L ( W [ α ] , b [ α ] ) ≤ log k + α ∆ max + eϵ ; 2. for all α ∈

Original: [ α 4 , 1] , the loss is monotonically decreasing.
Revised: [ α 4 , 1] , the loss is monotonically decreasing.

Original: Proof of Theorem 5.
Revised: Proof of Theorem 5.

Original: This theorem directly follows from Lemma 6 and Lemma 7.
Revised: This theorem directly follows from Lemma 6 and Lemma 7.

Original: □
Revised: □

Original: Next, we prove the initial loss plateau in Lemma 6 and the monotonicity in Lemma 7.
Revised: Next, we prove the initial loss plateau in Lemma 6 and the monotonicity in Lemma 7.

Original: Lemma 6 (Loss Plateau) .
Revised: Lemma 6 (Loss Plateau) .

Original: In the same setting as in Theorem 5, for any ϵ > 0 , there exists α 3 = ϵ 1 /r W max such that for all α ∈ [0 , α 3 ]
Revised: In the same setting as in Theorem 5, for any ϵ > 0 , there exists α 3 = ϵ 1 /r W max such that for all α ∈ [0 , α 3 ]

Original: [Equation]
Revised: [Equation]

Original: We show that for α ∈ [0 , α 3 ] , the weights W [ α ] is negligible and the bias dominates, which then gives a lower bound and an upper bound of the loss.
Revised: We show that for α ∈ [0 , α 3 ] , the weights W [ α ] is negligible and the bias dominates, which then gives a lower bound and an upper bound of the loss.

Original: Proof of Lemma 6.
Revised: Proof of Lemma 6.

Original: Since α ≤ α 3 = ϵ 1 /r W max and δ ≤ O (cid:0) ϵ 1 /r (cid:1) , we have
Revised: Since α ≤ α 3 = ϵ 1 /r W max and δ ≤ O (cid:0) ϵ 1 /r (cid:1) , we have

Original: [Equation]
Revised: [Equation]

Original: for all i ∈ [ k ] , x ∈ S .
Revised: for all i ∈ [ k ] , x ∈ S .

Original: We can divide the dataset S into N/k disjoint subsets { P l } N/kl =1 where each P l contains exactly one sample from each class.
Revised: We can divide the dataset S into N/k disjoint subsets { P l } N/kl =1 where each P l contains exactly one sample from each class.

Original: Next, we bound the total loss of each subset P l .
Revised: Next, we bound the total loss of each subset P l .

Original: Without loss of generality, let’s consider subset P 1 and suppose x ( i ) is the i -th class sample in this subset.
Revised: Without loss of generality, let’s consider subset P 1 and suppose x ( i ) is the i -th class sample in this subset.

Original: For convenience, we denote the total loss of samples in P 1 as L 1 ( W [ α ] , b [ α ] ) .
Revised: For convenience, we denote the total loss of samples in P 1 as L 1 ( W [ α ] , b [ α ] ) .

Original: Lower bounding L 1 ( W [ α ] , b [ α ] ) .
Revised: Lower bounding L 1 ( W [ α ] , b [ α ] ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses the HM-GM inequality
Revised: where the last inequality uses the HM-GM inequality

Original: [Equation]
Revised: [Equation]

Original: We can then upper bound
Revised: We can then upper bound

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: as follows

Original: [Equation] W [ α ] j, : , x [Equation] i )
Revised: [ k

Original: [Equation]
Revised: [Equation]

Original: ≤ [Equation] [Equation] j
Revised: as follows

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]

Original: [Equation]

Original: [Equation] ( αb [Equation] ) (cid:80) j ∈ [ k ] exp (αb
Revised: exp (

Original: [Equation]
Revised: [Equation]

Original: exp ( eϵ ) = exp ( eϵ ) .

Original: Plugging back to the lower bound of L 1 ( W [ α ] , b [ α ] ) , we have[Equation] k exp [Equation]
Revised: Plugging back to the lower bound of L 1 ( W [ α ] , b [ α ] ) , we have

Original: [Equation]

Original: =

Original: [Equation]

Original: log

Original: [Equation]
Revised: [Equation]

Original: Upper bounding L 1 ( W [ α ] , b [ α ] ) .
Revised: Upper bounding L 1 ( W [ α ] , b [ α ] ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The above analysis applies for every subset P l , so we have
Revised: The above analysis applies for every subset P l , so we have

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: Next we show that when α is reasonably large, we have f [ α ] i ( e i ) − f
Revised: Next we show that when α is reasonably large, we have f [ α ] i ( e i ) − f

Original: [ α ] j ( e i ) increasing for all
Revised: [ α ] j ( e i ) increasing for all

Original: i ̸
Revised: i ̸

Original: = j , which then implies that the loss is decreasing.
Revised: = j , which then implies that the loss is decreasing.

Original: Lemma 7 (Loss Monotonicity) .
Revised: Lemma 7 (Loss Monotonicity) .

Original: In the same setting as in Theorem 5, there exists α 4 =
Revised: In the same setting as in Theorem 5, there exists α 4 =

Original: (1 + O ( δ )) 1 r − 1 (cid:0) R max r (cid:1) 1 r − 1 such that the loss is monotonically decreasing for α ∈ [ α 4 , 1] .
Revised: (1 + O ( δ )) 1 r − 1 (cid:0) R max r (cid:1) 1 r − 1 such that the loss is monotonically decreasing for α ∈ [ α 4 , 1] .

Original: Proof of Lemma 7.
Revised: Proof of Lemma 7.

Original: To prove that the loss is monotonically decreasing, we only need to show that for any i ∈ [ k ] and any x ∈ S i , f
Revised: To prove that the loss is monotonically decreasing, we only need to show that for any i ∈ [ k ] and any x ∈ S i , f

Original: [ α ] i ( x ) − f
Revised: [ α ] i ( x ) − f

Original: [ α ] j ( x ) is monotonically increasing for j ̸ = i.
Revised: [ α ] j ( x ) is monotonically increasing for j ̸ = i.

Original: Same as in Lemma 5, it’s easy to prove that for x ∈ S k , f k ( x ) − f j ( x ) with j ̸ = k monotonically increases for α ∈ [0 , 1] .
Revised: Same as in Lemma 5, it’s easy to prove that for x ∈ S k , f k ( x ) − f j ( x ) with j ̸ = k monotonically increases for α ∈ [0 , 1] .

Original: So we focus on other classes.
Revised: So we focus on other classes.

Original: For i ̸
Revised: For i ̸

Original: = k, we show [Table] ∂∂α (cid:16) 
Revised: = k, we show that ∂∂α (cid:16) f

Original: [Table] [Table] ] i ( [Table]
Revised: [ α ] i ( x ) − f

Original: [Table] [Table] [Table] for x ∈ S i when α ≥ α 
Revised: [ α ] j ( x ) (cid:17) > 0 for x ∈ S i when α ≥ α 4 ,

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: > 0 , where the second last inequality uses (cid:68) W (0) i, : , x (cid:69) / (cid:68) W ( T ) i, : , x (cid:69) ≤ O (cid:16) δ/ ∆ 1 /r min (cid:17) .
Revised: > 0 , where the second last inequality uses (cid:68) W (0) i, : , x (cid:69) / (cid:68) W ( T ) i, : , x (cid:69) ≤ O (cid:16) δ/ ∆ 1 /r min (cid:17) .

Original: The last inequality requires
Revised: The last inequality requires

Original: [Equation]
Revised: [Equation]

Original: which is satisfied as long as α
Revised: which is satisfied as long as α

Original: [Equation]
Revised: [Equation]

Original: where R max = 
Revised: where R max = max i

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: B. P LATEAU FOR DEEP FULLY - CONNECTED NETWORKS
Revised: B. P LATEAU FOR DEEP FULLY - CONNECTED NETWORKS

Original: In this section, we consider fully-connected neural networks as defined in Section 3 and prove that both the error and loss curves have plateau.
Revised: In this section, we consider fully-connected neural networks as defined in Section 3 and prove that both the error and loss curves have plateau.

Original: We restate Theorem 1 as follows.
Revised: We restate Theorem 1 as follows.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose the network is defined as in Equation (1) and suppose the weights satisfy
Revised: Suppose the network is defined as in Equation (1) and suppose the weights satisfy

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13) ≤ V max for all layers i ∈
Revised: (cid:13)(cid:13)(cid:13) ≤ V max for all layers i ∈

Original: [ r ] .
Revised: [ r ] .

Original: On a k -class balanced dataset whose inputs have ℓ 2 norm at most 1 , if Assumption 1 holds, for any ϵ > 0 , as long as δ < min (cid:16) ϵ 1 /r r , 1 r 2 , (cid:0) 12 e (cid:1) 2 r − 2 (cid:17) , there exist α 1 =[Equation]
Revised: On a k -class balanced dataset whose inputs have ℓ 2 norm at most 1 , if Assumption 1 holds, for any ϵ > 0 , as long as δ < min (cid:16) ϵ 1 /r r , 1 r 2 , (cid:0) 12 e (cid:1) 2 r − 2 (cid:17) , there exist α 1 =

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: for all α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k ; 2.
Revised: for all α ∈ [ α 1 , α 2 ] , the error is 1 − 1 /k ; 2.

Original: for all α ∈ [0 , α 3 ] , we have log k − 2 eϵ ≤ 1 N L (cid:16)(cid:110)V [ [Equation] i [Equation]
Revised: for all α ∈ [0 , α 3 ] , we have log k − 2 eϵ ≤ 1 N L (cid:16)(cid:110)

Original: ≤ log

Original: [Equation]
Revised: [Equation]

Original: ∆ max
Revised: ∆ max

Original: [Equation]
Revised: [Equation]

Original: 2 eϵ
Revised: eϵ

Original: [Equation]
Revised: [Equation]

Original: where N is the number of training examples
Revised: where N is the number of training examples.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: This theorem directly follows from Lemma 8 and Lemma 9.
Revised: This theorem directly follows from Lemma 8 and Lemma 9.

Original: □
Revised: □

Original: We separately prove the plateau of error interpolation in Lemma 8 and the plateau of loss interpolation in Lemma 9.
Revised: We separately prove the plateau of error interpolation in Lemma 8 and the plateau of loss interpolation in Lemma 9.

Original: Then, Theorem 1 is simply a combination of Lemma 8 and Lemma 9.
Revised: Then, Theorem 1 is simply a combination of Lemma 8 and Lemma 9.

Original: For convenience, we denote h ( x ) :
Revised: For convenience, we denote h ( x ) :

Original: =
Revised: =

Original: V r σ ( V r − 1 · · · σ ( V 1 x ) · · · ) in the proof.
Revised: V r σ ( V r − 1 · · · σ ( V 1 x ) · · · ) in the proof.

Original: Lemma 8.
Revised: Lemma 8.

Original: In the setting of Theorem 1, there exist α 1 =
Revised: In the setting of Theorem 1, there exist α 1 =

Original: [Equation]
Revised: [Equation]

Original: and [Equation] such that the error is 1 − 1 /k for any interpolation point α ∈
Revised: such that the error is 1 − 1 /k for any interpolation point α ∈

Original: [ α 1 , α 2 ] .
Revised: [ α 1 , α 2 ] .

Original: Proof of Lemma 8.
Revised: Proof of Lemma 8.

Original: Recall that the network output under input x is g ( x ) :=
Revised: Recall that the network output under input x is g ( x ) :=

Original: V r σ ( V r − 1 · · · σ ( V 1 x ) · · · ) + b .
Revised: V r σ ( V r − 1 · · · σ ( V 1 x ) · · · ) + b .

Original: Similar as in the proof of Lemma 4, we only need to show that for all α ∈ [ α 1 , α 2 ] , we have
Revised: Similar as in the proof of Lemma 4, we only need to show that for all α ∈ [ α 1 , α 2 ] , we have

Original: [Equation]
Revised: [Equation]

Original: for all i ̸ =
Revised: for all i ̸ =

Original: k and all samples x , which immediately implies the error is 1 − 1 /k.
Revised: k and all samples x , which immediately implies the error is 1 − 1 /k.

Original: For α ∈ (cid:104) α 1 , √ δ V max (cid:17) .
Revised: For α ∈ (cid:104) α 1 , √ δ V max (cid:17) .

Original: If α 1 = δ ∆ min ≥ √ δ
Revised: If α 1 = δ ∆ min ≥ √ δ

Original: V max , we only need to consider the case when α ∈ (cid:104) √
Revised: V max , we only need to consider the case when α ∈ (cid:104) √

Original: δ
Revised: δ

Original: V max , α 2 (cid:105) .
Revised: V max , α 2 (cid:105) .

Original: So here we assume
Revised: So here we assume

Original: [Equation]
Revised: [Equation]

Original: V max .
Revised: V max .

Original: We can lower bound g
Revised: We can lower bound g

Original: [ α ] k ( x ) − g
Revised: [ α ] k ( x ) − g

Original: [ α ] i ( x ) as
Revised: [ α ] i ( x ) as

Original: [Equation]
Revised: [Equation]

Original: where the first inequality holds because b [ α ] k − b [ α ] i ≥ α ∆ min and (cid:12)(cid:12) (cid:12) h
Revised: where the first inequality holds because b [ α ] k − b [ α ] i ≥ α ∆ min and (cid:12)(cid:12) (cid:12) h

Original: [ α ] k ( x ) (cid:12)(cid:12) (cid:12) , (cid:12)(cid:12) (cid:12) h
Revised: [ α ] k ( x ) (cid:12)(cid:12) (cid:12) , (cid:12)(cid:12) (cid:12) h

Original: [ α ] i ( x ) (cid:12)(cid:12) (cid:12) ≤ (cid:81)
Revised: [ α ] i ( x ) (cid:12)(cid:12) (cid:12) ≤ (cid:81)

Original: j ∈
Revised: j ∈

Original: [ r ] (cid:13)(cid:13)(cid:13) (1 − α ) V (0) j
Revised: [ r ] (cid:13)(cid:13)(cid:13) (1 − α ) V (0) j

Original: + αV ( T ) j (cid:13)(cid:13)(cid:13) .
Revised: + αV ( T ) j (cid:13)(cid:13)(cid:13) .

Original: The second inequality uses (cid:13)(cid:13)(cid:13) (1 − α ) V (0) j
Revised: The second inequality uses (cid:13)(cid:13)(cid:13) (1 − α ) V (0) j

Original: + αV ( T ) j (cid:13)(cid:13)(cid:13) ≤ (1 − α )
Revised: + αV ( T ) j (cid:13)(cid:13)(cid:13) ≤ (1 − α )

Original: (cid:13)(cid:13)(cid:13) V (0)
Revised: (cid:13)(cid:13)(cid:13) V (0)

Original: j (cid:13)(cid:13)(cid:13) + α (cid:13)(cid:13)(cid:13) V ( T ) j (cid:13)(cid:13)(cid:13) ≤
Revised: j (cid:13)(cid:13)(cid:13) + α (cid:13)(cid:13)(cid:13) V ( T ) j (cid:13)(cid:13)(cid:13) ≤

Original: [Equation]
Revised: [Equation]

Original: + [Equation] max .
Revised: + αV max .

Original: Since α ∈ (cid:104) δ ∆ min , √ δ V max (cid:17) , we have
Revised: Since α ∈ (cid:104) δ ∆ min , √ δ V max (cid:17) , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the second inequality assumes δ ≤ 1 /r 2 and the last inequality assumes δ < (cid:0) 1 2 e (cid:1) 2 r − 2 .
Revised: where the second inequality assumes δ ≤ 1 /r 2 and the last inequality assumes δ < (cid:0) 1 2 e (cid:1) 2 r − 2 .

Original: For α ∈ (cid:104) √ δ V max , α 2 (cid:105) .
Revised: For α ∈ (cid:104) √ δ V max , α 2 (cid:105) .

Original: Similar as above, we only need to show that α ∆ min − 2 ( δ + αV max ) r > 0 .
Revised: Similar as above, we only need to show that α ∆ min − 2 ( δ + αV max ) r > 0 .

Original: Since α ≥ √ δ V max , we have δ ≤ √ δαV max .
Revised: Since α ≥ √ δ V max , we have δ ≤ √ δαV max .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds as long as α ≤ α
Revised: where the second inequality holds as long as α ≤ α

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: (cid:17)

Revised: rr

Original: [Equation]
Revised: [Equation]

Revised: (cid:16) ∆ min

Revised: [Equation]

Revised: r max (cid:17) 1 r − 1 .

Original: □
Revised: □

Original: Next, we show that for α ∈ [0 , ϵ 1 /r V max ] , the loss cannot decrease by much.
Revised: Next, we show that for α ∈ [0 , ϵ 1 /r V max ] , the loss cannot decrease by much.

Original: Similar as in Lemma 6, we prove that the signal is very small and the logit is dominated by the bias term.
Revised: Similar as in Lemma 6, we prove that the signal is very small and the logit is dominated by the bias term.

Original: This then gives a lower and upper bounds for the loss.
Revised: This then gives a lower and upper bounds for the loss.

Original: Lemma 9.
Revised: Lemma 9.

Original: In the setting of Theorem 1, there exists α 3 = ϵ 1 /r
Revised: In the setting of Theorem 1, there exists α 3 = ϵ 1 /r

Original: V max such that for all α ∈ [0 , α 3 ]
Revised: V max such that for all α ∈ [0 , α 3 ]

Original: [Equation]
Revised: [Equation]

Original: where N is the number of samples.
Revised: where N is the number of samples.

Original: Proof of Lemma 9.
Revised: Proof of Lemma 9.

Original: Since α ≤ α 3 = ϵ 1 /r V max and δ ≤ ϵ 1 /r r , we have
Revised: Since α ≤ α 3 = ϵ 1 /r V max and δ ≤ ϵ 1 /r r , we have

Original: [Equation]
Revised: [Equation]

Original: for all input x.
Revised: for all input x.

Original: Similar as in the proof of Lemma 6, we can show that
Revised: Similar as in the proof of Lemma 6, we can show that

Original: [Equation]
Revised: [Equation]

Original: ∆ max + 2 eϵ, where we have an additional factor of 2 before eϵ because now the signal can be positive or negative.
Revised: max + 2 eϵ, where we have an additional factor of 2 before eϵ because now the signal can be positive or negative.

Original: Here N is the number of samples.
Revised: Here N is the number of samples.

Original: □
Revised: □

Original: C P ROOF OF TRAINING DYNAMICS
Revised: C P ROOF OF TRAINING DYNAMICS

Original: In this section, we give the complete proof of Theorem 2.
Revised: In this section, we give the complete proof of Theorem 2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose the neural network, dataset and optimization procedure are as defined in
Revised: Suppose the neural network, dataset and optimization procedure are as defined in

Original: Section 4.
Revised: Section 4.

Original: Suppose initialization scale δ ≤ Θ(1) , noise level σ ≤ (cid:101) Θ(1) , dimension d ≥ (cid:101) Θ(1 /δ 2 r − 2 ) and number of samples N ≥ (cid:101) Θ(

Revised: Suppose initialization scale δ ≤ Θ(1) , noise level σ ≤ (cid:101) Θ(1) , dimension d ≥ (cid:101) Θ(1 /δ 2 r − 2 ) and number of samples N ≥ (cid:101) Θ(

Original: /δ
Revised: /δ

Original: r − 1 ) , with probability at least 0 .
Revised: r − 1 ) , with probability at least 0 .

Original: 99 in the initialization, there exists time T =
Revised: 99 in the initialization, there exists time T =

Original: Θ(log(1 /δ )
Revised: Θ(log(1 /δ )

Original: /δ
Revised: /δ

Original: r − 2 ) such that we have
Revised: r − 2 ) such that we have

Original: zero error: for all different i, j ∈ [ k ] and for all x ∈ S i , f ( T ) i
Revised: zero error: for all different i, j ∈ [ k ] and for all x ∈ S i , f ( T ) i

Original: [Equation] Ω
Revised: ( x ) ≥ f ( T ) j

Original: [Equation] ); 2. bias gap: b ( T ) i ∗ − max i ̸
Revised: ( x ) + Ω(1); 2. bias gap: b ( T ) i ∗ − max i ̸

Original: = i ∗ b ( T ) i ≥ Ω(1) with i ∗ = arg max i ∈
Revised: = i ∗ b ( T ) i ≥ Ω(1) with i ∗ = arg max i ∈

Original: [ k ] b ( T ) i .
Revised: [ k ] b ( T ) i .

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: This theorem directly follows from Proposition 1.
Revised: This theorem directly follows from Proposition 1.

Original: □
Revised: □

Original: We consider the r -homogeneous-weight network as defined in Section 4.
Revised: We consider the r -homogeneous-weight network as defined in Section 4.

Original: Our simple model simulates a depth- r ReLU/linear network with bias on the output layer, in the sense that the weights signal is r -homogeneous while the bias is 1 -homogeneous in the parameters.
Revised: Our simple model simulates a depth- r ReLU/linear network with bias on the output layer, in the sense that the weights signal is r -homogeneous while the bias is 1 -homogeneous in the parameters.

Original: Next, we prove Proposition 1 while leaving the proof of supporting lemmas into Section C.1.
Revised: Next, we prove Proposition 1 while leaving the proof of supporting lemmas into Section C.1.

Original: Through the proof of Proposition 1, we restate the lemmas when we use it for the convenience of readers.
Revised: Through the proof of Proposition 1, we restate the lemmas when we use it for the convenience of readers.

Original: Proposition 1 (Induction Hypothesis) .
Revised: Proposition 1 (Induction Hypothesis) .

Original: In the same setting of Theorem 2, with probability at least 0.
Revised: In the same setting of Theorem 2, with probability at least 0.

Original: in initialization, there exist time points 

Revised: in initialization, there exist time points

Original: =: s 1 < t 1 < s 2 < t 2 < · · · < s k − 1 < t k − 1 < s k :
Revised: =: s 1 < t 1 < s 2 < t 2 < · · · < s k − 1 < t k − 1 < s k :

Original: = T with t i − s i
Revised: = T with t i − s i

Original: = Θ(log(1 /δ )
Revised: = Θ(log(1 /δ )

Original: /δ
Revised: /δ

Original: r − 2 ) and s i +1 − t i = Θ(1) for i ∈ [ k −
Revised: r − 2 ) and s i +1 − t i = Θ(1) for i ∈ [ k −

Original: 1] such that for any t ∈  s i 
Revised: 1] such that for any t ∈ [ s i , s i +1 ] , 1.

Original: , s i +1 ] , 1. (classes not yet learned) for any class j, j ′ ≥
Revised: (classes not yet learned) for any class j, j ′ ≥

Original: i +1 , we have (1) b ( t ) j ≥ max i ′ ∈
Revised: i +1 , we have (1) b ( t ) j ≥ max i ′ ∈

Original: [ k ] b ( t ) i ′ − O
Revised: [ k ] b ( t ) i ′ − O

Original: ( δ r ) , (2) (cid:12)(cid:12)(cid:12) b ( t ) j − b ( t ) j ′ (cid:12)(cid:12)(cid:12) ≤ O ( δ r ) and (3) W ( t ) j,j ≤ O ( δ ) ;
Revised: ( δ r ) , (2) (cid:12)(cid:12)(cid:12) b ( t ) j − b ( t ) j ′ (cid:12)(cid:12)(cid:12) ≤ O ( δ r ) and (3) W ( t ) j,j ≤ O ( δ ) ;

Original: (classes already learned) for any class j ≤ i − 1 , we have (1) b ( t ) j ≤ max i ′ ∈
Revised: (classes already learned) for any class j ≤ i − 1 , we have (1) b ( t ) j ≤ max i ′ ∈

Original: [ k ] b ( t ) i ′ − Ω(1) , (2)
Revised: [ k ] b ( t ) i ′ − Ω(1) , (2)

Original: f
Revised: f

Original: ( t ) j ( x ) ≥ f ( t ) i ′ ( x ) + Ω(1) for i ′
Revised: ( t ) j ( x ) ≥ f ( t ) i ′ ( x ) + Ω(1) for i ′

Original: ̸
Revised: ̸

Original: = j, x ∈ S j and (3) W ( t ) j,j ≥ Ω(1) ; 3.
Revised: = j, x ∈ S j and (3) W ( t ) j,j ≥ Ω(1) ; 3.

Original: (parameters movement) (1) for any j ∈ [ k ] , Θ( δ ) =
Revised: (parameters movement) (1) for any j ∈ [ k ] , Θ( δ ) =

Original: W (0) j,j < W ( t ) j,j , (2) for any distinct j, j ′ ∈
Revised: W (0) j,j < W ( t ) j,j , (2) for any distinct j, j ′ ∈

Original: [ k ] , 0 < W ( t ) j,j ′ ≤ O ( δ ) and (3) for any j, j ′ ∈
Revised: [ k ] , 0 < W ( t ) j,j ′ ≤ O ( δ ) and (3) for any j, j ′ ∈

Original: [ k ] and any x ∈ S j ′ , (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ min (cid:16) O ( δ ) , W ( t ) j,j ′ (cid:17) .
Revised: [ k ] and any x ∈ S j ′ , (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ min (cid:16) O ( δ ) , W ( t ) j,j ′ (cid:17) .

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: Through the proof, we assume the conditions in Theorem 2 hold in all the lemmas without explicitly stated.
Revised: Through the proof, we assume the conditions in Theorem 2 hold in all the lemmas without explicitly stated.

Original: At the initialization, we have the following properties with probability at least 0 . 99 .
Revised: At the initialization, we have the following properties with probability at least 0 . 99 .

Original: Lemma 10 (Initialization) .
Revised: Lemma 10 (Initialization) .

Original: With probability at least 0 .
Revised: With probability at least 0 .

Original: 99 in the initialization, we have
Revised: 99 in the initialization, we have

Original: for all j, j ′ ∈ [ k ] , W (0) j,j ′ =
Revised: for all j, j ′ ∈ [ k ] , W (0) j,j ′ =

Original: Θ( δ ) ; 2.
Revised: Θ( δ ) ; 2.

Original: for all distinct j, j ′ ∈ [ k ] , (cid:12) (cid:12)(cid:12) W (0) j,j − W (0) j ′ ,j ′ (cid:12) (cid:12)(cid:12)
Revised: for all distinct j, j ′ ∈ [ k ] , (cid:12) (cid:12)(cid:12) W (0) j,j − W (0) j ′ ,j ′ (cid:12) (cid:12)(cid:12)

Original: =
Revised: =

Original: Θ( δ ); 3.
Revised: Θ( δ ); 3.

Original: for all x ∈ S , ∥ ξ x ∥ ≤ O ( σ ); 4. for all distinct x, x ′ ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , ¯ ξ x ′ (cid:11)(cid:12)(cid:12) ≤ 
Revised: for all x ∈ S , ∥ ξ x ∥ ≤ O ( σ ); 4. for all distinct x, x ′ ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , ¯ ξ x ′ (cid:11)(cid:12)(cid:12) ≤ O

Original: O (cid:18) √ 
Revised: (cid:18) √ log(

Original: [Equation]

Original: N )
Revised: N )

Original: [Equation]
Revised: [Equation]

Original:  for all j ∈ [ k ] and all x ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , e j (cid:11)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) ¯ ξ x , ¯ 
Revised: d (cid:19) for all j ∈ [ k ] and all x ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , e j (cid:11)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) ¯ ξ x , ¯ W (0) j, :

Original: [Equation] (0) j, : (cid:69)(cid:12)(cid:12)(cid:12) 
Revised: (cid:69)(cid:12)(cid:12)(cid:12) ≤ O (cid:18) √ log( N )

Original: [Equation]
Revised: [Equation]

Original: √ [Equation]
Revised: d (cid:19) .

Original: Without loss of generality, we assume W (0)1 , 1 > W (0)2 , 2 > · · · > W (0) k,k .
Revised: Without loss of generality, we assume W (0)1 , 1 > W (0)2 , 2 > · · · > W (0) k,k .

Original: It’s not hard to verify that the induction hypothesis holds at the initialization 4 .
Revised: It’s not hard to verify that the induction hypothesis holds at the initialization 4 .

Original: For any i ∈ [ k − 1] , assuming the induction hypothesis holds for time [0 , s i ] , now we prove that it continues to hold in [ s i , s i +1 ] .
Revised: For any i ∈ [ k − 1] , assuming the induction hypothesis holds for time [0 , s i ] , now we prove that it continues to hold in [ s i , s i +1 ] .

Original: Next, we first prove the first two properties in the Proposition 1 and leave the last one at the end.
Revised: Next, we first prove the first two properties in the Proposition 1 and leave the last one at the end.

Original: The learning of x ∈ S i can be divided into four stages:
Revised: The learning of x ∈ S i can be divided into four stages:

Original: Stage 0 for t ∈ [ s i , t i ] with t i − s i
Revised: Stage 0 for t ∈ [ s i , t i ] with t i − s i

Original: =
Revised: =

Original: O (log(1 /δ )
Revised: O (log(1 /δ )

Original: /δ
Revised: /δ

Original: r − 2 ) .
Revised: r − 2 ) .

Original: During this stage, W ( t ) i,i grows to a small constant µ 0 .
Revised: During this stage, W ( t ) i,i grows to a small constant µ 0 .

Original: Stage 1 for t ∈ [ t i , t ( w ) i ] with t ( w ) i − t i
Revised: Stage 1 for t ∈ [ t i , t ( w ) i ] with t ( w ) i − t i

Original: = O (1) .
Revised: = O (1) .

Original: In this stage, W ( t ) i,i grows from µ 0 to a large constant µ 1 . 3.
Revised: In this stage, W ( t ) i,i grows from µ 0 to a large constant µ 1 . 3.

Original: Stage 2 for t ∈ [ t ( w ) i , t ( u ) i ] with t ( u ) i − t ( w ) i
Revised: Stage 2 for t ∈ [ t ( w ) i , t ( u ) i ] with t ( u ) i − t ( w ) i

Original: = O (1) .
Revised: = O (1) .

Original: At the end of this stage, we have min x ∈S
Revised: At the end of this stage, we have min x ∈S

Original: i u ( t ) i ( x ) ≥ 1 − µ 2 for a small constant µ 2 . 4.
Revised: i u ( t ) i ( x ) ≥ 1 − µ 2 for a small constant µ 2 . 4.

Original: Stage 3 for t ∈ [ t ( u ) i , t ( b ) i ] with t ( b ) i − t ( u ) i
Revised: Stage 3 for t ∈ [ t ( u ) i , t ( b ) i ] with t ( b ) i − t ( u ) i

Original: = O (1) , where t ( b ) i
Revised: = O (1) , where t ( b ) i

Original: = s i +1 .
Revised: = s i +1 .

Original: During this stage, we have b ( t ) i − b ( t ) k decreases to − µ 3 with µ 3 a positive constant.
Revised: During this stage, we have b ( t ) i − b ( t ) k decreases to − µ 3 with µ 3 a positive constant.

Original: Next, we consider these four stages one by one.
Revised: Next, we consider these four stages one by one.

Original: Stage 0.
Revised: Stage 0.

Original: We show that W ( t ) i,i increases faster than W ( t ) i +1 ,i +1 so that W ( t ) i,i reaches a constant while
Revised: We show that W ( t ) i,i increases faster than W ( t ) i +1 ,i +1 so that W ( t ) i,i reaches a constant while

Original: W ( t ) i +1 ,i +1 is still O ( δ ) .
Revised: W ( t ) i +1 ,i +1 is still O ( δ ) .

Original: We use the following lemma to characterize the increasing rate of W ( t ) i +1 ,i + and W ( t ) i,i .
Revised: We use the following lemma to characterize the increasing rate of W ( t ) i +1 ,i + and W ( t ) i,i .

Original: Lemma 11.
Revised: Lemma 11.

Original: For any j ∈ [ k ] , we have
Revised: For any j ∈ [ k ] , we have

Original: [Equation]
Revised: [Equation]

Original: If min x ∈S j (1 − u j ( x )) ≥ Ω(1) , we further have
Revised: If min x ∈S j (1 − u j ( x )) ≥ Ω(1) , we further have

Original: [Equation]
Revised: [Equation]

Original:  x 
Revised: It’s not hard to verify that min x ∈S i (cid:16) 1 − u ( t ) i ( x ) (cid:17)

Original: [Equation]
Revised: [Equation]

Original: It’s not hard to verify that min x ∈S i (cid:16) 1 − u ( t ) i ( x ) (cid:17) , min x ∈S i +1 (cid:16) 1 − u ( t ) i +1 ( x ) (cid:17) ≥ Ω(1) , so we have 
Revised: min x ∈S i +1 (cid:16) 1 − u ( t ) i +1 ( x ) (cid:17) ≥ Ω(1) , so we have [Equation] t

Original: [Equation]
Revised: [Equation]

Original: i,
Revised: i

Original: [Equation]
Revised: [Equation]

Original: We can upper bound 1 − u ( t ) i +1 ( x ) for any x ∈ S
Revised: We can upper bound 1 − u ( t ) i +1 ( x ) for any x ∈ S

Original: i +1 as follows,
Revised: i +1 as follows,

Original: [Equation]
Revised: [Equation]

Original: where the inequality uses (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) i ′ [Equation] : , x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( δ ) for every i ′ ∈ [ k ] .
Revised: where the inequality uses (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) i ′ , : , x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( δ ) for every i ′ ∈ [ k ] .

Original: We can lower bound 1 − u ( t ) i ( x ) for x ∈ S i as follows,
Revised: We can lower bound 1 − u ( t ) i ( x ) for x ∈ S i as follows,

Original: [Equation]
Revised: [Equation]

Original: − [Equation] The first inequality uses (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) i ′ , : , x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( µ 0 ) for every i ′ ∈ [ k ] .
Revised: The first inequality uses (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) i ′ , : , x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( µ 0 ) for every i ′ ∈ [ k ] .

Original: The second inequality uses b ( t ) i ≤ b ( t ) i +

Revised: The second inequality uses b ( t ) i ≤ b ( t ) i +

Original: + O ( δ r ) , which is guaranteed by the following lemma.
Revised: + O ( δ r ) , which is guaranteed by the following lemma.

Original: Lemma 2 (Bias Gap Control I) .
Revised: Lemma 2 (Bias Gap Control I) .

Original: For any different j ′ , j ∈ [ k ] , if W j ′ ,j ′ ≥ W j,j , W j,j ≤ O ( δ ) and b j ′ − b j ≥ O
Revised: For any different j ′ , j ∈ [ k ] , if W j ′ ,j ′ ≥ W j,j , W j,j ≤ O ( δ ) and b j ′ − b j ≥ O

Original: ( δ r ) , b j ≥ max i ′ ∈
Revised: ( δ r ) , b j ≥ max i ′ ∈

Original: [ k ] b i ′ − O
Revised: [ k ] b i ′ − O

Original: ( δ r ) , we have ˙
Revised: ( δ r ) , we have ˙

Original: b j ′ − ˙
Revised: b j ′ − ˙

Original: b j < 0 .
Revised: b j < 0 .

Original: According to Lemma 10, we know there exists constant C > 1 such that W (0)
Revised: According to Lemma 10, we know there exists constant C > 1 such that W (0)

Original: i,i ≥ CW (0) i +1 ,i +1 .
Revised: i,i ≥ CW (0) i +1 ,i +1 .

Original: Choose constant S such that S 1 r − 2 = √ C and W (0) i,
Revised: Choose constant S such that S 1 r − 2 = √ C and W (0) i,

Original: i ≥ S 1 r − 2 √ CW (0)
Revised: i ≥ S 1 r − 2 √ CW (0)

Original: i +1 ,i +1 .
Revised: i +1 ,i +1 .

Original: Choosing µ 0 as small constants and σ ≤ O (1 / √ log N ) , δ ≤ O (1) , we have
Revised: Choosing µ 0 as small constants and σ ≤ O (1 / √ log N ) , δ ≤ O (1) , we have

Original: [Equation]
Revised: [Equation]

Original: We can also lower bound 1 − u ( t ) i ( x ) for x ∈ S i by a constant,
Revised: We can also lower bound 1 − u ( t ) i ( x ) for x ∈ S i by a constant,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds because µ 0 is a small constant and b ( t ) i +1 ≥ max
Revised: where the last inequality holds because µ 0 is a small constant and b ( t ) i +1 ≥ max

Original: i ′ ∈
Revised: i ′ ∈

Original: [ k ] b ( t ) i ′ − O
Revised: [ k ] b ( t ) i ′ − O

Original: ( δ r ) .
Revised: ( δ r ) .

Original: Lemma 12 (Adapted from Lemma C.19 in Allen-Zhu & Li (2020)) .
Revised: Lemma 12 (Adapted from Lemma C.19 in Allen-Zhu & Li (2020)) .

Original: Let r ≥ 3 be a constant and let { W ( t )
Revised: Let r ≥ 3 be a constant and let { W ( t )

Original: i,i , W ( t ) j,j } t ≥ 0 be two positive sequences updated as
Revised: i,i , W ( t ) j,j } t ≥ 0 be two positive sequences updated as

Original: [Equation]
Revised: [Equation]

Original: Suppose W (0) i,i ≥ W (0) j,j S 1 r − 2 (1 + Ω(1)) , then we must have for every A
Revised: Suppose W (0) i,i ≥ W (0) j,j S 1 r − 2 (1 + Ω(1)) , then we must have for every A

Original: = O (1) , let t i be the first time such that W ( t i )
Revised: = O (1) , let t i be the first time such that W ( t i )

Original: i,i ≥ A, then
Revised: i,i ≥ A, then

Original: [Equation]
Revised: [Equation]

Original: Then, according to Lemma 12, we know that there exists t i =
Revised: Then, according to Lemma 12, we know that there exists t i =

Original: O (log(1 /δ )
Revised: O (log(1 /δ )

Original: /δ
Revised: /δ

Original: r − 2 ) such that W ( t i ) i,i = µ 0 and W ( t i )
Revised: r − 2 ) such that W ( t i ) i,i = µ 0 and W ( t i )

Original: i +1 ,i +1 ≤ O ( δ ) .
Revised: i +1 ,i +1 ≤ O ( δ ) .

Original: By similar argument, we also know W ( t i ) j,j ≤ O ( δ ) for any j ≥ i + 1 .
Revised: By similar argument, we also know W ( t i ) j,j ≤ O ( δ ) for any j ≥ i + 1 .

Original: Stage 1.
Revised: Stage 1.

Original: In this stage, we show that W ( t ) i,i grows to a large constant µ 1 within constant time.
Revised: In this stage, we show that W ( t ) i,i grows to a large constant µ 1 within constant time.

Original: Since
Revised: Since

Original: W ( t ) i,i ≤ µ 

Revised: W ( t ) i,i ≤ µ

Original: and b ( t ) i,i ≤ b ( t ) i +1 ,i +

Revised: and b ( t ) i,i ≤ b ( t ) i +1 ,i +

Original: + O
Revised: + O

Original: ( δ r ) , we have
Revised: ( δ r ) , we have

Original: [Equation]

Original: t

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ) , for all x ∈ S i .
Revised: for all x ∈ S i .

Original: This further implies,
Revised: This further implies,

Original: [Equation]
Revised: [Equation]

Original: where the inequality also uses W ( t ) i,i ≥ µ 0 .
Revised: where the inequality also uses W ( t ) i,i ≥ µ 0 .

Original: Since the increasing rate is at least a constant, we know
Revised: Since the increasing rate is at least a constant, we know

Original: W ( t ) i,i grows to µ 1 in constant time.
Revised: W ( t ) i,i grows to µ 1 in constant time.

Original: For any j ≥ i + 1 , since the increasing rate of W ( t ) j,j is merely
Revised: For any j ≥ i + 1 , since the increasing rate of W ( t ) j,j is merely

Original: O ( δ r − 1 ) , we know W ( t ) j,j remains as O ( δ ) through Stage 1.
Revised: O ( δ r − 1 ) , we know W ( t ) j,j remains as O ( δ ) through Stage 1.

Original: Stage 2.
Revised: Stage 2.

Original: In this stage, we prove that u ( t ) i ( x ) for any x ∈ S i grows to 1 − µ 2 with µ 2 a small constant.
Revised: In this stage, we prove that u ( t ) i ( x ) for any x ∈ S i grows to 1 − µ 2 with µ 2 a small constant.

Original: We use the following lemma to characterize the increasing rate of f ( t ) i ( x ) − f ( t ) j ( x ) .
Revised: We use the following lemma to characterize the increasing rate of f ( t ) i ( x ) − f ( t ) j ( x ) .

Original: Lemma 13.
Revised: Lemma 13.

Original: For any x ∈ S i and any j ̸
Revised: For any x ∈ S i and any j ̸

Original: =
Revised: =

Original: i, if 1 − u i ( x ) ≥ Ω(1) , we have
Revised: i, if 1 − u i ( x ) ≥ Ω(1) , we have

Original: [Equation]
Revised: [Equation]

Original: Since u ( t ) i ( x ) ≤ 1 − µ 2 , we know 1 − u ( t ) i ( x ) ≥ Ω(1) .
Revised: Since u ( t ) i ( x ) ≤ 1 − µ 2 , we know 1 − u ( t ) i ( x ) ≥ Ω(1) .

Original: For any j ̸ = i, we have
Revised: For any j ̸ = i, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds because W ( t ) i,i ≥ µ 1 with µ 1 a large enough constant.
Revised: where the last inequality holds because W ( t ) i,i ≥ µ 1 with µ 1 a large enough constant.

Original: The next lemma guarantees that at the beginning of Stage 2, we have b ( t ) i − b ( t ) j ≥ − O (1) , which then implies f ( t ) i ( x ) − f ( t ) j ( x ) ≥ − O (1) .
Revised: The next lemma guarantees that at the beginning of Stage 2, we have b ( t ) i − b ( t ) j ≥ − O (1) , which then implies f ( t ) i ( x ) − f ( t ) j ( x ) ≥ − O (1) .

Original: Lemma 14 (Bias Gap Control III) .
Revised: Lemma 14 (Bias Gap Control III) .

Original: For any different i, j ∈ [ k ] , if W i,i ≤ O (1) , W j,j ≤ O ( δ ) and b i − b j ≤ − O (1) , we have
Revised: For any different i, j ∈ [ k ] , if W i,i ≤ O (1) , W j,j ≤ O ( δ ) and b i − b j ≤ − O (1) , we have

Original: [Equation]
Revised: [Equation]

Original: Let C be a constant such that f ( t ) i ( x ) − f ( t )
Revised: Let C be a constant such that f ( t ) i ( x ) − f ( t )

Original: j ( x ) ≥ C for every j ̸
Revised: j ( x ) ≥ C for every j ̸

Original: = i implies u i ( x ) ≥ 1 − µ 2 .
Revised: = i implies u i ( x ) ≥ 1 − µ 2 .

Original: Since at the at the beginning of Stage 2, we have f ( t ) i ( x ) − f ( t ) j ( x ) ≥ − O (1) , within constant time, we have f ( t ) i ( x ) − f ( t ) j ( x ) ≥ C for every j ̸
Revised: Since at the at the beginning of Stage 2, we have f ( t ) i ( x ) − f ( t ) j ( x ) ≥ − O (1) , within constant time, we have f ( t ) i ( x ) − f ( t ) j ( x ) ≥ C for every j ̸

Original: = i and u i ( x ) ≥ 1 − µ 2 .
Revised: = i and u i ( x ) ≥ 1 − µ 2 .

Original: Lemma 15 (Accuracy Monotonicity) .
Revised: Lemma 15 (Accuracy Monotonicity) .

Original: Given any positive constant C 2 , there exists
Revised: Given any positive constant C 2 , there exists

Original: positive constant
Revised: positive constant

Original: C 1 such that for all different i, j ∈ [ k ] , as long as W i,i ≥ C 1 and f i ( x ) − f j ( x ) ≤ C 2 for any x ∈ S i , we have ∂ ( f i ( x ) − f j ( x )) ∂t > 0 .
Revised: C 1 such that for all different i, j ∈ [ k ] , as long as W i,i ≥ C 1 and f i ( x ) − f j ( x ) ≤ C 2 for any x ∈ S i , we have ∂ ( f i ( x ) − f j ( x )) ∂t > 0 .

Original: According to Lemma 15, by choosing large enough µ 1 , we can ensure that f ( t ) i ( e i ) − f ( t ) j ( e i ) ≥ C and u i ( e i ) ≥ 1 − µ 2 throughout the training.
Revised: According to Lemma 15, by choosing large enough µ 1 , we can ensure that f ( t ) i ( e i ) − f ( t ) j ( e i ) ≥ C and u i ( e i ) ≥ 1 − µ 2 throughout the training.

Original: Note that once W ( t ) i,i rises to µ 1 , it will stay at least µ 1 − O
Revised: Note that once W ( t ) i,i rises to µ 1 , it will stay at least µ 1 − O

Original: ( δ ) throughout the training, according to the gradient lower bound in Lemma 11.
Revised: ( δ ) throughout the training, according to the gradient lower bound in Lemma 11.

Original: Stage 3.
Revised: Stage 3.

Original: In this stage, we prove that within constant time we have b ( t ) i − b ( t ) k ≤ − µ 3 .
Revised: In this stage, we prove that within constant time we have b ( t ) i − b ( t ) k ≤ − µ 3 .

Original: The following lemma shows that b ( t ) i − b ( t ) k decreases in at least a constant rate.
Revised: The following lemma shows that b ( t ) i − b ( t ) k decreases in at least a constant rate.

Original: Lemma 3 (Bias Gap Control II) .
Revised: Lemma 3 (Bias Gap Control II) .

Original: There exist small positive constants C 1 , C 2 such that for any j ∈ [ k − 1] and any x ∈ S j , if 1 − u j ( x ) ≤ C 1 , W k,k ≤ O ( δ ) and b j − b k ≥ − C 2 , we have ˙
Revised: There exist small positive constants C 1 , C 2 such that for any j ∈ [ k − 1] and any x ∈ S j , if 1 − u j ( x ) ≤ C 1 , W k,k ≤ O ( δ ) and b j − b k ≥ − C 2 , we have ˙

Original: b j − ˙
Revised: b j − ˙

Original: b k < − Ω(1) .
Revised: b k < − Ω(1) .

Original: Choosing µ 2 = C 1 , µ 3 = C 2 where C 1 , C 2 are from Lemma 3, so we know that b ( t ) i − b ( t ) k decreases at a constant rate until b ( t ) i − b ( t ) k ≤ − µ 3 .
Revised: Choosing µ 2 = C 1 , µ 3 = C 2 where C 1 , C 2 are from Lemma 3, so we know that b ( t ) i − b ( t ) k decreases at a constant rate until b ( t ) i − b ( t ) k ≤ − µ 3 .

Original: At time t ( u ) i , we know that b t ( u ) i i − b t ( u ) i k ≤ O ( δ r ) .
Revised: At time t ( u ) i , we know that b t ( u ) i i − b t ( u ) i k ≤ O ( δ r ) .

Original: So within constant time, we have b
Revised: So within constant time, we have b

Original: s
Revised: s

Original: i
Revised: i

Original: +

Revised: +

Original: i −
Revised: i −

Original: b
Revised: b

Original: s i +1 k
Revised: s i +1 k

Original: = − µ 3 .
Revised: = − µ 3 .

Original: By Lemma 3, we also know that for any t ≥ s i +1 , we have b ( t ) i − b ( t ) k ≤ − µ 3 .
Revised: By Lemma 3, we also know that for any t ≥ s i +1 , we have b ( t ) i − b ( t ) k ≤ − µ 3 .

Original: The following lemma shows that b ( t ) k is close to the maximum bias.
Revised: The following lemma shows that b ( t ) k is close to the maximum bias.

Original: Lemma 1 (Coupling Biases) .
Revised: Lemma 1 (Coupling Biases) .

Original: Assuming W j ′ ,j ′ , W j,j ≤ O ( δ ) and b j ′ , b j ≥ max i ′ ∈
Revised: Assuming W j ′ ,j ′ , W j,j ≤ O ( δ ) and b j ′ , b j ≥ max i ′ ∈

Original: [ k ] b i ′ − O
Revised: [ k ] b i ′ − O

Original: ( δ r ) , we have ˙
Revised: ( δ r ) , we have ˙

Original: b j ′ − ˙
Revised: b j ′ − ˙

Original: b j > 0 if b j ′ − b j ≤ − µδ r , and ˙ b j ′ − ˙
Revised: b j > 0 if b j ′ − b j ≤ − µδ r , and ˙ b j ′ − ˙

Original: b
Revised: b

Original: j < 

Revised: j <

Original: if b j ′ − b j ≥ + µδ r for some positive constant µ.
Revised: if b j ′ − b j ≥ + µδ r for some positive constant µ.

Original: Combining Lemma 1 and Lemma 2, we know that throughout the training b ( t ) k ≥ max i ′ ∈
Revised: Combining Lemma 1 and Lemma 2, we know that throughout the training b ( t ) k ≥ max i ′ ∈

Original: [ k ] b ( t ) i ′ −
Revised: [ k ] b ( t ) i ′ −

Original: O ( δ r ) .
Revised: O ( δ r ) .

Original: Therefore, we have b ( t ) i − max i ′ ∈
Revised: Therefore, we have b ( t ) i − max i ′ ∈

Original: [ k ] b ( t ) i ′ ≤ − Ω(1) for t ≥ s i +1 .
Revised: [ k ] b ( t ) i ′ ≤ − Ω(1) for t ≥ s i +1 .

Original: Finally, let’s bound the movement of different parameters.
Revised: Finally, let’s bound the movement of different parameters.

Original: Monotonicity of diagonal terms: For j ∈ [ k − 1] , according to Lemma 11 we know W ( t ) j,j can only start decreasing when it exceeds a large constant and can only decrease by at most O ( δ ) through the algorithm.
Revised: Monotonicity of diagonal terms: For j ∈ [ k − 1] , according to Lemma 11 we know W ( t ) j,j can only start decreasing when it exceeds a large constant and can only decrease by at most O ( δ ) through the algorithm.

Original: By choosing δ ≤ O (1) , we can ensure that W (0) j,j < W ( t ) j,j for any t. For W ( t ) k,k , we know it monotonically increases since we always have 1 − u ( t )
Revised: By choosing δ ≤ O (1) , we can ensure that W (0) j,j < W ( t ) j,j for any t. For W ( t ) k,k , we know it monotonically increases since we always have 1 − u ( t )

Original: k ( x ) ≥ Ω(1) for x ∈ S k .
Revised: k ( x ) ≥ Ω(1) for x ∈ S k .

Original: This is because W ( t ) k,k remains as small as O ( δ ) through the algorithm and b ( t )
Revised: This is because W ( t ) k,k remains as small as O ( δ ) through the algorithm and b ( t )

Original: k − b ( t ) k − 1 ≤ O (1) .
Revised: k − b ( t ) k − 1 ≤ O (1) .

Original: Bounding non-diagonal terms: We use the following lemma to prove that (cid:101) Ω( δ ) <
Revised: Bounding non-diagonal terms: We use the following lemma to prove that (cid:101) Ω( δ ) <

Original: W ( t ) j,j ′ ≤ O ( δ ) for j ̸ = j ′ .
Revised: W ( t ) j,j ′ ≤ O ( δ ) for j ̸ = j ′ .

Original: Lemma 16.
Revised: Lemma 16.

Original: For any j ̸ = j ′ , we have T ˙ W j,j ′ ≤ O ( δ ) .
Revised: For any j ̸ = j ′ , we have T ˙ W j,j ′ ≤ O ( δ ) .

Original: Furthermore, there exists absolute constant µ > 

Revised: Furthermore, there exists absolute constant µ >

Original: such that if 0 < W j,j ′<
Revised: such that if 0 < W j,j ′

Original: [Equation]
Revised: [Equation]

Original: (1 /δ ) , we have T ˙
Revised: ) , we have T ˙

Original: [Equation]
Revised: [Equation]

Original: j,j ′ ≥ [Equation] The first property in Lemma 16 guarantees that the increasing rate is so small that that the total increase within T time is only O ( δ ) , which then implies that W ( t ) j,j ′ ≤ O ( δ ) through the training.
Revised: The first property in Lemma 16 guarantees that the increasing rate is so small that that the total increase within T time is only O ( δ ) , which then implies that W ( t ) j,j ′ ≤ O ( δ ) through the training.

Original: The second property in Lemma 16 guarantees that once W ( t ) j,j ′ falls below µδ log 1 r − 2 (1 /δ ) , its decreasing rate is so small that that the total decrease within T time is only µδ 2log 1 r − 2 (1 /δ ) , which then implies that
Revised: The second property in Lemma 16 guarantees that once W ( t ) j,j ′ falls below µδ log 1 r − 2 (1 /δ ) , its decreasing rate is so small that that the total decrease within T time is only µδ 2log 1 r − 2 (1 /δ ) , which then implies that

Original: W ( t ) j,j ′ > (cid:101) Ω( δ ) through the training.
Revised: W ( t ) j,j ′ > (cid:101) Ω( δ ) through the training.

Original: Bounding noise correlations: The following lemma shows that the total change of (cid:68) W ( t ) j, : , ξ x (cid:69) within
Revised: Bounding noise correlations: The following lemma shows that the total change of (cid:68) W ( t ) j, : , ξ x (cid:69) within

Original: T time is only O ( √ log Nσδ ) .
Revised: T time is only O ( √ log Nσδ ) .

Original: Since at initialization, we know (cid:12)(cid:12)(cid:12)(cid:68) W (0)
Revised: Since at initialization, we know (cid:12)(cid:12)(cid:12)(cid:68) W (0)

Original: j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( √ log Nσδ )
Revised: j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( √ log Nσδ )

Original: , we conclude that (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( √ log Nσδ ) throughout the training.
Revised: , we conclude that (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( √ log Nσδ ) throughout the training.

Original: Since W ( t ) j,j ′ ≥ (cid:101) Ω( δ ) , as long as σ ≤ (cid:101) O (1) , we also have (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ W j,j ′ for x ∈ S j ′ .
Revised: Since W ( t ) j,j ′ ≥ (cid:101) Ω( δ ) , as long as σ ≤ (cid:101) O (1) , we also have (cid:12)(cid:12)(cid:12)(cid:68) W ( t ) j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ W j,j ′ for x ∈ S j ′ .

Original: Lemma 17.
Revised: Lemma 17.

Original: For every j ∈ [ k ] and every x ∈ S , we have
Revised: For every j ∈ [ k ] and every x ∈ S , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: · T ≤ O (cid:16)(cid:112) log Nσδ (cid:17)

Original: □
Revised: □

Original: C. P ROOF OF LEMMAS
Revised: C. P ROOF OF LEMMAS

Original: Lemma 10 (Initialization) .
Revised: Lemma 10 (Initialization) .

Original: With probability at least 0 .
Revised: With probability at least 0 .

Original: 99 in the initialization, we have
Revised: 99 in the initialization, we have

Original: for all j, j ′ ∈ [ k ] , W (0) j,j ′ =
Revised: for all j, j ′ ∈ [ k ] , W (0) j,j ′ =

Original: Θ( δ ) ; 2.
Revised: Θ( δ ) ; 2.

Original: for all distinct j, j ′ ∈ [ k ] , (cid:12)(cid:12)(cid:12) W (0) j,j − W (0) j ′ ,j ′ (cid:12)(cid:12)(cid:12)
Revised: for all distinct j, j ′ ∈ [ k ] , (cid:12)(cid:12)(cid:12) W (0) j,j − W (0) j ′ ,j ′ (cid:12)(cid:12)(cid:12)

Original: =
Revised: =

Original: Θ( δ ); 3.
Revised: Θ( δ ); 3.

Original: for all x ∈ S , ∥ ξ x ∥ ≤ O ( σ ); 4. for all distinct x, x ′ ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , ¯ ξ x ′ (cid:11)(cid:12)(cid:12) ≤ O (cid:18) √ log( N ) √ d (cid:19) .
Revised: for all x ∈ S , ∥ ξ x ∥ ≤ O ( σ ); 4. for all distinct x, x ′ ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , ¯ ξ x ′ (cid:11)(cid:12)(cid:12) ≤ O (cid:18) √ log( N ) √ d (cid:19) .

Original: for all j ∈ [ k ] and all x ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , e j (cid:11)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) ¯ ξ x , ¯ W (0) j, : (cid:69)(cid:12)(cid:12)(cid:12) ≤ O (cid:18) √ log( N ) √ d (cid:19) .
Revised: for all j ∈ [ k ] and all x ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , e j (cid:11)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) ¯ ξ x , ¯ W (0) j, : (cid:69)(cid:12)(cid:12)(cid:12) ≤ O (cid:18) √ log( N ) √ d (cid:19) .

Original: Without loss of generality, we assume W (0)1 , 1 > W (0)2 , 2 > · · · > W (0) k,k
Revised: Without loss of generality, we assume W (0)1 , 1 > W (0)2 , 2 > · · · > W (0) k,k

Original: Proof of Lemma 10.
Revised: Proof of Lemma 10.

Original: Recall that each W (0) j,j ′ is independently sampled from N (0 , δ 2 ) before taking the absolute value.
Revised: Recall that each W (0) j,j ′ is independently sampled from N (0 , δ 2 ) before taking the absolute value.

Original: By standard Gaussian concentration inequality, we know for any j, j ′ ∈ [ k ] , with probability at least 1 −1 1000 k 2 , W
Revised: By standard Gaussian concentration inequality, we know for any j, j ′ ∈ [ k ] , with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: ,j ′ ≤ O ( δ ) .
Revised: k 2 , [Equation]

Original: By anti-concentration inequality of Gaussian polynomials, we know for any j, j ′ ∈ [ k ] , with probability at least 1 −
Revised: By anti-concentration inequality of Gaussian polynomials, we know for any j, j ′ ∈ [ k ] , with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: Also by anti-concentration inequality of Gaussian polynomials, we know for any distinct j, j ′ ∈ [ k ] , with probability at least 1 −
Revised: Also by anti-concentration inequality of Gaussian polynomials, we know for any distinct j, j ′ ∈ [ k ] , with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: which implies (cid:12)(cid:12)(cid:12) W (0) j,j − W (0) j ′ ,j ′ (cid:12)(cid:12)(cid:12) ≥ Ω( δ ) assuming W (0) j,j , W (0) j ′ ,j ′ =
Revised: which implies (cid:12)(cid:12)(cid:12) W (0) j,j − W (0) j ′ ,j ′ (cid:12)(cid:12)(cid:12) ≥ Ω( δ ) assuming W (0) j,j , W (0) j ′ ,j ′ =

Original: Θ( δ ) .
Revised: Θ( δ ) .

Original: By the norm concentration of random vectors with independent Gaussian entries, for each x ∈ S , we have with probability at least 1 −
Revised: By the norm concentration of random vectors with independent Gaussian entries, for each x ∈ S , we have with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: as long as d ≥ O
Revised: as long as d ≥ O

Original: log N ) .
Revised: log N ) .

Original: By the concentration of standard Gaussian variable, for any distinct x, x ′ ∈ S , we have with probability at least 1 −
Revised: By the concentration of standard Gaussian variable, for any distinct x, x ′ ∈ S , we have with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: Similarly, for any x and any e j , we have with probability at least 1 −
Revised: Similarly, for any x and any e j , we have with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: for any x and any ¯ W (0) j, : , we have with probability at least 1 −
Revised: for any x and any ¯ W (0) j, : , we have with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: Taking a union bound over all these events, we know with probability at least 0 .
Revised: Taking a union bound over all these events, we know with probability at least 0 .

Original: 99 in the initialization, we have
Revised: 99 in the initialization, we have

Original: for all j, j ′ ∈ [ k ] , W (0) i,j = Θ( δ ) ; 2.
Revised: for all j, j ′ ∈ [ k ] , W (0) i,j = Θ( δ ) ; 2.

Original: for all distinct j, j ′ ∈ [ k ] , (cid:12)(cid:12)(cid:12) W (0) j,j − W (0) j ′ ,j ′ (cid:12)(cid:12)(cid:12)
Revised: for all distinct j, j ′ ∈ [ k ] , (cid:12)(cid:12)(cid:12) W (0) j,j − W (0) j ′ ,j ′ (cid:12)(cid:12)(cid:12)

Original: =
Revised: =

Original: Θ( δ ); 3.
Revised: Θ( δ ); 3.

Original: for all x ∈ S , ∥ ξ x ∥ ≤ O ( σ ); 4. for all distinct x, x ′ ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , ¯ ξ x ′ (cid:11)(cid:12)(cid:12) ≤ O (cid:18) √ log( N [Equation]
Revised: for all x ∈ S , ∥ ξ x ∥ ≤ O ( σ ); 4. for all distinct x, x ′ ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , ¯ ξ x ′ (cid:11)(cid:12)(cid:12) ≤ O (cid:18) √ log( N ) √ d (cid:19) .

Original: d (cid:19) . for all j ∈ [ k ] and all x ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , e j (cid:11)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) ¯ ξ x , ¯ W (0) j, : (cid:69)(cid:12)(cid:12)(cid:12) ≤ O (cid:18)
Revised: for all j ∈ [ k ] and all x ∈ S , (cid:12)(cid:12)(cid:10) ¯ ξ x , e j (cid:11)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:68) ¯ ξ x , ¯ W (0) j, : (cid:69)(cid:12)(cid:12)(cid:12) ≤ O (cid:18)

Original: [Equation]
Revised: [Equation]

Revised: ( N ) √

Original: [Equation]
Revised: [Equation]

Revised: (cid:19) .

Revised: □

Original: Lemma 12 (Adapted from Lemma C.19 in Allen-Zhu & Li (2020)) .
Revised: Lemma 12 (Adapted from Lemma C.19 in Allen-Zhu & Li (2020)) .

Original: Let r ≥ 3 be a constant and let { W ( t )
Revised: Let r ≥ 3 be a constant and let { W ( t )

Original: i,i , W ( t ) j,j } t ≥ 0 be two positive sequences updated as
Revised: i,i , W ( t ) j,j } t ≥ 0 be two positive sequences updated as

Original: [Equation]
Revised: [Equation]

Original: for [Equation] Suppose W (0) i,i ≥ W (0) j,j S 1 r − 2 (1 + Ω(1)) , then we must have for every A
Revised: Suppose W (0) i,i ≥ W (0) j,j S 1 r − 2 (1 + Ω(1)) , then we must have for every A

Original: = O (1) , let t i be the first time such that W ( t i )
Revised: = O (1) , let t i be the first time such that W ( t i )

Original: i,i ≥ A, then
Revised: i,i ≥ A, then

Original: W ( t i ) j,j ≤ O ( W (0) j,j ) .
Revised: W ( t i ) j,j ≤ O ( W (0) j,j ) .

Original: Proof of Lemma 12.
Revised: Proof of Lemma 12.

Original: This lemma directly follows from Lemma C.19 in Allen-Zhu & Li (2020) by taking the continuous time limit and setting k as a constant.
Revised: This lemma directly follows from Lemma C.19 in Allen-Zhu & Li (2020) by taking the continuous time limit and setting k as a constant.

Original: □
Revised: □

Original: Lemma 1 (Coupling Biases) .
Revised: Lemma 1 (Coupling Biases) .

Original: Assuming W j ′ ,j ′ , W j,j ≤ O ( δ ) and b j ′ , b j ≥ max i ′ ∈
Revised: Assuming W j ′ ,j ′ , W j,j ≤ O ( δ ) and b j ′ , b j ≥ max i ′ ∈

Original: [ k ] b i ′ − O
Revised: [ k ] b i ′ − O

Original: ( δ r ) , we have ˙
Revised: ( δ r ) , we have ˙

Original: b j ′ − ˙
Revised: b j ′ − ˙

Original: b j > 0 if b j ′ − b j ≤ − µδ r , and ˙ b j ′ − ˙
Revised: b j > 0 if b j ′ − b j ≤ − µδ r , and ˙ b j ′ − ˙

Original: b
Revised: b

Original: j < 

Revised: j <

Original: if b j ′ − b j ≥ + µδ r for some positive constant µ.
Revised: if b j ′ − b j ≥ + µδ r for some positive constant µ.

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Let’s first write down the time derivative on b j ′ ,
Revised: Let’s first write down the time derivative on b j ′ ,

Original: [Equation]
Revised: [Equation]

Original: For any x ∈ S , we can bound
Revised: For any x ∈ S , we can bound

Original: [Equation]
Revised: [Equation]

Original: as follows,
Revised: as follows,

Original: [Equation]
Revised: [Equation]

Original: where we uses |⟨ W j ′ , : , x ⟩| ≤ O ( δ ) + O ( √
Revised: where we uses |⟨ W j ′ , : , x [Equation] | [Equation]

Revised: √

Original: [Equation]
Revised: [Equation]

Original: ) ≤ O
Revised: ) ≤ O

Original: [Equation]
Revised: [Equation]

Original: δ ) assuming σ ≤ 1 / √ log N.
Revised: δ ) assuming σ ≤ 1 / √ log N.

Original: The similar bound also holds for
Revised: The similar bound also holds for

Original: [Equation]
Revised: [Equation]

Original: If b j ′ − b j ≥ µδ r , we can now upper bound ˙
Revised: If b j ′ − b j ≥ µδ r , we can now upper bound ˙

Original: b j ′ − ˙ b j as follows,
Revised: b j ′ − ˙ b j as follows,

Original: [Equation]
Revised: [Equation]

Original: When x ∈ S j ∪ S j ′ , we can lower bound exp( b j 
Revised: When x ∈ S j ∪ S j ′ , we can lower bound exp( b j )

Original: [Equation]
Revised: [Equation]

Original: [ k ] exp ( ⟨
Revised: [ k ] exp ( ⟨

Original: [Equation]
Revised: [Equation]

Original: i ′ , : [Equation] x ⟩ r + b i ′ ) as follows,
Revised: i ′ , : ,x ⟩ r + b i ′ ) as follows,

Original: [Equation]
Revised: [Equation]

Original: where the first inequality uses |⟨
Revised: where the first inequality uses |⟨

Original: W i ′ , : , x ⟩| ≤ δ and the second inequality assumes b j ≥ max i ′ ∈
Revised: W i ′ , : , x ⟩| ≤ δ and the second inequality assumes b j ≥ max i ′ ∈

Original: [ k ] b i ′ −
Revised: [ k ] b i ′ −

Original: O ( δ r ) and δ is at most some small constant.
Revised: O ( δ r ) and δ is at most some small constant.

Original: Therefore, if b j ′ − b j ≥ µδ r , we have
Revised: Therefore, if b j ′ − b j ≥ µδ r , we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality chooses µ as a large enough constant.
Revised: where the second inequality chooses µ as a large enough constant.

Original: Similarly, we can prove that if b j ′ − b j ≤ − µδ r , we have
Revised: Similarly, we can prove that if b j ′ − b j ≤ − µδ r , we have

Original: [Equation]
Revised: [Equation]

Original: □
Revised: □

Original: Lemma 2 (Bias Gap Control I) .
Revised: Lemma 2 (Bias Gap Control I) .

Original: For any different j ′ , j ∈ [ k ] , if W j ′ ,j ′ ≥ W j,j , W j,j ≤ O ( δ ) and b j ′ − b j ≥ O
Revised: For any different j ′ , j ∈ [ k ] , if W j ′ ,j ′ ≥ W j,j , W j,j ≤ O ( δ ) and b j ′ − b j ≥ O

Original: ( δ r ) , b j ≥ max i ′ ∈
Revised: ( δ r ) , b j ≥ max i ′ ∈

Original: [ k ] b i ′ − O
Revised: [ k ] b i ′ − O

Original: ( δ r ) , we have ˙
Revised: ( δ r ) , we have ˙

Original: b j ′ − ˙
Revised: b j ′ − ˙

Original: b j < 0 .
Revised: b j < 0 .

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: We can write down ˙
Revised: We can write down ˙

Original: b j ′ − ˙
Revised: b j ′ − ˙

Original: b j as follows,
Revised: b j as follows,

Original: [Equation]
Revised: [Equation]

Original: We first prove that for any x ∈ S j ′ , we have u j ( x ) − u j ′ ( x ) ≤ 0 .
Revised: We first prove that for any x ∈ S j ′ , we have u j ( x ) − u j ′ ( x ) ≤ 0 .

Original: We can upper bound f j ( x ) and lower bound f j ′ ( x ) as follows,
Revised: We can upper bound f j ( x ) and lower bound f j ′ ( x ) as follows,

Original: [Equation]
Revised: [Equation]

Original: The bound on f j ( x ) holds because ⟨ W j, : ,x ⟩ = W j
Revised: The bound on f j ( x ) holds because ⟨ W j, : ,

Original: [Equation]
Revised: [Equation]

Original: O
Revised: = W

Original: [Equation]
Revised: [Equation]

Original: The bound on f j ′ ( x ) holds because ⟨ W j
Revised: The bound on f j ′ ( x ) holds because ⟨ W j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]

Original: >

Original: [Equation]
Revised: [Equation]

Original: With the above two bounds, we know that u j ( x ) − u j ′ ( x ) ≤ 0 as long as b j ′ − b j ≥ O ( δ r ) .
Revised: With the above two bounds, we know that u j ( x ) − u j ′ ( x ) ≤ 0 as long as b j ′ − b j ≥ O ( δ r ) .

Original: Same as in the proof of Lemma 1, for each x ∈ S \ S j ′ , we can bound u j ′ ( x ) , u j ( x ) as follows,
Revised: Same as in the proof of Lemma 1, for each x ∈ S \ S j ′ , we can bound u j ′ ( x ) , u j ( x ) as follows,

Original: [Equation]
Revised: [Equation]

Original: Therefore, if b j ′ − b j ≥ µδ r , we can further upper bound ˙
Revised: Therefore, if b j ′ − b j ≥ µδ r , we can further upper bound ˙

Original: b j ′ − ˙ b j as follows,
Revised: b j ′ − ˙ b j as follows,

Original: [Equation]
Revised: [Equation]

Original: Similar as in Lemma 1, we can show that
Revised: Similar as in Lemma 1, we can show that

Original: [Equation]
Revised: [Equation]

Original: 1) due to W j,j ≤ O ( δ ) and b j ≥ max i ′ ∈
Revised: 1) due to W j,j ≤ O ( δ ) and b j ≥ max i ′ ∈

Original: [ k ] b i ′ − O
Revised: [ k ] b i ′ − O

Original: ( δ r ) .
Revised: ( δ r ) .

Original: So, finally we have
Revised: So, finally we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality chooses µ as a large enough constant.
Revised: where the last inequality chooses µ as a large enough constant.

Original: □
Revised: □

Original: Lemma 15 (Accuracy Monotonicity) .
Revised: Lemma 15 (Accuracy Monotonicity) .

Original: Given any positive constant C 2 , there exists
Revised: Given any positive constant C 2 , there exists

Original: positive constant
Revised: positive constant

Original: C 1 such that for all different i, j ∈ [ k ] , as long as W i,i ≥ C 1 and f i ( x ) − f j ( x ) ≤ C 2 for any x ∈ S i , we have ∂ ( f i ( x ) − f j ( x )) ∂t > 0 .
Revised: C 1 such that for all different i, j ∈ [ k ] , as long as W i,i ≥ C 1 and f i ( x ) − f j ( x ) ≤ C 2 for any x ∈ S i , we have ∂ ( f i ( x ) − f j ( x )) ∂t > 0 .

Original: Proof of Lemma 15.
Revised: Proof of Lemma 15.

Original: Since f i ( x ) − f j ( x ) ≤ C 2 , we know 1 − u i ( x ) ≥ Ω(1) .
Revised: Since f i ( x ) − f j ( x ) ≤ C 2 , we know 1 − u i ( x ) ≥ Ω(1) .

Original: This immediately implies min x ′ ∈S
Revised: This immediately implies min x ′ ∈S

Original: i (1 − u i ( x ′ )) ≥ Ω(1) since | u i ( x ) − u i ( x ′ )
Revised: i (1 − u i ( x ′ )) ≥ Ω(1) since | u i ( x ) − u i ( x ′ )

Original: | ≤ O ( δ ) .
Revised: | ≤ O ( δ ) .

Original: According to Lemma 13, we can bound ∂
Revised: According to Lemma 13, we can bound ∂

Original: ( f i ( e i ) − f j ( e i ))
Revised: ( f i ( e i ) − f j ( e i ))

Original: ∂t as follows,
Revised: ∂t as follows,

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds because W i,i ≥ C 1 with C 1 a large enough constant.
Revised: where the second inequality holds because W i,i ≥ C 1 with C 1 a large enough constant.

Original: □
Revised: □

Original: Lemma 3 (Bias Gap Control II) .
Revised: Lemma 3 (Bias Gap Control II) .

Original: There exist small positive constants C 1 , C 2 such that for any j ∈ [ k − 1] and any x ∈ S j , if 1 − u j ( x ) ≤ C 1 , W k,k ≤ O ( δ ) and b j − b k ≥ − C 2 , we have ˙
Revised: There exist small positive constants C 1 , C 2 such that for any j ∈ [ k − 1] and any x ∈ S j , if 1 − u j ( x ) ≤ C 1 , W k,k ≤ O ( δ ) and b j − b k ≥ − C 2 , we have ˙

Original: b j − ˙
Revised: b j − ˙

Original: b k < − Ω(1) .
Revised: b k < − Ω(1) .

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: Since 1 − u j ( x ) ≤ C 1 for some x ∈ S j , we know 1 − u j ( x ′ ) ≤ C 1 + O ( δ ) for every x ′ ∈ S j .
Revised: Since 1 − u j ( x ) ≤ C 1 for some x ∈ S j , we know 1 − u j ( x ′ ) ≤ C 1 + O ( δ ) for every x ′ ∈ S j .

Original: We can write down ˙
Revised: We can write down ˙

Original: b j − ˙
Revised: b j − ˙

Original: b k as follows,
Revised: b k as follows,

Original: [Equation]
Revised: [Equation]

Original: First, we upper bound u k ( x ′ ) − u j ( x ′ ) for every x ′ ∈ S j as follows,
Revised: First, we upper bound u k ( x ′ ) − u j ( x ′ ) for every x ′ ∈ S j as follows,

Original: [Equation]
Revised: [Equation]

Original: Same as in the proof of Lemma 1, for each x ′ ∈ S \ S j , we can bound u j ( x ′ ) , u k ( x ′ ) as follows,[Equation] b
Revised: Same as in the proof of Lemma 1, for each x ′ ∈ S \ S j , we can bound u j ( x ′ ) , u k ( x ′ ) as follows,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we can upper bound u k ( x ′ )
Revised: Therefore, we can upper bound u k ( x ′ )

Original: − u j ( x ′ ) as follows,
Revised: − u j ( x ′ ) as follows,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses b k − b j ≤ C 2 .
Revised: where the last inequality uses b k − b j ≤ C 2 .

Original: Above all, we can upper bound ˙
Revised: Above all, we can upper bound ˙

Original: b j − ˙ b k as follows,
Revised: b j − ˙ b k as follows,

Original: [Equation]
Revised: [Equation]

Original: ) , where the second inequality holds as long as C 1 , C 2 , δ are at most some small constants.
Revised: , where the second inequality holds as long as C 1 , C 2 , δ are at most some small constants.

Original: □
Revised: □

Original: Lemma 14 (Bias Gap Control III) .
Revised: Lemma 14 (Bias Gap Control III) .

Original: For any different i, j ∈ [ k ] , if W i,i ≤ O (1) , W j,j ≤ O ( δ ) and b i − b j ≤ − O (1) , we have ˙
Revised: For any different i, j ∈ [ k ] , if W i,i ≤ O (1) , W j,j ≤ O ( δ ) and b i − b j ≤ − O (1) , we have ˙

Original: b i − ˙
Revised: b i − ˙

Original: b j > 0 .
Revised: b j > 0 .

Original: Proof of Lemma 14.
Revised: Proof of Lemma 14.

Original: We can write down ˙
Revised: We can write down ˙

Original: b i − ˙
Revised: b i − ˙

Original: b j as follows,
Revised: b j as follows,

Original: [Equation]
Revised: [Equation]

Original: Next, we lower bound u j ( x ) − u i ( x ) for every x ∈ S ,
Revised: Next, we lower bound u j ( x ) − u i ( x ) for every x ∈ S ,

Original: [Equation]
Revised: [Equation]

Original: So as long as b j − b i > O (1) , we have u j ( x ) − u
Revised: So as long as b j − b i > O (1) , we have u j ( x ) − u

Original: i ( x ) > 0 for all x ∈ S , which then implies ˙
Revised: i ( x ) > 0 for all x ∈ S , which then implies ˙

Original: b i − ˙
Revised: b i − ˙

Original: b j > 0 .
Revised: b j > 0 .

Original: □
Revised: □

Original: Lemma 17.
Revised: Lemma 17.

Original: For every j ∈ [ k ] and every x ∈ S , we have
Revised: For every j ∈ [ k ] and every x ∈ S , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 17.
Revised: Proof of Lemma 17.

Original: For each j ∈ [ k ] , we have
Revised: For each j ∈ [ k ] , we have

Original: [Equation]
Revised: [Equation]

Original:  and 
Revised: and (cid:68)

Original: [Equation]
Revised: [Equation]

Original: (cid:11)
Revised: (cid:11)

Original: We know that (cid:12)(cid:12)(cid:10) x, ¯ ξ x (cid:11)(cid:12)(cid:12) ≤ O ( σ + √ log N/ √ d ) .
Revised: We know that (cid:12)(cid:12)(cid:10) x, ¯ ξ x (cid:11)(cid:12)(cid:12) ≤ O ( σ + √ log N/ √ d ) .

Original: For any x ′ ̸ = x, we have (cid:12)(cid:12)(cid:10) x ′ , ¯ ξ x (cid:11)(cid:12)(cid:12) ≤
Revised: For any x ′ ̸ = x, we have (cid:12)(cid:12)(cid:10) x ′ , ¯ ξ x (cid:11)(cid:12)(cid:12) ≤

Original: O (( σ √ log N ) / √ d + √ log N/ √ d ) ≤ O ( √ log N/ √ d ) as long as σ ≤ 1 .
Revised: O (( σ √ log N ) / √ d + √ log N/ √ d ) ≤ O ( √ log N/ √ d ) as long as σ ≤ 1 .

Original: According to Lemma 18, we know that for x ′ ∈ S j , we have (cid:12)(cid:12)(cid:12) (1 − u j ( x ′ ))
Revised: According to Lemma 18, we know that for x ′ ∈ S j , we have (cid:12)(cid:12)(cid:12) (1 − u j ( x ′ ))

Original: ⟨ W j, : , x ′ ⟩ r − 1 (cid:12)(cid:12)(cid:12) ≤ O (1) .
Revised: ⟨ W j, : , x ′ ⟩ r − 1 (cid:12)(cid:12)(cid:12) ≤ O (1) .

Original: For x ′ ∈ S \ S i , we have (cid:12)(cid:12)(cid:12) u
Revised: For x ′ ∈ S \ S i , we have (cid:12)(cid:12)(cid:12) u

Original: i ( x ′ ) ⟨ W j, : , x ′ ⟩ r − 1 (cid:12)(cid:12)(cid:12) ≤ O ( δ r − 1 ) since |⟨ W j, : , x ′ ⟩| ≤ O ( δ ) +
Revised: i ( x ′ ) ⟨ W j, : , x ′ ⟩ r − 1 (cid:12)(cid:12)(cid:12) ≤ O ( δ r − 1 ) since |⟨ W j, : , x ′ ⟩| ≤ O ( δ ) +

Original: O ( √ log Nδσ ) ≤ O ( δ ) assuming σ ≤ 1 / √ log N.
Revised: O ( √ log Nδσ ) ≤ O ( δ ) assuming σ ≤ 1 / √ log N.

Original: Therefore, we can bound (cid:12)(cid:12)(cid:12)(cid:68) ˙
Revised: Therefore, we can bound (cid:12)(cid:12)(cid:12)(cid:68) ˙

Original: W j, : , ¯ ξ x (cid:69)(cid:12)(cid:12)(cid:12) as follows, (cid:12)(cid:12)(cid:12)(cid:68) ˙
Revised: W j, : , ¯ ξ x (cid:69)(cid:12)(cid:12)(cid:12) as follows, (cid:12)(cid:12)(cid:12)(cid:68) ˙

Original: W j, : , ¯ ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O (cid:18) σ
Revised: W j, : , ¯ ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O (cid:18) σ

Original: N + √ log N √ d
Revised: N + √ log N √ d

Original: (cid:19)
Revised: (cid:19)

Original: Since T ≤ O (log(1 /δ ) /δ
Revised: Since T ≤ O (log(1 /δ ) /δ

Original: r − 2 ) , N ≥ log(1 /δ )
Revised: r − 2 ) , N ≥ log(1 /δ )

Original: /δ
Revised: /δ

Original: r − 1 and d ≥ log 2 (1 /δ ) /δ
Revised: r − 1 and d ≥ log 2 (1 /δ ) /δ

Original: 2 r − 2 , we know (cid:12)(cid:12)(cid:12)(cid:68) ˙
Revised: 2 r − 2 , we know (cid:12)(cid:12)(cid:12)(cid:68) ˙

Original: W j, : , ¯ ξ x (cid:69)(cid:12)(cid:12)(cid:12) · T ≤ O ( (cid:112) log Nδ ) .
Revised: W j, : , ¯ ξ x (cid:69)(cid:12)(cid:12)(cid:12) · T ≤ O ( (cid:112) log Nδ ) .

Original: □
Revised: □

Original: Lemma 18.
Revised: Lemma 18.

Original: For any i ∈ [ k ] and x ∈ S i , if (1 − u i ( x )) ⟨ W i, : , x ⟩ r − 1 ≥ Θ(1) , we have
Revised: For any i ∈ [ k ] and x ∈ S i , if (1 − u i ( x )) ⟨ W i, : , x ⟩ r − 1 ≥ Θ(1) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 18.
Revised: Proof of Lemma 18.

Original: We can write 1 − u
Revised: We can write 1 − u

Original: i ( x ) as
Revised: i ( x ) as

Original: [Equation]
Revised: [Equation]

Original: f j
Revised: f j

Original: [Equation]
Revised: [Equation]

Revised: x )

Revised: [Equation]

Original: exp( f i ( x )) .
Revised: +exp( f i ( x )) .

Original: Next, we prove that for any j ′ ̸
Revised: Next, we prove that for any j ′ ̸

Original: = i, we have
Revised: = i, we have

Original: [Equation]
Revised: [Equation]

Original: This derivative can be written the sum of two terms:
Revised: This derivative can be written the sum of two terms:

Original: [Equation]
Revised: [Equation]

Original: For the first term
Revised: For the first term

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the second term, we have
Revised: For the second term, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses f i ( x ) 
Revised: where the last inequality uses f i ( x ) − f

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Combining the bounds on both terms, as long as ⟨ W i, : , x ⟩ is larger than certain constant (which is guaranteed by (1 − u i ( x )) ⟨ W i, : , x ⟩ r − 1 ≥ Θ(1) ), we know ddt (cid:16) (1 − u
Revised: Combining the bounds on both terms, as long as ⟨ W i, : , x ⟩ is larger than certain constant (which is guaranteed by (1 − u i ( x )) ⟨ W i, : , x ⟩ r − 1 ≥ Θ(1) ), we know ddt (cid:16) (1 − u

Original: i ( x )) ⟨ W i, : , x ⟩ r − 1 (cid:17) < 0 .
Revised: i ( x )) ⟨ W i, : , x ⟩ r − 1 (cid:17) < 0 .

Original: □
Revised: □

Original: Lemma 16.
Revised: Lemma 16.

Original: For any j ̸ = j ′ , we have T ˙ W j,j ′ ≤ O ( δ ) .
Revised: For any j ̸ = j ′ , we have T ˙ W j,j ′ ≤ O ( δ ) .

Original: Furthermore, there exists absolute constant µ > 0 such that if 0 < W j,j ′ < µδ log 1 r − 2 (1 /δ ) , we have T ˙ W j,j ′ ≥ − µδ 2log 1 r − 2 (1 /δ ) .
Revised: Furthermore, there exists absolute constant µ > 0 such that if 0 < W j,j ′ < µδ log 1 r − 2 (1 /δ ) , we have T ˙ W j,j ′ ≥ − µδ 2log 1 r − 2 (1 /δ ) .

Original: Proof of Lemma 16.
Revised: Proof of Lemma 16.

Original: We can write down the derivative of W j,j ′ as follows,
Revised: We can write down the derivative of W j,j ′ as follows,

Original: [Equation]
Revised: [Equation]

Original: The bound on the first term relies on (1 − u j ( x )) ⟨ W j, : , x ⟩ r − 1 ≤ O (1) and ⟨ e j ′ , x ⟩ = ± O (cid:16) σ √ log N √ d (cid:17) for x ∈ S j , where (1 − u j ( x )) ⟨ W j, : , x ⟩ r − 1 ≤ O (1) is guaranteed by Lemma 18.
Revised: The bound on the first term relies on (1 − u j ( x )) ⟨ W j, : , x ⟩ r − 1 ≤ O (1) and ⟨ e j ′ , x ⟩ = ± O (cid:16) σ √ log N √ d (cid:17) for x ∈ S j , where (1 − u j ( x )) ⟨ W j, : , x ⟩ r − 1 ≤ O (1) is guaranteed by Lemma 18.

Original: The bound on the second term uses ⟨ W j, : , x ⟩ = W j,j ′ ±
Revised: The bound on the second term uses ⟨ W j, : , x ⟩ = W j,j ′ ±

Original: O (cid:0) √ log Nδσ (cid:1) and ⟨ e j ′ , x ⟩ = 1 ±
Revised: O (cid:0) √ log Nδσ (cid:1) and ⟨ e j ′ , x ⟩ = 1 ±

Original: O (cid:16) σ √ log N √ d (cid:17) for x ∈ S j ′ .
Revised: O (cid:16) σ √ log N √ d (cid:17) for x ∈ S j ′ .

Original: The bound on the third term uses ⟨ W j, : , x ⟩ = O ( δ ) and ⟨ e j ′ , x ⟩ = ± O (cid:16) σ √ log N √ d (cid:17) for x ∈ S \ ( S j ∪ S j ′ ) .
Revised: The bound on the third term uses ⟨ W j, : , x ⟩ = O ( δ ) and ⟨ e j ′ , x ⟩ = ± O (cid:16) σ √ log N √ d (cid:17) for x ∈ S \ ( S j ∪ S j ′ ) .

Original: To prove the upper bound of the derivative, we have
Revised: To prove the upper bound of the derivative, we have

Original: [Equation]
Revised: [Equation]

Original: where we use W j,j ′ ±
Revised: where we use W j,j ′ ±

Original: O (cid:0) √ log Nδσ (cid:1) ≥ 0 .
Revised: O (cid:0) √ log Nδσ (cid:1) ≥ 0 .

Original: Since T = O (log(1 /δ )
Revised: Since T = O (log(1 /δ )

Original: /δ
Revised: /δ

Original: r − 2 ) , we have
Revised: r − 2 ) , we have

Original: T ˙ W j,j ′ ≤ O ( δ ) , as long as d ≥ O
Revised: T ˙ W j,j ′ ≤ O ( δ ) , as long as d ≥ O

Original: ( log N log 2 (1 /δ )
Revised: ( log N log 2 (1 /δ )

Original: δ 2 r −) .
Revised: δ 2 r −) .

Original: We show that there exists absolute constant µ > 

Revised: We show that there exists absolute constant µ >

Original: such that if 0 < W j,j ′ <
Revised: such that if 0 < W j,j ′ <

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 /δ ) , we have
Revised: 1 /δ ) , we have

Original: T ˙ W j,j ′ ≥ −
Revised: T ˙ W j,j ′ ≥ −

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: δ ) , which holds as long as ˙ W j, [Equation] ′ ≥ − O (cid:18)
Revised: δ ) , which holds as long as ˙ W j,j ′ ≥ − O (cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19) .
Revised: (cid:19) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The first inequality assumes σ ≤ O
Revised: The first inequality assumes σ ≤ O

Original: [Equation]
Revised: [Equation]

Original: The second inequality assumes d ≥
Revised: The second inequality assumes d ≥

Original: O ( log N log [Equation] [Equation]
Revised: O ( log N log 2 r − 2 r − 2 (1 /δ )

Original: [Equation] [Equation]
Revised: µ 2 r − 2 δ 2 r −) .

Original: The third inequality chooses µ as a small enough constant.
Revised: The third inequality chooses µ as a small enough constant.

Original: □
Revised: □

Original: Lemma 11.
Revised: Lemma 11.

Original: For any j ∈ [ k ] , we have
Revised: For any j ∈ [ k ] , we have

Original: [Equation]
Revised: [Equation]

Original: If min x ∈S j (1 − u j ( x )) ≥ Ω(1) , we further have (cid:16) [Equation] − O [Equation]
Revised: If min x ∈S j (1 − u j ( x )) ≥ Ω(1) , we further have (cid:16) 1 − O ( (cid:112) log Nσ ) (cid:17)

Original: log Nσ )

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 11.
Revised: Proof of Lemma 11.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: the second equality uses |
Revised: the second equality uses |

Original: [Equation]
Revised: [Equation]

Original: (cid:16) √ log Nσ √ d (cid:17) and W j,j ′ ≤
Revised: √ log Nσ √ d (cid:17) and W j,j ′ ≤

Original: O ( δ ) for j ̸ = j ′ .
Revised: O ( δ ) for j ̸ = j ′ .

Original: Therefore, if min x ∈S j (1 − u j ( x )) ≥ Ω(1) , we know
Revised: Therefore, if min x ∈S j (1 − u j ( x )) ≥ Ω(1) , we know

Original: [Equation]
Revised: [Equation]

Original: And we always have
Revised: And we always have

Original: [Equation]
Revised: [Equation]

Original: Lemma 13.
Revised: Lemma 13.

Original: For any x ∈ S i and any j ̸
Revised: For any x ∈ S i and any j ̸

Original: =
Revised: =

Original: i, if 1 − u i ( x ) ≥ Ω(1) , we have
Revised: i, if 1 − u i ( x ) ≥ Ω(1) , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 13.
Revised: Proof of Lemma 13.

Original: Recall that f i ( x ) =
Revised: Recall that f i ( x ) =

Original: ⟨ W i, : , x ⟩ r + b i , so we have
Revised: ⟨ W i, : , x ⟩ r + b i , so we have

Original: [Equation]
Revised: [Equation]

Original: where in the last inequality we uses ˙
Revised: where in the last inequality we uses ˙

Original: W i,i ≥ Ω( W r − 1 i,i ) ≥ Ω( δ r ) and (cid:12)(cid:12)(cid:12)(cid:68) ˙
Revised: W i,i ≥ Ω( W r − 1 i,i ) ≥ Ω( δ r ) and (cid:12)(cid:12)(cid:12)(cid:68) ˙

Original: W i, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12)
Revised: W i, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) ≤

Original: O ( δ r − 1 ) .
Revised: O ( δ r − 1 ) .

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: where we uses | W j,i | ≤ O ( δ ) ,
Revised: where we uses | W j,i | ≤ O ( δ ) ,

Original: [Equation]
Revised: [Equation]

Original: ˙
Revised: ˙

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: ,i
Revised: ,i

Original: [Equation]
Revised: [Equation]

Original: ≤ O ( δ r − 1 ) and (cid:12)(cid:12)(cid:12)(cid:68) ˙
Revised: ≤ O ( δ r − 1 ) and (cid:12)(cid:12)(cid:12)(cid:68) ˙

Original: W j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( δ r − 1 ) .
Revised: W j, : , ξ x (cid:69)(cid:12)(cid:12)(cid:12) ≤ O ( δ r − 1 ) .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: D A DDITIONAL EXPERIMENTS
Revised: D A DDITIONAL EXPERIMENTS

Original: In this section, we describe the detailed setting of our experiments and also include additional experiment results.
Revised: In this section, we describe the detailed setting of our experiments and also include additional experiment results.

Original: MNIST & Fashion-MNIST.
Revised: MNIST & Fashion-MNIST.

Original: Unless specified otherwise, we use a depth-10 and widthfully-connected ReLU neural network (FCN10) for MNIST and Fashion-MNIST.
Revised: Unless specified otherwise, we use a depth-10 and widthfully-connected ReLU neural network (FCN10) for MNIST and Fashion-MNIST.

Original: We use Kaiming initialization for the weights and set all bias terms as zero.
Revised: We use Kaiming initialization for the weights and set all bias terms as zero.

Original: We use a small initialization by scaling the weights of each layer by (0 . 001) 1 /h
Revised: We use a small initialization by scaling the weights of each layer by (0 . 001) 1 /h

Original: so the output is scaled by 0 .
Revised: so the output is scaled by 0 .

Original: 001 , where h is the network depth.
Revised: 001 , where h is the network depth.

Original: We train the network using SGD with learning rate 0 . 01 and momentum 0 .
Revised: We train the network using SGD with learning rate 0 . 01 and momentum 0 .

Original: 9 for 100 epochs.
Revised: 9 for 100 epochs.

Original: CIFAR-10 & CIFAR-

Revised: CIFAR-10 & CIFAR-

Original: We use VGG-16 (without batch normalization) for CIFAR-10 and
Revised: We use VGG-16 (without batch normalization) for CIFAR-10 and

Original: CIFAR-100.
Revised: CIFAR-100.

Original: We use Kaiming initialization for the weights and set all bias terms as zero.
Revised: We use Kaiming initialization for the weights and set all bias terms as zero.

Original: We run
Revised: We run

Original: SGD with momentum 0 .
Revised: SGD with momentum 0 .

Original: 9 and weight decay 1e − 4 for 100 epochs.
Revised: 9 and weight decay 1e − 4 for 100 epochs.

Original: For the learning rate, we start from 0 .
Revised: For the learning rate, we start from 0 .

Original: 01 and reduce it by a factor of 0 .
Revised: 01 and reduce it by a factor of 0 .

Original: 1 at the 60 -th epoch and 90 -th epoch.
Revised: 1 at the 60 -th epoch and 90 -th epoch.

Original: We linearly interpolate using 50 evenly spaced points between the network at initialization and the network at the end of training.
Revised: We linearly interpolate using 50 evenly spaced points between the network at initialization and the network at the end of training.

Original: We evaluate error and loss on the train set.
Revised: We evaluate error and loss on the train set.

Original: For each setting, we repeat the experiments three times from different random seeds and plot the mean and deviation.
Revised: For each setting, we repeat the experiments three times from different random seeds and plot the mean and deviation.

Original: Note in Figure 1, to contrast the convex curve and plateau curve, we have used FCN4 with standard initialization on MNIST, and VGG-16 with 0 .
Revised: Note in Figure 1, to contrast the convex curve and plateau curve, we have used FCN4 with standard initialization on MNIST, and VGG-16 with 0 .

Original: 001 initialization on CIFAR-10.
Revised: 001 initialization on CIFAR-10.

Original: Our code is based on the implementation from Lucas et al.
Revised: Our code is based on the implementation from Lucas et al.

Original: Each trial of our [Figure]
Revised: Each trial of our experiment

Original: A
Revised: D.

Original: LL

Original: [Figure]
Revised: [Figure]

Original:  V . S . LAST BIAS V . S . NO BIAS
Revised: A LL BIAS V . S . LAST BIAS V . S . NO BIAS

Original: [Figure]
Revised: [Figure]

Original: Figure 9 shows that on both Fashion-MNIST and CIFAR-10, having bias on the last layer or on all layers can create longer plateau in error curve, while does not significantly affect the loss curve.
Revised: Figure 9 shows that on both Fashion-MNIST and CIFAR-10, having bias on the last layer or on all layers can create longer plateau in error curve, while does not significantly affect the loss curve.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Figure 10 shows that on both Fashion-MNIST and CIFAR-10, applying homogeneous interpolation on biases can significantly reduce the plateau on error interpolation curve
Revised: Figure 10 shows that on both Fashion-MNIST and CIFAR-10, applying homogeneous interpolation on biases can significantly reduce the plateau on error interpolation curve

Original: [Figure]
Revised: [Figure]

Original: D. D IFFERENT INITIALIZATIONS
Revised: D. D IFFERENT INITIALIZATIONS

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Smaller initialization creates longer plateau in both error and loss curves.
Revised: Smaller initialization creates longer plateau in both error and loss curves.

Original: See Figure 11 for MNIST,
Revised: See Figure 11 for MNIST,

Original: CIFAR-100 with last bias; see Figure 12 for Fashion-MNIST, CIFAR-10 with all bias; see Figurefor Fashion-MNIST, CIFAR-10 with last bias
Revised: CIFAR-100 with last bias; see Figure 12 for Fashion-MNIST, CIFAR-10 with all bias; see Figurefor Fashion-MNIST, CIFAR-10 with last bias

Original: [Figure]
Revised: [Figure]

Original: D. D IFFERENT DEPTHS
Revised: D. D IFFERENT DEPTHS

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: with different depth on Fashion-MNIST and CIFARwith all bias.
Revised: different depth on Fashion-MNIST and CIFARwith all bias.

Original: We use 0.001 initialization scale for VGG-16 on CIFAR-10.
Revised: We use 0.001 initialization scale for VGG-16 on CIFAR-10.

Original: [Figure]
Revised: [Figure]

Original: Deeper networks create longer plateau in both error and loss curves.
Revised: Deeper networks create longer plateau in both error and loss curves.

Original: See Figure 14 for MNIST,
Revised: See Figure 14 for MNIST,

Original: CIFAR-100 with last bias; see Figure 15 for Fashion-MNIST, CIFAR-10 with all bias; see Figurefor Fashion-MNIST, CIFAR-10 with last bias.
Revised: CIFAR-100 with last bias; see Figure 15 for Fashion-MNIST, CIFAR-10 with all bias; see Figurefor Fashion-MNIST, CIFAR-10 with last bias.

Original: D. B IAS DYNAMICS
Revised: D. B IAS DYNAMICS

Original: [Figure]
Revised: [Figure]

Original: In Figure
Revised: In Figure

Original: [Figure]
Revised: [Figure]

Original: we give two more examples on two-class MNIST in which the later learned class has larger bias.
Revised: we give two more examples on two-class MNIST in which the later learned class has larger bias.

Original: [Figure]
Revised: [Figure]

Revised: Train loss for each class and bias term dynamics on MNIST { 7 , 8 , 9 } .

Original: In Figure 18, although class 9 is learned last, class 7 gets the largest bias after training.
Revised: In Figure 18, although class 9 is learned last, class 7 gets the largest bias after training.

Original: Let S be the set of all samples for number 7,8,9 and let S 7 , S 9 , S 9 be the set of samples for each class.
Revised: Let S be the set of all samples for number 7,8,9 and let S 7 , S 9 , S 9 be the set of samples for each class.

Original: For convenience, we use u i,j to denote 1 | S | (cid:80) x ∈ S j u i ( x ) , where u i ( x ) is the softmax output for class i under input x. Then, we can write down the derivative on three bias terms:
Revised: For convenience, we use u i,j to denote 1 | S | (cid:80) x ∈ S j u i ( x ) , where u i ( x ) is the softmax output for class i under input x. Then, we can write down the derivative on three bias terms:

Original: [Equation]
Revised: [Equation]

Original: According to the per-class loss, we know that (cid:80) x ∈ S 7 − log ( u 7 ( x )) < (cid:80) x ∈ S 9 − log ( u 9 ( x )) , which intuitively implies that (cid:80) x ∈ S 7 u 7 ( x ) > (cid:80) x ∈ S 9 u 9 ( x ) that is u 7 , 7 > u 9 , 9 .
Revised: According to the per-class loss, we know that (cid:80) x ∈ S 7 − log ( u 7 ( x )) < (cid:80) x ∈ S 9 − log ( u 9 ( x )) , which intuitively implies that (cid:80) x ∈ S 7 u 7 ( x ) > (cid:80) x ∈ S 9 u 9 ( x ) that is u 7 , 7 > u 9 , 9 .

Original: This tends to drive bsmaller than b 9 .
Revised: This tends to drive bsmaller than b 9 .

Original: However, because u 9 , 8 > u 7 , 8 , we actually have ˙ b 9 < ˙ b 7 .
Revised: However, because u 9 , 8 > u 7 , 8 , we actually have ˙ b 9 < ˙ b 7 .

Original: So eventually b 9 becomes smaller than b 7 .
Revised: So eventually b 9 becomes smaller than b 7 .

Original: Intuitively, class 9 is more correlated with class 8 , so u 9 , 8 > u 7 , 8 .
Revised: Intuitively, class 9 is more correlated with class 8 , so u 9 , 8 > u 7 , 8 .