Original: U NDERSTANDING THE ROLE OF IMPORTANCE WEIGHTING FOR DEEP LEARNING
Revised: U NDERSTANDING THE ROLE OF IMPORTANCE WEIGHTING FOR DEEP LEARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The recent paper by Byrd & Lipton (2019), based on empirical observations, raises a major concern on the impact of importance weighting for the over-parameterized deep learning models.
Revised: The recent paper by Byrd & Lipton (2019), based on empirical observations, raises a major concern on the impact of importance weighting for the over -parameterized deep learning models.

Original: They observe that as long as the model can separate the training data, the impact of importance weighting diminishes as the training proceeds.
Revised: They observe that as long as the model can separate the training data, the impact of importance weighting diminishes as the training proceeds.

Original: Nevertheless, there lacks a rigorous characterization of this phenomenon.
Revised: Nevertheless, there lacks a rigorous characterization of this phenomenon.

Original: In this paper, we provide formal characterizations and theoretical justiﬁcations on the role of importance weighting with respect to the implicit bias of gradient descent and margin-based learning theory.
Revised: In this paper we provide formal characterizations and theoretical justiﬁcations on the role of importance weighting with respect to the implicit bias of gradient descent and margin-based learning theory.

Original: We reveal both the optimization dynamics and generalization performance under deep learning models.
Revised: We reveal both the optimization dynamics and generalization performance under deep learning models.

Original: Our work not only explains the various novel phenomenons observed for importance weighting in deep learning, but also extends to the studies where the weights are being optimized as part of the model, which applies to a number of topics under active research.
Revised: Our work not only explains the various novel phenomenons observed for importance weighting in deep learning, but also extends to the studies where the weights are being optimized as part of the model, which applies to a number of topics under active research

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Importance weighting is a standard tool used to estimate a quantity under a target distribution while only the  source distribution is accessible.
Revised: Importance weighting is a standard tool for estimating a quantity under a target distribution while only the samples from some source distribution is accessible.

Original: It has been drawing extensive attention in the communities of statistics and machine learning.
Revised: It has been drawing extensive attention in the communities of statistics and machine learning.

Original: Causal inference for deep learning investigates heavily on the propensity score weighting method that applies the off-policy optimization with counterfactual estimator (Gilotte et al., 2018; Jiang & Li, 2016), modelling with observational feedback (Schnabel et al., 2016) and learning from controlled intervention (Swaminathan & Joachims, 2015).
Revised: Causal inference for deep learning investigates heavily on the propensity score weighting method that applies the off-policy optimization with counterfactual estimator (Gilotte et al., 2018; Jiang & Li, 2016), modelling with observational feedback (Schnabel et al., 2016; Xu et al., 2020) and learning from controlled intervention (Swaminathan & Joachims, 2015).

Original: The importance weights are also applied to characterize distribution shifts  in such as  domain adaptation (Azizzadenesheli et al., 2019; Lipton et al., 2018) and learning from noisy labels (Song et al., 2020).
Revised: The importance weighting methods are also applied to characterize distribution shifts for deep learning models (Fang et al., 2020), with modern applications in such as the domain adaptation (Azizzadenesheli et al., 2019; Lipton et al., 2018) and learning from noisy labels (Song et al., 2020).

Original: Other usages include curriculum learning Bengio et al. and knowledge distillation Hinton et al. (2015) where the weights characterize the model conﬁdence on each sample.
Revised: Other usages include curriculum learning (Bengio et al., 2009) and knowledge distillation (Hinton et al., 2015), where the weights characterize the model conﬁdence on each sample.

Original: To reduce the discrepancy between thetarget distribution and the source  distribution for model training, a standard routine is to minimize a weighted risk (Rubinstein & Kroese, 2016).
Revised: To reduce the discrepancy between the source and target distribution for model training, a standard routine is to minimize a weighted risk (Rubinstein & Kroese, 2016).

Original: Many techniques have been developed to this end. A common strategy is to re-weight classes proportionally to the inverse of their frequency (Huang et al., 2016; 2019; Wang et al., 2017).
Revised: Many techniques have been developed to this end, and the common strategy is re-weighting the classes proportionally to the inverse of their frequencies (Huang et al., 2016; 2019; Wang et al., 2017).

Original:  Cui et al.
Revised: For example, Cui et al.

Original: (2019) proposes re-weighting by the inverse of effective number of samples.
Revised: (2019) proposes re-weighting by the inverse of effective number of samples.

Original: The focal loss (Lin et al., 2017) down-weights the well-classiﬁed examples; and  Li et al.
Revised: The focal loss (Lin et al., 2017) down-weights the well-classiﬁed examples, and the work by Li et al.

Original: (2019) suggests an improved technique which down-weights examples based on the magnitude of the gradients.
Revised: (2019) suggests an improved technique which down-weights examples based on the magnitude of the gradients.

Original: Despite the empirical successes of various re-weighting methods, it is  not clear how importance weighting lays inﬂuence from a theoretical perspective.
Revised: Despite the empirical successes of various re-weighting methods, it is ultimately not clear how importance weighting lays inﬂuence from the theoretical standpoint.

Original: A recent study  Byrd & Lipton (2019) observes from experiments that there is little impact of importance weights on the converged deep neural network, if the data can be separated by the model using gradient descent.
Revised: The recent study of Byrd & Lipton (2019) observes from experiments that there is little impact of importance weights on the converged deep neural network, if the data can be separated by the model using gradient descent.

Original: They connect this phenomenon to the implicit bias of gradient descent (Soudry et al., 2018)
Revised: They connect this phenomenon to the implicit bias of gradient descent (Soudry et al., 2018)

Original: - a novel topic that studies why over-parameterized models trained on separable data is biased toward solutions that generalize well.
Revised: - a novel topic that studies why over-parameterized models trained on separable data is biased toward solutions that generalize well.

Original: Implicit bias of gradient descent has been observed and studied for linear model (Soudry et al., 2018; Ji & Telgarsky, 2018b), linear neural network (Ji & Telgarsky, 2018a;
Revised: Implicit bias of gradient descent has been observed and studied for linear model (Soudry et al., 2018; Ji & Telgarsky, 2018b), linear neural network (Ji & Telgarsky, 2018a;

Original: Gunasekar et al., 2018), two-layer neural network with homogeneous activation (Chizat & Bach, 2020) and smooth neural networks (Nacson et al., 2019; Lyu & Li, 2019).
Revised: Gunasekar et al., 2018), two-layer neural network with homogeneous activation (Chizat & Bach, 2020) and smooth neural networks (Nacson et al., 2019; Lyu & Li, 2019).

Original: These works reveal that the direction of the parameters (for linear predictor) and the normalized margin (for nonlinear predictor), regardless of the initialization, converge to that of a max-margin solution.
Revised: To summarize, those work reveals that the direction of the parameters (for linear predictor) and the normalized margin (for nonlinear predictor), regardless of the initialization, converge to that of a max-margin solution.

Original: The pivotal role of margin for deep learning models has been explored actively after the long journey of understanding the generalization of over-parameterized neural networks (Bartlett et al., 2017; Golowich et al., 2018; Neyshabur et al., 2018).
Revised: The pivotal role of margin for deep learning models has been explored actively after the long journey of understanding the generalization of over-parameterized neural networks (Bartlett et al., 2017; Golowich et al., 2018; Neyshabur et al., 2018).

Original: For instance, Wei et al.
Revised: For instance, Wei et al.

Original: (2019) studies the margin of the neural networks for separable data under weak regularization.
Revised: (2019) studies the margin of the neural networks for separable data under weak regularization.

Original: They show that the normalized margin also converges to the max-margin solution, and provide a generalization bound for a neural network that hinges on its margin.
Revised: They show that the normalized margin also converges to the max-margin solution, and provide a generalization bound for a neural network that hinges on its margin.

Original: Although there are rich understandings for the implicit bias of gradient descent and the margin-based generalization, very few efforts are dedicated to studying how they adjust to the weighted ERM setting.
Revised: Although there are rich understandings for the implicit bias of gradient descent and the margin-based generalization, very few efforts are dedicated to studying how they adjust to the weighted empiricalrisk minimization (ERM) setting.

Original: The established results do not directly apply since importance weighting changes both the optimization geometry and how the generalization is measured.
Revised: The established results do not directly transfer since importance weighting can change both the optimization geometry and how the generalization is measured.

Original: In this paper, we ﬁll in the gap by showing the impact of importance weighting on the implicit bias of gradient descent as well as the generalization performance.
Revised: In this paper, we ﬁll in the gap by showing the impact of importance weighting on the implicit bias of gradient descent as well as the generalization performance.

Original: By studying the optimization dynamics of linear models, we ﬁrst reveal the effect of importance  on the convergence speed for separable data.
Revised: By studying the optimization dynamics of linear models, we ﬁrst reveal the effect of importance weighting on the convergence speed under linearly separable data.

Original: When the data is not linearly separable, we also characterize the unique role of importance weighting on deﬁning the intercept term upon the implicit bias.
Revised: When the data is not linearly separable, we also characterize the unique role of importance weighting on deﬁning the intercept term upon the implicit bias.

Original: Then we investigate the non-linear neural network under a weak regularization as Wei et al.
Revised: We then investigate the non-linear neural network under a weak regularization as Wei et al.

Original: We provide a novel generalization bound that reﬂects how importance weighting leads to the interplay between the empirical risk and a compounding term that consists of the model complexity as well as the deviation between the sourcedistribution and the target distribution.
Revised: We provide a novel generalization bound that reﬂects how importance weighting leads to the interplay between the empirical risk and a compounding term that consists of the model complexity as well as the deviation between the source target distribution.

Original: Based on our theoretical results, we give several exploratory developments on importance weighting that are worthy of further investigations.
Revised: Based on our theoretical results, we discuss several exploratory developments on importance weighting that are worthy of further investigations.

Original: • A good set of weights for learning can be inversely proportional to the hard-to-classify extent.
Revised: • A good set of weights for learning can be inversely proportional to the hard-to-classify extent.

Original: For example, a sample that is close to (far from) an oracle decision boundary should have a large (small) weight.
Revised: For example, a sample that is close to (far from) the oracle decision boundary should have a large (small) weight.

Original: • If the importance weights are jointly trained according to a weight model, the impact of the weight model eventually diminishes after showing strong correlation with the hard-toclassify extent such as margin.
Revised: • If the importance weights are jointly trained according to a weighting model, the impact of the weighting model eventually diminishes after showing strong correlation with the hard-to-classify extent such as margin.

Original: • The usefulness of explicit regularization on weighted ERM can be studied, via their impact on the margin, on balancing the empirical loss and the distribution divergence.
Revised: • The usefulness of explicit regularization on weighted ERM can be studied, via their impact on the margin, on balancing the empirical loss and the distribution divergence.

Original: In summary, our contribution are three folds.
Revised: In summary, our contribution are three folds.

Original: • We characterize the impact of importance weighting on the implicit bias of gradient descent.
Revised: • We characterize the impact of importance weighting on the implicit bias of gradient descent.

Original: We ﬁnd a generalization bound that hinges on the importance weights.
Revised: We ﬁnd a generalization bound that hinges on the importance weights.

Original: In ﬁnite-step training, the role of importance weighting on the generalization bound is reﬂected in how the margin is affected, and how it balances  source and target distribution.
Revised: For ﬁnite-step training, the role of importance weighting on the generalization bound is reﬂected in how the margin is affected, and how it balances the source and target distribution.

Original: • We propose several exploratory topics for importance weighting that worth further investigating from both the application and theoretical perspective.
Revised: • We propose several exploratory topics for importance weighting that worth further investigating from both the application and theoretical perspective.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: In Section 2, we introduce the background, preliminary results and the experimental setup.
Revised: In Section 2, we introduce the background, preliminary results and the experimental setup.

Original: In Section 3 and 4, we demonstrate the inﬂuence of the importance weighting for linear and non-linear models in terms of the implicit bias of gradient descent and the generalization performance.
Revised: In Section 3 and 4, we demonstrate the inﬂuence of the importance weighting for linear and non-linear models in terms of the implicit bias of gradient descent and the generalization performance.

Original: We then discuss the extended investigations in Section 5.
Revised: We then discuss the extended investigations in Section 5.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We use bold-font letters for vectors and matrices, uppercase letters for random variables and distributions, and (cid:107) · (cid:107) to denote (cid:96) 2 norm when no confusion arises.
Revised: We use bold-font letters for vectors and matrices, uppercase letters for random variables and distributions, and (cid:107) · (cid:107) to denote (cid:96) 2 norm when no confusion arises.

Original: We denote the training data by
Revised: We denote the training data by

Original: D = { w i , x i , y i } ni =1 where x i ∈
Revised: D = { w i , x i , y i } ni =1 where x i ∈

Original: X , y i is binary or categorical and the importance weight is bounded: w i ∈ [1 /M, M ] for some M > 1 .
Revised: X denotes the features, y i is binary or categorical, and the importance weight is bounded such that: w i ∈ [1 /M, M ] for some M > 1 .

Original: Note that the importance weight is often deﬁned with respect to the source distribution P s from which the training data is drawn, and the target distribution P t .
Revised: We mention that the importance weights are often deﬁned with respect to the source distribution P s from which the training data is drawn, and the target distribution P t .

Original: We do not make this assumption here because importance weighting is often applied for more general purposes.
Revised: We do not make this assumption here because importance weighting is often applied for more general purposes.

Original: Therefore, w i can be deﬁned arbitrarily.
Revised: Therefore, w i can be deﬁned arbitrarily.

Original: We use f ( θ , x ) to denote the predictor and deﬁne F = { f ( θ , · ) | θ ∈ Θ ⊂ R d } .
Revised: We use f ( θ , x ) to denote the predictor and deﬁne F = { f ( θ , · ) | θ ∈ Θ ⊂ R d } .

Original: For the sake of notation, we mostly study the binary setting: y i ∈ {− 1 , +1 } with f ( θ , x ) ∈ R .
Revised: For the sake of notation, we focus on the binary setting: y i ∈ {− 1 , +1 } with f ( θ , x ) ∈ R .

Revised: However, it will become clear later that our results can be easily extended to the multi-class setting.

Original: Consider the weighted empirical risk minimization (ERM) task with the risk: L ( θ ; w )
Revised: Consider the weighted empirical risk minimization (ERM) task with the risk given by L ( θ ; w )

Original: =
Revised: =

Original: 1 /n (cid:80) ni
Revised: 1 /n (cid:80) ni

Original: =

Revised: =

Original: w i (cid:96) (cid:0) y i f
Revised: w i (cid:96) (cid:0) y i f

Original: ( θ , x i ) (cid:1) for some non-negative loss function (cid:96) ( · ) .
Revised: ( θ , x i ) (cid:1) for some non-negative loss function (cid:96) ( · ) .

Original: The weight-agnostic counterpart is given by: L ( θ )= 1 /n (cid:80) ni
Revised: The weight-agnostic counterpart is denoted by: L ( θ )

Original:  =1 (cid:96) ( y i f
Revised: = 1 /n (cid:80) ni =1 (cid:96) ( y i f

Original: ( θ , x i )) .
Revised: ( θ , x i )) .

Original: We focus particularly on the exponential loss (cid:96) ( u ) =
Revised: We focus particularly on the exponential loss (cid:96) ( u ) =

Original: exp( − u ) and log loss (cid:96) ( u ) = {+log(
+}
Revised: exp( − u ) and log loss (cid:96) ( u ) = {+log(
+}

Original: log(1 + exp( − u )) .
Revised: + exp( − u )) .

Original: For the multi-class problem where y i ∈ [ k ] , we extend our setup using the softmax function where the logits are now given by { f j ( θ , x ) } kj =1 .
Revised: For the multi-class problem where y i ∈ [ k ] , we extend our setup using the softmax function where the logits are now given by { f j ( θ , x ) } kj =1 .

Original: For optimization, we consider using gradient descent to minimize the total loss: θ ( t +1) ( w )= θ ( t ) ( w ) − η [Equation] L (
Revised: For optimization, we consider using gradient descent to minimize the total loss: θ ( t +1) ( w )

Original: [Equation]

Original:  ; w ) (cid:12)(cid:12)[Equation] = θ( [Equation]
Revised: = θ ( t ) ( w ) − η t ∇ L ( θ ; w ) (cid:12)(cid:12) θ

Original:  ) ( w ) , where the learning rate η t can be constant or step-dependent.
Revised: = θ ( t ) ( w ) , where the learning rate η t can be constant or step-dependent.

Original: From parameter norm divergence to support vectors.
Revised: From parameter norm divergence to support vectors.

Original: Suppose D is separated by f ( θ ( t ) , x ) at some point.
Revised: Suppose D is separated by f ( θ ( t ) , x ) after some point during training.

Original: The key factor that contributes the implicit bias for both linear and non-linear predictor under weak regularization 1 is that the norm of the parameters diverges on separable data, i.e. lim t →∞ (cid:107) θ ( t ) (cid:107) 2 = ∞ , as a consequence of using gradient descent.
Revised: The key factor that contributes the implicit bias for both linear and non-linear predictor under weak regularization 1 is that the norm of the parameters diverges after separation, i.e. lim t →∞ (cid:107) θ ( t ) (cid:107) 2 = ∞ , as a consequence of using gradient descent.

Original: Now we examine (cid:107) θ ( t ) ( w ) (cid:107) 2 .
Revised: Now we examine (cid:107) θ ( t ) ( w ) (cid:107) 2 .

Original: The heuristic is that if (cid:96) ( . ) is exponential-like, multiplying by w i only changes its tail property by a constant and do not affect the asymptotic behavior.
Revised: The heuristic is that if (cid:96) ( · ) is exponential-like, multiplying by w i only changes its tail property by a constant and do not affect the asymptotic behavior.

Original: In particular, the necessary conditions for norm divergence under gradient descent can be summarized by:
Revised: In particular, the necessary conditions for norm divergence under gradient descent can be summarized by:

Original: • C1 .
Revised: • C1 .

Original: The loss function (cid:96) ( · ) has a exponential tail behavior (that we formalize in Appendix A.1) such that lim u →∞ (cid:96) ( − u ) =
Revised: The loss function (cid:96) ( · ) has a exponential tail behavior (that we formalize in Appendix A.1) such that lim u →∞ (cid:96) ( − u ) =

Original: lim u →∞ ∇ (cid:96) ( − u )
Revised: lim u →∞ ∇ (cid:96) ( − u )

Original: = 0 ; • C2 .
Revised: = 0 ; • C2 .

Original: The predictor f ( θ , x ) is α -homogeneous such that f ( c · θ , x ) =
Revised: The predictor f ( θ , x ) is α -homogeneous such that f ( c · θ , x ) =

Original: c
Revised: c

Original: α f ( θ , x ) , ∀ c > 0 .
Revised: α f ( θ , x ) , ∀ c > 0 .

Original: In addition, we need certain regularities from f ( θ , x ) to ensure the existence of critical points and the convergence of gradient descent:
Revised: In addition, we need certain regularities from f ( θ , x ) to ensure the existence of critical points and the convergence of gradient descent:

Original: • C3 .
Revised: • C3 .

Original: for any x ∈ X , f ( · , x ) is β -smooth and l -Lipschitz on R d .
Revised: for any x ∈ X , f ( · , x ) is β -smooth and l -Lipschitz on R d .

Original: C1 can be satisﬁed by the exponential loss, log loss and cross entropy loss under the multi-class setting.
Revised: C1 can be satisﬁed by the exponential loss, log loss and cross entropy loss under the multi-class setting.

Original: For standard deep learning models such as MLP, C2 implies that the activation functions are homogeneous such as ReLU and LeakyReLU, and bias terms are disallowed.
Revised: For standard deep learning models such as MLP, C2 implies that the activation functions are homogeneous such as ReLU and LeakyReLU, and bias terms are disallowed.

Original: C3 is a common technical assumptions whose practical implications we discuss in Appendix A.1.
Revised: C3 is a common technical assumptions whose practical implications we discuss in Appendix A.1.

Original: Among the three necessary conditions, importance weighting only affects C1 by a constant, so its impact on the norm divergence diminishes in the asymptotic regime.
Revised: Among the three necessary conditions, importance weighting only affects C1 by a constant, so its impact on the norm divergence diminishes in the asymptotic regime.

Original: The formal statement is provided as below.
Revised: The formal statement is provided as below.

Original: Claim 1.
Revised: Claim 1.

Original: There exists a constant learning rate for gradient descent, such that for any w ∈ [1 /M, M ] n , with the weak regularization, lim t →∞ (cid:13)(cid:13) θ ( t ) ( w ) (cid:13)(cid:13)
Revised: There exists a constant learning rate for gradient descent, such that for any w ∈ [1 /M, M ] n , with the weak regularization, lim t →∞ (cid:13)(cid:13) θ ( t ) ( w ) (cid:13)(cid:13)

Original: = ∞ under C1 - C3 .
Revised: = ∞ under C1 - C3 .

Original: Compared with the previous work, we extend the norm divergence result not only to weighted
Revised: Compared with the previous work, we extend the norm divergence result not only to weighted

Original: ERM but a more general setting where weak regularization is considered.
Revised: ERM but a more general setting where weak regularization is considered.

Original: We defer the proof to
Revised: We defer the proof to

Original: Appendix A.1.
Revised: Appendix A.1.

Original: A direct consequence of parameter norm divergence is that both the risk and gradient is dominated by the terms with the smallest margin, i.e.  min i y i
Revised: A direct consequence of parameter norm divergence is that both the risk and gradient is dominated by the terms with the smallest margin, i.e. arg min i y i

Original: f
Revised: f

Original: ( θ , x i ) , which are also referred to as the "support vectors".
Revised: ( θ , x i ) , which are also referred to as the "support vectors".

Original: To see this point, notice that both the risk and gradient has the form  (cid:80) i C i exp (cid:0) − y i
Revised: To make sense of this point, notice that both the risk and gradient has the form of: (cid:80) i C i exp (cid:0) − y i

Original: f
Revised: f

Original: ( θ , x i ) (cid:1)  where C i are low-order terms.
Revised: ( θ , x i ) (cid:1) , where C i are low-order terms.

Original: Since f ( θ , x i ) =
Revised: Since f ( θ , x i ) =

Original: (cid:107) θ (cid:107) 2 f (cid:0) θ / (cid:107) θ (cid:107) 2 , x i (cid:1) under C2 , we have lim t →∞ exp (cid:0)
Revised: (cid:107) θ (cid:107) 2 f (cid:0) θ / (cid:107) θ (cid:107) 2 , x i (cid:1) due to the homogeneous assumption in C2 , it holds that:

Original:  − y i f
Revised: lim t →∞ exp (cid:0) − y i f

Original: ( θ ( t ) ( w ) , x i ) (cid:1) → 0 .
Revised: ( θ ( t ) ( w ) , x i ) (cid:1) → 0 .

Original: Therefore, the decision boundaries may share certain characteristics with the support vector machine (SVM) since they "rely on" the same support vectors.
Revised: Therefore, the decision boundaries may share certain characteristics with the support vector machine (SVM) since they rely on the same support vectors.

Original: As a matter of fact, the current understandings on the implicit bias of gradient descent mostly build on the connection with hard-margin 
Revised: As a matter of fact, the current understandings on the implicit bias of gradient descent mostly build on the connection with hard-margin SVM:

Original: [Equation]
Revised: [Equation]

Original: whose solution path coincides with the max-margin problem: max (cid:107)θ (cid:107)
Revised: whose optimization path coincides with the max-margin problem: max (cid:107)

Original: [Equation]
Revised: [Equation]

Original: 1min i =1 ,...,n y i f
Revised: 1 ,...,n y i [Equation]

Original: ( θ , x i ) , as shown by Nacson et al.
Revised: , x [Equation] , as shown by Nacson et al.

Original: We use θ ∗ to denote the optimal solution and γ ∗ = γ ( θ ∗ ) :
Revised: We use θ ∗ to denote the optimal solution and γ ∗ = γ ( θ ∗ ) :

Original: = min i y i f
Revised: = min i y i f

Original: ( θ ∗ , x i ) to denote the correspondingmaximum margin.
Revised: ( θ ∗ , x i ) to denote the corresponding margin.

Original: Implicit bias of gradient descent.
Revised: Implicit bias of gradient descent.

Original: We ﬁrst consider the weight-agnostic setting.
Revised: We start by considering the weight-agnostic setting.

Original: When D is linear separable, it is reasonable to conjecture that the separating hyperplane under a linear f ( θ , · ) overlaps with the solution of hard-margin SVM.
Revised: When D is linear separable, it is reasonable to conjecture that the separating hyperplane under a linear f ( θ , · ) overlaps with the solution of hard-margin SVM.

Original: Soudry et al.
Revised: Soudry et al.

Original: and Ji & Telgarsky (2018b)
Revised: and Ji & Telgarsky (2018b)

Original: ﬁrst show that (cid:107) θ ( t ) (cid:107) converges in direction to θ ∗ ,
Revised: ﬁrst show that (cid:107) θ ( t ) (cid:107) converges in direction to θ ∗ ,

Original: i.e. lim t →∞ θ ( t ) /
Revised: i.e. lim t →∞ θ ( t ) /

Original: (cid:107) θ ( t ) (cid:107) 

Revised: (cid:107) θ ( t ) (cid:107)

Original: =
Revised: =

Original: θ ∗ , which is an equivalent statement since (cid:107) θ ( t ) (cid:107) 2 diverges.
Revised: θ ∗ , which are equivalent statements since (cid:107) θ ( t ) (cid:107)diverges.

Original: For nonlinear predictors, however, the parameter direction is less meaningful.
Revised: For nonlinear predictors, however, the parameter direction is less meaningful.

Original: Instead, it is observed that neural networks often achieve perfect separation of the training data.
Revised: Instead, it has been pointed out that neural networks often achieve perfect separation of the training data (Zhang et al., 2016).

Original: Therefore, we are more interested in the margin whose pivoting role for the generalization of neural networks is well understood (Neyshabur et al., 2017; Bartlett et al., 2017; Golowich et al., 2018).
Revised: Therefore, we are more interested in the margin whose pivoting role for the generalization of neural networks is studied extensively (Neyshabur et al., 2017; Bartlett et al., 2017; Golowich et al., 2018).

Original: Speciﬁcally, it has been show in Nacson et al.
Revised: Speciﬁcally, it has been show in Nacson et al.

Original: and
Revised: and

Original: Lyu & Li (2019) that the normalized margin, deﬁned as ˜ γ ( θ ( t ) ) :
Revised: Lyu & Li (2019) that the normalized margin , deﬁned by ˜ γ ( θ ( t ) ) :

Original: = γ (cid:0) θ ( t ) / (cid:107) θ ( t ) (cid:107) 2 (cid:1) , converges to the maximum margin γ ∗ without regularization.
Revised: = γ (cid:0) θ ( t ) / (cid:107) θ ( t ) (cid:107) 2 (cid:1) , converges to the maximum margin γ ∗ without regularization.

Original: It becomes clear at this point that to understand the role of importance weighting for deep learning, we must characterize the impact of weights on the implicit bias since they reveal the optimization geometry and generalization performance.
Revised: It becomes clear at this point that to understand the role of importance weighting for deep learning, we must characterize the impact of weights on the implicit bias since they reveal the optimization geometry and generalization performance.

Original: Formally, we address the following critical questions.
Revised: Formally, we address the following critical questions.

Original: • Q1 .
Revised: • Q1 .

Original: Does importance weighting modify the convergence results (convergence in direction for linear predictor and in normalized margin for nonlinear predictor)?
Revised: Does importance weighting modify the convergence results (convergence in direction for linear predictor and in normalized margin for nonlinear predictor)?

Original: • If the convergence results are not changed, then: – Q2 .
Revised: • If the convergence results remain unchanged, then: – Q2 .

Original: in what way are the weights affecting the optimization outcome; – Q3 .
Revised: in what way is importance weighting affecting the optimization process; – Q3 .

Original: how can importance weighting inﬂuence the generalization from the source distribution to  target distribution?
Revised: how does importance weighting inﬂuence the generalization from the source distribution to the target distribution?

Original: Experiment setup.
Revised: Experiment setup.

Original: Throughout this paper, we use the regular regression model as linear predictor.
Revised: Throughout this paper, we use the regular regression model as linear predictor.

Original: Nonlinear predictor is given the two-layer MLP with ﬁve hidden units and ReLU as  activation function.
Revised: The nonlinear predictor is given a two-layer MLP with ﬁve hidden units and ReLU as the activation function.

Original: The models are trained with gradient descent using 0.1 as learning rate.
Revised: All the models are trained with gradient descent using 0.1 as learning rate.

Original: We use the exponential loss and random initialization.
Revised: We use the exponential loss and the standard normal initialization.

Original: The data for our illustrative experiments are shown in Figure 1, whichconsists of different setups that correspond to  our major topics.
Revised: The generated datasets for our illustrative experiments are shown in Figure 1, which correspond to the different settings of our major topics.

Original:  I MPORTANCE WEIGHTING FOR LINEAR PREDICTOR
Revised:  I MPORTANCE WEIGHTING FOR LINEAR PREDICTOR

Original: We begin with the linear predictors which allows more reﬁned analysis on the gradient dynamics.
Revised: We begin with the linear predictors which allows more reﬁned analysis on the gradient dynamics.

Original: Without loss of generality, we assume using the exponential loss.
Revised: Without loss of generality, we assume using the exponential loss.

Original: Also, we do not consider the weak regularization here since its practical impact on linear model is trivial when λ → 0 (Rosset et al., 2004a;b), but it is not the case for nonlinear predictors.
Revised: Also, we do not consider the weak regularization here since its practical impact on linear model is trivial when λ → 0 (Rosset et al., 2004a;b), but it is not the case for nonlinear predictors.

Original: One sophistication with linear predictor is that the data may not be perfectly separated, as opposed to the nonlinear case where neural networks can in theory separate any non-degenerate data.
Revised: One sophistication with linear predictor is that the data may not be perfectly separated, as opposed to the nonlinear case where neural networks can in theory separate any non-degenerate data.

Original: With this kept in mind, we ﬁrst assume D is linear separable and characterize the new convergence result in the following proposition.
Revised: With this kept in mind, we ﬁrst assume D is linear separable and characterize the new convergence result in the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: With a constant learning rate η t (cid:46) β − 1 , for any w ∈[1 /M, M ] n , we have
Revised: With a constant learning rate η t (cid:46) β − 1 , we consider normalizing the weights w ∈

Revised: [ 1 M , M ] n such that (cid:80) i w i = 1 without loss of generality, it holds that:

Original: [Equation]
Revised: [Equation]

Original: where p ∗ =
Revised: where p ∗ =

Original: [ p ∗ 1 , . .
Revised: [ p ∗ 1 , . .

Original: . , p ∗ n ] , p ∗ i ≥ 0 and (cid:80) ni

Original: =1 p ∗ i = 1 is the dual optimal for the hard-margin SVM where θ ∗ =
Revised: . , p ∗ n ] characterizes the dual optimal for the hard-margin SVM such that θ ∗ =

Original: (cid:80) n i =

Revised: (cid:80) ni =

Revised: y i x i · p ∗ i and satisﬁes: p ∗ i ≥ 0 and (cid:80) ni

Original: y i x i · p ∗ i , and D KL is the Kullback-Leibler divergence.
Revised: =1 p ∗ i = 1 . Here, D KL is the Kullback-Leibler divergence.

Original: We leave the proof to Appendix A.2.
Revised: We leave the proof to Appendix A.2.

Original: We ﬁnd that importance weighting does not change the convergence result as well as the 1  log t rate, however, it affects the convergence speed under the ﬁnite-step optimization.
Revised: We ﬁnd that importance weighting does not change the convergence result as well as the 1 / log t convergence rate. However, it does affect the convergence speed under the ﬁnite-step optimization.

Original: In particular, we show that the extra constant term induced by importance weighting is given by the KL-divergence between the  weights and the dual optimal of the hard-margin SVM, where samples with small margin usually have larger values.
Revised: In particular, we show that the extra constant term induced by importance weighting is given by the KL-divergence between the (normalized) weights and the dual optimal of the hard-margin SVM, where samples with smaller margins usually have larger values.

Original: Therefore, we are able to accelerate gradient descent in ﬁnite-step optimization by matchingthe weights with  inverse margin.
Revised: Therefore, importance weighting may accelerate gradient descent in ﬁnite-step optimization by matching weights with the inverse margin.

Original: As we show in Figure 2a and 2b, this type of weight design("inverse margin weighted") is able to accelerate the convergence and bring better performance under ﬁnite-step optimization.
Revised: As we show in Figure 2a and 2b, this type of "inversemargin weighted" design is able to accelerate the convergence and bring better performance under ﬁnite-step optimization.

Original: When D is not linear separable, the key insight is that we can always partition D into D sep ∪ D non-sep , where D sep is the maximal linear separable subset as suggested in Ji & Telgarsky (2018b).
Revised: When D is not linearly separable, the key insight is that we can always partition D into D sep ∪D non-sep , where D sep is the maximal linear separable subset as suggested in Ji & Telgarsky (2018b).

Original: Let Π non-sep be the (orthogonal) projection onto the subspace S spanned by the x  in D non-sep , and  Π sep be the projection onto the orthogonal complement S ⊥ .
Revised: Let Π non-sep be the (orthogonal) projection onto the subspace S spanned by the x i in D non-sep , and let Π sep be the projection onto the orthogonal complement S ⊥ .

Original: Now we can study  two projected parts independently since by the deﬁnition, we have θ ( t ) ( w ) =
Revised: The partition allows us to study the two projected parts independently since by the construction, we have θ ( t ) ( w ) =

Original: Π non-sep θ ( t ) ( w ) +
Revised: Π non-sep θ ( t ) ( w ) +

Original: Π sep θ ( t ) ( w ) .
Revised: Π sep θ ( t ) ( w ) .

Original: It is intuitive that  Π sep θ ( t ) ( w ) behaves similarly to the linear separable case  in Proposition 1, so we  focus on the properties of Π non-sep θ ( t ) ( w )  which we summarize in the follow proposition.
Revised: It is intuitive that the optimization path of Π sep θ ( t ) ( w ) behaves similarly to the linear separable case as in Proposition 1, so we can focus on the properties of Π non-sep θ ( t ) ( w ) , which we summarize in the follow proposition.

Original: Proposition 2 (Informal) .
Revised: Proposition 2 (Informal) .

Original: Let L non-sep ( θ , w ) be the weighted risk deﬁned on the non-separable subset, then with the constant learning rate:
Revised: Let L non-sep ( θ , w ) be the weighted risk deﬁned on the non-separable subset, then with the constant learning rate:

Original: • ˜ θ ( w ) =
Revised: • ˜ θ ( w ) =

Original: arg min θ L non-sep ( θ , w ) is uniquely deﬁned and (cid:13)(cid:13)
Revised: arg min θ L non-sep ( θ , w ) is uniquely deﬁned and (cid:13)(cid:13)

Original: ˜ θ ( w ) (cid:13)(cid:13)
Revised: ˜ θ ( w ) (cid:13)(cid:13)

Original: 2 =
Revised: 2 =

Original: O (1) ; • (cid:12)(cid:12) Π non-sep θ ( t ) ( w ) − ˜ θ ( w ) (cid:12)(cid:12) (cid:46) C (cid:0)(cid:13)(cid:13)
Revised: O (1) ; • (cid:12)(cid:12) Π non-sep θ ( t ) ( w ) − ˜ θ ( w ) (cid:12)(cid:12) (cid:46) C (cid:0)(cid:13)(cid:13)

Original: ˜ θ ( w ) (cid:13)(cid:13) 2 (cid:1)
Revised: ˜ θ ( w ) (cid:13)(cid:13) 2 (cid:1)

Original: + log 2 t/γ sep t , where γ sep is the maximum margin on D sep and C (cid:0)(cid:13)(cid:13)
Revised: + log 2 t/γ sep t , where γ sep is the maximum margin on D sep and C (cid:0)(cid:13)(cid:13)

Original: ˜ θ ( w ) (cid:1)(cid:13)(cid:13) 2 (cid:1)
Revised: ˜ θ ( w ) (cid:1)(cid:13)(cid:13) 2 (cid:1)

Original: =
Revised: =

Original: O (1) .
Revised: O (1) .

Original: The formal statement, which involves how D sep is deﬁned, is deferred to Appendix A.2 together with the proof.
Revised: The formal statement, which involves how D sep is deﬁned, is deferred to Appendix A.2 together with the proof.

Original: We see that importance weighting uniquely deﬁnes the solution ˜ θ ( w ) on the non-separable subset of the data, to which Π non-sep θ ( t ) ( w ) converges.
Revised: Proposition 2 informs that importance weighting uniquely deﬁnes the solution ˜ θ ( w ) on the non-separable subset of the data, to which Π non-sep θ ( t ) ( w ) converges.

Original: Hence, we expect lim t →∞ θ ( t ) ( w ) =
Revised: Hence, we expect lim t →∞ θ ( t ) ( w ) =

Original: ˜ θ ( w ) + θ ∗ sep , where θ ∗ sep is the solution on the separable subset D sep and thus its direction does not depend on w as we show above.
Revised: ˜ θ ( w ) + θ ∗ sep , where θ ∗ sep is the solution on the separable subset D sep and thus its direction does not depend on w as implied by Proposition 1.

Original: We can therefore think of ˜ θ ( w ) as the intercept term where the weight controls how the intercept shifts on the subspace of the non-separable data, which we illustrate  in Figure 3.
Revised: We can therefore think of ˜ θ ( w ) as the intercept term where the weight controls how the intercept shifts on the subspace of the non-separable data. We also illustrate this ﬁnding in Figure 3.

Original: By far, we provide an in-depth understanding and our theoretical results fully explain the observations made in Byrd & Lipton (2019) on how importance weighting affects the inductive bias of gradient descent using linear predictors.
Revised: By far, we provide an in-depth understanding and our theoretical results fully explain the observations made in Byrd & Lipton (2019) on how importance weighting affects the inductive bias of gradient descent using linear predictors.

Original:  I MPORTANCE WEIGHTING FOR NONLINEAR PREDICTOR
Revised:  I MPORTANCE WEIGHTING FOR NONLINEAR PREDICTOR

Original: Now we investigate the inﬂuence of importance weighting ona non-linear predictor, e.g, a neural network.
Revised: Now we investigate the inﬂuence of importance weighting on non-linear predictors, e.g, the neural network.

Original: For nonlinear predictors, we are more interested in the regularized setting:
Revised: For nonlinear predictors, we are more interested in the regularized setting:

Original: [Equation]
Revised: [Equation]

Original: where r > 0 is ﬁxed, λ is the regularization coefﬁcient.
Revised: where r > 0 is ﬁxed, λ is the regularization coefﬁcient.

Original: Denote by θ λ ( w ) ∈ arg min L λ ( θ , w ) .
Revised: We use the notation: θ λ ( w ) ∈ arg min L λ ( θ , w ) .

Original: Recall that γ ∗ :=
Revised: Recall that γ ∗ :=

Original: max (cid:107) θ (cid:107)≤ 1 min i y i f
Revised: max (cid:107) θ (cid:107)≤ 1 min i y i f

Original: ( θ , x i ) .
Revised: ( θ , x i ) .

Original: Unlike the linear case, characterizing the gradient dynamics for nonlinear predictor is often insurmountable.
Revised: Unlike the linear case, characterizing the gradient dynamics for nonlinear predictor is often insurmountable.

Original: Therefore, we mainly considereither the asymptotic regime or the regime with sufﬁciently large t .
Revised: Therefore, we mainly consider the asymptotic regime or the regime with sufﬁciently large t .

Original: We omit the superscript in θ ( t ) when there is no confusion.
Revised: We omit the superscript in θ ( t ) when there is no confusion.

Original: The only assumption we need to make is that:
Revised: The only assumption we need to make is that:

Original: A1 .
Revised: A1 .

Original: the data is separated by f at some point during gradient descent, i e. ∃ t > 0 s
Revised: the data is separated by f at some point during gradient descent, i.e. ∃ t > 0 s.t.

Original: t. y if
Revised: y i

Original:  ( θ ( t ) , x i ) > 0 , ∀ i = 1 , . .
Revised: f ( θ ( t ) , x i ) > 0 , ∀ i = 1 , . .

Original: . , n .
Revised: . , n .

Original: In addition, y i f ( θ ∗ , x i ) ≥ γ ∗ > 0 for each i .
Revised: In addition, y i f ( θ ∗ , x i ) ≥ γ ∗ > 0 for each i .

Original: In Section 4.1, we show that by solving the equation 3 with an inﬁnitesimal (weak) regularizer, the solver leads to the optimal margin γ ∗ , regardless of the choice of the importance weighting.
Revised: In Section 4.1, we show that by solving the equation 3 with an inﬁnitesimal (weak) regularizer, gradient descent leads to the optimal margin γ ∗ , regardless of the choice of the importance weights.

Original: In Section 4.2, we show that the the importance weighting affects the generalization bound via a multiplication factor as well as the margin in the ﬁnite-sample scenario.
Revised: In Section 4.2, we show that the the importance weighting affects the generalization bound via a multiplication factor as well as the margin in the ﬁnite-sample scenario.

Original: 4. M ARGIN IS INVARIANT TO IMPORTANCE WEIGHTING UNDER WEAK REGULARIZATION
Revised: 4. M ARGIN IS INVARIANT TO IMPORTANCE WEIGHTING UNDER WEAK REGULARIZATION

Original: We show that for any bounded w , ˜ γ ( θ λ ( w )) converges to γ ∗ as λ decreases to zero.
Revised: We show that for any bounded w , ˜ γ ( θ λ ( w )) converges to γ ∗ as λ decreases to zero.

Original: In practice, however, we might not obtain θ λ ( w ) in limited time.
Revised: In practice, however, we might not obtain θ λ ( w ) in limited time.

Original: It is shown that as long as equation 3 is close enough to its optimum, the margin of the associative θ (cid:48) ( w ) (under ﬁnite-step optimization) is lower bounded by γ ∗ multiplied by a non-trivial factor.
Revised: It is shown that as long as equation 3 is close enough to its optimum, the margin of the associative θ (cid:48) ( w ) (under ﬁnite-step optimization) is lower bounded by γ ∗ multiplied by a non-trivial factor.

Original: Formally,
Revised: Formally,

Original: Proposition 3.
Revised: Proposition 3.

Original: Suppose C1 , C2 , A1 hold.
Revised: Suppose C1 , C2 , A1 hold.

Original: For any w ∈ [1 /M, M ] n , it follows that
Revised: For any w ∈ [1 /M, M ] n , it follows that

Original: • (Asymptotic) lim λ
Revised: • (Asymptotic) lim λ

Original: → 0 ˜ γ ( θ λ ( w ))
Revised: → 0 ˜ γ ( θ λ ( w ))

Original: → γ ∗ .
Revised: → γ ∗ .

Original: • (Finite steps) There exists a λ :=
Revised: • (Finite steps) There exists a λ :=

Original: λ ( r, α, γ ∗ , w , c ) such that for θ (cid:48) ( w ) with L λ ( θ (cid:48) ( w ); w ) ≤ τL λ ( θ λ ( w ); w ) and τ ≤ 2 , the associated margin ˜ γ
Revised: λ ( r, α, γ ∗ , w , c ) such that for θ (cid:48) ( w ) with L λ ( θ (cid:48) ( w ); w ) ≤ τL λ ( θ λ ( w ); w ) and τ ≤ 2 , the associated margin ˜ γ

Original: ( θ (cid:48) ( w )) satisﬁes ˜ γ ( θ (cid:48) ( w )) ≥ c · γ ∗ τ α/r , where 110 ≤ c < 1 .
Revised: ( θ (cid:48) ( w )) satisﬁes ˜ γ ( θ (cid:48) ( w )) ≥ c · γ ∗ τ α/r , where 110 ≤ c < 1 .

Original: This result is adapted from Wei et al.
Revised: This result is adapted from Wei et al.

Original: (2019), which relies on Claim 1.
Revised: (2019), which relies on Claim 1.

Original: The proof is delegated to
Revised: The proof is relegated to

Original: Appendix A.4.1.
Revised: Appendix A.4.1.

Original: We see that importance weightingalso does not affect the asymptotic margin as  λ is sufﬁciently small.
Revised: We see that importance weighting does not affect the asymptotic margin as well when λ is sufﬁciently small.

Original: To get the intuition, note that when (cid:107) θ λ ( w ) (cid:107) is large enough and λ is small enough to be ignored, L λ ( θ λ ( w ) , w ) ≈ exp( −(cid:107) θ λ ( w ) (cid:107) α γ λ ) , which favors a large margin.
Revised: To get the intuition, note that when (cid:107) θ λ ( w ) (cid:107) is large enough and λ is small enough to be ignored, L λ ( θ λ ( w ) , w ) ≈ exp (cid:0) −(cid:107) θ λ ( w ) (cid:107) α γ λ (cid:1) , which favors a large margin.

Original: In addition, even if L λ ( θ , w ) has not yet converged but close enough to its optimum, the corresponding normalized margin has a good lower bound for θ (cid:48) ( w ) .
Revised: In addition, even if L λ ( θ , w ) has not yet converged but close enough to its optimum, the corresponding normalized margin gives a reasonable lower bound for θ (cid:48) ( w ) .

Original: Note that the importance weighting does not only affect the choice of λ to guarantee this.
Revised: We point out that this result does not rely on the choice of λ .

Original: The assumption L λ ( θ (cid:48) ( w ); w ) ≤ τL λ ( θ λ ( w ); w ) has already accounted for the major inﬂuence of importance weighting in terms of  optimization.
Revised: The assumption L λ ( θ (cid:48) ( w ); w ) ≤ τL λ (cid:0) θ λ ( w ); w (cid:1) has already accounted for the major inﬂuence of importance weighting in terms of the optimization.

Original: That is, with a "good" set of importance weights, we can achieve this criteria (by approaching global optimum) faster.
Revised: That is, with a "good" set of importance weights, we can achieve this criteria (by approaching global optimum) faster.

Original: We leave detailed discussions to Section 5.
Revised: We leave detailed discussions to Section 5.

Original: Figure 2d  demonstrates that the choice of the importance weights has a signiﬁcant inﬂuence on the convergence speed of L λ ( θ (cid:48) , w ) .
Revised: Figure 2d also demonstrates that the choice of the importance weights has a signiﬁcant inﬂuence on the convergence speed for the non-linear predictor (given by an feedforward neural network).

Original: 4. I MPORTANCE WEIGHTING AFFECTS THE GENERALIZATION BOUND
Revised: 4. I MPORTANCE WEIGHTING AFFECTS THE GENERALIZATION BOUND

Original: Proposition 3 concentrates on the behavior of the margin corresponding to  optimum of L λ ( θ ; w ) , thus does not rely on the sample size.
Revised: Proposition 3 conjectures on the behavior of the margin corresponding to the optimum of L λ ( θ ; w ) , which does not rely on the sample size.

Original: To bridge the connection betweenthe importance weights and the behavior of f  in the ﬁnite-sample setting, we investigate the generalization bound of f when the training sample distribution deviates from the testing sample distribution.
Revised: To bridge the connection between importance weighting and the behavior of f ( θ , · ) in the ﬁnite-sample setting, we investigate the generalization bound of f when the training sample distribution deviates from the testing sample distribution.

Original: Let P s be the source distribution and P t be the target distribution with the corresponding densities p s ( · ) and p t ( · ) .
Revised: Let P s be the source distribution and P t be the target distribution with the corresponding densities p s ( · ) and p t ( · ) .

Original: Assume  P s and P t have the same support.
Revised: Assume that P s and P t have the same support.

Original: We consider the Pearson χ 2 -divergence to measure the difference between P s and P t , i.e., D ( P || Q ) =
Revised: We consider the Pearson χ 2divergence to measure the difference between P s and P t , i.e., D ( P t (cid:107) P t ) =

Original: (cid:82) [( dPdQ ) 2 − 1] dQ .
Revised: (cid:82) (cid:2) ( dP s /dP t ) 2 − 1 (cid:3) dP s .

Original: The training covariates x 1 , . . .
Revised: The training covariates x 1 , . . .

Original: , x n are generated from P s , and the testing covariates are generated from P t .
Revised: , x n are generated from P s , and the testing covariates are generated from P t .

Original: Denote by p train and p test the joint distribution of ( x , y ) for the training data and the testing data respectively.
Revised: Denote by p train and p test the joint distribution of ( x , y ) for the training data and the testing data, respectively.

Original: We minimize equation 3 over an H -layer feedforward neural network  f NN ( θ , x ) :=
Revised: We minimize equation 3 over the H -layer feedforward neural network given by f NN ( θ , x ) :=

Original: W h σ ( W h − 1 σ ( · · · σ ( W 1 x ) · · · )) , where θ =
Revised: W h σ ( W h − 1 σ ( · · · σ ( W 1 x ) · · · )) , where θ =

Original: [ W 1 , · · · , W H ]  and σ  is an element-wise activation function.
Revised: [ W 1 , · · · , W H ] are the parameter matrices and σ ( · ) is the element-wise activation function such as ReLU.

Original: Denote  η ( x ) =
Revised: Denote by η ( x ) =

Original: p t ( x ) p s ( x ) .
Revised: p t ( x ) /p s ( x ) .

Original: We show  the generalization bound is affected bythe importance sampling via the interplay between the empirical risk that hinges on η and a term that depends on the model complexity and the deviation of the target distribution from the source distribution.
Revised: We show that the generalization performance is affected by importance weighting via the interplay between the empirical risk that hinges on η , as well as a term that depends on the model complexity and the deviation of the target distribution from the source distribution.

Original: Theorem 1 (1) .
Revised: Theorem 1 (1) .

Original: Assume σ is 1 -Lipschitz and 1 -positive homogeneous.
Revised: Assume σ is 1 -Lipschitz and 1 -positive homogeneous.

Original: Then with probability at least 1 − δ , we have
Revised: Then with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original:  where (I) is the empirical risk, (II) reﬂects the compounding effect of the model complexity of the class of H -layer neural networks and the deviation between target distribution and source distribution , (cid:15)
Revised: [Equation] where (I) is the empirical risk, (II) reﬂects the compounding effect of the model complexity of the class of H -layer neural networks and the deviation between target distribution and source distribution , (cid:15)

Original: [Equation]
Revised: [Equation]

Original: is a small quantity compared to (I) and (II).
Revised: is a small quantity compared to (I) and (II).

Original: Here C :=
Revised: Here, C :=

Original: sup x (cid:107) x (cid:107) ; γ is any positive value.
Revised: sup x (cid:107) x (cid:107) and γ can take any positive value.

Original: The proof is deferred to Appendix A.4.2.
Revised: The proof is deferred to Appendix A.4.2.

Original: Compared to Wei et al.
Revised: Compared to Wei et al.

Original: (2019), the empirical risk (I) hinges on η and there is an additional multiplier factor (cid:112) D
Revised: (2019), the empirical risk (I) hinges on η and there is an additional multiplier factor (cid:112) D

Original: ( P t ||
Revised: ( P t ||

Original: P s )
Revised: P s )

Original: + 1 on (II).
Revised: + 1 on (II).

Original: From thebelow two points, we argue that the role of importance weighting on the generalization bound in Theorem 1 is not only reﬂected in how the margin is affected, but also how it balances source and target distribution: 1.
Revised: In the two discussions below, we argue that the role of importance weighting on the generalization bound in Theorem 1 is not only reﬂected in how the margin is affected, but also how it balances source and target distribution: 1.

Original: Suppose θ ( w ) enables f NN to separate the data.
Revised: Suppose θ ( w ) enables f NN to separate the data.

Original: Let γ θ ( w ) :
Revised: Let γ θ ( w ) :

Original: = min i y i f
Revised: = min i y i f

Original: NN ( θ ( w ) / (cid:107) θ ( w ) (cid:107) , x i ) .
Revised: NN (cid:0) θ ( w ) / (cid:107) θ ( w ) (cid:107) , x i (cid:1) .

Original: In the generalization bound of Theorem 1,  let γ = γ θ ( w ) . Then (I) vanishes and only (II) remains.
Revised: In the generalization bound of Theorem 1, if we let γ = γ θ ( w ) , then (I) vanishes and only (II) remains.

Original: Then the importance weights affects the generalization bound via γ θ ( w ) in ﬁnite steps as discussed in Section 4.1.
Revised: In this case, the importance weights affects the generalization bound via γ θ ( w ) in ﬁnite steps as discussed in Section 4.1.

Original: In ﬁnite training steps, a good set of weights w can lead to a large γ θ ( w ) ,  thus giving a better generalization upper bound.
Revised: Under ﬁnite training steps, a good set of weights w can lead to an improved γ θ ( w ) , and thus giving a better generalization performance.

Original: Also note that Theorem 1 holds for non-separated cases as well.
Revised: Also note that Theorem 1 holds for the non-separable cases as well.

Original: Note that (II) is a strictly decreasing function,  (I) is a non-decreasing step function with respect to γ .
Revised: We point out that (II) is a strictly decreasing function, while (I) is a non-decreasing step function with respect to γ .

Original: There must exists a trade-off γ that minimizes the sum of (I) and (II), which is usually attained at some γ > γ θ ( w ) .
Revised: Therefore, there must exists a trade-off γ that minimizes the sum of (I) and (II), which is usually attained at some γ > γ θ ( w ) .

Original: When γ grows, some samples will activate I ( y i f
Revised: When γ grows, certain samples will activate I ( y i f

Original: NN ( θ ( w ) / (cid:107) θ ( w ) (cid:107) , x i ) < γ ) .
Revised: NN ( θ ( w ) / (cid:107) θ ( w ) (cid:107) , x i ) < γ ) and inﬂate (I).

Original: We hope  that an initially activated indicator in (I) corresponds to a small η ( x i )  while one with a large η ( x i (cid:48) ) has a large value of y i (cid:48)
Revised: The hope is that an initially activated sample (indicator term) in (I) corresponds to a small η ( x i ) , while one with a large η ( x i (cid:48) ) has a large value of y i (cid:48)

Original: f NN ( θ ( w ) / (cid:107) θ ( w ) (cid:107) , x i (cid:48) ) , thus will be activated later.
Revised: f NN ( θ ( w ) / (cid:107) θ ( w ) (cid:107) , x i (cid:48) ) and thus will be activated later.

Original: This can be achieved by aligning w with η because a large weight on sample i forces the decision boundary of  f NN ( θ ( w ) / (cid:107) θ ( w ) (cid:107) , · ) to be away from this point.
Revised: This can be achieved by aligning w with η because a large weight on sample i forces the decision boundary of to drift away from this data point and gives a larger value of y i f NN ( θ ( w ) / (cid:107) θ ( w ) (cid:107) , x i ) .

Original: Therefore, the generalization bound with w aligning with η can be smaller than that with w deviating from η .
Revised: Therefore, the generalization bound with w aligning with η can be smaller than that with w deviating from η .

Original: The empirical results in Figure 2c provides the numerical evidence that reﬂects the strong effects of importance weighting on the generalization behavior.
Revised: The empirical results in Figure 2c provides the numerical evidence that reﬂects the strong effects of importance weighting on the generalization behavior.

Original:  E XTENSION
Revised:  E XTENSION

Original: What makes a good set of weights for learning?
Revised: What makes a good set of weights for learning?

Original: We show in both Section 3 and 4 that importance weighting can affect how fast the classiﬁer separates the data and converges to the max-margin solution.
Revised: We show in both Section 3 and 4 that importance weighting can affect how fast the classiﬁer separates the data and converges to the max-margin solution.

Original: We also justify how the small-margin support vectors, who can think of as the hard-to-classify data points, are of signiﬁcant importance.
Revised: We also justify how the small-margin support vectors, who can think of as the hard-to-classify data points, are of signiﬁcant importance.

Original: Imagine that we have access to an oracle that outputs the distance of each sample to the max-margin decision boundary.
Revised: Imagine that we have access to an oracle that outputs the distance of each sample to the max-margin decision boundary.

Original: It is intuitive that by putting more weights on the small-margin samples, we "inform" gradient descent of their importance from the beginning and therefore accelerates the optimization.
Revised: It is intuitive that by putting more weights on the small-margin samples, we "inform" gradient descent of their importance from the beginning and therefore accelerates the optimization.

Original: We also provide a rigorous result for linear predictor in Proposition 1.
Revised: We also provide a rigorous result for linear predictor in Proposition 1.

Original: Our high-level intuition justiﬁes a number of topics where people use various methods to measure the hardness of classifying a sample and use that as the weight, explicitly or implicitly.
Revised: Our high-level intuition justiﬁes a number of methodologies where people use various methods to measure the hardness of classifying a sample and use that as the weight, explicitly or implicitly.

Original: Examplesof such work include the curriculum learning (Bengio et al., 2009), mentor net (Jiang et al., 2018), co-teaching (Han et al., 2018) and knowledge distillation (Li et al., 2017; Hinton et al., 2015), where auxiliary models are employed (replacing the oracle) to represent the hardness of each data point.
Revised: Examples include the curriculum learning (Bengio et al., 2009), mentor net (Jiang et al., 2018), co-teaching (Han et al., 2018) and knowledge distillation (Li et al., 2017; Hinton et al., 2015), where auxiliary models are employed (replacing the oracle) to represent the hardness of each data point.

Original: The effect of jointly optimize a weighting model
Revised: The effect of jointly optimize a weighting model

Original: It is not unusual that the importance weights, when depending on another model, is jointly trained with the classiﬁer to achieve an better overall performance, such as the counterfactual modelling (Schnabel et al., 2016) and learning from noisy labels (Song et al., 2020).
Revised: It is not unusual that the importance weights, when depending on another model, is jointly trained with the classiﬁer to achieve an better overall performance, such as the counterfactual modelling (Schnabel et al., 2016; Xu et al., 2020) and learning from noisy labels (Song et al., 2020).

Original: We study the general setup:
Revised: For the illustration purpose, we consider the following setup:

Original: [Equation]
Revised: [Equation]

Original: According to our main results, it is not difﬁcult to conjecture that if the data is separable by f , the convergence of f to the max-margin solution  still holds as the weight model g ( ψ , x i ) gradually concentrates to a constant for all i = 1 , . . . , n .
Revised: where g ( ψ , x i ) is the weighting model. By our main results, it is not difﬁcult to conjecture that if the data is separable by f , the convergence of f to the max-margin solution will still hold and the weighting model g ( ψ , x i ) will concentrate to a constant for all i = 1 , . . . , n .

Revised: This is because the general convergence results are agnostic to the weights, so the weighting model will eventually be nulliﬁed.

Original: Also, during the beginning phase of training, the learnt weights may correlate negatively to the margin, and that correlation will diminish eventually.
Revised: Also, during the beginning phase of training, the learnt weights may correlate negatively to the margin (as it helps to speed up the convergence), and the correlation will diminish eventually as the weights converge to the same constant.

Original: The conjecture is supported by the empirical evidence that we show in Figure 4.
Revised: The above conjectures are supported by the empirical evidence that we discuss in Figure 4.

Revised: Therefore, jointly optimizing the weighting model may not change the convergence result but the speed of convergence.

Original: Interaction with explicit regularizations
Revised: Interaction with explicit regularizations

Original: Deep learning models are often trained with explicit regularization.
Revised: Deep learning models are often trained with explicit regularization.

Original: To see how they interact with importance weighting, we ﬁrst check weather they alter the norm divergence in Claim 1.
Revised: To see how they interact with importance weighting, we ﬁrst check weather they alter the norm divergence in Claim 1.

Original: It is obvious that both the early stopping and strong regularization on (cid:107) θ (cid:107) prohibits the norm divergence, so f ( θ , · ) will not achieve the max-margin solution or even separate the training data.
Revised: It is obvious that both the early stopping and strong regularization on (cid:107) θ (cid:107) prohibits the norm divergence, so f ( θ , · ) will not achieve the max-margin solution or even separate the training data.

Original: In such cases, as it has been observed by Byrd & Lipton (2019), the impact of importance weighting on θ λ ( w ) and ˜ γ ( θ λ ( w )) will be signiﬁcant.
Revised: In such cases, as it has been observed by Byrd & Lipton (2019), the impact of importance weighting on θ λ ( w ) and g ( x = σ ( ψ (cid:124) x i + b ) + σ ( · ) sigmoid issues

Revised: ˜ γ ( θ λ ( w )) will be signiﬁcant.

Original: However, this may not help generalization according to our arguments in Section 4.2.
Revised: However, this may not help generalization according to our arguments in Section 4.2, since the margins will be altered as well.

Original: Indeed, Zhang et al.
Revised: Indeed, Zhang et al.

Original: (2016) shows that explicit regularizations may not lead to better generalization for neural networks.
Revised: (2016) shows that explicit regularizations may not lead to better generalization for neural networks.

Original: For the weighted ERM, Theorem 1 provides a powerful tool to characterize the trade-off induced by explicit regularizations via the lens of margin.
Revised: For the weighted ERM, Theorem 1 provides a powerful tool to characterize the trade-off induced by explicit regularizations via the lens of margin.

Original: Dropout, as an counter example, does not prohibit norm divergence and may not interfere with our main conclusions.
Revised: Dropout, as an counter example, does not prohibit norm divergence and may not interfere with our main conclusions.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this paper, we study the impact of importance weighting on the implicit bias of gradient descent as well as the generalization performance.
Revised: In this paper, we study the impact of importance weighting on the implicit bias of gradient descent as well as the generalization performance.

Original: Based on our theoretical ﬁndings, we propose the following future directions that are worth investigating from both the application and theoretical perspective: 1)
Revised: Based on our theoretical ﬁndings, we propose the following future directions that are worth investigating from both the application and theoretical perspective: 1)

Original: Is there an optimal way to construct importance weights using such as the oracle margin? 2)
Revised: Is there an optimal way to construct importance weights using such as the oracle margin? 2)

Original: How to correctly understand and utilize the role of a jointly-trained weighting model?
Revised: How to correctly understand and utilize the role of a jointly-trained weighting model?

Original: What is the combined effect of importance weighting and explicit regularizations for deep learning models?
Revised: What is the combined effect of importance weighting and explicit regularizations for deep learning models?

Original: A A PPENDIX
Revised: A A PPENDIX

Original: We provide the omitted discussions, proofs, and extra numerical results in the appendix.
Revised: We provide the omitted discussions, proofs, and extra numerical results in the appendix.

Original: A. S UPPLEMENTARY MATERIAL FOR S ECTION
Revised: A. S UPPLEMENTARY MATERIAL FOR S ECTION

Original: We discuss the exponential-tail behavior for loss functions, the piratical implication of condition C and the proof of Claim 1.
Revised: We discuss the exponential-tail behavior for loss functions, the piratical implication of condition C and the proof of Claim 1.

Original: A.1. L OSS FUNCTION WITH EXPONENTIAL - TAIL BEHAVIOR
Revised: A.1. L OSS FUNCTION WITH EXPONENTIAL - TAIL BEHAVIOR

Original: Having a exponential decay on the tail of the loss function is essential for realizing the implicit bias of gradient descent, since we need (cid:96) ( u ) behave like exp( − u ) as u → ∞ .
Revised: Having a exponential decay on the tail of the loss function is essential for realizing the implicit bias of gradient descent, since we need (cid:96) ( u ) behave like exp( − u ) as u → ∞ .

Original: Soudry et al.
Revised: Soudry et al.

Original: ﬁrst propose the notion of tight exponential tail , where the negative loss derivative − (cid:96) (cid:48) ( u ) behave like: − (cid:96) [Equation] 1 + [Equation]
Revised: ﬁrst propose the notion of tight exponential tail , where the negative loss derivative − (cid:96) (cid:48) ( u ) behave like: − (cid:96) (cid:48) ( u ) (cid:46) (cid:0) 1 + exp( − c 1 u ) (cid:1) e − u and − (cid:96) (cid:48) ( u ) (cid:38) (cid:0) 1 − exp( − c 2 u ) (cid:1) e − u , for sufﬁciently large u , where c 1 and c 2 are positive constants.

Original: [Equation]

Original: e

Original: [Equation]

Original: u and − (cid:96)

Original: [Equation]

Original: u

Original: [Equation]

Original: −

Original: [Equation]

Original: e − u , for sufﬁciently large u , where c 1 and c 2 are positive constants.

Original: There is also a smoothness assumption on (cid:96) ( · ) .
Revised: There is also a smoothness assumption on (cid:96) ( · ) .

Original: It is obvious that under this deﬁnition, the tail behavior of the loss function is constraint from both sides by exponential-type functions.
Revised: It is obvious that under this deﬁnition, the tail behavior of the loss function is constraint from both sides by exponential-type functions.

Original: There is a more general (and perhaps more direct) deﬁnition of exponential-tail loss function Lyu &
Revised: There is a more general (and perhaps more direct) deﬁnition of exponential-tail loss function Lyu &

Original: Li (2019), where (cid:96) ( u ) = exp( − f ( u )) , such that:
Revised: Li (2019), where (cid:96) ( u ) = exp( − f ( u )) , such that:

Original: • f is smooth and f (cid:48)
Revised: • f is smooth and f (cid:48)

Original: ( u ) ≥ 0 , ∀ u ; • there exists c > 0 such that f (cid:48) ( u ) u is non-decreasing for u > c and f (cid:48) ( u ) u → ∞ as u → ∞ .
Revised: ( u ) ≥ 0 , ∀ u ; • there exists c > 0 such that f (cid:48) ( u ) u is non-decreasing for u > c and f (cid:48) ( u ) u → ∞ as u → ∞ .

Original: It is easy to verify that the exponential loss, log loss and cross-entropy loss satisfy both deﬁnitions.
Revised: It is easy to verify that the exponential loss, log loss and cross-entropy loss satisfy both deﬁnitions.

Original: Since our focus is not to study the implicit bias of gradient descent, it sufﬁce to work with the above loss functions.
Revised: Since our focus is not to study the implicit bias of gradient descent, it sufﬁce to work with the above loss functions.

Original: A.1. P RACTICAL IMPLICATIONS OF CONDITION C 
Revised: A.1. P RACTICAL IMPLICATIONS OF CONDITION C

Original: C3 asserts the Lipschitz and smoothness properties.
Revised: C3 asserts the Lipschitz and smoothness properties.

Original: The Lipschitz condition is rather mild assumption for neural networks, and several recent paper are dedicated to obtaining the Lipschitz constant of certain deep learning models (Fazlyab et al., 2019; Virmaux & Scaman, 2018).
Revised: The Lipschitz condition is rather mild assumption for neural networks, and several recent paper are dedicated to obtaining the Lipschitz constant of certain deep learning models (Fazlyab et al., 2019; Virmaux & Scaman, 2018).

Original: The β -smooth condition, on the other hand, is more technical-driven such that we can analyze the gradient descent.
Revised: The β -smooth condition, on the other hand, is more technical-driven such that we can analyze the gradient descent.

Original: In practice, neural networks with ReLU activation do not satisfy the smoothness condition.
Revised: In practice, neural networks with ReLU activation do not satisfy the smoothness condition.

Original: However, there are smooth homogeneous activation functions, such as the quadratic activation σ ( x ) =
Revised: However, there are smooth homogeneous activation functions, such as the quadratic activation σ ( x ) =

Original: x 2 and higher-order ReLU activation σ ( x ) =
Revised: x 2 and higher-order ReLU activation σ ( x ) =

Original: ReLU ( x ) c for c > 2 .
Revised: ReLU ( x ) c for c > 2 .

Original: Still, in our experiments, we use ReLU as the activation function for its convenience.
Revised: Still, in our experiments, we use ReLU as the activation function for its convenience.

Original: A.1. P ROOF FOR C LAIM
Revised: A.1. P ROOF FOR C LAIM

Original: Soudry et al.
Revised: Soudry et al.

Original: and Ji & Telgarsky (2018b) show norm divergence for linear predictors, and the follow-up work by Ji & Telgarsky (2018a)
Revised: and Ji & Telgarsky (2018b) show norm divergence for linear predictors, and the follow-up work by Ji & Telgarsky (2018a)

Original: ; Gunasekar et al.
Revised: ; Gunasekar et al.

Original: (2018) extend the result to linear neural networks.
Revised: (2018) extend the result to linear neural networks.

Original: For nonlinear predictors such as multi-layer neural network with homogeneous activation, Nacson et al.
Revised: For nonlinear predictors such as multi-layer neural network with homogeneous activation, Nacson et al.

Original: and Lyu & Li (2019) prove the norm divergence for gradient descent in the absence of explicit regularization.
Revised: and Lyu & Li (2019) prove the norm divergence for gradient descent in the absence of explicit regularization.

Original: Rosset et al. (2004a) and Wei et al.
Revised: Rosset et al. (2004a) and Wei et al.

Original: (2019) considers the weak regularization for linear and nonlinear predictors, however, they only study the property of the critical points instead of the gradient descent sequence.
Revised: (2019) considers the weak regularization for linear and nonlinear predictors, however, they only study the property of the critical points instead of the gradient descent sequence.

Original: Proof.
Revised: Proof.

Original: We ﬁrst state a technical lemma that characterizes the dynamics of gradient descent.
Revised: We ﬁrst state a technical lemma that characterizes the dynamics of gradient descent.

Original: Lemma A.1 (Theorem E.10 of Lyu & Li (2019)) .
Revised: Lemma A.1 (Theorem E.10 of Lyu & Li (2019)) .

Original: Under the conditions that:
Revised: Under the conditions that:

Original: • (cid:96) ( · ) is given by the exponential loss, and (cid:96) ◦ f ( · , x ) is a smooth function on R d for all x ∈ X ;
Revised: • (cid:96) ( · ) is given by the exponential loss, and (cid:96) ◦ f ( · , x ) is a smooth function on R d for all x ∈ X ;

Original: • f ( θ , x ) is α -homogeneous as in C2 ; • the data is separated by f during gradient descent at some point t 0 ; • the learning rate satisfy η t :=
Revised: • f ( θ , x ) is α -homogeneous as in C2 ; • the data is separated by f during gradient descent at some point t 0 ; • the learning rate satisfy η t :=

Original: η 0 (cid:46) (cid:16) L ( θ ( t ) ; w ) log (cid:0) 1 /L
Revised: η 0 (cid:46) (cid:16) L ( θ ( t ) ; w ) log (cid:0) 1 /L

Original: ( θ ( t ) ; w ) (cid:1) 3 − 2 /α (cid:17) −  for all t ,
Revised: ( θ ( t ) ; w ) (cid:1) 3 − 2 /α (cid:17) − 1 for all t ,

Original: then under exponential loss we have:
Revised: then under exponential loss we have:

Original: [Equation]
Revised: [Equation]

Original: To use the results of Lemma A.1, we simply need to show two things for weak regularization:
Revised: To use the results of Lemma A.1, we simply need to show two things for weak regularization:

Original: • the total risk is still smooth and we still can achieve zero risk; • there exists a critical (stationary) point such that lim λ
Revised: • the total risk is still smooth and we still can achieve zero risk; • there exists a critical (stationary) point such that lim λ

Original: → 0 L λ ( θ ∗ ; w ) = 0 .
Revised: → 0 L λ ( θ ∗ ; w ) = 0 .

Original: Notice that the risk without regularization is a smooth function in terms of θ for all x , since the composition of smooth functions is still smooth.
Revised: Notice that the risk without regularization is a smooth function in terms of θ for all x , since the composition of smooth functions is still smooth.

Original: It is easy to see that adding a weak regularization, e.g. λ (cid:107) θ (cid:107) r 2 for r > 1 , does not alter the smoothness condition as λ → 0 .
Revised: It is easy to see that adding a weak regularization, e.g. λ (cid:107) θ (cid:107) r 2 for r > 1 , does not alter the smoothness condition as λ → 0 .

Original: However, the weak (cid:96)regularization will make the total risk non-smooth, and therefore we have excluded it from our discussion.
Revised: However, the weak (cid:96)regularization will make the total risk non-smooth, and therefore we have excluded it from our discussion.

Original: For the second point, it is obvious that (cid:107) θ (cid:107) 2 → ∞ is a critical point under exponential loss when λ → 0 .
Revised: For the second point, it is obvious that (cid:107) θ (cid:107) 2 → ∞ is a critical point under exponential loss when λ → 0 .

Original: Recall that:
Revised: Recall that:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Therefore, for both the loss function and gradient, the main term decreases exponentially fast as (cid:107) θ (cid:107)increases, while the remainder terms are only polynomial in (cid:107) θ (cid:107) 2 , so we can always ﬁnd a small enough λ that satisfy: lim λ → 0 lim (cid:107) θ (cid:107)→∞ L λ ( θ ; w ) = 0 and lim λ
Revised: Therefore, for both the loss function and gradient, the main term decreases exponentially fast as (cid:107) θ (cid:107)increases, while the remainder terms are only polynomial in (cid:107) θ (cid:107) 2 , so we can always ﬁnd a small enough λ that satisfy: lim λ → 0 lim (cid:107) θ (cid:107)→∞ L λ ( θ ; w ) = 0 and lim λ

Original: → 0 lim
Revised: → 0 lim

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: ∇ L λ ( θ ; w ) = 0 , in the same fashion as we show in the (A.1) below.
Revised: ∇ L λ ( θ ; w ) = 0 , in the same fashion as we show in the (A.1) below.

Original: From a standard result of gradient descent on smooth function, which we summarize in Lemma A.2, gradient descent will always converge to a critical (stationary) point for the weighted ERM problem.
Revised: From a standard result of gradient descent on smooth function, which we summarize in Lemma A.2, gradient descent will always converge to a critical (stationary) point for the weighted ERM problem.

Original: Lemma A.2 (Lemma 10 of Soudry et al.
Revised: Lemma A.2 (Lemma 10 of Soudry et al.

Original: Let L λ
Revised: Let L λ

Original: ( θ ; w ) be a B ( w ) -smooth non-negative objective.
Revised: ( θ ; w ) be a B ( w ) -smooth non-negative objective.

Original: With a constant learning rate η 0 (cid:46) B ( w ) − 1 , the gradient descent sequence satisﬁes:
Revised: With a constant learning rate η 0 (cid:46) B ( w ) − 1 , the gradient descent sequence satisﬁes:

Original: • lim t →∞ (cid:80) ti =1 (cid:13)(cid:13) ∇ L λ ( θ ( t ) ; w ) (cid:13)(cid:13) < ∞ ; • lim t →∞ ∇ L λ ( θ ( t ) ; w ) = 0 .
Revised: • lim t →∞ (cid:80) ti =1 (cid:13)(cid:13) ∇ L λ ( θ ( t ) ; w ) (cid:13)(cid:13) < ∞ ; • lim t →∞ ∇ L λ ( θ ( t ) ; w ) = 0 .

Original: Now we need to show that under appropriate learning rate, which is speciﬁed in Lemma A.1, gradient descent converges to the stationary point that corresponds to the zero risk under weak regularization.
Revised: Now we need to show that under appropriate learning rate, which is speciﬁed in Lemma A.1, gradient descent converges to the stationary point that corresponds to the zero risk under weak regularization.

Original: Using the result from Lemma A.1, notice that if L λ ( θ ( t ) ; w ) does not decrease to 0, then the denominator L λ ( θ ( t ) ; w ) 2 (cid:0) log
Revised: Using the result from Lemma A.1, notice that if L λ ( θ ( t ) ; w ) does not decrease to 0, then the denominator L λ ( θ ( t ) ; w ) 2 (cid:0) log

Original: [Equation]
Revised: [Equation]

Original: is bounded
Revised: is bounded

Original: from below.
Revised: from below.

Original: However, there exists a constant learning rate such that (cid:80) ti = t 0 η i → ∞ as t → ∞ , which leads to contradiction.
Revised: However, there exists a constant learning rate such that (cid:80) ti = t 0 η i → ∞ as t → ∞ , which leads to contradiction.

Original: Therefore, for weighted ERM with weak regularization, gradient descent converges to the stationary point where L λ ( θ ( t ) ; w )
Revised: Therefore, for weighted ERM with weak regularization, gradient descent converges to the stationary point where L λ ( θ ( t ) ; w )

Original: = 0 .
Revised: = 0 .

Original: Finally, we show to make L λ ( θ ( t ) ; w ) → 0 , we must have (cid:107) θ ( t ) ( w ) (cid:107) → ∞ .
Revised: Finally, we show to make L λ ( θ ( t ) ; w ) → 0 , we must have (cid:107) θ ( t ) ( w ) (cid:107) → ∞ .

Original: We show by contradiction.
Revised: We show by contradiction.

Original: Suppose (cid:107) θ ( t ) ; w ) (cid:107) is bounded from above by some constant C > 0 , for all λ < ˜ λ that we choose later.
Revised: Suppose (cid:107) θ ( t ) ; w ) (cid:107) is bounded from above by some constant C > 0 , for all λ < ˜ λ that we choose later.

Original: So the loss function for each sample i is bounded below by a positive value that depends on C : w i exp( − y i
Revised: So the loss function for each sample i is bounded below by a positive value that depends on C : w i exp( − y i

Original: f
Revised: f

Original: ( θ ( t ) , x )) ≥ l ( C ) > 0 .
Revised: ( θ ( t ) , x )) ≥ l ( C ) > 0 .

Original: Hence, let K :=
Revised: Hence, let K :=

Original: ˜ λ − 1 / ( r +1) , then
Revised: ˜ λ − 1 / ( r +1) , then

Original: [Equation]
Revised: [Equation]

Original: and it easy obvious that RHS → 0 for a sufﬁciently small ˜ λ , which contradicts l ( C ) > 0 .
Revised: and it easy obvious that RHS → 0 for a sufﬁciently small ˜ λ , which contradicts l ( C ) > 0 .

Original: Hence, we have (cid:107) θ ( t ) ( w ) (cid:107) → ∞ for all all λ < ˜ λ , which completes the proof.
Revised: Hence, we have (cid:107) θ ( t ) ( w ) (cid:107) → ∞ for all all λ < ˜ λ , which completes the proof.

Original: A. S UPPLEMENTARY MATERIAL FOR S ECTION
Revised: A. S UPPLEMENTARY MATERIAL FOR S ECTION

Original: We provide the proofs for Proposition 1 and 2 in this part of the appendix.
Revised: We provide the proofs for Proposition 1 and 2 in this part of the appendix.

Original: A.2. P ROOF FOR P ROPOSITION
Revised: A.2. P ROOF FOR P ROPOSITION

Original: Proof.
Revised: Proof.

Original: We ﬁrst characterize the 1 / log t rate using asymptotic arguments similar to that of Soudry et al.
Revised: We ﬁrst characterize the 1 / log t rate using asymptotic arguments similar to that of Soudry et al.

Original: The key purpose here is to rigorously show that importance weighting plays a negligible role in the asymptotic regime.
Revised: The key purpose here is to rigorously show that importance weighting plays a negligible role in the asymptotic regime.

Original: Let δ ( t ) be the residual term at step t :
Revised: Let δ ( t ) be the residual term at step t :

Original: [Equation]
Revised: [Equation]

Original: To show the 1 / log t rate, we simply need to prove that (cid:107) δ
Revised: To show the 1 / log t rate, we simply need to prove that (cid:107) δ

Original: ( t, w ) (cid:107) is bounded for any w ∈ [1 /M, M ] n .
Revised: ( t, w ) (cid:107) is bounded for any w ∈ [1 /M, M ] n .

Original: Notice that
Revised: Notice that

Original: [Equation]

Original: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst term, we have:
Revised: For the ﬁrst term, we have:

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation]

Original: where in the last line we use:
Revised: where in the last line we use:

Original: • ∀ u > 0 , log(1 + u ) ≤ u ; • ( θ ∗ ) (cid:124) ∇ L (cid:0) θ ( t ) ( w ); w (cid:1) =
Revised: • ∀ u > 0 , log(1 + u ) ≤ u ; • ( θ ∗ ) (cid:124) ∇ L (cid:0) θ ( t ) ( w ); w (cid:1) =

Original: (cid:80) i − w i exp( − y
Revised: (cid:80) i − w i exp( − y

Original: i θ ∗ x i )
Revised: i θ ∗ x i )

Original: y
Revised: y

Original: i θ ∗ x i ≤ 0 because θ ∗
Revised: i θ ∗ x i ≤ 0 because θ ∗

Original: separates the data.
Revised: separates the data.

Original: Also, from the ﬁrst conclusion of Lemma A.2, we see that (cid:13)(cid:13) ∇ L (cid:0) θ ( t ) ( w ); w (cid:1)(cid:13)(cid:13) =
Revised: Also, from the ﬁrst conclusion of Lemma A.2, we see that (cid:13)(cid:13) ∇ L (cid:0) θ ( t ) ( w ); w (cid:1)(cid:13)(cid:13) =

Original: o (1 /t ) , so (cid:13)(cid:13)
Revised: o (1 /t ) , so (cid:13)(cid:13)

Original: δ ( t [Equation] 1 , w ) {+− δ ( t, w ) (cid:13)(cid:13) 
 = o+}
Revised: δ ( t + 1 , w ) {+− δ ( t, w ) (cid:13)(cid:13) 
 = o+}

Original: [Equation]
Revised: [Equation]

Original: [-t, w ) (cid:13)(cid:13) 
-]
Revised: [-t, w ) (cid:13)(cid:13) 
-]

Original: = o (1 /t ) and the running sum converges to some ﬁnite number:
Revised: 1 /t ) and the running sum converges to some ﬁnite number:

Original: [Equation]
Revised: [Equation]

Original: We see that the role of the weights is totally negligible because θ ∗ separates the data (the second bullet point above).
Revised: We see that the role of the weights is totally negligible because θ ∗ separates the data (the second bullet point above).

Original: The same argument applies to the second term 2 (cid:0) δ ( t +1 , w ) − δ ( t, w ) (cid:1) (cid:124) δ ( t, w ) , where w plays no part as long as θ ∗ separates the data.
Revised: The same argument applies to the second term 2 (cid:0) δ ( t +1 , w ) − δ ( t, w ) (cid:1) (cid:124) δ ( t, w ) , where w plays no part as long as θ ∗ separates the data.

Original: The detailed proof is technical, and we refer to Lemma 6 of Soudry et al.
Revised: The detailed proof is technical, and we refer to Lemma 6 of Soudry et al.

Original: (2018), which states that:
Revised: (2018), which states that:

Original: [Equation]
Revised: [Equation]

Original: Therefore, by tensorization we see that:
Revised: Therefore, by applying tensorization, it holds that:

Original: [Equation]
Revised: [Equation]

Original: hence (cid:13)(cid:13) δ ( t, w ) (cid:13)(cid:13) is bounded and
Revised: hence (cid:13)(cid:13) δ ( t, w ) (cid:13)(cid:13) is bounded and

Original: [Equation]
Revised: [Equation]

Original: It is now obvious that under the asymptotic characterization of (A.2), the weights only play a negligible role since θ ∗ separate the data.
Revised: It is now obvious that under the asymptotic characterization of (A.2), the weights only play a negligible role since θ ∗ separate the data.

Original: However, the deﬁnition of δ under (A.2) also prohibits us from studying the ﬁnite-step behavior since it absorbs all the constant effects.
Revised: However, the deﬁnition of δ under (A.2) also prohibits us from studying the ﬁnite-step behavior since it absorbs all the constant factors.

Original: Now we use the Fenchel-Young inequality to give a more precise characterization of the convergence speed.
Revised: Now we use the Fenchel-Young inequality to give a more precise characterization of the convergence speed.

Original: First of all, recall the max-margin problem for linear predictor has a dual representation for separable data according to the KKT condition for separable problem:
Revised: First of all, recall the max-margin problem for linear predictor has a dual representation for separable data according to the KKT condition for separable problem:

Original: [Equation]
Revised: [Equation]

Original: where p ∗ i is the dual optimal such that
Revised: where p ∗ i is the dual optimal such that

Original: [Equation]
Revised: [Equation]

Original: Now, we directly work with (cid:12)(cid:12)(cid:12) 
Revised: Now, we directly work with (cid:12)(cid:12)(cid:12) θ ( t

Original: [Equation]
Revised: [Equation]

Revised: w

Original: [Equation]
Revised: [Equation]

Original:  w 
Revised: [Equation] ∗ (cid:12)(cid:12)(cid:12) : [Equation] ( w )

Original: [Equation]
Revised: [Equation]

Revised: w

Revised: [Equation]

Original: and from (A.4) and Fenchel-Young inequality we have:
Revised: and from (A.4) and Fenchel-Young inequality we have:

Original: [Equation]
Revised: [Equation]

Original: where g is a convex function with it conjugate function given by g ∗ .
Revised: where g is a convex function with it conjugate function given by g ∗ .

Original: To build the connections with the loss function and risk, we choose g , so g ( u ) = log 1 n (cid:80)
Revised: To build the connections with the loss function and risk [Equation] we choose g such that g ( u ) = log 1 n (cid:80)

Original: i w i exp( u i ) , so with u i = − y i
Revised: i w i exp( u i ) . As a consequence, by letting u i = − y i

Original: x ( (cid:124) )
Revised: x ( (cid:124) )

Original: i θ ( t ) and u = [ u 1 , . .
Revised: i θ ( t ) and u = [ u 1 , . .

Original: . , u n ] , we have g ( u ) =
Revised: . , u n ] , we have g ( u ) =

Original: L ( θ ( t ) ; w ) .
Revised: L ( θ ( t ) ; w ) .

Original: By simple linear algebra, the conjugate function g ∗ ( p ) is given by:
Revised: With simple algebraic computations [Equation] the conjugate function g ∗ ( p ) is given by:

Original: [Equation]
Revised: [Equation]

Original: Plugging the above results to (A.5):
Revised: Plugging the above results to (A.5):

Original: [Equation]
Revised: [Equation]

Original: According the convergence analysis of Adaboost, we have the following technical lemma.
Revised: According the convergence analysis of Adaboost, we have the following technical lemma.

Original: Lemma A.3 (Schapire & Freund (2013)) .
Revised: Lemma A.3 (Schapire & Freund (2013)) .

Original: Suppose (cid:96) is convex, (cid:96) (cid:48) ≤ (cid:96) , and (cid:96) (cid:48)(cid:48) ≤ (cid:96) , with a linear predictor and a sufﬁciently small learning rate such that η t L ( θ ( t ) ) ≤ 1 , then:
Revised: Suppose (cid:96) is convex, (cid:96) (cid:48) ≤ (cid:96) , and (cid:96) (cid:48)(cid:48) ≤ (cid:96) , with a linear predictor and a sufﬁciently small learning rate such that η t L ( θ ( t ) ) ≤ 1 , then:

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: , (A

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , (A.7) and thusL
Revised: 7) and thus

Original: [Equation]
Revised: [Equation]

Original: j<t
Revised: j

Original: [Equation]
Revised: [Equation]

Original: η [Equation] [Equation] (cid:107)∇ L ( θ ( j
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: ) (cid:107) 2 L ( θ ( j
Revised: (A

Original: [Equation]
Revised: [Equation]

Original: ) (cid:17) 2 (cid:17) .

Original: (A.8)

Original: Also, (cid:107) θ ( t +1) (cid:107) ≤ (cid:80)
Revised: Also, (cid:107) θ ( t +1) (cid:107) ≤ (cid:80)

Original: j<t η t L ( θ ( j ) )
Revised: j<t η t L ( θ ( j ) )

Original: (cid:107)∇ L ( θ ( j ) ) (cid:107)
Revised: (cid:107)∇ L ( θ ( j ) ) (cid:107)

Original: L ( θ ( j ) ) .
Revised: L ( θ ( j ) ) .

Original: To use the results in Lemma A.3, we deﬁne the following shorthand notations.
Revised: To use the results in Lemma A.3, we deﬁne the following shorthand notations.

Original: Let a t ( w ) :
Revised: Let a t ( w ) :

Original: =
Revised: =

Original: η t L ( θ ( t ) ; w ) and b t ( w ) :=
Revised: η t L ( θ ( t ) ; w ) and b t ( w ) :=

Original: (cid:107)∇ L ( θ ( t ) ( w ); w ) (cid:107)
Revised: (cid:107)∇ L ( θ ( t ) ( w ); w ) (cid:107)

Original: L ( θ ( t ) ( w ); w ) .
Revised: L ( θ ( t ) ( w ); w ) .

Original: Now, (A.6) can be further given by:
Revised: Now, (A.6) can be further given by:

Original: [Equation]
Revised: [Equation]

Original: Notice that Lemma A.3 also imply:
Revised: Notice that Lemma A.3 also imply:

Original: [Equation]
Revised: [Equation]

Original: i
Revised: =

Original: [Equation]
Revised: [Equation]

Original: , which is bounded from above by 2 M .
Revised: , which is bounded from above by 2 M .

Original: Finally, it is easy to verify that b t ( w ) ≥ γ ∗ , and Lemma A. also gives that (cid:107) θ ( t ) ( w ) (cid:107) ≤ (cid:80) i<
Revised: Finally, it is easy to verify that b t ( w ) ≥ γ ∗ , and Lemma A. also implies that (cid:107) θ ( t ) ( w ) (cid:107) ≤ (cid:80) i<

Original: t a i ( w ) b i ( w ) , and we simply (A.9) to:
Revised: t a i ( w ) b i ( w ) . Finally, we simplify (A.9) to:

Original: [Equation]
Revised: [Equation]

Original: and obtain the desired result.
Revised: and obtain the desired result.

Original: A. P ROOF FOR P ROPOSITION
Revised: A. P ROOF FOR P ROPOSITION

Original: We ﬁrst present a greedy approach for the construction of the maximal separable subset D sep , which is proposed by Ji & Telgarsky (2018b).
Revised: We ﬁrst present a greedy approach for the construction of the maximal separable subset D sep , which is proposed by Ji & Telgarsky (2018b).

Original: For each sample ( x i , y i ) , if there exists a θ i such that
Revised: For each sample ( x i , y i ) , if there exists a θ i such that

Original: y
Revised: y

Original: i θ (cid:124) i x i > 0 and min j =1 ,...,n y j θ (cid:124) i x j ≥ 0 , we add it to D sep .
Revised: i θ (cid:124) i x i > 0 and min j =1 ,...,n y j θ (cid:124) i x j ≥ 0 , we add it to D sep .

Original: Otherwise, we add it to D non-sep .
Revised: Otherwise, we add it to D non-sep .

Original: To see why this approach work, ﬁrst notice that by choosing θ ∗ sep =
Revised: To see why this approach work, ﬁrst notice that by choosing θ ∗ sep =

Original: (cid:80) i ∈D θ
Revised: (cid:80) i ∈D θ

Original: i , θ ∗ sep separates the data in D sep .
Revised: i , θ ∗ sep separates the data in D sep .

Original: Then we check it is indeed maximal: for any θ that is correct on any ( x i , y i ) in D non-sep , there must also exist another ( x j , y j ) in D non-sep
Revised: Then we check it is indeed maximal: for any θ that is correct on any ( x i , y i ) in D non-sep , there must also exist another ( x j , y j ) in D non-sep

Original: so y i θ (cid:124)
Revised: so y i θ (cid:124)

Original: i x i < 0 , or otherwise ( x i , y i ) would have been in D sep .
Revised: i x i < 0 , or otherwise ( x i , y i ) would have been in D sep .

Original: It has been shown in Ji & Telgarsky (2018b) that the risk is strongly convex on D non-sep under conditions that are satisﬁed by our setting.
Revised: It has been shown in Ji & Telgarsky (2018b) that the risk is strongly convex on D non-sep under conditions that are satisﬁed by our setting.

Original: Lemma A.4 (Theorem 2.1 of Ji & Telgarsky (2018b)) .
Revised: Lemma A.4 (Theorem 2.1 of Ji & Telgarsky (2018b)) .

Original: If (cid:96) is twice differentiable, (cid:96) (cid:48)(cid:48) > 0 , l ≥ 0 and lim u →∞ (cid:96) ( u ) = 0 , then L ( θ )
Revised: If (cid:96) is twice differentiable, (cid:96) (cid:48)(cid:48) > 0 , l ≥ 0 and lim u →∞ (cid:96) ( u ) = 0 , then L ( θ )

Original: =
Revised: =

Original: (cid:80) i 1 n (cid:96) ( y i θ (cid:124) x i ) is strongly convex on D non-sep .
Revised: (cid:80) i 1 n (cid:96) ( y i θ (cid:124) x i ) is strongly convex on D non-sep .

Original: Now we provide the proof for Proposition 2.
Revised: Now we provide the proof for Proposition 2.

Original: Proof.
Revised: Proof.

Original: The ﬁrst part is a direct consequence of Lemma A.4, that L ( θ ; w ) = 1 n (cid:80) i w i exp( − y i θ (cid:124) x i ) is strongly convex on D non-sep .
Revised: The ﬁrst part is a direct consequence of Lemma A.4, that L ( θ ; w ) = 1 n (cid:80) i w i exp( − y i θ (cid:124) x i ) is strongly convex on D non-sep .

Original: Therefore, the optimum ˜ θ ( w ) is uniquely deﬁned and (cid:107) ˜ θ ( w ) (cid:107) =
Revised: Therefore, the optimum ˜ θ ( w ) is uniquely deﬁned and (cid:107) ˜ θ ( w ) (cid:107) =

Original: O (1) .
Revised: O (1) .

Original: To show the second part, we leverage a standard argument for gradient descent with smoothness condition.
Revised: To show the second part, we leverage a standard argument for gradient descent with smoothness condition.

Original: Lemma A.5 (Bubeck (2014)) .
Revised: Lemma A.5 (Bubeck (2014)) .

Original: Suppose L ( θ ) is convex and β -smooth.
Revised: Suppose L ( θ ) is convex and β -smooth.

Original: Then with learning rate η t ≤ β/ 2 , the sequence of gradient descent satisﬁes:
Revised: Then with learning rate η t ≤ β/ 2 , the sequence of gradient descent satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Then for any z ∈ R d :
Revised: Then for any z ∈ R d :

Original: [Equation]
Revised: [Equation]

Original: = η i1 − βη
Revised: i

Original:  i/(cid:0) L ( θ ( i ) )
Revised: [Equation] [Equation] i

Original: − L ( z ) (cid:1) .
Revised: [Equation]

Original: It is immediately clear that we may choose the z in Lemma A.5 such that it combines the optimal from D sep and D non-sep .
Revised: It is immediately clear that we may choose the z in Lemma A.5 such that it combines the optimal from D sep and D non-sep .

Original: In particular, we have shown that the optimal on D non-sep is uniquely given by ˜ θ ( w ) .
Revised: In particular, we have shown that the optimal on D non-sep is uniquely given by ˜ θ ( w ) .

Original: For D sep we assume the max-margin linear predictor is given by θ ∗ sep (so (cid:107) θ ∗ sep (cid:107) = 1 ).
Revised: For D sep we assume the max-margin linear predictor is given by θ ∗ sep (so (cid:107) θ ∗ sep (cid:107) = 1 ).

Original: Therefore, according to Proposition 1, the optimum is given by log t · θ ∗ sep .
Revised: Therefore, according to Proposition 1, the optimum is given by log t · θ ∗ sep .

Original: Now deﬁne
Revised: Now deﬁne

Original: [Equation]
Revised: [Equation]

Original: where we add the extra constant γ sep , which is the maximum margin on the separable subset of the data, to simplify the following bound.
Revised: where we add the extra constant γ sep , which is the maximum margin on the separable subset of the data, to simplify the following bound.

Original: Without loss of generality, we assume the features are bounded in (cid:107) · (cid:107) 2 norm such that (cid:107) x i (cid:107) 2 ≤ 1 .
Revised: Without loss of generality, we assume the features are bounded in (cid:107) · (cid:107) 2 norm such that (cid:107) x i (cid:107) 2 ≤ 1 .

Original: As a consequence:
Revised: As a consequence:

Original: [Equation]
Revised: [Equation]

Original: where we use L non-sep and L sep to denote the risk associated with D non-sep and D sep .
Revised: where we use L non-sep and L sep to denote the risk associated with D non-sep and D sep .

Original: To invoke Lemma
Revised: To invoke Lemma

Original: A.5, ﬁrst note that the required smoothness condition is guaranteed by Lemma A.3, i.e. in each step, the risk is η t L ( θ ( t ) )
Revised: A.5, ﬁrst note that the required smoothness condition is guaranteed by Lemma A.3, i.e. in each step, the risk is η t L ( θ ( t ) )

Original: -smooth.
Revised: -smooth.

Original: Without loss of generality, we assume η t L ( θ ( t ) ) ≤ η t .
Revised: Without loss of generality, we assume η t L ( θ ( t ) ) ≤ η t .

Original: Therefore, according to Lemma A.5, we have:
Revised: Therefore, according to Lemma A.5, we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, by our choice of z as well as (A.10), we obtain the bound in terms of the risk:
Revised: Therefore, by our choice of z as well as the result in (A.10) [Equation] we obtain the bound in terms of the risk:

Original: [Equation]
Revised: [Equation]

Original: Since we assume a constant learning rate, (cid:80) i<
Revised: Since we assume a constant learning rate [Equation] when (cid:80) i<

Original: t η i
Revised: t η i

Original: = O ( t ) we  simplify the above result with:
Revised: = O ( t ) we can simplify the above result to: [Equation] t

Original: [Equation]
Revised: [Equation]

Original: Finally, from Lemma A.4 we known L ( θ ; w ) is strongly convex (which we assume to be ω -stronglyconvex).
Revised: Finally [Equation] from Lemma A.4 we known L ( θ ; w ) is strongly convex (which we assume to be ω -stronglyconvex).

Original: So the convergence in terms of the risk can be transformed to parameters:
Revised: So the convergence in terms of the risk can be transformed to parameters:

Original: [Equation]
Revised: [Equation]

Original: which leads to our desired results.
Revised: which leads to our desired results.

Original: A. S UPPLEMENTARY MATERIAL FOR S ECTION
Revised: A. S UPPLEMENTARY MATERIAL FOR S ECTION

Original: In this section, we establish the detailed proofs of Proposition 3 and Theorem 1.
Revised: In this section, we establish the detailed proofs of Proposition 3 and Theorem 1.

Original: Recall  the loss function we are interested in:
Revised: Recall that the loss function we are interested in is:

Original: [Equation]
Revised: [Equation]

Original: Denote θ λ ( w ) ∈ arg min
Revised: Denote θ λ ( w ) ∈ arg min

Original: L λ ( θ , w ) , θ ∗ = arg max θ : (cid:107) θ (cid:107)≤ 1 max
Revised: L λ ( θ , w ) , θ ∗ = arg max θ : (cid:107) θ (cid:107)≤ 1 max

Original: i
Revised: i

Original: y
Revised: y

Original: i f
Revised: i f

Original: ( θ , x i )) .
Revised: ( θ , x i )) .

Original: Let γ λ ( w ) =
Revised: Let γ λ ( w ) =

Original: max i y i f(
Revised: max i y i f

Original: [Equation]

Original:  λ ( w ) / (cid:107) θ λ ( w [Equation] (cid:107) [Equation]
Revised: ( θ λ ( w ) / (cid:107) θ λ ( w ) (cid:107) , x i ) , γ ∗ = max i y i

Original: x

Original: [Equation]

Original: ) , γ ∗ = max [Equation] y i f[Equation]
Revised: f

Original:  ∗ [Equation] x [Equation]
Revised: ( θ ∗ , x i ) .

Original: A.4. P ROOF OF P ROPOSITION 3.
Revised: A.4. P ROOF OF P ROPOSITION 3.

Original: We ﬁrst restate the proposition.
Revised: We ﬁrst restate the proposition.

Original: Proposition A.1.
Revised: Proposition A.1.

Original: Suppose C1 , C2 , A1 hold.
Revised: Suppose C1 , C2 , A1 hold.

Original: For any w ∈ [1 /M, M ] n , it follows that
Revised: For any w ∈ [1 /M, M ] n , it follows that

Original: • (Asymptotic) lim λ
Revised: • (Asymptotic) lim λ

Original: → 0 γ λ ( w ) → γ ∗ .
Revised: → 0 γ λ ( w ) → γ ∗ .

Original: • (Finite steps)
Revised: • (Finite steps)

Original: There exists a λ :=
Revised: There exists a λ : =

Original: λ ( r, α, γ ∗ , w , c ) such that for θ (cid:48) ( w ) with L λ ( θ (cid:48) ( w ); w ) ≤ τL λ ( θ λ ( w ); w ) and τ ≤ 2 , the associated margin ˜ γ 
Revised: λ ( r, α, γ ∗ , w , c ) such that for θ (cid:48) ( w ) with L λ ( θ (cid:48) ( w ); w ) ≤ τL λ ( θ λ ( w ); w ) and τ ≤ 2 , the associated margin ˜ γ ( θ (cid:48) ( w )) satisﬁes ˜ γ ( θ (cid:48)

Original: (θ (cid:48) ( w )) satisﬁes ˜ γ ( θ (cid:48) ( w )) ≥ c · γ ∗ τ α/r , where110 ≤ c <
Revised: ( w )) ≥ c · γ ∗ τ α/r , where ≤ c <

Original: Proof of the Asymptotic part :
Revised: Proof of the Asymptotic part :

Original: Proof.
Revised: Proof.

Original: We ﬁrst take consider the exponential loss (cid:96) ( u ) =
Revised: We ﬁrst take consider the exponential loss (cid:96) ( u ) =

Original: exp( − u ) .
Revised: exp( − u ) .

Original: The log loss (cid:96) ( u ) = log(1 + exp( − u )) can be shown in a similar fashion.
Revised: The log loss (cid:96) ( u ) = log(1 + exp( − u )) can be shown in a similar fashion.

Original: Suppose the weights w = ( w 1 , . . .
Revised: Suppose the weights w = ( w 1 , . . .

Original: w n ) are normalized so that (cid:80) ni
Revised: w n ) are normalized so that (cid:80) ni

Original: =

Revised: =

Original: w i = 1 and w i ≥ 0 .
Revised: w i = 1 and w i ≥ 0 .

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: where A > 0 , and we disregard the 1 /n term in L λ for the sake of notation.
Revised: where A > 0 , and we disregard the 1 /n term in L λ for the sake of notation.

Original: In addition, we have the lower bound
Revised: In addition, we have the lower bound

Original: [Equation]
Revised: [Equation]

Original: where i (cid:48) =
Revised: where i (cid:48) =

Original: arg min i
Revised: arg min i

Original: y
Revised: y

Original: i f
Revised: i f

Original: ( θ ; x i )) , w [ n ] = min
Revised: ( θ ; x i )) , w [ n ] = min

Original: i w i .
Revised: i w i .

Original: By taking A = (cid:107) θ λ ( w ) (cid:107) , θ = θ ∗ in the upper bound and A = 1 , θ
Revised: By taking A = (cid:107) θ λ ( w ) (cid:107) , θ = θ ∗ in the upper bound and A = 1 , θ

Original: = θ λ ( w ) in the lower bound , it follows that
Revised: = θ λ ( w ) in the lower bound , it follows that

Original: [Equation]
Revised: [Equation]

Original: It implies that
Revised: It implies that

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: By Claim 1 that (cid:107) θ λ ( w ) (cid:107) → ∞ as λ → 0 (or Lemma C.4 in Wei et al.
Revised: By Claim 1 that (cid:107) θ λ ( w ) (cid:107) → ∞ as λ → 0 (or Lemma C.4 in Wei et al.

Original: , the above inequality implies that γ λ ( w ) → γ ∗ as λ → 0 .
Revised: , the above inequality implies that γ λ ( w ) → γ ∗ as λ → 0 .

Original: Proof of the Finite steps part
Revised: Proof of the Finite steps part

Original: Proof.
Revised: Proof.

Original: Consider A =
Revised: Consider A =

Original: [ 1 γ ∗ log(( γ ∗ ) r/α /λ )] 1 /α , it follows that
Revised: [ 1 γ ∗ log(( γ ∗ ) r/α /λ )] 1 /α , it follows that

Original: [Equation]
Revised: [Equation]

Original: [ Upper Bound A.4.1 ]
Revised: [ Upper Bound A.4.1 ]

Original: [Equation]
Revised: [Equation]

Original: (A.15)
Revised: (A.15)

Original: Then by the lower bound A.4.1, it follows that
Revised: Then by the lower bound A.4.1, it follows that

Original: [Equation]
Revised: [Equation]

Original: [Equation] where γ (cid:48) ( w ) = max i y i f ( w (cid:48) / (cid:107) w (cid:48) (cid:107) , x i ) .
Revised: ( w )) ≤ L λ ( θ (cid:48) ( w ) , w ) ≤ A. 15 , where γ (cid:48) ( w ) = max i y i f ( w (cid:48) / (cid:107) w (cid:48) (cid:107) , x i ) .

Original: Note λ (cid:107) θ (cid:48) ( w ) (cid:107) r ≤ A. 15 .
Revised: Note λ (cid:107) θ (cid:48) ( w ) (cid:107) r ≤ A. 15 .

Original: It implies that 
Revised: It implies that γ

Original: [Equation]
Revised: [Equation]

Original: Note that the numerator is at the scale log( 1 λ / log 1 λ ) and the denominator is at the scale log 1 λ .
Revised: Note that the numerator is at the scale log( 1 λ / log 1 λ ) and the denominator is at the scale log 1 λ .

Original: So for sufﬁciently small λ = λ ( r, α, γ ∗ , w , c ) , we have γ (cid:48) ( w ) ≥ c · γ ∗ τ α/r , where 110 ≤ c < 1 .
Revised: So for sufﬁciently small λ = λ ( r, α, γ ∗ , w , c ) , we have γ (cid:48) ( w ) ≥ c · γ ∗ τ α/r , where 110 ≤ c < 1 .

Original: We leave the details of ﬁnding out the dependency of λ ( r, α, γ ∗ , w , c ) on c to the readers, which is simply the basic analysis.
Revised: We leave the details of ﬁnding out the dependency of λ ( r, α, γ ∗ , w , c ) on c to the readers, which is simply the basic analysis.

Original: A.4. P ROOF OF T HEOREM
Revised: A.4. P ROOF OF T HEOREM

Original: When the training distribution p train deviates from the testing distribution p test , we develop the generalization bound that characterizes this deviation.
Revised: When the training distribution p train deviates from the testing distribution p test , we develop the generalization bound that characterizes this deviation.

Original: Denote by p s and p t the respective densities of x from the training data and the testing data.
Revised: Denote by p s and p t the respective densities of x from the training data and the testing data.

Original: Let D ( P t || P s )
Revised: Let D ( P t (cid:107) P s )

Original: =
Revised: =

Original: (cid:82) (( p t ( x ) p s ( x ) ) 2 − 1) ps ( x ) dx
Revised: (cid:82) (cid:0) ( p t ( x ) p s ( x ) ) 2 − 1 (cid:1) p

Original:  and η ( x i ) =
Revised: s ( x ) dx and η ( x i ) =

Original: p t ( x i ) p s ( x i ) .
Revised: p t ( x i ) p s ( x i ) .

Original: We ﬁrst restate Theorem 1:
Revised: We ﬁrst restate Theorem 1:

Original: Theorem A.1.
Revised: Theorem A.1.

Original: Assume σ is 1 -Lipschitz and 1 -positive homogeneous.
Revised: Assume σ is 1 -Lipschitz and 1 -positive homogeneous.

Original: Then with probability at least 1 − δ , we have
Revised: Then with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original:  where (I) is the empirical risk, (II) reﬂects the compounding effect of the model complexity of the class of H -layer neural networks and the deviation of the target distribution from the source distribution , (cid:15) 
Revised: , where (I) is the empirical risk, (II) reﬂects the compounding effect of the model complexity of the class of H -layer neural networks and the deviation of the target distribution from the source distribution , (cid:15) ( [Equation]

Original: [Equation]
Revised: [Equation]

Original: is a small quantity compared to (I) and (II).
Revised: is a small quantity compared to (I) and (II).

Original: Here C :=
Revised: Here C :=

Original: sup x (cid:107) x (cid:107) ; γ is any positive value.
Revised: sup x (cid:107) x (cid:107) ; γ is any positive value.

Original: To prove Theorem A.1, we ﬁrst establish a few lemmas.
Revised: To prove Theorem A.1, we ﬁrst establish a few lemmas.

Original: Lemma A.6.
Revised: Lemma A.6.

Original: Consider an arbitrary function class F such that ∀ f ∈ F we have (cid:80) x ∈X | f ( x ) | ≤ C .
Revised: Consider an arbitrary function class F such that ∀ f ∈ F we have (cid:80) x ∈X | f ( x ) | ≤ C .

Original: Then, with probability at least 1 − δ over the sample, for all margins γ > 0 and all f ∈ F we have,[Equation] n
Revised: Then, with probability at least 1 − δ over the sample, for all margins γ > 0 and all f ∈ F we have,

Original: [Equation]
Revised: [Equation]

Original: where R n, η ( F ) = E (cid:104) sup f ∈F 1 n (cid:80) ni
Revised: where R n, η ( F ) = E (cid:104) sup f ∈F 1 n (cid:80) ni

Original: =1 η ( x i ) f ( x i ) (cid:15) i (cid:105) is the weighted Rademacher complexity ( (cid:15)
Revised: =1 η ( x i ) f ( x i ) (cid:15) i (cid:105) is the weighted Rademacher complexity ( (cid:15)

Original: i ’s are i.i.d Rademacher variables).
Revised: i ’s are i.i.d Rademacher variables).

Original: Proof.
Revised: Proof.

Original: This lemma is adapted from Theorem 1 of Koltchinskii et al.
Revised: This lemma is adapted from Theorem 1 of Koltchinskii et al.

Original: by considering the deviation of the testing distribution from the training distribution.
Revised: by considering the deviation of the testing distribution from the training distribution.

Original: Then it is obtained following
Revised: Then it is obtained following

Original: Theorem 5 of Kakade et al.
Revised: Theorem 5 of Kakade et al.

Original: Lemma A.7.
Revised: Lemma A.7.

Original: Let F H be the class of real-valued networks of depth H over the domain X , where each parameter matrix W h has Frobenius norm at most M F ( h ) , and with an activation that is 1 -Lipschitz, positive-homogeneous.
Revised: Let F H be the class of real-valued networks of depth H over the domain X , where each parameter matrix W h has Frobenius norm at most M F ( h ) , and with an activation that is 1 -Lipschitz, positive-homogeneous.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where C := sup x ∈X (cid:107) x (cid:107) .
Revised: where C := sup x ∈X (cid:107) x (cid:107) .

Original: Proof.
Revised: Proof.

Original: From Theorem 1 of Golowich et al.
Revised: From Theorem 1 of Golowich et al.

Original: (2018), we arrive at
Revised: (2018), we arrive at

Original: [Equation]
Revised: [Equation]

Original: where M = (cid:81)
Revised: where M = (cid:81)

Original: Hh =1 M F ( h ) .
Revised: Hh =1 M F ( h ) .

Original: Consider Z :=
Revised: Consider Z :=

Original: M · (cid:107) (cid:80) ni
Revised: M · (cid:107) (cid:80) ni

Original: =1 (cid:15)
Revised: =1 (cid:15)

Original: i η ( x i ) x
Revised: i η ( x i ) x

Original: i (cid:107) that is a random function of the n Rademacher variables.
Revised: i (cid:107) that is a random function of the n Rademacher variables.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: By Jensen’s inequality, we have
Revised: By Jensen’s inequality, we have

Original: [Equation]
Revised: [Equation]

Original: In addition, we note that
Revised: In addition, we note that

Original: [Equation]
Revised: [Equation]

Original: By the bounded-difference condition (Boucheron et al., 2013), Z is a sub-Gaussian with variance factor v = 14 (cid:80) ni 
Revised: By the bounded-difference condition (Boucheron et al., 2013), Z is a sub-Gaussian with variance factor v = 14 (cid:80) ni = [Equation] [Equation] [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Revised: So

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking λ = √ 2log(2)
Revised: Taking λ = √ 2log(2)

Original: [Equation]
Revised: H

Original: M √ (cid:80) 
Revised: M √ (cid:80) ni

Original: [Equation]

Original: =1 η ( x i ) 2 (cid:107) 
Revised: =1 η ( x i ) 2 (cid:107) x

Original: [Equation]

Original: i (cid:107) 2 , it follows that
Revised: i (cid:107) 2 , it follows that

Original: [Equation]
Revised: [Equation]

Original: (A.17)
Revised: [Equation]

Original: By law of large number, 1 n (cid:80) ni =1 η ( x i ) 2 = D ( P t || P s ) + [-
-] + o ( 1 √ n ) .
Revised: By law of large number, 1 n (cid:80) ni =1 η ( x i ) 2 = D ( P t (cid:107) P s ) + [-
-]1 + o ( 1 √ n ) .

Original: The desired result follows.
Revised: The desired result follows.

Original: Lemma A.8.
Revised: Lemma A.8.

Original: Suppose f NN ( θ , · ) is a H -layer neural network and C
Revised: Suppose f NN ( θ , · ) is a H -layer neural network and C

Original: = sup x ∈X (cid:107) x (cid:107) 2 .
Revised: = sup x ∈X (cid:107) x (cid:107) 2 .

Original: Then, There exists another parameter ˜ θ
Revised: Then, There exists another parameter ˜ θ

Original: s.t. f NN ( θ / (cid:107) θ (cid:107) , x ) =
Revised: s.t. f NN ( θ / (cid:107) θ (cid:107) , x ) =

Original: f NN (˜ θ , x ) , for any x ∈ X and that
Revised: f NN (˜ θ , x ) , for any x ∈ X and that

Original: • the parameter matrix of each layer of f NN (˜ θ , · ) has a Frobenius norm no larger than 1 / √ H .
Revised: • the parameter matrix of each layer of f NN (˜ θ , · ) has a Frobenius norm no larger than 1 / √ H .

Original: • sup x ∈X f NN (˜ θ , · ) ≤ C .
Revised: • sup x ∈X f NN (˜ θ , · ) ≤ C .

Original: Proof.
Revised: Proof.

Original: This lemma are obtained by reorganizing the proof of Lemma D3 and the proof of Proposition
Revised: This lemma are obtained by reorganizing the proof of Lemma D3 and the proof of Proposition

Original: D.1 of Wei et al.
Revised: D.1 of Wei et al.

Original: Proof of Theorem A. 
Revised: Proof of Theorem A.

Original: Proof.
Revised: Proof.

Original: Theorem A.1 follows by Lemma A.6, A.7 and A.8.
Revised: Theorem A.1 follows by Lemma A.6, A.7 and A.8.