Original: R ESONANCE IN W EIGHT S PACE : C OVARIATE S HIFT C AN D RIVE D IVERGENCE OF SGD WITH M OMENTUM
Revised: R ESONANCE IN W EIGHT S PACE : C OVARIATE S HIFT C AN D RIVE D IVERGENCE OF SGD WITH M OMENTUM

Original: ABSTRACT
Revised: A

Original: Most convergence guarantees for stochastic gradient descent with momentum (SGDm) rely on iid sampling.
Revised: Most convergence guarantees for stochastic gradient descent with momentum (SGDm) rely on iid sampling.

Original: Yet, SGDm is often used outside this regime, in settings with temporally correlated input samples such as continual learning and reinforcement learning.
Revised: Yet, SGDm is often used outside this regime, in settings with temporally correlated input samples such as continual learning and reinforcement learning.

Original: Existing work has shown that SGDm with a decaying stepsize can converge under Markovian temporal correlation.
Revised: Existing work has shown that SGDm with a decaying stepsize can converge under Markovian temporal correlation.

Original: In this work, we show that SGDm under covariate shift with a fixed step-size can be unstable and diverge.
Revised: In this work, we show that SGDm under covariate shift with a fixed step-size can be unstable and diverge.

Original: In particular, we show SGDm under covariate shift is a parametric oscillator, and so can suffer from a phenomenon known as resonance.
Revised: In particular, we show SGDm under covariate shift is a parametric oscillator, and so can suffer from a phenomenon known as resonance.

Original: We approximate the learning system as a time varying system of ordinary differential equations, and leverage existing theory to characterize the system’s divergence/convergence as resonant/nonresonant modes.
Revised: We approximate the learning system as a time varying system of ordinary differential equations, and leverage existing theory to characterize the system’s divergence/convergence as resonant/nonresonant modes.

Original: The theoretical result is limited to the linear setting with periodic covariate shift, so we empirically supplement this result to show that resonance phenomena persist even under non-periodic covariate shift, nonlinear dynamics with neural networks, and optimizers other than SGDm.
Revised: The theoretical result is limited to the linear setting with periodic covariate shift, so we empirically supplement this result to show that resonance phenomena persist even under non-periodic covariate shift, nonlinear dynamics with neural networks, and optimizers other than SGDm.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Stochastic gradient descent (SGD) (Robbins & Monro, 1951) – and its variants such as Adagrad (Duchi et al., 2011), ADAM (Kingma & Ba, 2014) and RMSprop (Hinton et al., 2012) – are very widely used optimization algorithms across machine learning.
Revised: Stochastic gradient descent (SGD) (Robbins & Monro, 1951) – and its variants such as Adagrad (Duchi et al., 2011), ADAM (Kingma & Ba, 2014) and RMSprop (Hinton et al., 2012) – are very widely used optimization algorithms across machine learning.

Original: SGD is conceptually straightforward, easy to implement, and often performs well in practice.
Revised: SGD is conceptually straightforward, easy to implement, and often performs well in practice.

Original: Among the variants of SGD, accelerated versions based on Polyak’s or Nesterov’s acceleration (Polyak, 1964; Nesterov, 1983), known generally as Stochastic Gradient Descent with Momentum (SGDm), are used widely due to the improvements in convergence rate they offer.
Revised: Among the variants of SGD, accelerated versions based on Polyak’s or Nesterov’s acceleration (Polyak, 1964; Nesterov, 1983), known generally as Stochastic Gradient Descent with Momentum (SGDm), are used widely due to the improvements in convergence rate they offer.

Original: SGDm can give up to a quadratic speedup to SGD on many functions, and is in fact optimal among all methods having only information about the gradient at consecutive iterates for convex and Lipschitz continuous optimization problems (Nesterov, 2004; Goh, 2017).
Revised: SGDm can give up to a quadratic speedup to SGD on many functions, and is in fact optimal among all methods having only information about the gradient at consecutive iterates for convex and Lipschitz continuous optimization problems (Nesterov, 2004; Goh, 2017).

Original: SGDm has the same computational complexity as SGD, but exhibits superior convergence rates under reasonable assumptions (Su et al., 2014).
Revised: SGDm has the same computational complexity as SGD, but exhibits superior convergence rates under reasonable assumptions (Su et al., 2014).

Original: These convergence results for SGDm, however, rely on independent and identically distributed (iid) sampling.
Revised: These convergence results for SGDm, however, rely on independent and identically distributed (iid) sampling.

Original: Little is known about the convergence properties of SGDm under non-iid sampling, yet the non-iid setting is critical.
Revised: Little is known about the convergence properties of SGDm under non-iid sampling, yet the non-iid setting is critical.

Original: In many machine learning problems it is expensive or impossible to obtain iid samples.
Revised: In many machine learning problems it is expensive or impossible to obtain iid samples.

Original: In online learning (Rakhlin et al., 2010) and reinforcement learning (RL) (Sutton & Barto, 2018), the data becomes available in a sequential order and there is a temporal dependence among the samples.
Revised: In online learning (Rakhlin et al., 2010) and reinforcement learning (RL) (Sutton & Barto, 2018), the data becomes available in a sequential order and there is a temporal dependence among the samples.

Original: There is a particularly strong temporal dependence in RL, where observed states are sampled according to the transition dynamics of the underlying Markov decision process (MDP).
Revised: There is a particularly strong temporal dependence in RL, where observed states are sampled according to the transition dynamics of the underlying Markov decision process (MDP).

Original: Federated learning (Hsieh et al., 2020) and time-series learning (Kuznetsov & Mohri, 2014) provide further examples of when non-iid sampling is essential to the learning problem.
Revised: Federated learning (Hsieh et al., 2020) and time-series learning (Kuznetsov & Mohri, 2014) provide further examples of when non-iid sampling is essential to the learning problem.

Original: Without momentum, SGD’s convergence rate has been examined under non-iid sampling with the stochastic approximation framework in (Benveniste et al., 2012; Kushner & Yin, 2003), and more recently under specific assumptions of ergodicity (Duchi et al., 2012) or Markovian sampling (Nagaraj et al., 2020; Doan et al., 2020b; Sun et al., 2018).
Revised: Without momentum, SGD’s convergence rate has been examined under non-iid sampling with the stochastic approximation framework in (Benveniste et al., 2012; Kushner & Yin, 2003), and more recently under specific assumptions of ergodicity (Duchi et al., 2012) or Markovian sampling (Nagaraj et al., 2020; Doan et al., 2020b; Sun et al., 2018).

Original: Convergence rates under Markovian sampling are also known for ADAM-type algorithms when applied to policy gradient and temporal difference learning (Xiong et al., 2020).
Revised: Convergence rates under Markovian sampling are also known for ADAM-type algorithms when applied to policy gradient and temporal difference learning (Xiong et al., 2020).

Original: To our knowledge, however, there has been little work on providing convergence rates or guarantees for SGDm under non-iid sampling.
Revised: To our knowledge, however, there has been little work on providing convergence rates or guarantees for SGDm under non-iid sampling.

Original: In (Doan et al., 2020a), a progress bound is provided for SGDm under Markovian sampling based on mixing time—the time required for a distribution’s convergence toward its stationary distribution—along with a convergence rate guarantee under decaying step-sizes.
Revised: In (Doan et al., 2020a), a progress bound is provided for SGDm under Markovian sampling based on mixing time—the time required for a distribution’s convergence toward its stationary distribution—along with a convergence rate guarantee under decaying step-sizes.

Original: In this work, we assume a fixed step-size, since it is a common choice in the online setting, especially when the practitioner is unsure of mixing rate or stationarity.
Revised: In this work, we assume a fixed step-size, since it is a common choice in the online setting, especially when the practitioner is unsure of mixing rate or stationarity.

Original: There is a broad literature using ordinary differential equations (ODEs) to analyze gradient descent methods by approximating descent updates as continuous-time flows.
Revised: There is a broad literature using ordinary differential equations (ODEs) to analyze gradient descent methods by approximating descent updates as continuous-time flows.

Original: Early work is comprehensively discussed in (Kushner & Yin, 2003; Benveniste et al., 2012).
Revised: Early work is comprehensively discussed in (Kushner & Yin, 2003; Benveniste et al., 2012).

Original: Despite the age and establishment of the linear setting, the gradient flow lens continues to reveal new insights (e.g. implicit rank reduction (Arora et al., 2019)) and new perspectives on old insights (e.g. regularization of early stopping (Ali et al., 2019)).
Revised: Despite the age and establishment of the linear setting, the gradient flow lens continues to reveal new insights (e.g. implicit rank reduction (Arora et al., 2019)) and new perspectives on old insights (e.g. regularization of early stopping (Ali et al., 2019)).

Original: Recent work has paid particular attention to the flow induced by momentum accelerated methods (Su et al., 2014;
Revised: Recent work has paid particular attention to the flow induced by momentum accelerated methods (Su et al., 2014;

Original: Wibisono et al., 2016; Wilson et al., 2016;
Revised: Wibisono et al., 2016; Wilson et al., 2016;

Original: Scieur et al., 2017; Muehlebach &
Revised: Scieur et al., 2017; Muehlebach &

Original: Jordan, 2021; Diakonikolas & Orecchia, 2019; Muehlebach & Jordan, 2019; Li et al., 2017; Simsekli et al., 2020; Betancourt et al., 2018; Kovachki & Stuart, 2019; Attouch et al., 2018; Shi et al., 2019;
Revised: Jordan, 2021; Diakonikolas & Orecchia, 2019; Muehlebach & Jordan, 2019; Li et al., 2017; Simsekli et al., 2020; Betancourt et al., 2018; Kovachki & Stuart, 2019; Attouch et al., 2018; Shi et al., 2019;

Original: Siegel, 2019; Zhang et al., 2018; Berthier et al., 2021; Kovachki & Stuart, 2021).
Revised: Siegel, 2019; Zhang et al., 2018; Berthier et al., 2021; Kovachki & Stuart, 2021).

Original: In these works, it is demonstrated that linear regression under iid sampling with SGDm can be represented as an ODE resembling a harmonic oscillator, sometimes with a time-decaying damping coefficient.
Revised: In these works, it is demonstrated that linear regression under iid sampling with SGDm can be represented as an ODE resembling a harmonic oscillator, sometimes with a time-decaying damping coefficient.

Original: Contributions:
Revised: Contributions:

Original: In this work, we show that non-iid sampling, due to covariate shift, induces a related system: the parametric oscillator, a harmonic oscillator having coefficients which vary over time in a manner capable of exponentially exciting the system (Mumford, 1960; Csörg˝o & Hatvani, 2010;
Revised: In this work, we show that non-iid sampling, due to covariate shift, induces a related system: the parametric oscillator, a harmonic oscillator having coefficients which vary over time in a manner capable of exponentially exciting the system (Mumford, 1960; Csörg˝o & Hatvani, 2010;

Original: Halanay, 1966).
Revised: Halanay, 1966).

Original: We characterize the underlying time-varying ODE, which requires a novel approach to incorporate covariate shift, and develop specific conditions on covariate shift that lead to divergence in SGDm.
Revised: We characterize the underlying time-varying ODE, which requires a novel approach to incorporate covariate shift, and develop specific conditions on covariate shift that lead to divergence in SGDm.

Original: We provide an empirical design, with synthetic data, to systematically test the response of the learning system to different covariate shift frequencies.
Revised: We provide an empirical design, with synthetic data, to systematically test the response of the learning system to different covariate shift frequencies.

Original: We empirically validate that resonancedriven divergence occurs in a learning system which aligns well with theoretical assumptions.
Revised: We empirically validate that resonancedriven divergence occurs in a learning system which aligns well with theoretical assumptions.

Original: We follow with similar demonstrations on learning systems which progressively relax further and further away from our theoretical assumptions, including non-periodic covariate shift, learning with neural networks and optimizers other than SDGm.
Revised: We follow with similar demonstrations on learning systems which progressively relax further and further away from our theoretical assumptions, including non-periodic covariate shift, learning with neural networks and optimizers other than SDGm.

Original: The implications of this work are that resonance can occur in learning systems with momentum, causing instability or poor accuracy, simply due to certain temporal correlations in the input sequence.
Revised: The implications of this work are that resonance can occur in learning systems with momentum, causing instability or poor accuracy, simply due to certain temporal correlations in the input sequence.

Original: Our results provide a novel direction to better understand the properties of our learning systems for time varying inputs—namely beyond iid sampling—and point to a gap in the robustness of our algorithms that merits further investigation.
Revised: Our results provide a novel direction to better understand the properties of our learning systems for time varying inputs—namely beyond iid sampling—and point to a gap in the robustness of our algorithms that merits further investigation.

Original:  P ROBLEM S ETTING
Revised:  P ROBLEM S ETTING

Original: We investigate the effect of non-iid sampling on a model optimized using SGDm.
Revised: We investigate the effect of non-iid sampling on a model optimized using SGDm.

Original: We assume labelled training data sampled from discreteindexed, real-valued stochastic processes { X k } k ∈ N and { Y k } k ∈ N such that { X k } k ∈ N has a unique stationary distribution Π .
Revised: We assume labelled training data sampled from discreteindexed, real-valued stochastic processes { X k } k ∈ N and { Y k } k ∈ N such that { X k } k ∈ N has a unique stationary distribution Π .

Original: Y k is a function of
Revised: Y k is a function of

Original: X k with zero-mean observation noise,
Revised: X k with zero-mean observation noise,

Original: Y k
Revised: Y k

Original: = f ( X k )
Revised: = f ( X k )

Original: + ϵ k .
Revised: + ϵ k .

Original: The learning algorithm does not have access to iid samples from Π .
Revised: The learning algorithm does not have access to iid samples from Π .

Original: Instead, the learning algorithm may only update parameters θ k at time k using samples z k
Revised: Instead, the learning algorithm may only update parameters θ k at time k using samples z k

Original: = ( x k , y k ) , where X k ’s marginal over time converges to Π .
Revised: = ( x k , y k ) , where X k ’s marginal over time converges to Π .

Original: The goal is to train a model with parameters θ to minimize an objective function L ( z ; θ ) with respect to the stationary distribution Π : θ ∗ =
Revised: The goal is to train a model with parameters θ to minimize an objective function L ( z ; θ ) with respect to the stationary distribution Π : θ ∗ =

Original: arg min θ ∈ R d E Π [ L ( z ; θ )]
Revised: arg min θ ∈ R d E Π [ L ( z ; θ )]

Original: = (cid:82) R L ( z ; θ )
Revised: = (cid:82) R L ( z ; θ )

Original: d Π( x ) .
Revised: d Π( x ) .

Original: An example of this problem setting is illustrated in Figure 1.
Revised: An example of this problem setting is illustrated in Figure 1.

Original: This setting is identical to that explored in prior work on Markovian sampling (Sun et al., 2018;
Revised: This setting is identical to that explored in prior work on Markovian sampling (Sun et al., 2018;

Original: Xiong et al., 2020; Nagaraj et al., 2020), but we do not require that our stochastic processes adhere to the Markov property.
Revised: Xiong et al., 2020; Nagaraj et al., 2020), but we do not require that our stochastic processes adhere to the Markov property.

Original: Note, also, that the underlying functional relationship between the inputs, X k , and the targets, Y k , does not change over time.
Revised: Note, also, that the underlying functional relationship between the inputs, X k , and the targets, Y k , does not change over time.

Original: That is, the non-iid sampling is a result of covariate shift rather than a changing relationship between the inputs and targets.
Revised: That is, the non-iid sampling is a result of covariate shift rather than a changing relationship between the inputs and targets.

Original: We consider Polyak’s Heavy Ball method (Polyak, 1964) with the formulation from Sutskever et al.
Revised: We consider Polyak’s Heavy Ball method (Polyak, 1964) with the formulation from Sutskever et al.

Original: (2013), namely SGD with momentum (SGDm)
Revised: (2013), namely SGD with momentum (SGDm)

Original: [Equation]
Revised: [Equation]

Original: where η ≥ 0 is the learning rate and µ ∈ (0 , 1) the momentum coefficient.
Revised: where η ≥ 0 is the learning rate and µ ∈ (0 , 1) the momentum coefficient.

Original: Our goal is to understand the behavior of SGDm when learning on this non-iid sequence z k .
Revised: Our goal is to understand the behavior of SGDm when learning on this non-iid sequence z k .

Original:  C OVARIATE S HIFT AS A D RIVING F ORCE
Revised:  C OVARIATE S HIFT AS A D RIVING F ORCE

Original: In this section we characterize SGDm as a discretization of a particular parametric oscillator in continuous time.
Revised: In this section we characterize SGDm as a discretization of a particular parametric oscillator in continuous time.

Original: A parametric oscillator resembles a harmonic oscillator ODE, but has time-varying system coefficients which are capable of driving the system.
Revised: A parametric oscillator resembles a harmonic oscillator ODE, but has time-varying system coefficients which are capable of driving the system.

Original: It is well understood that parametric oscillators can suffer from global solution instability due to coefficients oscillating at particular frequencies (Halanay, 1966), a condition known as parametric resonance .
Revised: It is well understood that parametric oscillators can suffer from global solution instability due to coefficients oscillating at particular frequencies (Halanay, 1966), a condition known as parametric resonance .

Original: We show parametric resonance conditions sufficient to induce exponential divergence in SGDm.
Revised: We show parametric resonance conditions sufficient to induce exponential divergence in SGDm.

Original: We provide proof sketches and defer complete proofs to Appendix A.4.
Revised: We provide proof sketches and defer complete proofs to Appendix A.4.

Original: Notation: Capital letters are matrices and random variables.
Revised: Notation: Capital letters are matrices and random variables.

Original: { X k } is a stochastic process, shorthand for { X k } k ∈ N .
Revised: { X k } is a stochastic process, shorthand for { X k } k ∈ N .

Original: k and t index discrete and continuous time, respectively.
Revised: k and t index discrete and continuous time, respectively.

Original: When a discrete sequence and a function approximate each other, they share the same symbol and are differentiated by subscript k and function argument t , e.g. the sequence { θ k } and the function θ ( t ) .
Revised: When a discrete sequence and a function approximate each other, they share the same symbol and are differentiated by subscript k and function argument t , e.g. the sequence { θ k } and the function θ ( t ) .

Original: P k is the joint distribution of X k and Y k .
Revised: P k is the joint distribution of X k and Y k .

Original: For loss function L ( z ; θ ) on training pair z
Revised: For loss function L ( z ; θ ) on training pair z

Original: = ( x, y ) and weights θ , we denote the time-varying expected gradient g k ( θ ) :
Revised: = ( x, y ) and weights θ , we denote the time-varying expected gradient g k ( θ ) :

Original: = E P k [ ∇ θ L ( z ; θ )]
Revised: = E P k [ ∇ θ L ( z ; θ )]

Original: 3. C OVARIATE S HIFT I NDUCES A L INEAR T IME -V ARYING E XPECTED G RADIENT
Revised: 3. C OVARIATE S HIFT I NDUCES A L INEAR T IME -V ARYING E XPECTED G RADIENT

Original: We begin by showing that linear least squares regression with covariate shift and a fixed target induces a linear time-varying expected loss gradient.
Revised: We begin by showing that linear least squares regression with covariate shift and a fixed target induces a linear time-varying expected loss gradient.

Original: That is, the gradient function g k ( θ )
Revised: That is, the gradient function g k ( θ )

Original: =
Revised: =

Original: B k ( θ − θ ∗ ) for some matrix B k that varies with time, due to covariate shift.
Revised: B k ( θ − θ ∗ ) for some matrix B k that varies with time, due to covariate shift.

Original: Assumption 1.
Revised: Assumption 1.

Original: The covariate generating process { X k } k ∈ N is not identically distributed:
Revised: The covariate generating process { X k } k ∈ N is not identically distributed:

Original: Corr ( X k 1 , X k 1 ) ̸
Revised: Corr ( X k 1 , X k 1 ) ̸

Original: =
Revised: =

Original: Corr ( X k 2 , X k 2 ) for some k 1 , k 2 ∈ N .
Revised: Corr ( X k 2 , X k 2 ) for some k 1 , k 2 ∈ N .

Original: Assumption 2.
Revised: Assumption 2.

Original: The targets Y k are a fixed linear function of X k with iid zero-mean observation noise, ϵ k
Revised: The targets Y k are a fixed linear function of X k with iid zero-mean observation noise, ϵ k

Original: such that E [ ϵ k ] = 0 .
Revised: such that E [ ϵ k ] = 0 .

Original: That is, Y k
Revised: That is, Y k

Original: =
Revised: =

Original: ⟨ θ ∗ , X k ⟩ + ϵ k where θ ∗ ∈ R d is fixed for all k .
Revised: ⟨ θ ∗ , X k ⟩ + ϵ k where θ ∗ ∈ R d is fixed for all k .

Original: Proposition 1.
Revised: Proposition 1.

Original: Under Assumptions 1, 2, a linear model with weights θ k making predictions (cid:98) Y k
Revised: Under Assumptions 1, 2, a linear model with weights θ k making predictions (cid:98) Y k

Original: =
Revised: =

Original: ⟨ θ k , X k ⟩ with a mean squared error (MSE) objective will induce time-varying linear expected loss gradients g k ( θ k )
Revised: ⟨ θ k , X k ⟩ with a mean squared error (MSE) objective will induce time-varying linear expected loss gradients g k ( θ k )

Original: =
Revised: =

Original: B k
Revised: B k

Original: ( θ k − θ ∗ ) for all k ∈ N , where B k ∈ R d × d and B k 1 ̸ = B k 2 for some k 1 , k 2 .
Revised: ( θ k − θ ∗ ) for all k ∈ N , where B k ∈ R d × d and B k 1 ̸ = B k 2 for some k 1 , k 2 .

Original: Proof Sketch: We fix a time step k and take the expectation of the MSE loss over observation noise, and over the distribution of X k .
Revised: Proof Sketch: We fix a time step k and take the expectation of the MSE loss over observation noise, and over the distribution of X k .

Original: A time-varying expected loss surface remains, and the linear setting implies the loss gradient is a linear function for all steps k , with B k depending on X k :
Revised: A time-varying expected loss surface remains, and the linear setting implies the loss gradient is a linear function for all steps k , with B k depending on X k :

Original: [Equation]
Revised: [Equation]

Original: 3. ODE C ORRESPONDENCE
Revised: 3. ODE C ORRESPONDENCE

Original: Next, we show that the iterates { θ k } generated by SGDm are a first order numerical integration of a particular ODE.
Revised: Next, we show that the iterates { θ k } generated by SGDm are a first order numerical integration of a particular ODE.

Original: The procedure is similar to (Muehlebach & Jordan, 2021), but due to the time-varying loss gradient, we must pay specific attention to the conditions on the ODE necessary to have integration consistency, .
Revised: The procedure is similar to (Muehlebach & Jordan, 2021), but due to the time-varying loss gradient, we must pay specific attention to the conditions on the ODE necessary to have integration consistency, .

Original: Let B ( t ) be a matrix-valued function, Lipschitz continuous in t , such that { B k } are samples spaced √ η apart, i.e. B k = B ( √ ηk ) .
Revised: Let B ( t ) be a matrix-valued function, Lipschitz continuous in t , such that { B k } are samples spaced √ η apart, i.e. B k = B ( √ ηk ) .

Original: The existence and uniqueness of this
Revised: The existence and uniqueness of this

Original: B ( t ) are ensured if we assume the { X k } are sampled from an underlying continuous-time X ( t ) .
Revised: B ( t ) are ensured if we assume the { X k } are sampled from an underlying continuous-time X ( t ) .

Original: Proposition 2.
Revised: Proposition 2.

Original: The SGDm iterates { θ k } numerically integrate the ODE system in equation 3 with integration step √ η and first order consistency.
Revised: The SGDm iterates { θ k } numerically integrate the ODE system in equation 3 with integration step √ η and first order consistency.

Original: [Equation]
Revised: [Equation]

Original: Proof Sketch: First, the ODE in equation 3 is converted to first order linear form, and split into a sum of two separate systems.
Revised: Proof Sketch: First, the ODE in equation 3 is converted to first order linear form, and split into a sum of two separate systems.

Original: The sum’s terms have implicit and explicit Euler integrations; using operator splitting, these can be composed into a consistent numerical integrator for linear time-varying systems.
Revised: The sum’s terms have implicit and explicit Euler integrations; using operator splitting, these can be composed into a consistent numerical integrator for linear time-varying systems.

Original: Then, it is shown that the composed integrator is precisely equivalent to SGDm when the integration time step is √ η . □
Revised: Then, it is shown that the composed integrator is precisely equivalent to SGDm when the integration time step is √ η . □

Original: The first order consistency guarantee in Proposition 2 means that when the step-size η is small and initial conditions agree between continuous and discrete time, i.e. θ 

Revised: The first order consistency guarantee in Proposition 2 means that when the step-size η is small and initial conditions agree between continuous and discrete time, i.e. θ

Original: = θ (0) , the continuous and discrete time trajectories approximate each other as θ k ≈ θ ( k √ η ) , where the difference between them depends on the integration time step h = √
Revised: = θ (0) , the continuous and discrete time trajectories approximate each other as θ k ≈ θ ( k √ η ) , where the difference between them depends on the integration time step h = √

Original: η (Hairer et al., 2006).
Revised: η (Hairer et al., 2006).

Original: Therefore, assuming θ 0 = θ (0) , the difference accrued in one step of k (local error) is θ 1 = θ ( √ η ) + O ( η ) and over arbitrarily many steps (global error) is θ k
Revised: Therefore, assuming θ 0 = θ (0) , the difference accrued in one step of k (local error) is θ 1 = θ ( √ η ) + O ( η ) and over arbitrarily many steps (global error) is θ k

Original: = θ ( √ ηk )
Revised: = θ ( √ ηk )

Original: + O ( ηk ) .
Revised: + O ( ηk ) .

Original: 3. P ARAMETRIC R ESONANCE FOR ODE C ONVERGENCE AND D IVERGENCE
Revised: 3. P ARAMETRIC R ESONANCE FOR ODE C ONVERGENCE AND D IVERGENCE

Original: The conditions sufficient for convergence and divergence in equation 3 may be shown using established dynamical systems theory.
Revised: The conditions sufficient for convergence and divergence in equation 3 may be shown using established dynamical systems theory.

Original: The conditions sufficient for divergence are precisely the conditions for parametric resonance, when B ( t ) is periodic.
Revised: The conditions sufficient for divergence are precisely the conditions for parametric resonance, when B ( t ) is periodic.

Original: Note that periodicity in the first or second moment of { X k } is sufficient to induce periodicity in B ( t ) .
Revised: Note that periodicity in the first or second moment of { X k } is sufficient to induce periodicity in B ( t ) .

Original: As per elementary results in linear ODE theory, the system in equation 3 can be transformed into a linear time-varying first order form
Revised: As per elementary results in linear ODE theory, the system in equation 3 can be transformed into a linear time-varying first order form

Original: [Equation]
Revised: [Equation]

Original: where solution trajectories θ ( t ) of equation 3 are embedded in solution trajectories ξ ( t ) of equation 4.
Revised: where solution trajectories θ ( t ) of equation 3 are embedded in solution trajectories ξ ( t ) of equation 4.

Original: Equation 4 admits a fundamental solution matrix 1 ψ ( t ) such that the spectral radius ρ of ψ ( T ) characterizes ODE instability, which implies divergence for SGDm, in Theorem 1.
Revised: Equation 4 admits a fundamental solution matrix 1 ψ ( t ) such that the spectral radius ρ of ψ ( T ) characterizes ODE instability, which implies divergence for SGDm, in Theorem 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: When B ( t ) is periodic such that B ( t )
Revised: When B ( t ) is periodic such that B ( t )

Original: =
Revised: =

Original: B ( t + T ) for some T > 0 , the spectral radius ρ of ψ ( T ) characterizes the stability of solution trajectories of equation 3 as follows:
Revised: B ( t + T ) for some T > 0 , the spectral radius ρ of ψ ( T ) characterizes the stability of solution trajectories of equation 3 as follows:

Original: • ρ > 1 = ⇒ trivial solution θ ( t )
Revised: • ρ > 1 = ⇒ trivial solution θ ( t )

Original: =
Revised: =

Original: θ ∗ is unstable.
Revised: θ ∗ is unstable.

Original: All other solutions diverge as θ ( t ) → ∞ exponentially with rate ρ .
Revised: All other solutions diverge as θ ( t ) → ∞ exponentially with rate ρ .

Original: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as θ ( t ) → θ ∗ exponentially with rate ρ .
Revised: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as θ ( t ) → θ ∗ exponentially with rate ρ .

Original: Proof Sketch: ODE 3’s stability depends on the relationship between the coefficient 1 − µ √ η and the expected gradient signal because they determine the spectral radius of ψ ( T ) .
Revised: Proof Sketch: ODE 3’s stability depends on the relationship between the coefficient 1 − µ √ η and the expected gradient signal because they determine the spectral radius of ψ ( T ) .

Original: The spectral radius ρ is the convergence/divergence rate per period T towards/away from the stationary point θ ∗ .
Revised: The spectral radius ρ is the convergence/divergence rate per period T towards/away from the stationary point θ ∗ .

Original: □
Revised: □

Original: The spectral radius conditions characterize when ODE solution trajectories θ ( t ) will converge or diverge, and Proposition 2 tells us that these solution trajectories are approximations of discrete time
Revised: The spectral radius conditions characterize when ODE solution trajectories θ ( t ) will converge or diverge, and Proposition 2 tells us that these solution trajectories are approximations of discrete time

Original: SGDm trajectories { θ k } under identical initialization.
Revised: SGDm trajectories { θ k } under identical initialization.

Original: But does convergence or divergence of θ ( t ) imply the same for SGDm?
Revised: But does convergence or divergence of θ ( t ) imply the same for SGDm?

Original: Experiment 4.1 empirically suggests that the approximation is sufficient, since the boundary at ρ = 1 in Figure 2a agrees with both continuous and discrete time.
Revised: Experiment 4.1 empirically suggests that the approximation is sufficient, since the boundary at ρ = 1 in Figure 2a agrees with both continuous and discrete time.

Original: However, given Theorem 1 and Proposition 2, we have a theoretical guarantee of SGDm’s divergence, but not its convergence.
Revised: However, given Theorem 1 and Proposition 2, we have a theoretical guarantee of SGDm’s divergence, but not its convergence.

Original: This is because the divergence rate of θ ( t ) is exponential, and Proposition 2 provides a bound on long tail behaviour as θ k = θ ( √ ηk ) + O ( ηk ) .
Revised: This is because the divergence rate of θ ( t ) is exponential, and Proposition 2 provides a bound on long tail behaviour as θ k = θ ( √ ηk ) + O ( ηk ) .

Original: Since the approximation error is linear in time k , and the divergence rate is exponential, the divergence rate dominates, and we are guaranteed that a diverging θ ( t ) corresponds to a diverging { θ k } .
Revised: Since the approximation error is linear in time k , and the divergence rate is exponential, the divergence rate dominates, and we are guaranteed that a diverging θ ( t ) corresponds to a diverging { θ k } .

Original: However, the same argument does not imply that a convergent θ ( t ) corresponds to a convergent { θ k } , because the linear error bound technically permits the discrete trajectory to escape θ ∗ at a linear rate.
Revised: However, the same argument does not imply that a convergent θ ( t ) corresponds to a convergent { θ k } , because the linear error bound technically permits the discrete trajectory to escape θ ∗ at a linear rate.

Original: Empirically, we see agreement for both convergent and divergent cases, but we defer theoretical proof to future work.
Revised: Empirically, we see agreement for both convergent and divergent cases, but we defer theoretical proof to future work.

Original: Summary: There is a chain of dependencies starting from the non-i.i.d.
Revised: Summary: There is a chain of dependencies starting from the non-i.i.d.

Original: sampling in { X k } , and ending at the monodromy’s spectral radius.
Revised: sampling in { X k } , and ending at the monodromy’s spectral radius.

Original: [Equation]
Revised: [Equation]

Original: In order, we have the discrete stochastic process { X k } from which training inputs are sampled, and its underlying continuous stochastic process X ( t ) .
Revised: In order, we have the discrete stochastic process { X k } from which training inputs are sampled, and its underlying continuous stochastic process X ( t ) .

Original: If { X k } and X ( t ) were i.i.d.
Revised: If { X k } and X ( t ) were i.i.d.

Original: , then the matrix B would be constant, but the non-i.d.d.
Revised: , then the matrix B would be constant, but the non-i.d.d.

Original: nature means B ( t ) is a function of time.
Revised: nature means B ( t ) is a function of time.

Original: The matrix A ( t ) is the matrix describing SGDm’s continuous time dynamics as a linear time-varying ODE ˙ ξ
Revised: The matrix A ( t ) is the matrix describing SGDm’s continuous time dynamics as a linear time-varying ODE ˙ ξ

Original: = A ( t ) ξ , and the matrix B ( t ) is a simply a submatrix of A ( t ) .
Revised: = A ( t ) ξ , and the matrix B ( t ) is a simply a submatrix of A ( t ) .

Original: Since we have a linear ODE, the space of solution trajectories is spanned by the columns of the ODE’s fundamental solution matrix ψ ( t ) .
Revised: Since we have a linear ODE, the space of solution trajectories is spanned by the columns of the ODE’s fundamental solution matrix ψ ( t ) .

Original: Moreover, since we have a periodic A ( t ) with period T , the stability of all solutions can be determined simply from the largest eigenvalue of ψ ( T ) , a.k.a.
Revised: Moreover, since we have a periodic A ( t ) with period T , the stability of all solutions can be determined simply from the largest eigenvalue of ψ ( T ) , a.k.a.

Original: its spectral radius ρ .
Revised: its spectral radius ρ .

Original: The intuition behind ρ ’s importance comes from the following fact: after each elapsed period T , the phase space (including weight space) is subject to the linear transformation ψ ( T ) .
Revised: The intuition behind ρ ’s importance comes from the following fact: after each elapsed period T , the phase space (including weight space) is subject to the linear transformation ψ ( T ) .

Original: So as time increases, say n periods, the linear transformation is applied iteratively as ψ ( T ) n , so it is clear that any eigenvalue larger than unity means all solutions eventually diverge exponentially.
Revised: So as time increases, say n periods, the linear transformation is applied iteratively as ψ ( T ) n , so it is clear that any eigenvalue larger than unity means all solutions eventually diverge exponentially.

Original: This is precisely the parametric resonance condition.
Revised: This is precisely the parametric resonance condition.

Original: For a detailed example of all these quantities characterizing the divergence of a simple learning system, see Appendix A.1.
Revised: For a detailed example of all these quantities characterizing the divergence of a simple learning system, see Appendix A.1.

Original: Technical Novelty:
Revised: Technical Novelty:

Original: There are two key novel technical contributions to obtain the results above.
Revised: There are two key novel technical contributions to obtain the results above.

Original: The ODE correspondence for non-iid data in combination with momentum methods is new.
Revised: The ODE correspondence for non-iid data in combination with momentum methods is new.

Original: The non-iid data, which induces time variation in ODE dynamics, is subtle to address, because elementary
Revised: The non-iid data, which induces time variation in ODE dynamics, is subtle to address, because elementary

Original: ODE correspondence (i.e. consistency guarantees) require time invariant ODE dynamics.
Revised: ODE correspondence (i.e. consistency guarantees) require time invariant ODE dynamics.

Original: Our proof specifically addresses this difficulty using operator splitting theory from recent numerical integration literature.
Revised: Our proof specifically addresses this difficulty using operator splitting theory from recent numerical integration literature.

Original: (2) Once we have this ODE, it is straightforward to recognize that it is a parametric oscillator.
Revised: (2) Once we have this ODE, it is straightforward to recognize that it is a parametric oscillator.

Original: The utility of this connection, however, is that it opens up entirely new avenues for analysis, since there is deeply established literature on the dynamics of these systems.
Revised: The utility of this connection, however, is that it opens up entirely new avenues for analysis, since there is deeply established literature on the dynamics of these systems.

Original: The primary second technical novelty, therefore, is to leverage this connection, and use Floquet theory to give precise divergence conditions for a given η and µ , under periodic covariate shift.
Revised: The primary second technical novelty, therefore, is to leverage this connection, and use Floquet theory to give precise divergence conditions for a given η and µ , under periodic covariate shift.

Original:  V ALIDATING T HEORY & A BLATING T OWARDS C ONDITIONS IN THE W ILD
Revised:  V ALIDATING T HEORY & A BLATING T OWARDS C ONDITIONS IN THE W ILD

Original: We now empirically evaluate the effect of resonance in learning problems.
Revised: We now empirically evaluate the effect of resonance in learning problems.

Original: First, Experiment 4.
Revised: First, Experiment 4.

Original: validates the theoretical predictions by investigating the dynamics of a learning problem which closely matches the theoretical assumptions in Section 3.
Revised: validates the theoretical predictions by investigating the dynamics of a learning problem which closely matches the theoretical assumptions in Section 3.

Original: Specifically, we assess whether or not the spectral radius ρ predicts optimizer convergence or divergence.
Revised: Specifically, we assess whether or not the spectral radius ρ predicts optimizer convergence or divergence.

Original: Then, the remaining Experiments 4.2 - 4.6 test for this phenomena in settings beyond our theory.
Revised: Then, the remaining Experiments 4.2 - 4.6 test for this phenomena in settings beyond our theory.

Original: Each experiment makes a single step away from a theoretical assumption, first relaxing the periodicity assumption, then using stochastic rather than expected gradients and then moving to nonlinear models (neural networks) and optimizers (ADAM).
Revised: Each experiment makes a single step away from a theoretical assumption, first relaxing the periodicity assumption, then using stochastic rather than expected gradients and then moving to nonlinear models (neural networks) and optimizers (ADAM).

Original: Across all experiments, input samples at each training step k are drawn from Gaussian distributions with diagonal covariance matrices, i.e. X k ∼ N (¯ x k , cI d × d ) , and we induce covariate shift by constructing a time-varying mean sequence { ¯ x k } .
Revised: Across all experiments, input samples at each training step k are drawn from Gaussian distributions with diagonal covariance matrices, i.e. X k ∼ N (¯ x k , cI d × d ) , and we induce covariate shift by constructing a time-varying mean sequence { ¯ x k } .

Original: Each mean sequence is designed such that (a) its frequency content can be swept with a single parameter f or T and (b) iid sampling is induced by setting f = 0 or T = 0 , so that each experiment has an iid baseline for comparison.
Revised: Each mean sequence is designed such that (a) its frequency content can be swept with a single parameter f or T and (b) iid sampling is induced by setting f = 0 or T = 0 , so that each experiment has an iid baseline for comparison.

Original: The specification of ¯ x k is provided in Table 1 for each experiment.
Revised: The specification of ¯ x k is provided in Table 1 for each experiment.

Original: This empirical design allows us to systematically test response to different frequencies.
Revised: This empirical design allows us to systematically test response to different frequencies.

Original: See Appendix A.5 for specification of all experiment details, including visual depictions of all synthetic { X k } and their frequency content.
Revised: See Appendix A.5 for specification of all experiment details, including visual depictions of all synthetic { X k } and their frequency content.

Original: 4. V ALIDATING T HEORY
Revised: 4. V ALIDATING T HEORY

Original: We start in a setting as close as possible to the theoretical predictions, with linear regression for a quadratic loss, and covariate shift such that the mean E [ X k ] varies as a strict sinusoid.
Revised: We start in a setting as close as possible to the theoretical predictions, with linear regression for a quadratic loss, and covariate shift such that the mean E [ X k ] varies as a strict sinusoid.

Original: We perform regression in two weights (i.e. inputs X k and labels Y k are scalar-valued).
Revised: We perform regression in two weights (i.e. inputs X k and labels Y k are scalar-valued).

Original: We draw 20 samples from each X k , so that the loss gradient for each time step k is close to its expected value.
Revised: We draw 20 samples from each X k , so that the loss gradient for each time step k is close to its expected value.

Original: Since we have only two weights, the learning system’s underlying ODE can be easily specified, such that the fundamental solution matrix can be numerically computed.
Revised: Since we have only two weights, the learning system’s underlying ODE can be easily specified, such that the fundamental solution matrix can be numerically computed.

Original: As per Theorem 1, we can use the spectral radius of the fundamental solution matrix evaluated at time T to make theoretical predictions of where the system should converge or diverge.
Revised: As per Theorem 1, we can use the spectral radius of the fundamental solution matrix evaluated at time T to make theoretical predictions of where the system should converge or diverge.

Original: As depicted in Figure 2a, the theory agrees very well with empirical results.
Revised: As depicted in Figure 2a, the theory agrees very well with empirical results.

Original: This suggests that parametric resonance is indeed the dominant mechanism behind SGDm divergence under covariate shift.
Revised: This suggests that parametric resonance is indeed the dominant mechanism behind SGDm divergence under covariate shift.

Original: In the appendix, refer to example A.1 for the procedure used to compute the theoretical predictions (i.e. spectral radii ρ ), and Figure 6a for the full surface from which the contour lines in Figure 2a are rendered.
Revised: In the appendix, refer to example A.1 for the procedure used to compute the theoretical predictions (i.e. spectral radii ρ ), and Figure 6a for the full surface from which the contour lines in Figure 2a are rendered.

Original: Experiment
Revised: Experiment

Original: Sweep Param.
Revised: Sweep Param.

Original: Mean Sequence { ¯ x k } k ∈ N
Revised: Mean Sequence { ¯ x k } k ∈ N

Original: 4.1 Validating Theory
Revised: 4.1 Validating Theory

Original: f ∈ [0 , 0 .
Revised: f ∈ [0 , 0 .

Original: ¯ x k = 0 .
Revised: ¯ x k = 0 .

Original: 5 sin(2 πfk ) , T = 1 /f
Revised: 5 sin(2 πfk ) , T = 1 /f

Original: 4.2 Ablating Periodicity
Revised: 4.2 Ablating Periodicity

Original: f ∈ [0 , 0 .
Revised: f ∈ [0 , 0 .

Original: ¯ x k = ϕ 1 ¯ x k − 1 + ϕ 2 ¯ x k − 2 + ξ k ϕ 

Revised: ¯ x k = ϕ 1 ¯ x k − 1 + ϕ 2 ¯ x k − 2 + ξ k ϕ

Original: = 4 ϕ 2 ( ϕ 2 − 1) − 1 cos(2 πf )
Revised: = 4 ϕ 2 ( ϕ 2 − 1) − 1 cos(2 πf )

Original: ϕ 2 , ξ k , ¯ x 1 , ¯ x 2 see Table
Revised: ϕ 2 , ξ k , ¯ x 1 , ¯ x 2 see Table

Original: 4.3 Ablating Expected Gradient
Revised: 4.3 Ablating Expected Gradient

Original: T ∈ [0 , 120] ¯ x k = (cid:40)
Revised: T ∈ [0 , 120] ¯ x k = (cid:40)

Original: ξ [Table] ||
Revised: ξ 2 || ξ ||

Original: [Table]

Original: || if [Table] mod [Table]
Revised: if ⌊ 2 kT ⌋ ≡ 0 mod 1 − ξ 2 || ξ

Original: |
Revised: ||

Original: [Table] if [Table] ξ ∼ N (0 , I d × d ) iid 4.

Revised: if ⌊ 2 kT ⌋ ≡ 1 modξ ∼ N (0 , I d × d ) iid 4.

Original: Ablating Periodicity Further
Revised: Ablating Periodicity Further

Original: T ∈ [0 , 50] ¯ x k = ξ i where i = (cid:22) k
Revised: T ∈ [0 , 50] ¯ x k = ξ i where i = (cid:22) k

Original: T (cid:23) ξ i ∼ N (0 , vI d × d ) iid4.
Revised: T (cid:23) ξ i ∼ N (0 , vI d × d ) iid

Original: 5 Ablating Optimizer Linearity
Revised: 4.5 Ablating Optimizer Linearity

Original: T ∈ [0 , 100] ¯ x k same as above.
Revised: T ∈ [0 , 100] ¯ x k same as above.

Original:  Ablating Model Linearity
Revised: 4.6 Ablating Model Linearity

Original: T ∈ [0 , 100] ¯ x k same as above.
Revised: T ∈ [0 , 100] ¯ x k same as above.

Original: [Figure]
Revised: [Figure]

Original: 4. A BLATING P ERIODICITY
Revised: 4. A BLATING P ERIODICITY

Original: We repeat Experiment 4.1, but with the mean of X k varying stochastically instead of deterministically.
Revised: We repeat Experiment 4.1, but with the mean of X k varying stochastically instead of deterministically.

Original: The result in Theorem 1 assumes the expected gradient varies periodically over time given fixed weights θ .
Revised: The result in Theorem 1 assumes the expected gradient varies periodically over time given fixed weights θ .

Original: But even with aperiodic and/or stochastic time variation, our LTV system equation 4 might be similarly susceptible to instability; the theory does not have periodicity as a necessary condition of divergence.
Revised: But even with aperiodic and/or stochastic time variation, our LTV system equation 4 might be similarly susceptible to instability; the theory does not have periodicity as a necessary condition of divergence.

Original: To test this, we replace our periodic covariate shift mean with a mean that moves according to an AR(2) process.
Revised: To test this, we replace our periodic covariate shift mean with a mean that moves according to an AR(2) process.

Original: The AR(2) process is tuned to have a frequency peak exactly at the frequency of the sinusoidal covariate shift of Experiment 4.1, to change only the nature of the shift—aperiodicity—rather than the frequency.
Revised: The AR(2) process is tuned to have a frequency peak exactly at the frequency of the sinusoidal covariate shift of Experiment 4.1, to change only the nature of the shift—aperiodicity—rather than the frequency.

Original: We can see from the heatmap in Figure 2b that under this setting we observe very similar resonance behaviour in the learning system, which aligns well with the identical predicted stability regions.
Revised: We can see from the heatmap in Figure 2b that under this setting we observe very similar resonance behaviour in the learning system, which aligns well with the identical predicted stability regions.

Original: Note that both theoretical and empirical results in Figures 2a and 2b suggest that sufficiently low momentum values µ mitigate the resonance phenomenon, which agrees with the role it plays in the
Revised: Note that both theoretical and empirical results in Figures 2a and 2b suggest that sufficiently low momentum values µ mitigate the resonance phenomenon, which agrees with the role it plays in the

Original: ODE system: decreasing µ increases damping.
Revised: ODE system: decreasing µ increases damping.

Original: We also observe that resonance regions shrink as step size η is decreased; see the Appendix for plots demonstrating this trend.
Revised: We also observe that resonance regions shrink as step size η is decreased; see the Appendix for plots demonstrating this trend.

Original: Analytically characterizing the bounds of stable µ, η in terms of system properties is an interesting future direction.
Revised: Analytically characterizing the bounds of stable µ, η in terms of system properties is an interesting future direction.

Original: 4. A BLATING E XPECTED G RADIENT
Revised: 4. A BLATING E XPECTED G RADIENT

Original: Here we perform linear regression with periodic mean covariate shift, and ablate the number of samples drawn from each X k to show the effect of increasing noise in the gradient signal.
Revised: Here we perform linear regression with periodic mean covariate shift, and ablate the number of samples drawn from each X k to show the effect of increasing noise in the gradient signal.

Original: Linear regression is performed mapping R 5 → R , with the weights learned via SGDm.
Revised: Linear regression is performed mapping R 5 → R , with the weights learned via SGDm.

Original: The mean sequence ¯ x k oscillates with strict periodicity between ± ¯ x , where ¯ x is a unit norm 5-vector randomly chosen for each run.
Revised: The mean sequence ¯ x k oscillates with strict periodicity between ± ¯ x , where ¯ x is a unit norm 5-vector randomly chosen for each run.

Original: i.e. the mean signal is a square wave in R 5 with period T .
Revised: i.e. the mean signal is a square wave in R 5 with period T .

Original: Linear regression implies a quadratic loss, and periodic covariate shift implies a loss with periodic time variation in expectation, so we are very near to the setting in which Theorem 1 is directly applicable.
Revised: Linear regression implies a quadratic loss, and periodic covariate shift implies a loss with periodic time variation in expectation, so we are very near to the setting in which Theorem 1 is directly applicable.

Original: However, as we ablate from 5 samples to 1 sample drawn from each X k , we move further away from the expected gradient, towards the fully stochastic gradient setting.
Revised: However, as we ablate from 5 samples to 1 sample drawn from each X k , we move further away from the expected gradient, towards the fully stochastic gradient setting.

Original: As we can see in Figure 3 resonance is dampened by stochasticity in the gradient signal, but nonetheless is still present.
Revised: As we can see in Figure 3 resonance is dampened by stochasticity in the gradient signal, but nonetheless is still present.

Original: These findings are further supported in Appendix A.3, where Experiments 4.
Revised: These findings are further supported in Appendix A.3, where Experiments 4.

Original: and 4.2 are also ablated from expected to stochastic gradient signals.
Revised: and 4.2 are also ablated from expected to stochastic gradient signals.

Original: 4. A BLATING P ERIODICITY F URTHER
Revised: 4. A BLATING P ERIODICITY F URTHER

Original: We further depart from strictly periodic covariate shift by having piece-wise stationary means and nonsmooth changes: we randomly sample a new mean from a normal distribution N (0 , v 2 ) after every T update steps.
Revised: We further depart from strictly periodic covariate shift by having piece-wise stationary means and nonsmooth changes: we randomly sample a new mean from a normal distribution N (0 , v 2 ) after every T update steps.

Original: We further investigate the impact of this covariate shift under (a) increasing variance v 2 and (b) increasing dimensionality of the inputs.
Revised: We further investigate the impact of this covariate shift under (a) increasing variance v 2 and (b) increasing dimensionality of the inputs.

Original: We observe divergence characteristic of the parametric resonance, with smaller T having a highly divergent response.
Revised: We observe divergence characteristic of the parametric resonance, with smaller T having a highly divergent response.

Original: In Figure 4a we can see that a higher variance results in much more instability, though even for smaller variance the distance to θ ∗ is still on the order of 10 3 .
Revised: In Figure 4a we can see that a higher variance results in much more instability, though even for smaller variance the distance to θ ∗ is still on the order of 10 3 .

Original: This results makes sense, as the variance is connected to the driving signal amplitude, on which parametric resonance has a strong dependence.
Revised: This results makes sense, as the variance is connected to the driving signal amplitude, on which parametric resonance has a strong dependence.

Original: We also observe a strong dependence on the number of input space dimensions, in Figure 4b.
Revised: We also observe a strong dependence on the number of input space dimensions, in Figure 4b.

Original: This aligns with the fact that the expected norm of samples drawn from a multivariate
Revised: This aligns with the fact that the expected norm of samples drawn from a multivariate

Original: Gaussian increases with dimensionality, even with a fixed covariance scale (of 0.1).
Revised: Gaussian increases with dimensionality, even with a fixed covariance scale (of 0.1).

Original: This result suggests that resonance phenomena may actually be exacerbated for higher-dimensional inputs, which is the setting we expect to see in practice.
Revised: This result suggests that resonance phenomena may actually be exacerbated for higher-dimensional inputs, which is the setting we expect to see in practice.

Original: 4. A BLATING O PTIMIZER L INEARITY
Revised: 4. A BLATING O PTIMIZER L INEARITY

Original: So far our experiments have ablated periodicity, input dimensionality, and gradient stochasticity.
Revised: So far our experiments have ablated periodicity, input dimensionality, and gradient stochasticity.

Original: As per the conditions in Section 3, each of these ablated systems still corresponds to a discretization of an LTV ODE.
Revised: As per the conditions in Section 3, each of these ablated systems still corresponds to a discretization of an LTV ODE.

Original: We now ask whether resonance can be observed in systems which do not correspond to a discretization of our ODE.
Revised: We now ask whether resonance can be observed in systems which do not correspond to a discretization of our ODE.

Original: To investigate this hypothesis, we replace SGDm with ADAM as an update rule for our learning algorithm.
Revised: To investigate this hypothesis, we replace SGDm with ADAM as an update rule for our learning algorithm.

Original: We no longer have the ODE representation of this learning system, but can empirically investigate the behaviour of the system under covariate shift.
Revised: We no longer have the ODE representation of this learning system, but can empirically investigate the behaviour of the system under covariate shift.

Original: We use the same stochastic mean switching problem as the previous experiment, where we regress from 5 dimensions to 1 with input samples.
Revised: We use the same stochastic mean switching problem as the previous experiment, where we regress from 5 dimensions to 1 with input samples.

Original: Again, we measure the distance || θ k − θ ∗ || , and we vary the ADAM parameter β 1 .
Revised: Again, we measure the distance || θ k − θ ∗ || , and we vary the ADAM parameter β 1 .

Original: Similar to the SGDm optimizer, we can see in Figure 4c that there is a band of mean switching intervals T for which convergence is worse.
Revised: Similar to the SGDm optimizer, we can see in Figure 4c that there is a band of mean switching intervals T for which convergence is worse.

Original: Unlike SGDm, there is no divergence.
Revised: Unlike SGDm, there is no divergence.

Original: Proper parametric resonance induces exponential divergence, which is why the previous results were presented with Euclidean distances between weights on the order of 10 19 .
Revised: Proper parametric resonance induces exponential divergence, which is why the previous results were presented with Euclidean distances between weights on the order of 10 19 .

Original: Note that in Figure 4c, the frequency response in weight space distance is instead measured on the order of 10 0 .
Revised: Note that in Figure 4c, the frequency response in weight space distance is instead measured on the order of 10 0 .

Original: [Figure]
Revised: [Figure]

Original: 4. A BLATING M ODEL L INEARITY
Revised: 4. A BLATING M ODEL L INEARITY

Original: Our final step away from ODE equation 3 abandons the quadratic loss surface.
Revised: Our final step away from ODE equation 3 abandons the quadratic loss surface.

Original: We fit a nonlinear target function using a fully connected ReLU network, reusing the previous covariate shift.
Revised: We fit a nonlinear target function using a fully connected ReLU network, reusing the previous covariate shift.

Original: In previous experiments we measured the distance || θ k − θ ∗ || .
Revised: In previous experiments, we measured the distance || θ k − θ ∗ || .

Original: Now a unique target is not guaranteed, so frequency response is in terms of loss.
Revised: Now a unique target is not guaranteed, so frequency response is in terms of loss.

Original: In order to amplify frequency response, 10 samples were drawn from each X k to reduce gradient signal noise.
Revised: In order to amplify frequency response, 10 samples were drawn from each X k to reduce gradient signal noise.

Original: Figure 5 shows a band of mean switching intervals T which damage convergence, which is characteristic of resonance.
Revised: Figure 5 shows a band of mean switching intervals T which damage convergence, which is characteristic of resonance.

Original: Though we do not see divergence, the impact on accuracy is problematic.
Revised: Though we do not see divergence, the impact on accuracy is problematic.

Original: In this case, models with loss > 0 .
Revised: In this case, models with loss > 0 .

Original: 3 fit extremely poorly, but those with < 0 .
Revised: 3 fit extremely poorly, but those with < 0 .

Original: 05 perform well.
Revised: 05 perform well.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: This work is a first step towards understanding frequency response due to covariate shift.
Revised: This work is a first step towards understanding frequency response due to covariate shift.

Original: Here we discuss future directions, limitations, and practicality.
Revised: Here we discuss future directions, limitations, and practicality.

Original: Limitations and Next Steps: In our theoretical analysis, we take expectation over observation noise and over the distribution of inputs X k , so the gradient coefficient matrix B ( t ) in the ODE equation 3 is deterministic.
Revised: Limitations and Next Steps: In our theoretical analysis, we take expectation over observation noise and over the distribution of inputs X k , so the gradient coefficient matrix B ( t ) in the ODE equation 3 is deterministic.

Original: This allows rigorous characterization of resonance for periodic B ( t ) .
Revised: This allows rigorous characterization of resonance for periodic B ( t ) .

Original: Experiments 4.2 and 4.4 demonstrate that resonance still drives divergence when B ( t ) is nondeterministic, so rigorous exploration in future work might employ stochastic differential equations.
Revised: Experiments 4.2 and 4.4 demonstrate that resonance still drives divergence when B ( t ) is nondeterministic, so rigorous exploration in future work might employ stochastic differential equations.

Original: In particular, Experiment 4.2 aligns exactly with the theoretical predictions, suggesting that the resonance is in response to the frequency content of X t , not its periodicity.
Revised: In particular, Experiment 4.2 aligns exactly with the theoretical predictions, suggesting that the resonance is in response to the frequency content of X t , not its periodicity.

Original: For the sake of consistency across experiments, our non-iid sampling is always induced by Gaussian
Revised: For the sake of consistency across experiments, our non-iid sampling is always induced by Gaussian

Original: X k with fixed diagonal covariance matrix.
Revised: X k with fixed diagonal covariance matrix.

Original: But Assumption 1 is far more relaxed, suggesting resonance is possible with time-varying covariance and non-Gaussian distributions, which may be explored in future empirical work
Revised: But Assumption 1 is far more relaxed, suggesting resonance is possible with time-varying covariance and non-Gaussian distributions, which may be explored in future empirical work

Original: When assessing a new phenomenon like covariate shift resonance, assessment on simple problems is a critical first step, and we have endeavored to be comprehensive in that assessment.
Revised: When assessing a new phenomenon like covariate shift resonance, assessment on simple problems is a critical first step, and we have endeavored to be comprehensive in that assessment.

Original: But we only scratch the surface of more complex optimizers and nonlinear models.
Revised: But we only scratch the surface of more complex optimizers and nonlinear models.

Original: Similarly, for the sake of experimental control and interpretability, we assess resonance on synthetic data instead of real data.
Revised: Similarly, for the sake of experimental control and interpretability, we assess resonance on synthetic data instead of real data.

Original: While it is obvious that many sources of real data have nontrivial frequency content (e.g. audio samples, machine sensors, etc.)
Revised: While it is obvious that many sources of real data have nontrivial frequency content (e.g. audio samples, machine sensors, etc.)

Original: it is not yet clear when realistic frequency content will resonate with the system trying to learn from it.
Revised: it is not yet clear when realistic frequency content will resonate with the system trying to learn from it.

Original: Optimizers, models, and data all pose exciting future directions for this work: to extend into the frequency response of more complex optimizers and models, and into the frequency content of real data.
Revised: Optimizers, models, and data all pose exciting future directions for this work: to extend into the frequency response of more complex optimizers and models, and into the frequency content of real data.

Original: Finally, the theory in this work is for linear models.
Revised: Finally, the theory in this work is for linear models.

Original: This is the correct setting to understand first, and is of independent interest.
Revised: This is the correct setting to understand first, and is of independent interest.

Original: Our theory applies to any linear models that are augmented with a nonlinear basis (such as a Fourier basis, polynomials or radial basis functions) by examining the frequency content of the basis output signal.
Revised: Our theory applies to any linear models that are augmented with a nonlinear basis (such as a Fourier basis, polynomials or radial basis functions) by examining the frequency content of the basis output signal.

Original: Practical Implications: One potential conclusion from our results is that though resonance could potentially arise in our learning systems, it may not be problematic.
Revised: Practical Implications: One potential conclusion from our results is that though resonance could potentially arise in our learning systems, it may not be problematic.

Original: First, the heatmaps in Figures 2a and 2b indicate that SGDm converges much more often than it diverges, and that we can practically avoid resonance simply by reducing the momentum parameter µ .
Revised: First, the heatmaps in Figures 2a and 2b indicate that SGDm converges much more often than it diverges, and that we can practically avoid resonance simply by reducing the momentum parameter µ .

Original: As per Appendix A.2, we find that sufficiently low µ does mitigate resonance.
Revised: As per Appendix A.2, we find that sufficiently low µ does mitigate resonance.

Original: Second, we may simply be able to avoid covariate shift in the first place, by keeping buffers of data and shuffling samples.
Revised: Second, we may simply be able to avoid covariate shift in the first place, by keeping buffers of data and shuffling samples.

Original: However, the reality is not so simple.
Revised: However, the reality is not so simple.

Original: Higher levels of momentum are typically more effective in practice.
Revised: Higher levels of momentum are typically more effective in practice.

Original: There is a tension between the level of momentum and avoiding resonance, that cannot easily be addressed by simply picking low momentum by default.
Revised: There is a tension between the level of momentum and avoiding resonance, that cannot easily be addressed by simply picking low momentum by default.

Original: Instead, smarter approaches which recognize potential resonance and then reduce momentum may be a much more effective way to guard against this issue; the development of such methods can leverage the insights in this work.
Revised: Instead, smarter approaches which recognize potential resonance and then reduce momentum may be a much more effective way to guard against this issue; the development of such methods can leverage the insights in this work.

Original: In addition, relying on well-behaved or shuffled inputs may be impractical in certain settings.
Revised: In addition, relying on well-behaved or shuffled inputs may be impractical in certain settings.

Original: Algorithms in signal processing and the streaming literature often process data sequentially.
Revised: Algorithms in signal processing and the streaming literature often process data sequentially.

Original: Reinforcement learning approaches use sliding window buffers (in replay), where covariate shift may still persist across buffers.
Revised: Reinforcement learning approaches use sliding window buffers (in replay), where covariate shift may still persist across buffers.

Original: Further, it is even possible that adversarial attacks could be designed to exploit this weakness of SGDm, causing poor performance or even instability from attackers attempting to generate data that results in resonant behavior.
Revised: Further, it is even possible that adversarial attacks could be designed to exploit this weakness of SGDm, causing poor performance or even instability from attackers attempting to generate data that results in resonant behavior.

Original: Our work also suggests avenues towards the development of diagnostic tools.
Revised: Our work also suggests avenues towards the development of diagnostic tools.

Original: Indeed, our experimental approach to investigate this phenomena mimics those in the modal analysis literature (Avitabile, 2017).
Revised: Indeed, our experimental approach to investigate this phenomena mimics those in the modal analysis literature (Avitabile, 2017).

Original: We treat the input sampling process as an input signal which is noisy and/or stochastic, vary the signal’s frequency content across a wide band, and measure the learning system’s output as a response across input frequencies.
Revised: We treat the input sampling process as an input signal which is noisy and/or stochastic, vary the signal’s frequency content across a wide band, and measure the learning system’s output as a response across input frequencies.

Original: This is analogous to a mechanical engineer triggering an oscillatory vibration within a machine’s engine compartment, sweeping across frequencies, and measuring the resulting vibration amplitude in the machine’s housing.
Revised: This is analogous to a mechanical engineer triggering an oscillatory vibration within a machine’s engine compartment, sweeping across frequencies, and measuring the resulting vibration amplitude in the machine’s housing.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: To our knowledge, this work is the first to investigate SGDm under covariate shift from the ODE perspective.
Revised: To our knowledge, this work is the first to investigate SGDm under covariate shift from the ODE perspective.

Original: This perspective reveals new insights, in particular the revelation that SGDm under covariate shift numerically integrates a parametric oscillator, rather than the simple undriven harmonic oscillator induced by iid sampling.
Revised: This perspective reveals new insights, in particular the revelation that SGDm under covariate shift numerically integrates a parametric oscillator, rather than the simple undriven harmonic oscillator induced by iid sampling.

Original: We leverage dynamical systems theory to analyze the stability of a learning system using SGDm under covariate shift, and are able to provide conditions for exponential divergence due entirely to the frequency content induced by this non-iid sampling.
Revised: We leverage dynamical systems theory to analyze the stability of a learning system using SGDm under covariate shift, and are able to provide conditions for exponential divergence due entirely to the frequency content induced by this non-iid sampling.

Original: Despite its age and establishment, understanding SGDm is an ongoing process.
Revised: Despite its age and establishment, understanding SGDm is an ongoing process.

Original: We contribute to this process in a way that provides physical intuition, rather than a merely algebraic proof.
Revised: We contribute to this process in a way that provides physical intuition, rather than a merely algebraic proof.

Original: We hope that the connection we have drawn to parametric oscillation in dynamical systems theory will help to further propel understanding of non-iid sampling and SGDm.
Revised: We hope that the connection we have drawn to parametric oscillation in dynamical systems theory will help to further propel understanding of non-iid sampling and SGDm.

Original:  A CKNOWLEDGEMENTS
Revised:  A CKNOWLEDGEMENTS

Original: We would like to thank Yurij Salmaniw and Ryan Thiessen for their discussions regarding Floquet theory and helping us makes sense of our differential equations.
Revised: We would like to thank Yurij Salmaniw and Ryan Thiessen for their discussions regarding Floquet theory and helping us makes sense of our differential equations.

Original: We would also like to thank Csaba
Revised: We would also like to thank Csaba

Original: Szepesvari for pointing us to excellent resources for numerical integration, algorithmic convergence, stochastic processes, and various future directions for this work.
Revised: Szepesvari for pointing us to excellent resources for numerical integration, algorithmic convergence, stochastic processes, and various future directions for this work.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our work is theoretical and attempts to provide greater understanding of an optimization algorithm.
Revised: Our work is theoretical and attempts to provide greater understanding of an optimization algorithm.

Original: Due to the remoteness of this work from direct ethical concerns, it is difficult to reason about any ethical implications of the work.
Revised: Due to the remoteness of this work from direct ethical concerns, it is difficult to reason about any ethical implications of the work.

Original: We hope that the insights into SGDm could help to make machine learning algorithm more robust, but beyond that we are not aware of any ethical concerns or implications of this work.
Revised: We hope that the insights into SGDm could help to make machine learning algorithm more robust, but beyond that we are not aware of any ethical concerns or implications of this work.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We have endeavored to be very clear and thorough in our explanation of the theoretical results, and all assumptions and proofs are explicitly stated and contained in either the main body of the paper or in the appendix.
Revised: We have endeavored to be very clear and thorough in our explanation of the theoretical results and all assumptions and proofs are explicitly stated and contained in either the main body of the paper or in the appendix.

Original: The code for all of the experiments is included in the supplementary material and the instructions for reproduction are in the README file.
Revised: The code for all of the experiments is included in the supplementary material and the instructions for reproduction are in the README file.

Original: The data used is synthetic and code to generate the data is contained in the supplementary material.
Revised: The data used is synthetic and code to generate the data is contained in the supplementary material.

Original: All of the experimental details (e g. hyperparameter selection) are contained in tables in the paper and appendix and the supplementary material.
Revised: All of the experimental details (e g. hyperparameter selection) are contained in tables in the paper and appendix and the supplementary material.

Original: All of the experiments should be easily run on modest commodity hardware in the space of a few days, at most.
Revised: All of the experiments should be easily run on modest commodity hardware in the space of a few days, at most.

Original: A A PPENDIX A. A P ROCEDURE TO N UMERICALLY D ETERMINE S YSTEM S TABILITY
Revised: A A PPENDIX A. A P ROCEDURE TO N UMERICALLY D ETERMINE S YSTEM S TABILITY

Original: Here we expand on the example given at the end of Section 3.
Revised: Here we expand on the example given at the end of Section 3.

Original: As a reminder, we consider online linear least squares regression using SGDm with a fixed η, µ under a periodic covariate shift.
Revised: As a reminder, we consider online linear least squares regression using SGDm with a fixed η, µ under a periodic covariate shift.

Original: Data is sampled from the sequence of random variables { X k , Y k } k ∈ N where X k is normally distributed and shifts periodically in expectation.
Revised: Data is sampled from the sequence of random variables { X k , Y k } k ∈ N where X k is normally distributed and shifts periodically in expectation.

Original: Y k is an unchanging linear function of X k with zero-mean observation noise.
Revised: Y k is an unchanging linear function of X k with zero-mean observation noise.

Original: For ease of exposition we consider the one dimensional regression problem with
Revised: For ease of exposition we consider the one dimensional regression problem with

Original: X k ∈ R and Y k ∈ R , however the theory applies to any learning system representable as an LTV
Revised: X k ∈ R and Y k ∈ R , however the theory applies to any learning system representable as an LTV

Original: ODE satisfying the assumptions required for Theorem 1.
Revised: ODE satisfying the assumptions required for Theorem 1.

Original: Under this covaiate shift, ¯ x k varies sinusoidally in [ − a, a ] with period T = 1 f .
Revised: Under this covaiate shift, ¯ x k varies sinusoidally in [ − a, a ] with period T = 1 f .

Original: Note that although
Revised: Note that although

Original: X k is periodic in expectation it admits a stationary distribution.
Revised: X k is periodic in expectation it admits a stationary distribution.

Original: The full specification of the data generating process is as follows
Revised: The full specification of the data generating process is as follows

Original: [Equation]
Revised: [Equation]

Original: where θ ∗ =
Revised: where θ ∗ =

Original: [ θ ∗ 1 , θ ∗ 2 ] T are the target weights.
Revised: [ θ ∗ 1 , θ ∗ 2 ] T are the target weights.

Original: We consider a squared error loss function,
Revised: We consider a squared error loss function,

Original: [Equation]
Revised: [Equation]

Original: Hence, taking gradients at time k w.r.t.
Revised: Hence, taking gradients at time k w.r.t.

Original: θ k we have
Revised: θ k we have

Original: [Equation]
Revised: [Equation]

Original: Taking expected gradients,
Revised: Taking expected gradients,

Original: [Equation]
Revised: [Equation]

Original: In the notation of Proposition
Revised: In the notation of Proposition

Original: [Equation]
Revised: [Equation]

Original: we can write this as
Revised: we can write this as

Original: [Equation]
Revised: [Equation]

Original: If we let ¯ x ( t ) =
Revised: If we let ¯ x ( t ) =

Original: a cos(2 πfη − 12 t ) , then we induce an X ( t ) from which { X k } is sampled, as per
Revised: a cos(2 πfη − 12 t ) , then we induce an X ( t ) from which { X k } is sampled, as per

Original: Section A.4.3.
Revised: Section A.4.3.

Original: This results in B k
Revised: This results in B k

Original: = B ( √ ηk with the matrix-valued function B ( t )
Revised: = B ( √ ηk with the matrix-valued function B ( t )

Original: [Equation]
Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: From Proposition 2, we have that our learning system numerically integrates an LTV of the form¨
Revised: From Proposition 2, we have that our learning system numerically integrates an LTV of the form

Original: [Equation]
Revised: [Equation]

Original: with ( θ ( t ) − θ ∗ ) =

Original: (cid:20) θ 1 ( t

Original: [Equation]

Original: − θ ∗ 1 ( t

Original: [Equation]

Original: θ 2 ( t ) − θ 2 ∗ ( t ) (cid:21) (5)

Original: Note that B ( t ) is piecewise continuous and periodic with period T = 1 f .
Revised: Note that B ( t ) is piecewise continuous and periodic with period T = 1 f .

Original: This second order ODE can be represented as a system of first order ODEs with the transformation
Revised: This second order ODE can be represented as a system of first order ODEs with the transformation

Original: [Equation]
Revised: [Equation]

Original: so that the equation equation 5 can be written in standard first order linear form ˙ ξ =
Revised: so that the equation equation 5 can be written in standard first order linear form ˙ ξ =

Original: A ( t ) ξ :
Revised: A ( t ) ξ :

Original: [Equation]
Revised: [Equation]

Original: The matrix B ( t ) is piecewise continuous and periodic, which implies the matrix A ( t ) also shares these properties.
Revised: The matrix B ( t ) is piecewise continuous and periodic, which implies the matrix A ( t ) also shares these properties.

Original: The ODE equation 5 satisfies all the assumptions needed to make use of Theoremto determine the stability of solution trajectories.
Revised: The ODE equation 5 satisfies all the assumptions needed to make use of Theoremto determine the stability of solution trajectories.

Original: We can obtain a fundamental solution matrix, ψ ( t ) , of equation 6 satisfying initial conditions ψ (0)
Revised: We can obtain a fundamental solution matrix, ψ ( t ) , of equation 6 satisfying initial conditions ψ (0)

Original: =
Revised: =

Original: I 4 × 4 by numerical computation through the use of an ODE solver.
Revised: I 4 × 4 by numerical computation through the use of an ODE solver.

Original: Once we have this matrix ψ ( t ) , we evaluate it at time T = 1 f to obtain the system’s monodromy matrix.
Revised: Once we have this matrix ψ ( t ) , we evaluate it at time T = 1 f to obtain the system’s monodromy matrix.

Original: Now we only need to determine the spectral radius, ρ , of ψ ( T ) to determine the stability of solution trajectories for the system.
Revised: Now we only need to determine the spectral radius, ρ , of ψ ( T ) to determine the stability of solution trajectories for the system.

Original: Details regarding why this is true are given in the proof of
Revised: Details regarding why this is true are given in the proof of

Original: Theorem 1.
Revised: Theorem 1.

Original: We repeat this process, sweeping over values of µ , η , and f to determine the stability of solution trajectories for systems to triples of fixed values for these hyperparameters.
Revised: We repeat this process, sweeping over values of µ , η , and f to determine the stability of solution trajectories for systems to triples of fixed values for these hyperparameters.

Original: The results of this process are given in Figure 6 below.
Revised: The results of this process are given in Figure 6 below.

Original: A. R EDUCING R ESONANT R ESPONSES
Revised: A. R EDUCING R ESONANT R ESPONSES

Original: In experiments 4.1 and 4.2, we see that as the momentum parameter is reduced, the tendency to resonate is mitigated.
Revised: In experiments 4.1 and 4.2, we see that as the momentum parameter is reduced, the tendency to resonate is mitigated.

Original: This aligns with theoretical predictions, which (in the context of those particular experiments) suggest that a sufficiently low µ makes SGDm convergent across all frequencies in the band we evaluate.
Revised: This aligns with theoretical predictions, which (in the context of those particular experiments) suggest that a sufficiently low µ makes SGDm convergent across all frequencies in the band we evaluate.

Original: Intuitively, this aligns with the role µ plays in the ODE, as it appears in the following coefficient on the first order derivative of system equation 3 (i.e. the system’s ‘damping’ coefficient), repeated below with the damping coefficient α explicitly labelled:
Revised: Intuitively, this aligns with the role µ plays in the ODE, as it appears in the following coefficient on the first order derivative of system equation 3 (i.e. the system’s ‘damping’ coefficient), repeated below with the damping coefficient α explicitly labelled:

Original: [Equation]
Revised: [Equation]

Original: When α = 0 , the system has no damping (i.e. friction is zero, in the physical analogue) so resonant responses are maximized.
Revised: When α = 0 , the system has no damping (i.e. friction is zero, in the physical analogue) so resonant responses are maximized.

Original: As α increases, damping increases, and resonant responses are reduced.
Revised: As α increases, damping increases, and resonant responses are reduced.

Original: There are two ways to increase α : reduce µ or reduce η .
Revised: There are two ways to increase α : reduce µ or reduce η .

Original: A.2. R EDUCING THE MOMENTUM PARAMETER
Revised: A.2. R EDUCING THE MOMENTUM PARAMETER

Original: Figure 6b shows the theoretical heatmap across all possible momentum values, and across a much wider band of frequencies, which suggests a trend: setting µ to a sufficiently low value will completely mitigate resonance, though setting µ too low will worsen convergence rate.
Revised: Figure 6b shows the theoretical heatmap across all possible momentum values, and across a much wider band of frequencies, which suggests a trend: setting µ to a sufficiently low value will completely mitigate resonance, though setting µ too low will worsen convergence rate.

Original: For now, we will set aside the observation that µ too low worsens convergence rate, and we will now show that reducing µ will reliably dampen resonance across all other experiments.
Revised: For now, we will set aside the observation that µ too low worsens convergence rate, and we will now show that reducing µ will reliably dampen resonance across all other experiments.

Original: In 4.5, ADAM’s parameter β 1 was varied, and we see that reducing β 1 decreases the tendency to resonate.
Revised: In 4.5, ADAM’s parameter β 1 was varied, and we see that reducing β 1 decreases the tendency to resonate.

Original: Since β 1 is the nearest parameter to µ in SGDm, the desired trend has already been
Revised: Since β 1 is the nearest parameter to µ in SGDm, the desired trend has already been

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: demonstrated.
Revised: demonstrated.

Original: For the remainder of this section, we show the resonance-damping behaviour of µ in the remaining experiments: 4.3, 4.4, and 4.6.
Revised: For the remainder of this section, we show the resonance-damping behaviour of µ in the remaining experiments: 4.3, 4.4, and 4.6.

Original: In particular, from each experiment we choose the configuration which had the highest tendency to resonate, and we modify the experiment by running them with several decreasing values of momentum µ .
Revised: In particular, from each experiment we choose the configuration which had the highest tendency to resonate, and we modify the experiment by running them with several decreasing values of momentum µ .

Original: See Figure 7 for results.
Revised: See Figure 7 for results.

Original: In Section 4, these experiments used momentum µ = 0 .
Revised: In Section 4, these experiments used momentum µ = 0 .

Original: 95 , and here we run with µ ∈ [0 . 85 , 0 .
Revised: 95 , and here we run with µ ∈ [0 . 85 , 0 .

Original: 95] , with all other experimental parameters identical.
Revised: 95] , with all other experimental parameters identical.

Original: A.2. R EDUCING THE STEP - SIZE
Revised: A.2. R EDUCING THE STEP - SIZE

Original: Another way to reduce tendency to resonate is suggested by the damping coefficient equation 7: reducing step-size η .
Revised: Another way to reduce tendency to resonate is suggested by the damping coefficient equation 7: reducing step-size η .

Original: Here we repeat Experiments 4.1 and 4.2, including two smaller step-sizes η to empirically demonstrate that trend.
Revised: Here we repeat Experiments 4.1 and 4.2, including two smaller step-sizes η to empirically demonstrate that trend.

Original: Specifically, Figure 8 shows the resonance heatmap and spectral radius contour lines for regression in two weights with covariate shift, identically to Experiments 4.
Revised: Specifically, Figure 8 shows the resonance heatmap and spectral radius contour lines for regression in two weights with covariate shift, identically to Experiments 4.

Original: and 4.2.
Revised: and 4.2.

Original: The left column shows sinusoidal covariate shift, and the right column the AR(2) covariate shift.
Revised: The left column shows sinusoidal covariate shift, and the right column the AR(2) covariate shift.

Original: Each row corresponds to a fixed step-size η ∈ { 0 . 01 , 0 .
Revised: Each row corresponds to a fixed step-size η ∈ { 0 . 01 , 0 .

Original: 001 } , and it is clear that resonant, diverging regions are significantly reduced in size as step-size is decreased, with a narrower band of frequencies diverging, and the minimum momentum µ required for resonance increasing towards 1.
Revised: 001 } , and it is clear that resonant, diverging regions are significantly reduced in size as step-size is decreased, with a narrower band of frequencies diverging, and the minimum momentum µ required for resonance increasing towards 1.

Original: This trend is reflected in both the empirical heatmap results, as well as the theoretically predicted spectral radius contour lines.
Revised: This trend is reflected in both the empirical heatmap results, as well as the theoretically predicted spectral radius contour lines.

Original: A. E XPERIMENTS 4.1 AND 4.2 WITH S TOCHASTIC G RADIENTS
Revised: A. E XPERIMENTS 4.1 AND 4.2 WITH S TOCHASTIC G RADIENTS

Original: Experiment 4.1 1 serves as a simple setting, as close to the theory of Section 3 as possible.
Revised: Experiment 4.1 1 serves as a simple setting, as close to the theory of Section 3 as possible.

Original: In particular, each descent step made at time k uses the gradients induced by 20 samples from each
Revised: In particular, each descent step made at time k uses the gradients induced by 20 samples from each

Original: X k so that the gradient signal is much closer to the expected gradient than the stochastic setting.
Revised: X k so that the gradient signal is much closer to the expected gradient than the stochastic setting.

Original: Experiment 4.2 uses the same technique.
Revised: Experiment 4.2 uses the same technique.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Here we repeat Experiments 4.1 and 4.2, but drawing only a single sample from each X k , so that the gradient signal is fully stochastic.
Revised: Here we repeat Experiments 4.1 and 4.2, but drawing only a single sample from each X k , so that the gradient signal is fully stochastic.

Original: That is, we significantly increase gradient signal’s variance.
Revised: That is, we significantly increase gradient signal’s variance.

Original: As
Revised: As

Original: Figure 9 shows, the resonant response is significantly damped by increasing gradient signal variance.
Revised: Figure 9 shows, the resonant response is significantly damped by increasing gradient signal variance.

Original: We observe no other change in the empirical heatmap: regions of instability are neither introduced nor removed, and the existing regions do not change shape or location.
Revised: We observe no other change in the empirical heatmap: regions of instability are neither introduced nor removed, and the existing regions do not change shape or location.

Original: Only the size of instability regions is affected, which is most apparent under comparison to Figure 8, where the empirically observed instability regions are expanded to closer match the theoretical predictions derived from expected gradients.
Revised: Only the size of instability regions is affected, which is most apparent under comparison to Figure 8, where the empirically observed instability regions are expanded to closer match the theoretical predictions derived from expected gradients.

Original: Also the bright band of instability in all subfigures of Figure 9 are wider than those of 8.
Revised: Also the bright band of instability in all subfigures of Figure 9 are wider than those of 8.

Original: This aligns with intuition from the conventional setting with iid data, where increasing gradient variance reduces optimizer stability.
Revised: This aligns with intuition from the conventional setting with iid data, where increasing gradient variance reduces optimizer stability.

Original:  A.4. A DDITIONAL N OTATION
Revised: A. P ROOFS A.4. A DDITIONAL N OTATION

Original: Wherever we write the training input generating process { X k } , it is implicit that the last dimension is fixed at 1 if one wishes to describe linear models with a bias term.
Revised: Wherever we write the training input generating process { X k } , it is implicit that the last dimension is fixed at 1 if one wishes to describe linear models with a bias term.

Original: In this way, the notation ⟨ θ, X k ⟩ accommodates linear models with or without a bias term.
Revised: In this way, the notation ⟨ θ, X k ⟩ accommodates linear models with or without a bias term.

Original: We use ξ to denote the phase space coordinates of weights θ , meaning ξ = (cid:20) θ ˙ θ (cid:21) , where vectors θ, ˙ θ ∈ R
Revised: We use ξ to denote the phase space coordinates of weights θ , meaning ξ = (cid:20) θ ˙ θ (cid:21) , where vectors θ, ˙ θ ∈ R

Original: d are stacked so that the resulting ξ ∈ R 2 d .
Revised: d are stacked so that the resulting ξ ∈ R 2 d .

Original: We denote the d -dimensional zero vector as 0 d .
Revised: We denote the d -dimensional zero vector as 0 d .

Original: For an ODE system ˙ ξ ( t ) =
Revised: For an ODE system ˙ ξ ( t ) =

Original: f ( ξ ( t ) , t ) with arbitrary solution trajectory ξ ( t ) approximated by a sequence of iterates { ξ k } , we denote the integration time step as h , and the numerical flow as ϕ h,k , which is the map such that ξ k
Revised: f ( ξ ( t ) , t ) with arbitrary solution trajectory ξ ( t ) approximated by a sequence of iterates { ξ k } , we denote the integration time step as h , and the numerical flow as ϕ h,k , which is the map such that ξ k

Original: +

Revised: +

Original: =
Revised: =

Original: ϕ h,k ( ξ k ) .
Revised: ϕ h,k ( ξ k ) .

Original: We denote the k -th discrete timestep as t k , which we always use to refer to the k -th multiple of integration time step h , i.e. t k = hk .
Revised: We denote the k -th discrete timestep as t k , which we always use to refer to the k -th multiple of integration time step h , i.e. t k = hk .

Original: Similar to the main body, when a function of time θ ( t ) is approximated by a discrete time sequence { θ k } , they are denoted with the same symbol, and distinguished by parenthetical argument t and subscript k , respectively.
Revised: Similar to the main body, when a function of time θ ( t ) is approximated by a discrete time sequence { θ k } , they are denoted with the same symbol, and distinguished by parenthetical argument t and subscript k , respectively.

Original: At the risk of abusing notation, we will adhere to this convention and use { ˙ θ k } to be a discrete sequence approximating the function of time ˙ θ ( t ) , which refers to the time derivative of θ ( t ) .
Revised: At the risk of abusing notation, we will adhere to this convention and use { ˙ θ k } to be a discrete sequence approximating the function of time ˙ θ ( t ) , which refers to the time derivative of θ ( t ) .

Original: This means the symbol ˙
Revised: This means the symbol ˙

Original: θ k is the k -th iterate of a sequence approximating ˙
Revised: θ k is the k -th iterate of a sequence approximating ˙

Original: θ ( t ) .
Revised: θ ( t ) .

Original: We now restate Assumptions, Propositions, and the Theorem from Section 3, providing complete proofs.
Revised: We now restate Assumptions, Propositions, and the Theorem from Section 3, providing complete proofs.

Original: A.4. L INEAR T IME -V ARYING E XPECTED G RADIENT VIA C OVARIATE S HIFT
Revised: A.4. L INEAR T IME -V ARYING E XPECTED G RADIENT VIA C OVARIATE S HIFT

Original: Assumption 1.
Revised: Assumption 1.

Original: The covariate generating process { X k } k ∈ N is not identically distributed:
Revised: The covariate generating process { X k } k ∈ N is not identically distributed:

Original: [Equation]
Revised: [Equation]

Original:  ) for some k 1 , k 2 ∈ N .
Revised: = Corr ( X k 2 , X k 2 ) for some k 1 , k 2 ∈ N .

Original: Assumption 2.
Revised: Assumption 2.

Original: The targets Y k are a fixed linear function of X k with iid zero-mean observation noise, ϵ k
Revised: The targets Y k are a fixed linear function of X k with iid zero-mean observation noise, ϵ k

Original: such that E [ ϵ k ] = 0 .
Revised: such that E [ ϵ k ] = 0 .

Original: That is, Y k
Revised: That is, Y k

Original: =
Revised: =

Original: ⟨ θ ∗ , X k ⟩ + ϵ k where θ ∗ ∈ R d is fixed for all k .
Revised: ⟨ θ ∗ , X k ⟩ + ϵ k where θ ∗ ∈ R d is fixed for all k .

Original: Proposition 1.
Revised: Proposition 1.

Original: Under Assumptions 1, 2, a linear model with weights θ k making predictions (cid:98) Y k
Revised: Under Assumptions 1, 2, a linear model with weights θ k making predictions (cid:98) Y k

Original: =
Revised: =

Original: ⟨ θ k , X k ⟩ with a mean squared error (MSE) objective will induce time-varying linear expected loss gradients g k ( θ k )
Revised: ⟨ θ k , X k ⟩ with a mean squared error (MSE) objective will induce time-varying linear expected loss gradients g k ( θ k )

Original: =
Revised: =

Original: B k
Revised: B k

Original: ( θ k − θ ∗ ) for all k ∈ N , where B k ∈ R d × d and B k 1 ̸ = B k 2 for some k 1 , k 2 .
Revised: ( θ k − θ ∗ ) for all k ∈ N , where B k ∈ R d × d and B k 1 ̸ = B k 2 for some k 1 , k 2 .

Original: Proof.
Revised: Proof.

Original: We will show that the gradient of MSE loss ∇ θ k L ( Z ; θ k ) is a random variable whose expectation will take the linear form B k ( θ k − θ ∗ ) .
Revised: We will show that the gradient of MSE loss ∇ θ k L ( Z ; θ k ) is a random variable whose expectation will take the linear form B k ( θ k − θ ∗ ) .

Original: We start with gradient for arbitrary time step k ∇ θ k L ( Z ; θ k ) =
Revised: We start with gradient for arbitrary time step k ∇ θ k L ( Z ; θ k ) =

Original: ∇ θ k ( (cid:98) Y k − Y k )
Revised: ∇ θ k ( (cid:98) Y k − Y k )

Original: MSE def’n = ∇ θ k
Revised: MSE def’n = ∇ θ k

Original: [( ⟨ θ k , X k ⟩ − Y k ) 2 ] (cid:98) Y k def’n from Prop.= ∇ θ k
Revised: [( ⟨ θ k , X k ⟩ − Y k ) 2 ] (cid:98) Y k def’n from Prop.= ∇ θ k

Original: [( ⟨ θ k , X k ⟩ − ⟨ θ ∗ , X k ⟩ + ϵ k ) 2 ]
Revised: [( ⟨ θ k , X k ⟩ − ⟨ θ ∗ , X k ⟩ + ϵ k ) 2 ]

Original: Y k def’n from Ass.= ∇ θ k
Revised: Y k def’n from Ass.= ∇ θ k

Original: [( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) 2 ] ⟨· , ·⟩ distributivity = 2( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) ∇ θ k
Revised: [( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) 2 ] ⟨· , ·⟩ distributivity = 2( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) ∇ θ k

Original: [ ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ] chain rule = 2( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k )
Revised: [ ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ] chain rule = 2( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k )

Original: X k
Revised: X k

Original: = 2 ⟨ θ k − θ ∗ , X k ⟩ X k + 

Revised: = 2 ⟨ θ k − θ ∗ , X k ⟩ X k +

Original: ϵ k X k
Revised: ϵ k X k

Original: Taking expectation with respect to distribution P k :
Revised: Taking expectation with respect to distribution P k :

Original: [Equation]
Revised: [Equation]

Original: g def’n
Revised: g def’n

Original: [Equation]
Revised: [Equation]

Original: by equation* [Equation] linear
Revised: equation

Original: [Equation]
Revised: [Equation]

Original: indep
Revised: E

Original: [Equation]
Revised: [Equation]

Original: scalar and vector commute
Revised: scalar and vector commute

Original: [Equation]
Revised: [Equation]

Original: commutative

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ·⟩ matrix form
Revised: ·⟩ matrix form

Original: [Equation]
Revised: [Equation]

Original: matrix mult.
Revised: matrix mult.

Original: associative
Revised: associative

Original: [Equation]
Revised: [Equation]

Original: In the last step, we have defined the matrix B k as twice the expected outer product of X k with itself, which is precisely the autocorrelation matrix, and is a function of X k ’s first two moments as follows.
Revised: In the last step, we have defined the matrix B k as twice the expected outer product of X k with itself, which is precisely the autocorrelation matrix, and is a function of X k ’s first two moments as follows.

Original: [Equation]
Revised: [Equation]

Original: So if there exists k 1 , k 2 ∈ N such that the autocorrelation matrices differ
Revised: So if there exists k 1 , k 2 ∈ N such that the autocorrelation matrices differ

Original: [Equation]
Revised: [Equation]

Original: then B k 1 ̸ = B k 2 .
Revised: then B k 1 ̸ = B k 2 .

Original: Assumption 1 provides the necessary k 1 , k 2 , so the proposition is proved.
Revised: Assumption 1 provides the necessary k 1 , k 2 , so the proposition is proved.

Original: A.4. ODE C ORRESPONDENCE
Revised: A.4. ODE C ORRESPONDENCE

Original: In Section 3 of the main body, we defined B ( t ) , Lipschitz continuous in t , such that B k = B ( √ ηk ) .
Revised: In Section 3 of the main body, we defined B ( t ) , Lipschitz continuous in t , such that B k = B ( √ ηk ) .

Original: If we assume that { X k } are sampled at intervals of √ η from an underlying continuous-time stochastic process X ( t ) for which E P ( t ) [ X ( t )] and Cov ( X ( t ) , X ( t )) are Lipschitz continuous in t , this naturally induces a unique Lipschitz continuous B ( t ) .
Revised: If we assume that { X k } are sampled at intervals of √ η from an underlying continuous-time stochastic process X ( t ) for which E P ( t ) [ X ( t )] and Cov ( X ( t ) , X ( t )) are Lipschitz continuous in t , this naturally induces a unique Lipschitz continuous B ( t ) .

Original: Note P ( t ) is the distribution at time t , and we can extend equation 2 as follows.
Revised: Note P ( t ) is the distribution at time t , and we can extend equation 2 as follows.

Original: [Equation]
Revised: [Equation]

Original: This means that { B k } is sampled from a continuous B ( t ) , with subsequent B k spaced √ η apart.
Revised: This means that { B k } is sampled from a continuous B ( t ) , with subsequent B k spaced √ η apart.

Original: In essence, step-size η acts as a scaling constant, such that tne unit of t is ≈ 1 √ η discrete steps of k .
Revised: In essence, step-size η acts as a scaling constant, such that tne unit of t is ≈ 1 √ η discrete steps of k .

Original: (Exact when 1 √ η is an integer.)
Revised: (Exact when 1 √ η is an integer.)

Original: Proposition 2.
Revised: Proposition 2.

Original: The SGDm iterates { θ k } numerically integrate the ODE system equation 3 with integration step √ η and first order consistency.
Revised: The SGDm iterates { θ k } numerically integrate the ODE system equation 3 with integration step √ η and first order consistency.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof proceeds two parts.
Revised: The proof proceeds two parts.

Original: First we derive a first order operator-splitting integrator for the
Revised: First we derive a first order operator-splitting integrator for the

Original: ODE equation 3, then we show that it is equivalent to SGDm equation 1.
Revised: ODE equation 3, then we show that it is equivalent to SGDm equation 1.

Original: (Part 1: Operator-Splitting Integrator)
Revised: (Part 1: Operator-Splitting Integrator)

Original: Let ξ : R (cid:55)→ R 2 d be the vector valued function of time whose first d elements are θ ( t ) , and last d elements are ˙ θ ( t ) :ξ ( t )
Revised: Let ξ : R (cid:55)→ R 2 d be the vector valued function of time whose first d elements are θ ( t ) , and last d elements are ˙ θ ( t ) :

Original: = (cid:20) θ ( t ) − θ ∗ ˙
Revised: [Equation]

Original: θ ( t ) (cid:21) (9) i.e. ξ is a phase space transformation allowing us to rewrite ODE equation 3 in the form of equation 4, which can then be split into a sum of separate systems f
Revised: i.e. ξ is a phase space transformation allowing us to rewrite ODE equation 3 in the form of equation 4, which can then be split into a sum of separate systems f

Original: [1] , f [2] as follows
Revised: [1] , f [2] as follows

Original: [Equation]
Revised: [Equation]

Original: Let h > 0 be an integration time step, ϕ [1] h,k ( ξ k ) be the implicit Euler numerical flow of f [1] ( ξ ( t ) , t ) :
Revised: Let h > 0 be an integration time step, ϕ [1] h,k ( ξ k ) be the implicit Euler numerical flow of f [1] ( ξ ( t ) , t ) :

Original: [Equation]
Revised: [Equation]

Original: Let ϕ [2] h,k ( ξ k ) be explicit Euler numerical flow of f [2] ( ξ ( t ) , t ) :[Equation]
Revised: Let ϕ [2] h,k ( ξ k ) be explicit Euler numerical flow of f [2] ( ξ ( t ) , t ) :

Original: [Equation]
Revised: [Equation]

Original: The composed flow
Revised: The composed flow

Original: [Equation]
Revised: [Equation]

Original: 13) is a sequentially split operator, which has splitting error order 1 because equation 10 is time-varying linear system (Faragó et al., 2011).
Revised: (13) is a sequentially split operator, which has splitting error order 1 because equation 10 is time-varying linear system (Faragó et al., 2011).

Original: The operators being composed, implicit and explicit Euler, are both order 1 consistent with their respective systems (Iserles, 2008).
Revised: The operators being composed, implicit and explicit Euler, are both order 1 consistent with their respective systems (Iserles, 2008).

Original: The overall order of consistency of a split operator is the minimum of splitting error, and the orders of the composed flows (Csomós &
Revised: The overall order of consistency of a split operator is the minimum of splitting error, and the orders of the composed flows (Csomós &

Original: Faragó, 2008).
Revised: Faragó, 2008).

Original: So we have that equation 13 approximates equation 10 with order 1 consistency.
Revised: So we have that equation 13 approximates equation 10 with order 1 consistency.

Original: (Part 2: SGDm Equivalency)
Revised: (Part 2: SGDm Equivalency)

Original: We will now show that equation 13 is equivalent to SGDm when integration timestep h = √ η , where η is the SGDm step-size.
Revised: We will now show that equation 13 is equivalent to SGDm when integration timestep h = √ η , where η is the SGDm step-size.

Original: We start with the definition of ξ k +1 as the numerical flow of ξ k :
Revised: We start with the definition of ξ k +1 as the numerical flow of ξ k :

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]

Original: =

Original: [Equation]

Original: by equation
Revised: by equation

Original: [Equation]
Revised: [Equation]

Original: subst’n from equation
Revised: subst’n from equation

Original: [Equation]
Revised: [Equation]

Original: def’n of ξ k+
Revised: def’n of ξ k

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: matrix mult.
Revised: matrix mult.

Original: [Equation]
Revised: [Equation]

Original: subst’n from equation
Revised: subst’n from equation

Original: [Equation]
Revised: [Equation]

Original: matrix mult.
Revised: matrix mult.

Original: [Equation]
Revised: [Equation]

Original: simplification
Revised: simplification

Original: Recalling that ξ k = (cid:20) θ k − θ ∗ ˙ θ k (cid:21) [Equation] the above immediately provides the following two recurrence relations:
Revised: Recalling that ξ k = (cid:20) θ k − θ ∗ ˙ θ k (cid:21) , the above immediately provides the following two recurrence relations:

Original: [Equation]
Revised: [Equation]

Original: Now let integration time step h be the square root of SGDm step-size, h = √ η , and define v k = √
Revised: Now let integration time step h be the square root of SGDm step-size, h = √ η , and define v k = √

Original: η ˙ θ k .
Revised: η ˙ θ k .

Original: Substituting h, v k , we proceed via elementary algebra:
Revised: Substituting h, v k , we proceed via elementary algebra:

Original: [Equation]
Revised: [Equation]

Original: Note that B ( t k )
Revised: Note that B ( t k )

Original: =
Revised: =

Original: B k , and that B k defines the time-varying gradients induced by covariate shift and linear regression to a linear target (Proposition 1).
Revised: B k , and that B k defines the time-varying gradients induced by covariate shift and linear regression to a linear target (Proposition 1).

Original: Hence, under those conditions, we have arrived at
Revised: Hence, under those conditions, we have arrived at

Original: SGDm equation 1.
Revised: SGDm equation 1.

Original: The proof is complete.
Revised: The proof is complete.

Original: The reader may observe that, given a solution ξ ( t ) to the system ˙ ξ ( t ) =
Revised: The reader may observe that, given a solution ξ ( t ) to the system ˙ ξ ( t ) =

Original: A ( t ) ξ ( t ) , the proof above shows that the SGDm iterates { θ k } are precisely a first order numerical approximation of the first d dimensions of ξ ( t )
Revised: A ( t ) ξ ( t ) , the proof above shows that the SGDm iterates { θ k } are precisely a first order numerical approximation of the first d dimensions of ξ ( t )

Original: = (cid:20) θ ( t ) − θ ∗ ˙
Revised: = (cid:20) θ ( t ) − θ ∗ ˙

Original: θ ( t ) (cid:21) , but the remaining d dimensions are approximated only up to a scale factor √ η by iterates { v k } .
Revised: θ ( t ) (cid:21) , but the remaining d dimensions are approximated only up to a scale factor √ η by iterates { v k } .

Original: However, this is not an issue.
Revised: However, this is not an issue.

Original: Solutions θ ( t ) to the system equation 3 are embedded within the first d dimensions of ξ ( t ) , so the remaining d dimensions and iterates { v k } do not affect the result.
Revised: Solutions θ ( t ) to the system equation 3 are embedded within the first d dimensions of ξ ( t ) , so the remaining d dimensions and iterates { v k } do not affect the result.

Original: A.4. P ARAMETRIC R ESONANCE FOR ODE C ONVERGENCE AND D IVERGENCE
Revised: A.4. P ARAMETRIC R ESONANCE FOR ODE C ONVERGENCE AND D IVERGENCE

Original: Theorem 1.
Revised: Theorem 1.

Original: When B ( t ) is periodic such that B ( t )
Revised: When B ( t ) is periodic such that B ( t )

Original: =
Revised: =

Original: B ( t + T ) , the spectral radius ρ of ψ ( T ) characterizes the stability of solution trajectories of equation 3 as follows:
Revised: B ( t + T ) , the spectral radius ρ of ψ ( T ) characterizes the stability of solution trajectories of equation 3 as follows:

Original: • ρ > 1 = ⇒ trivial solution θ ( t )
Revised: • ρ > 1 = ⇒ trivial solution θ ( t )

Original: =
Revised: =

Original: θ ∗ is unstable.
Revised: θ ∗ is unstable.

Original: All other solutions diverge as θ ( t ) → ∞ exponentially with rate ρ .
Revised: All other solutions diverge as θ ( t ) → ∞ exponentially with rate ρ .

Original: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as θ ( t ) → θ ∗ exponentially with rate ρ .
Revised: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as θ ( t ) → θ ∗ exponentially with rate ρ .

Original: Proof.
Revised: Proof.

Original: The proof of this theorem relies heavily on the well-established mathematics of Floquet theory and the stability result is contained in Theorem 1.9 and Theorem 1.10 of (Halanay, 1966).
Revised: The proof of this theorem relies heavily on the well-established mathematics of Floquet theory and the stability result is contained in Theorem 1.9 and Theorem 1.10 of (Halanay, 1966).

Original: Theorem 1.10 states that for a system of differential equations of the form
Revised: Theorem 1.10 states that for a system of differential equations of the form

Original: [Equation]
Revised: [Equation]

Original: where A ( t ) is piecewise continuous, the stability of the trivial solution ξ ( t ) ≡ 0 is determined by the spectral radius of the system’s monodromy matrix M defined below
Revised: where A ( t ) is piecewise continuous, the stability of the trivial solution ξ ( t ) ≡ 0 is determined by the spectral radius of the system’s monodromy matrix M defined below

Original: [Equation]
Revised: [Equation]

Original: where ˙ ψ ( t ) =
Revised: where ˙ ψ ( t ) =

Original: A ( t ) ψ ( t ) (Monodromy Matrix)
Revised: A ( t ) ψ ( t ) ( Monodromy Matrix)

Original: Matrix-valued functions ψ ( t ) are the system’s fundamental solution matrix , and elementary existence results for linear systems allow one to choose a ψ ( t ) such that ψ (0)
Revised: Matrix-valued functions ψ ( t ) are the system’s fundamental solution matrix , and elementary existence results for linear systems allow one to choose a ψ ( t ) such that ψ (0)

Original: =
Revised: =

Original: I so that the monodromy matrix simplifies as
Revised: I so that the monodromy matrix simplifies as

Original: [Equation]
Revised: [Equation]

Original: Below we denote the spectral radius of ψ ( T ) (and hence of M ) as ρ .
Revised: Below we denote the spectral radius of ψ ( T ) (and hence of M ) as ρ .

Original: (First Order Linear Form, Stability via Floquet)
Revised: (First Order Linear Form, Stability via Floquet)

Original: Below we show that equation 3 can be transformed to the form equation 14 with A ( t ) continuous such that trivial solution stability of equation 14 is equivalent to stability of the solution θ ( t )
Revised: Below we show that equation 3 can be transformed to the form equation 14 with A ( t ) continuous such that trivial solution stability of equation 14 is equivalent to stability of the solution θ ( t )

Original: = θ ∗ .
Revised: = θ ∗ .

Original: Let ξ ( t ) be the phase space transformation of θ ( t ) , similarly to the proof of Proposition
Revised: Let ξ ( t ) be the phase space transformation of θ ( t ) , similarly to the proof of Proposition

Original: [Equation]
Revised: [Equation]

Original: so that for each t , θ ( t ) ∈ R d and ξ ( t ) ∈ R 2 d .
Revised: so that for each t , θ ( t ) ∈ R d and ξ ( t ) ∈ R 2 d .

Original: This means equation 3 (restated below)
Revised: This means equation 3 (restated below)

Original: [Equation]
Revised: [Equation]

Original: is equivalent to the following first order linear form
Revised: is equivalent to the following first order linear form

Original: [Equation]
Revised: [Equation]

Original: Since B ( t ) is periodic and continuous, which is stronger than the requisite piecewise continuity.
Revised: Since B ( t ) is periodic and continuous, which is stronger than the requisite piecewise continuity.

Original: Since all other submatrices of A ( t ) are constant, we have that A ( t ) is also periodic and (piecewise) continuous.
Revised: Since all other submatrices of A ( t ) are constant, we have that A ( t ) is also periodic and (piecewise) continuous.

Original: Now Theorem 1.10 in (Halanay, 1966) immediately provides the following
Revised: Now Theorem 1.10 in (Halanay, 1966) immediately provides the following

Original: • ρ > 1 = ⇒ trivial solution ξ ( t ) = 0 is unstable.
Revised: • ρ > 1 = ⇒ trivial solution ξ ( t ) = 0 is unstable.

Original: All other solutions diverge as ξ ( t ) → ∞ exponentially with rate ρ .
Revised: All other solutions diverge as ξ ( t ) → ∞ exponentially with rate ρ .

Original: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as ξ ( t ) →exponentially with rate ρ .
Revised: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as ξ ( t ) →exponentially with rate ρ .

Original: Since ξ ( t ) =
Revised: Since ξ ( t ) =

Original: [Equation]
Revised: [Equation]

Original: • The trivial solution ξ ( t ) = 0 is equivalent to θ ( t )
Revised: • The trivial solution ξ ( t ) = 0 is equivalent to θ ( t )

Original: = θ ∗ • ξ ( t ) → 0 is equivalent to θ ( t ) → θ ∗ • ξ ( t ) → ∞ is equivalent to θ ( t ) → ∞
Revised: = θ ∗ • ξ ( t ) → 0 is equivalent to θ ( t ) → θ ∗ • ξ ( t ) → ∞ is equivalent to θ ( t ) → ∞

Original: The theorem is proved.
Revised: The theorem is proved.

Original: A. D ETAILED DESCRIPTIONS OF EXPERIMENTS
Revised: A. D ETAILED DESCRIPTIONS OF EXPERIMENTS

Original: Below we provide details of the experiments from section 4 in the main body.
Revised: Below we provide details of the experiments from section 4 in the main body.

Original: The tables below contain details about the data generating process, learning algorithm and optimizer, and hyperparameter choices.
Revised: The tables below contain details about the data generating process, learning algorithm and optimizer, and hyperparameter choices.

Original: Experiments 4.1-4.4 introduce new data generating processes { X k } via their mean sequences { ¯ x k } .
Revised: Experiments 4.1-4.4 introduce new data generating processes { X k } via their mean sequences { ¯ x k } .

Original: For those experiments, a sample trajectory is visualized for several values of the process’ frequency tuning parameter, either f or T .
Revised: For those experiments, a sample trajectory is visualized for several values of the process’ frequency tuning parameter, either f or T .

Original: In addition to the sample trajectory, the frequency content of each process is shown in two ways: the power spectral density (PSD) of { X k } estimated via a long sample trajectory, as well as the PSD of the underlying mean sequence { ¯ x k } .
Revised: In addition to the sample trajectory, the frequency content of each process is shown in two ways: the power spectral density (PSD) of { X k } estimated via a long sample trajectory, as well as the PSD of the underlying mean sequence { ¯ x k } .

Original: As one would expect from the sampling relationship between X k and ¯ x k , the PSD of { X k } is simply the PSD of { ¯ x k } with a noise floor.
Revised: As one would expect from the sampling relationship between X k and ¯ x k , the PSD of { X k } is simply the PSD of { ¯ x k } with a noise floor.

Original: Experiments 4.5, 4.6 reuse the process from experiment 4.4, so the visual depiction is elided.
Revised: Experiments 4.5, 4.6 reuse the process from experiment 4.4, so the visual depiction is elided.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Figure]

Revised: .2 D ETAILS

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Revised: Remaining Parameters

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: for three values of frequency tuning parameter T .
Revised: for three values of frequency tuning parameter T .

Original: The signal is a square wave in a single dimension for the figure, but is a square wave randomly oriented in higher dimensions for theactual experiment.
Revised: The signal is a square wave in a single dimension for the figure, but is a square wave randomly oriented in higher dimensions for the experiment.

Original: There are multiple frequency peaks, with the frequency domain contracted by changing T .
Revised: There are multiple frequency peaks, with the frequency domain contracted by changing T .

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: d
Revised: +1 ) [Equation] c = 0 . [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: iid
Revised: N

Original: [Table] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: d +

Original: [Equation]
Revised: [Equation]

Original: θ ∗ ∼ N (0 , cI d +1 × d +1 ) where c
Revised: d +1 ) where c

Original: = 0 .
Revised: = 0 .

Original: [Equation]
Revised: [Equation]

Original: ∼ N (0 , 0 . 1) Model and Optimizer (cid:99) Y
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: = ⟨ θ [1: d ] , X
Revised: η [Equation] [Equation] [Equation]

Revised: µ

Original: [Equation]
Revised: [Equation]

Original: ⟩ + θ d + θ 0 ∼ N (0 , cI d +1 × d +1 ) where c = 0 .
Revised: [Equation] [Equation] [Equation] [Table]

Original: ( θ k ) k ∈ N ← SGDm ( η, µ ) η = 0 .
Revised: [Table] [Table] Figure 4b [Table]

Original: 01 , µ = 0 .k ∈
Revised: Figure

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: (Tables 4-7) for three values of frequency tuning parameter T .

Revised: The mean signal is essentially white noise ‘stretched’ according to T .

Revised: The signal is a single dimension for the figure, but is of higher dimensions for the actual experiment.

Revised: The frequency content is a vertical reflection of Figure(right,) with frequency troughs instead of peaks.

Revised: Note that the frequency content is the mildest of all settings, having the lowest and least-defined peaks, which aligns with intuition: there is nothing even remotely oscillatory in its definition.

Revised: It is reasonable to expect that many ‘natural’ processes would have more aggressive frequency content.

Original: A.5. E XPERIMENT 4.5 D ETAILS
Revised: A.5. E XPERIMENT 4.5 D ETAILS

Original: For visual depiction, see Figure 13.
Revised: For visual depiction, see Figure 13.

Original: [Table]

Original: to Table

Original: [Table]
Revised: [Table]

Original: A.5. E XPERIMENT 4.6 D ETAILS
Revised: A.5. E XPERIMENT 4.6 D ETAILS

Original: For visual depiction, see Figure 13.
Revised: For visual depiction, see Figure 13.

Original: [Table]
Revised: [Table]

Original: N ← SGDm (

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]