Original: R ESONANCE IN W EIGHT S PACE : C OVARIATE S HIFT C AN D RIVE D IVERGENCE OF SGD WITH M OMENTUM
Revised: R ESONANCE IN W EIGHT S PACE : C OVARIATE S HIFT C AN D RIVE D IVERGENCE OF SGD WITH M OMENTUM

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Most convergence guarantees for stochastic gradient descent with momentum (SGDm) rely on iid sampling.
Revised: Most convergence guarantees for stochastic gradient descent with momentum (SGDm) rely on iid sampling.

Original: Yet, SGDm is often used outside this regime, in settings with temporally correlated input samples such as continual learning and reinforcement learning.
Revised: Yet, SGDm is often used outside this regime, in settings with temporally correlated input samples such as continual learning and reinforcement learning.

Original: Existing work has shown that SGDm with a decaying stepsize can converge under Markovian temporal correlation
Revised: Existing work has shown that SGDm with a decaying stepsize can converge under Markovian temporal correlation.

Original: In this work, we show that SGDm under covariate shift with a ﬁxed step size can be unstable and diverge
Revised: In this work, we show that SGDm under covariate shift with a fixed step-size can be unstable and diverge.

Original: In particular , we show SGDm under covariate shift is a parametric oscillator, and so can suffer from a phenomenon known as resonance.
Revised: In particular, we show SGDm under covariate shift is a parametric oscillator, and so can suffer from a phenomenon known as resonance.

Original: We approximate the learning system as a time varying system of ordinary differential equations, and leverage existing theory to characterize the system’s divergence/convergence as resonant/nonresonant modes.
Revised: We approximate the learning system as a time varying system of ordinary differential equations, and leverage existing theory to characterize the system’s divergence/convergence as resonant/nonresonant modes.

Original: The theoretical result is limited to the linear setting with periodic covariate shift, so we empirically supplement this result to show that resonance phenomena persist even under non-periodic covariate shift nonlinear dynamics with neural networks and optimizers other than SGDm
Revised: The theoretical result is limited to the linear setting with periodic covariate shift, so we empirically supplement this result to show that resonance phenomena persist even under non-periodic covariate shift, nonlinear dynamics with neural networks, and optimizers other than SGDm.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Stochastic gradient descent (SGD) (Robbins & Monro, 1951) – and its variants such as Adagrad (Duchi et al., 2011), ADAM (Kingma & Ba, 2014) and RMSprop (Hinton et al., 2012) – are very widely used optimization algorithms across machine learning.
Revised: Stochastic gradient descent (SGD) (Robbins & Monro, 1951) – and its variants such as Adagrad (Duchi et al., 2011), ADAM (Kingma & Ba, 2014) and RMSprop (Hinton et al., 2012) – are very widely used optimization algorithms across machine learning.

Original: SGD is conceptually straightforward, easy to implement, and often performs well in practice.
Revised: SGD is conceptually straightforward, easy to implement, and often performs well in practice.

Original: Among the variants of SGD, accelerated versions based on Polyak’s or Nesterov’s acceleration (Polyak, 1964; Nesterov, 1983), known generally as Stochastic Gradient Descent with Momentum (SGDm), are used widely due to the improvements in convergence rate they offer.
Revised: Among the variants of SGD, accelerated versions based on Polyak’s or Nesterov’s acceleration (Polyak, 1964; Nesterov, 1983), known generally as Stochastic Gradient Descent with Momentum (SGDm), are used widely due to the improvements in convergence rate they offer.

Original: SGDm can give up to a quadratic speedup to SGD on many functions, and is in fact optimal among all methods having only information about the gradient at consecutive iterates for convex and Lipschitz continuous optimization problems (Nesterov, 2004; Goh, 2017).
Revised: SGDm can give up to a quadratic speedup to SGD on many functions, and is in fact optimal among all methods having only information about the gradient at consecutive iterates for convex and Lipschitz continuous optimization problems (Nesterov, 2004; Goh, 2017).

Original: SGDm has the same computational complexity as SGD, but exhibits superior convergence rates under reasonable assumptions (Su et al., 2014).
Revised: SGDm has the same computational complexity as SGD, but exhibits superior convergence rates under reasonable assumptions (Su et al., 2014).

Original: These convergence results for SGDm, however, rely on independent and identically distributed (iid) sampling.
Revised: These convergence results for SGDm, however, rely on independent and identically distributed (iid) sampling.

Original: Little is known about the convergence properties of SGDm under non-iid sampling, yet the non-iid setting is critical.
Revised: Little is known about the convergence properties of SGDm under non-iid sampling, yet the non-iid setting is critical.

Original: In many machine learning problems it is expensive or impossible to obtain iid samples.
Revised: In many machine learning problems it is expensive or impossible to obtain iid samples.

Original: In online learning (Rakhlin et al., 2010) and reinforcement learning (RL) (Sutton & Barto, 2018), the data becomes available in a sequential order and there is a temporal dependence among the samples.
Revised: In online learning (Rakhlin et al., 2010) and reinforcement learning (RL) (Sutton & Barto, 2018), the data becomes available in a sequential order and there is a temporal dependence among the samples.

Original: There is a particularly strong temporal dependence in RL, where observed states are sampled according to the transition dynamics of the underlying Markov decision process (MDP).
Revised: There is a particularly strong temporal dependence in RL, where observed states are sampled according to the transition dynamics of the underlying Markov decision process (MDP).

Original: Federated learning (Hsieh et al., 2020) and time-series learning (Kuznetsov & Mohri, 2014) provide further examples of when non-iid sampling is essential to the learning problem.
Revised: Federated learning (Hsieh et al., 2020) and time-series learning (Kuznetsov & Mohri, 2014) provide further examples of when non-iid sampling is essential to the learning problem.

Original: Without momentum, SGD’s convergence rate has been examined under non-iid sampling with the stochastic approximation framework in (Benveniste et al., 2012; Kushner & Yin, 2003), and more recently under speciﬁc assumptions of ergodicity (Duchi et al., 2012) or Markovian sampling (Nagaraj et al., 2020; Doan et al., 2020b; Sun et al., 2018).
Revised: Without momentum, SGD’s convergence rate has been examined under non-iid sampling with the stochastic approximation framework in (Benveniste et al., 2012; Kushner & Yin, 2003), and more recently under specific assumptions of ergodicity (Duchi et al., 2012) or Markovian sampling (Nagaraj et al., 2020; Doan et al., 2020b; Sun et al., 2018).

Original: Convergence rates under Markovian sampling are also known for ADAM-type algorithms when applied to policy gradient and temporal difference learning (Xiong et al., 2020).
Revised: Convergence rates under Markovian sampling are also known for ADAM-type algorithms when applied to policy gradient and temporal difference learning (Xiong et al., 2020).

Original: To our knowledge, however, there has been little work on providing convergence rates or guarantees for SGDm under non-iid sampling.
Revised: To our knowledge, however, there has been little work on providing convergence rates or guarantees for SGDm under non-iid sampling.

Original: In (Doan et al., 2020a), a progress bound is provided for SGDm under Markovian sampling based on mixing time—the time required for a distribution’s convergence toward its stationary distribution—along with a convergence rate guarantee under decaying step-sizes.
Revised: In (Doan et al., 2020a), a progress bound is provided for SGDm under Markovian sampling based on mixing time—the time required for a distribution’s convergence toward its stationary distribution—along with a convergence rate guarantee under decaying step-sizes.

Original: In this work, we assume a ﬁxed step-size, since it is a common choice in the online setting, especially when the practitioner is unsure of mixing rate or stationarity.
Revised: In this work, we assume a fixed step-size, since it is a common choice in the online setting, especially when the practitioner is unsure of mixing rate or stationarity.

Original: There is a broad literature using ordinary differential equations (ODEs) to analyze gradient descent methods by approximating descent updates as continuous-time ﬂows.
Revised: There is a broad literature using ordinary differential equations (ODEs) to analyze gradient descent methods by approximating descent updates as continuous-time flows.

Original: Early work is comprehensively discussed in (Kushner & Yin, 2003; Benveniste et al., 2012).
Revised: Early work is comprehensively discussed in (Kushner & Yin, 2003; Benveniste et al., 2012).

Original: Despite the age and establishment of the linear setting, the gradient ﬂow lens continues to reveal new insights (e.g. implicit rank reduction (Arora et al., 2019)) and new perspectives on old insights (e.g. regularization of early stopping (Ali et al., 2019)).
Revised: Despite the age and establishment of the linear setting, the gradient flow lens continues to reveal new insights (e.g. implicit rank reduction (Arora et al., 2019)) and new perspectives on old insights (e.g. regularization of early stopping (Ali et al., 2019)).

Original: Recent work has paid particular attention to the ﬂow induced by momentum accelerated methods (Su et al., 2014;
Revised: Recent work has paid particular attention to the flow induced by momentum accelerated methods (Su et al., 2014;

Original: Wibisono et al., 2016; Wilson et al., 2016;
Revised: Wibisono et al., 2016; Wilson et al., 2016;

Original: Scieur et al., 2017; Muehlebach &
Revised: Scieur et al., 2017; Muehlebach &

Original: Jordan, 2021; Diakonikolas & Orecchia, 2019; Muehlebach & Jordan, 2019; Li et al., 2017; Simsekli et al., 2020; Betancourt et al., 2018; Kovachki & Stuart, 2019; Attouch et al., 2018; Shi et al., 2019;
Revised: Jordan, 2021; Diakonikolas & Orecchia, 2019; Muehlebach & Jordan, 2019; Li et al., 2017; Simsekli et al., 2020; Betancourt et al., 2018; Kovachki & Stuart, 2019; Attouch et al., 2018; Shi et al., 2019;

Original: Siegel, 2019; Zhang et al., 2018; Berthier et al., 2021; Kovachki & Stuart, 2021).
Revised: Siegel, 2019; Zhang et al., 2018; Berthier et al., 2021; Kovachki & Stuart, 2021).

Original: In these works, it is demonstrated that linear regression under iid sampling with SGDm can be represented as an ODE resembling a harmonic oscillator, sometimes with a time-decaying damping coefﬁcient.
Revised: In these works, it is demonstrated that linear regression under iid sampling with SGDm can be represented as an ODE resembling a harmonic oscillator, sometimes with a time-decaying damping coefficient.

Original: Contributions:
Revised: Contributions:

Original: In this work, we show that non-iid sampling, due to covariate shift, induces a related system: the parametric oscillator, a harmonic oscillator having coefﬁcients which vary over time in a manner capable of exponentially exciting the system (Mumford, 1960; Csörg˝o & Hatvani, 2010;
Revised: In this work, we show that non-iid sampling, due to covariate shift, induces a related system: the parametric oscillator, a harmonic oscillator having coefficients which vary over time in a manner capable of exponentially exciting the system (Mumford, 1960; Csörg˝o & Hatvani, 2010;

Original: Halanay, 1966).
Revised: Halanay, 1966).

Original: We characterize the underlying time-varying ODE, which requires a novel approach to incorporate covariate shift, and develop speciﬁc conditions on covariate shift that lead to divergence in SGDm.
Revised: We characterize the underlying time-varying ODE, which requires a novel approach to incorporate covariate shift, and develop specific conditions on covariate shift that lead to divergence in SGDm.

Original: We provide an empirical design, with synthetic data, to systematically test the response of the learning system to different covariate shift frequencies.
Revised: We provide an empirical design, with synthetic data, to systematically test the response of the learning system to different covariate shift frequencies.

Original: We empirically validate that resonancedriven divergence occurs in a learning system which aligns well with theoretical assumptions.
Revised: We empirically validate that resonancedriven divergence occurs in a learning system which aligns well with theoretical assumptions.

Original: We follow with similar demonstrations on learning systems which progressively relax further and further away from our theoretical assumptions, including non-periodic covariate shift, learning with neural networks and optimizers other than SDGm.
Revised: We follow with similar demonstrations on learning systems which progressively relax further and further away from our theoretical assumptions, including non-periodic covariate shift, learning with neural networks and optimizers other than SDGm.

Original: The implications of this work are that resonance can occur in learning systems with momentum, causing instability or poor accuracy, simply due to certain temporal correlations in the input sequence.
Revised: The implications of this work are that resonance can occur in learning systems with momentum, causing instability or poor accuracy, simply due to certain temporal correlations in the input sequence.

Original: Our results provide a novel direction to better understand the properties of our learning systems for time varying inputs—namely beyond iid sampling—and point to a gap in the robustness of our algorithms that merits further investigation.
Revised: Our results provide a novel direction to better understand the properties of our learning systems for time varying inputs—namely beyond iid sampling—and point to a gap in the robustness of our algorithms that merits further investigation.

Original:  P ROBLEM S ETTING
Revised:  P ROBLEM S ETTING

Original: We investigate the effect of non-iid sampling on a model optimized using SGDm.
Revised: We investigate the effect of non-iid sampling on a model optimized using SGDm.

Original: We assume labelled training data sampled from discreteindexed, real-valued stochastic processes { X k } k ∈ N and { Y k } k ∈ N such that { X k } k ∈ N has a unique stationary distribution Π .
Revised: We assume labelled training data sampled from discreteindexed, real-valued stochastic processes { X k } k ∈ N and { Y k } k ∈ N such that { X k } k ∈ N has a unique stationary distribution Π .

Original: Y k is a function of
Revised: Y k is a function of

Original: X k with zero-mean observation noise,
Revised: X k with zero-mean observation noise,

Original: Y k
Revised: Y k

Original: = f ( X k ) + (cid:15) k .
Revised: = f ( X k ) + ϵ k .

Original: The learning algorithm does not have access to iid samples from Π .
Revised: The learning algorithm does not have access to iid samples from Π .

Original: Instead, the learning algorithm may only update parameters θ k at time k using samples z k
Revised: Instead, the learning algorithm may only update parameters θ k at time k using samples z k

Original: = ( x k , y k ) , where X k ’s marginal over time converges to Π .
Revised: = ( x k , y k ) , where X k ’s marginal over time converges to Π .

Original: The goal is to train a model with parameters θ to minimize an objective function L ( z ; θ ) with respect to the stationary distribution Π : θ ∗ =
Revised: The goal is to train a model with parameters θ to minimize an objective function L ( z ; θ ) with respect to the stationary distribution Π : θ ∗ =

Original: arg min θ ∈ R d E Π [ L ( z ; θ )]
Revised: arg min θ ∈ R d E Π [ L ( z ; θ )]

Original: = (cid:82) R L ( z ; θ )
Revised: = (cid:82) R L ( z ; θ )

Original: d Π( x ) .
Revised: d Π( x ) .

Original: An example of this problem setting is illustrated in Figure 1.
Revised: An example of this problem setting is illustrated in Figure 1.

Original: This setting is identical to that explored in prior work on Markovian sampling (Sun et al., 2018;
Revised: This setting is identical to that explored in prior work on Markovian sampling (Sun et al., 2018;

Original: Xiong et al., 2020; Nagaraj et al., 2020), but we do not require that our stochastic processes adhere to the Markov property.
Revised: Xiong et al., 2020; Nagaraj et al., 2020), but we do not require that our stochastic processes adhere to the Markov property.

Original: Note, also, that the underlying functional relationship between the inputs, X k , and the targets, Y k , does not change over time.
Revised: Note, also, that the underlying functional relationship between the inputs, X k , and the targets, Y k , does not change over time.

Original: That is, the non-iid sampling is a result of covariate shift rather than a changing relationship between the inputs and targets.
Revised: That is, the non-iid sampling is a result of covariate shift rather than a changing relationship between the inputs and targets.

Original: We consider Polyak’s Heavy Ball method (Polyak, 1964) with the formulation from Sutskever et al.
Revised: We consider Polyak’s Heavy Ball method (Polyak, 1964) with the formulation from Sutskever et al.

Original: (2013), namely SGD with momentum (SGDm)
Revised: (2013), namely SGD with momentum (SGDm)

Original: [Equation]
Revised: [Equation]

Original: where η ≥ 0 is the learning rate and µ ∈ (0 , 1) the momentum coefﬁcient.
Revised: where η ≥ 0 is the learning rate and µ ∈ (0 , 1) the momentum coefficient.

Original: Our goal is to understand the behavior of SGDm when learning on this non-iid sequence z k .
Revised: Our goal is to understand the behavior of SGDm when learning on this non-iid sequence z k .

Original:  C OVARIATE S HIFT AS A D RIVING F ORCE
Revised:  C OVARIATE S HIFT AS A D RIVING F ORCE

Original: In this section we characterize SGDm as a discretization of a particular parametric oscillator in continuous time.
Revised: In this section we characterize SGDm as a discretization of a particular parametric oscillator in continuous time.

Original: A parametric oscillator resembles a harmonic oscillator ODE, but has time-varying system coefﬁcients which are capable of driving the system.
Revised: A parametric oscillator resembles a harmonic oscillator ODE, but has time-varying system coefficients which are capable of driving the system.

Original: It is well understood that parametric oscillators can suffer from global solution instability due to coefﬁcients oscillating at particular frequencies (Halanay, 1966), a condition known as parametric resonance .
Revised: It is well understood that parametric oscillators can suffer from global solution instability due to coefficients oscillating at particular frequencies (Halanay, 1966), a condition known as parametric resonance .

Original: We show parametric resonance conditions sufﬁcient to induce exponential divergence in SGDm.
Revised: We show parametric resonance conditions sufficient to induce exponential divergence in SGDm.

Original: We provide proof sketches and defer complete proofs to Appendix A.3.
Revised: We provide proof sketches and defer complete proofs to Appendix A.4.

Original: Notation: Capital letters are matrices and random variables.
Revised: Notation: Capital letters are matrices and random variables.

Original: { X k } is a stochastic process, shorthand for { X k } k ∈ N .
Revised: { X k } is a stochastic process, shorthand for { X k } k ∈ N .

Original: k and t index discrete and continuous time, respectively.
Revised: k and t index discrete and continuous time, respectively.

Original: When a discrete sequence and a function approximate each other, they share the same symbol and are differentiated by subscript k and function argument t , e.g. the sequence { θ k } and the function θ ( t ) .
Revised: When a discrete sequence and a function approximate each other, they share the same symbol and are differentiated by subscript k and function argument t , e.g. the sequence { θ k } and the function θ ( t ) .

Original: P k is the joint distribution of X k and Y k .
Revised: P k is the joint distribution of X k and Y k .

Original: For loss function L ( z ; θ ) on training pair z
Revised: For loss function L ( z ; θ ) on training pair z

Original: = ( x, y ) and weights θ , we denote the time-varying expected gradient g k ( θ ) :
Revised: = ( x, y ) and weights θ , we denote the time-varying expected gradient g k ( θ ) :

Original: = E P k [ ∇ θ L ( z ; θ )]
Revised: = E P k [ ∇ θ L ( z ; θ )]

Original: 3. C OVARIATE S HIFT I NDUCES A L INEAR T IME -V ARYING E XPECTED G RADIENT
Revised: 3. C OVARIATE S HIFT I NDUCES A L INEAR T IME -V ARYING E XPECTED G RADIENT

Original: We begin by showing that linear least squares regression with covariate shift and a ﬁxed target induces a linear time-varying expected loss gradient.
Revised: We begin by showing that linear least squares regression with covariate shift and a fixed target induces a linear time-varying expected loss gradient.

Original: That is, the gradient function g k ( θ )
Revised: That is, the gradient function g k ( θ )

Original: =
Revised: =

Original: B k ( θ − θ ∗ ) for some matrix B k that varies with time, due to covariate shift.
Revised: B k ( θ − θ ∗ ) for some matrix B k that varies with time, due to covariate shift.

Original: Assumption 1.
Revised: Assumption 1.

Original: The covariate generating process { X k } k ∈ N is not identically distributed:
Revised: The covariate generating process { X k } k ∈ N is not identically distributed:

Original: Corr ( X k 1 , X k 1 ) (cid:54) =
Revised: Corr ( X k 1 , X k 1 ) ̸ =

Original: Corr ( X k 2 , X k 2 ) for some k 1 , k 2 ∈ N .
Revised: Corr ( X k 2 , X k 2 ) for some k 1 , k 2 ∈ N .

Original: Assumption 2.
Revised: Assumption 2.

Original: The targets Y k are a ﬁxed linear function of X k with iid zero-mean observation noise, (cid:15) k
Revised: The targets Y k are a fixed linear function of X k with iid zero-mean observation noise, ϵ k

Original: such that E [ (cid:15) k ] = 0 .
Revised: such that E [ ϵ k ] = 0 .

Original: That is, Y k
Revised: That is, Y k

Original: =(cid:104)
Revised: =

Original:  θ ∗ , X k (cid:105) + (cid:15) k where θ ∗ ∈ R d is ﬁxed for all k .
Revised: ⟨ θ ∗ , X k ⟩ + ϵ k where θ ∗ ∈ R d is fixed for all k .

Original: Proposition 1.
Revised: Proposition 1.

Original: Under Assumptions 1, 2, a linear model with weights θ k making predictions (cid:98) Y k
Revised: Under Assumptions 1, 2, a linear model with weights θ k making predictions (cid:98) Y k

Original: =(cid:104)
Revised: =

Original:  θ k , X k (cid:105) with a mean squared error (MSE) objective will induce time-varying linear expected loss gradients g k ( θ k )
Revised: ⟨ θ k , X k ⟩ with a mean squared error (MSE) objective will induce time-varying linear expected loss gradients g k ( θ k )

Original: =
Revised: =

Original: B k
Revised: B k

Original: ( θ k − θ ∗ ) for all k ∈ N , where B k ∈ R d × d and B k 1 (cid:54) = B k 2 for some k 1 , k 2 .
Revised: ( θ k − θ ∗ ) for all k ∈ N , where B k ∈ R d × d and B k 1 ̸ = B k 2 for some k 1 , k 2 .

Original: Proof Sketch: We ﬁx a time step k and take the expectation of the MSE loss over observation noise, and over the distribution of X k .
Revised: Proof Sketch: We fix a time step k and take the expectation of the MSE loss over observation noise, and over the distribution of X k .

Original: A time-varying expected loss surface remains, and the linear setting implies the loss gradient is a linear function for all steps k , with B k depending on X k :
Revised: A time-varying expected loss surface remains, and the linear setting implies the loss gradient is a linear function for all steps k , with B k depending on X k :

Original: [Equation]
Revised: [Equation]

Original: 3. ODE C ORRESPONDENCE
Revised: 3. ODE C ORRESPONDENCE

Original: Next, we show that the iterates { θ k } generated by SGDm are a ﬁrst order numerical integration of a particular ODE.
Revised: Next, we show that the iterates { θ k } generated by SGDm are a first order numerical integration of a particular ODE.

Original: The procedure is similar to (Muehlebach & Jordan, 2021), but due to the time-varying loss gradient, we must pay speciﬁc attention to the conditions on the ODE necessary to have integration consistency, .
Revised: The procedure is similar to (Muehlebach & Jordan, 2021), but due to the time-varying loss gradient, we must pay specific attention to the conditions on the ODE necessary to have integration consistency, .

Original: Let B ( t ) be a matrix-valued function, Lipschitz continuous in t , such that { B k } are samples spaced √ η apart, i.e. B k = B ( √ ηk ) .
Revised: Let B ( t ) be a matrix-valued function, Lipschitz continuous in t , such that { B k } are samples spaced √ η apart, i.e. B k = B ( √ ηk ) .

Original: The existence and uniqueness of this
Revised: The existence and uniqueness of this

Original: B ( t ) are ensured if we assume the { X k } are sampled from an underlying continuous-time X ( t ) .
Revised: B ( t ) are ensured if we assume the { X k } are sampled from an underlying continuous-time X ( t ) .

Original: Proposition 2.
Revised: Proposition 2.

Original: The SGDm iterates { θ k } numerically integrate the ODE system in equation 3 with integration step √ η and ﬁrst order consistency.
Revised: The SGDm iterates { θ k } numerically integrate the ODE system in equation 3 with integration step √ η and first order consistency.

Original: [Equation]
Revised: [Equation]

Original: Proof Sketch: First, the ODE in equation 3 is converted to ﬁrst order linear form, and split into a sum of two separate systems.
Revised: Proof Sketch: First, the ODE in equation 3 is converted to first order linear form, and split into a sum of two separate systems.

Original: The sum’s terms have implicit and explicit Euler integrations; using operator splitting, these can be composed into a consistent numerical integrator for linear time-varying systems.
Revised: The sum’s terms have implicit and explicit Euler integrations; using operator splitting, these can be composed into a consistent numerical integrator for linear time-varying systems.

Original: Then, it is shown that the composed integrator is precisely equivalent to SGDm when the integration time step is √ η . (cid:3)
Revised: Then, it is shown that the composed integrator is precisely equivalent to SGDm when the integration time step is √ η . □

Original: The ﬁrst order consistency guarantee in Proposition 2 means that when the step-size η is small and initial conditions agree between continuous and discrete time, i.e. θ 

Revised: The first order consistency guarantee in Proposition 2 means that when the step-size η is small and initial conditions agree between continuous and discrete time, i.e. θ

Original: = θ (0) , the continuous and discrete time trajectories approximate each other as θ k ≈ θ ( k √ η ) , where the difference between them depends on the integration time step h = √
Revised: = θ (0) , the continuous and discrete time trajectories approximate each other as θ k ≈ θ ( k √ η ) , where the difference between them depends on the integration time step h = √

Original: η (Hairer et al., 2006).
Revised: η (Hairer et al., 2006).

Original: Therefore, assuming θ 0 = θ (0) , the difference accrued in one step of k (local error) is θ 1 = θ ( √ η ) + O ( η ) and over arbitrarily many steps (global error) is θ k
Revised: Therefore, assuming θ 0 = θ (0) , the difference accrued in one step of k (local error) is θ 1 = θ ( √ η ) + O ( η ) and over arbitrarily many steps (global error) is θ k

Original: = θ ( √ ηk )
Revised: = θ ( √ ηk )

Original: + O ( ηk ) .
Revised: + O ( ηk ) .

Original: 3. P ARAMETRIC R ESONANCE FOR ODE C ONVERGENCE AND D IVERGENCE
Revised: 3. P ARAMETRIC R ESONANCE FOR ODE C ONVERGENCE AND D IVERGENCE

Original: The conditions sufﬁcient for convergence and divergence in equation 3 may be shown using established dynamical systems theory.
Revised: The conditions sufficient for convergence and divergence in equation 3 may be shown using established dynamical systems theory.

Original: The conditions sufﬁcient for divergence are precisely the conditions for parametric resonance, when B ( t ) is periodic.
Revised: The conditions sufficient for divergence are precisely the conditions for parametric resonance, when B ( t ) is periodic.

Original: Note that periodicity in the ﬁrst or second moment of { X k } is sufﬁcient to induce periodicity in B ( t ) .
Revised: Note that periodicity in the first or second moment of { X k } is sufficient to induce periodicity in B ( t ) .

Original: As per elementary results in linear ODE theory, the system in equation 3 can be transformed into a linear time-varying ﬁrst order form
Revised: As per elementary results in linear ODE theory, the system in equation 3 can be transformed into a linear time-varying first order form

Original: [Equation]
Revised: [Equation]

Original: where solution trajectories θ ( t ) of equation 3 are embedded in solution trajectories ξ ( t ) of equation 4.
Revised: where solution trajectories θ ( t ) of equation 3 are embedded in solution trajectories ξ ( t ) of equation 4.

Original: Equation 4 admits a fundamental solution matrix 1 ψ ( t ) such that the spectral radius ρ of ψ ( T ) characterizes ODE instability, which implies divergence for SGDm, in Theorem 1.
Revised: Equation 4 admits a fundamental solution matrix 1 ψ ( t ) such that the spectral radius ρ of ψ ( T ) characterizes ODE instability, which implies divergence for SGDm, in Theorem 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: When B ( t ) is periodic such that B ( t )
Revised: When B ( t ) is periodic such that B ( t )

Original: =
Revised: =

Original: B ( t + T ) for some T > 0 , the spectral radius ρ of ψ ( T ) characterizes the stability of solution trajectories of equation 3 as follows:
Revised: B ( t + T ) for some T > 0 , the spectral radius ρ of ψ ( T ) characterizes the stability of solution trajectories of equation 3 as follows:

Original: • ρ > 1 = ⇒ trivial solution θ ( t )
Revised: • ρ > 1 = ⇒ trivial solution θ ( t )

Original: =
Revised: =

Original: θ ∗ is unstable.
Revised: θ ∗ is unstable.

Original: All other solutions diverge as θ ( t ) → ∞ exponentially with rate ρ .
Revised: All other solutions diverge as θ ( t ) → ∞ exponentially with rate ρ .

Original: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as θ ( t ) → θ ∗ exponentially with rate ρ .
Revised: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as θ ( t ) → θ ∗ exponentially with rate ρ .

Original: Proof Sketch: ODE 3’s stability depends on the relationship between the coefﬁcient 1 − µ √ η and the expected gradient signal because they determine the spectral radius of ψ ( T ) .
Revised: Proof Sketch: ODE 3’s stability depends on the relationship between the coefficient 1 − µ √ η and the expected gradient signal because they determine the spectral radius of ψ ( T ) .

Original: The spectral radius ρ is the convergence/divergence rate per period T towards/away from the stationary point θ ∗ .
Revised: The spectral radius ρ is the convergence/divergence rate per period T towards/away from the stationary point θ ∗ .

Original: (cid:3)
Revised: □

Original: The spectral radius conditions characterize when ODE solution trajectories θ ( t ) will converge or diverge, and Proposition 2 tells us that these solution trajectories are approximations of discrete time
Revised: The spectral radius conditions characterize when ODE solution trajectories θ ( t ) will converge or diverge, and Proposition 2 tells us that these solution trajectories are approximations of discrete time

Original: SGDm trajectories { θ k } under identical initialization.
Revised: SGDm trajectories { θ k } under identical initialization.

Original: But does convergence or divergence of θ ( t ) imply the same for SGDm?
Revised: But does convergence or divergence of θ ( t ) imply the same for SGDm?

Original: Experiment 4.1 empirically suggests that the approximation is sufﬁcient, since the boundary at ρ = 1 in Figure 2a agrees with both continuous and discrete time.
Revised: Experiment 4.1 empirically suggests that the approximation is sufficient, since the boundary at ρ = 1 in Figure 2a agrees with both continuous and discrete time.

Original: However, given Theorem 1 and Proposition 2, we have a theoretical guarantee of SGDm’s divergence, but not its convergence.
Revised: However, given Theorem 1 and Proposition 2, we have a theoretical guarantee of SGDm’s divergence, but not its convergence.

Original: This is because the divergence rate of θ ( t ) is exponential, and Proposition 2 provides a bound on long tail behaviour as θ k = θ ( √ ηk ) + O ( ηk ) .
Revised: This is because the divergence rate of θ ( t ) is exponential, and Proposition 2 provides a bound on long tail behaviour as θ k = θ ( √ ηk ) + O ( ηk ) .

Original: Since the approximation error is linear in time k , and the divergence rate is exponential, the divergence rate dominates, and we are guaranteed that a diverging θ ( t ) corresponds to a diverging { θ k } .
Revised: Since the approximation error is linear in time k , and the divergence rate is exponential, the divergence rate dominates, and we are guaranteed that a diverging θ ( t ) corresponds to a diverging { θ k } .

Original: However, the same argument does not imply that a convergent θ ( t ) corresponds to a convergent { θ k } , because the linear error bound technically permits the discrete trajectory to escape θ ∗ at a linear rate.
Revised: However, the same argument does not imply that a convergent θ ( t ) corresponds to a convergent { θ k } , because the linear error bound technically permits the discrete trajectory to escape θ ∗ at a linear rate.

Original: Empirically, we see agreement for both convergent and divergent cases, but we defer theoretical proof to future work.
Revised: Empirically, we see agreement for both convergent and divergent cases, but we defer theoretical proof to future work.

Original: Summary: There is a chain of dependencies starting from the non-i.i.d.
Revised: Summary: There is a chain of dependencies starting from the non-i.i.d.

Original: sampling in { X k } , and ending at the monodromy’s spectral radius.
Revised: sampling in { X k } , and ending at the monodromy’s spectral radius.

Original: [Equation]
Revised: [Equation]

Original: In order, we have the discrete stochastic process { X k } from which training inputs are sampled, and its underlying continuous stochastic process X ( t ) .
Revised: In order, we have the discrete stochastic process { X k } from which training inputs are sampled, and its underlying continuous stochastic process X ( t ) .

Original: If { X k } and X ( t ) were i.i.d.
Revised: If { X k } and X ( t ) were i.i.d.

Original: , then the matrix B would be constant, but the non-i.d.d.
Revised: , then the matrix B would be constant, but the non-i.d.d.

Original: nature means B ( t ) is a function of time.
Revised: nature means B ( t ) is a function of time.

Original: The matrix A ( t ) is the matrix describing SGDm’s continuous time dynamics as a linear time-varying ODE ˙ ξ
Revised: The matrix A ( t ) is the matrix describing SGDm’s continuous time dynamics as a linear time-varying ODE ˙ ξ

Original: = A ( t ) ξ , and the matrix B ( t ) is a simply a submatrix of A ( t ) .
Revised: = A ( t ) ξ , and the matrix B ( t ) is a simply a submatrix of A ( t ) .

Original: Since we have a linear ODE, the space of solution trajectories is spanned by the columns of the ODE’s fundamental solution matrix ψ ( t ) .
Revised: Since we have a linear ODE, the space of solution trajectories is spanned by the columns of the ODE’s fundamental solution matrix ψ ( t ) .

Original: Moreover, since we have a periodic A ( t ) with period T , the stability of all solutions can be determined simply from the largest eigenvalue of ψ ( T ) , a.k.a.
Revised: Moreover, since we have a periodic A ( t ) with period T , the stability of all solutions can be determined simply from the largest eigenvalue of ψ ( T ) , a.k.a.

Original: its spectral radius ρ .
Revised: its spectral radius ρ .

Original: The intuition behind ρ ’s importance comes from the following fact: after each elapsed period T , the phase space (including weight space) is subject to the linear transformation ψ ( T ) .
Revised: The intuition behind ρ ’s importance comes from the following fact: after each elapsed period T , the phase space (including weight space) is subject to the linear transformation ψ ( T ) .

Original: So as time increases, say n periods, the linear transformation is applied iteratively as ψ ( T ) n , so it is clear that any eigenvalue larger than unity means all solutions eventually diverge exponentially.
Revised: So as time increases, say n periods, the linear transformation is applied iteratively as ψ ( T ) n , so it is clear that any eigenvalue larger than unity means all solutions eventually diverge exponentially.

Original: This is precisely the parametric resonance condition.
Revised: This is precisely the parametric resonance condition.

Original: For a detailed example of all these quantities characterizing the divergence of a simple learning system, see Appendix A.1.
Revised: For a detailed example of all these quantities characterizing the divergence of a simple learning system, see Appendix A.1.

Original: Technical Novelty:
Revised: Technical Novelty:

Original: There are two key novel technical contributions to obtain the results above.
Revised: There are two key novel technical contributions to obtain the results above.

Original: The ODE correspondence for non-iid data in combination with momentum methods is new.
Revised: The ODE correspondence for non-iid data in combination with momentum methods is new.

Original: The non-iid data, which induces time variation in ODE dynamics, is subtle to address, because elementary
Revised: The non-iid data, which induces time variation in ODE dynamics, is subtle to address, because elementary

Original: ODE correspondence (i.e. consistency guarantees) require time invariant ODE dynamics.
Revised: ODE correspondence (i.e. consistency guarantees) require time invariant ODE dynamics.

Original: Our proof speciﬁcally addresses this difﬁculty using operator splitting theory from recent numerical integration literature.
Revised: Our proof specifically addresses this difficulty using operator splitting theory from recent numerical integration literature.

Original: (2) Once we have this ODE, it is straightforward to recognize that it is a parametric oscillator.
Revised: (2) Once we have this ODE, it is straightforward to recognize that it is a parametric oscillator.

Original: The utility of this connection, however, is that it opens up entirely new avenues for analysis, since there is deeply established literature on the dynamics of these systems.
Revised: The utility of this connection, however, is that it opens up entirely new avenues for analysis, since there is deeply established literature on the dynamics of these systems.

Original: The primary second technical novelty, therefore, is to leverage this connection, and use Floquet theory to give precise divergence conditions for a given η and µ , under periodic covariate shift.
Revised: The primary second technical novelty, therefore, is to leverage this connection, and use Floquet theory to give precise divergence conditions for a given η and µ , under periodic covariate shift.

Original:  V ALIDATING T HEORY & A BLATING T OWARDS C ONDITIONS IN THE W ILD
Revised:  V ALIDATING T HEORY & A BLATING T OWARDS C ONDITIONS IN THE W ILD

Original: We now empirically evaluate the effect of resonance in learning problems.
Revised: We now empirically evaluate the effect of resonance in learning problems.

Original: First, Experiment 4.
Revised: First, Experiment 4.

Original: validates the theoretical predictions by investigating the dynamics of a learning problem which closely matches the theoretical assumptions in Section 3.
Revised: validates the theoretical predictions by investigating the dynamics of a learning problem which closely matches the theoretical assumptions in Section 3.

Original: Speciﬁcally, we assess whether or not the spectral radius ρ predicts optimizer convergence or divergence.
Revised: Specifically, we assess whether or not the spectral radius ρ predicts optimizer convergence or divergence.

Original: Then, the remaining Experiments 4.2 - 4. test for this phenomena in settings beyond our theory.
Revised: Then, the remaining Experiments 4.2 - 4.6 test for this phenomena in settings beyond our theory.

Original: Each experiment makes a single step away from a theoretical assumption, ﬁrst relaxing the periodicity assumption, then using stochastic rather than expected gradients and then moving tomore widely used nonlinear models (neural networks) and optimizers (ADAM).
Revised: Each experiment makes a single step away from a theoretical assumption, first relaxing the periodicity assumption, then using stochastic rather than expected gradients and then moving to nonlinear models (neural networks) and optimizers (ADAM).

Original: Across all experiments, input samples at each training step k are drawn from Gaussian distributions with diagonal covariance matrices, i.e. X k ∼ N (¯ x k , cI d × d ) , and we induce covariate shift by constructing a time-varying mean sequence { ¯ x k } .
Revised: Across all experiments, input samples at each training step k are drawn from Gaussian distributions with diagonal covariance matrices, i.e. X k ∼ N (¯ x k , cI d × d ) , and we induce covariate shift by constructing a time-varying mean sequence { ¯ x k } .

Original: Each mean sequence is designed such that (a) its frequency content can be swept with a single parameter f or T and (b) iid sampling is induced by setting f = 0 or T = 0 , so that each experiment has an iid baseline for comparison.
Revised: Each mean sequence is designed such that (a) its frequency content can be swept with a single parameter f or T and (b) iid sampling is induced by setting f = 0 or T = 0 , so that each experiment has an iid baseline for comparison.

Original: The speciﬁcation of ¯ x k is provided in Table 1 for each experiment.
Revised: The specification of ¯ x k is provided in Table 1 for each experiment.

Original: This empirical design allows us to systematically test response to different frequencies.
Revised: This empirical design allows us to systematically test response to different frequencies.

Original: See Appendix A.4 for speciﬁcation of all experiment details.
Revised: See Appendix A.5 for specification of all experiment details, including visual depictions of all synthetic { X k } and their frequency content.

Original: 4. V ALIDATING T HEORY
Revised: 4. V ALIDATING T HEORY

Original: We start in a setting as close as possible to the theoretical predictions, with linear regression for a quadratic loss, and covariate shift such that the mean E [ X k ] varies as a strict sinusoid.
Revised: We start in a setting as close as possible to the theoretical predictions, with linear regression for a quadratic loss, and covariate shift such that the mean E [ X k ] varies as a strict sinusoid.

Original: We perform regression in two weights (i.e. inputs X k and labels Y k are scalar-valued).
Revised: We perform regression in two weights (i.e. inputs X k and labels Y k are scalar-valued).

Original: We draw 20 samples from each X k , so that the loss gradient for each time step k is close to its expected value.
Revised: We draw 20 samples from each X k , so that the loss gradient for each time step k is close to its expected value.

Original: Since we have only two weights, the learning system’s underlying ODE can be easily speciﬁed, such that the fundamental solution matrix can be numerically computed.
Revised: Since we have only two weights, the learning system’s underlying ODE can be easily specified, such that the fundamental solution matrix can be numerically computed.

Original: As per Theorem 1, we can use the spectral radius of the fundamental solution matrix evaluated at time T to make theoretical predictions of where the system should converge or diverge.
Revised: As per Theorem 1, we can use the spectral radius of the fundamental solution matrix evaluated at time T to make theoretical predictions of where the system should converge or diverge.

Original: As depicted in Figure 2a, the theory agrees very well with empirical results.
Revised: As depicted in Figure 2a, the theory agrees very well with empirical results.

Original: This suggests that parametric resonance is indeed the dominant mechanism behind SGDm divergence under covariate shift.
Revised: This suggests that parametric resonance is indeed the dominant mechanism behind SGDm divergence under covariate shift.

Original: Refer to example A.1 for the procedure used to compute the theoretical predictions (i.e. spectral radii ρ ), and Figure 6a for the full surface from which the contour lines in Figure 2a are rendered.
Revised: In the appendix, refer to example A.1 for the procedure used to compute the theoretical predictions (i.e. spectral radii ρ ), and Figure 6a for the full surface from which the contour lines in Figure 2a are rendered.

Original: Experiment
Revised: Experiment

Original: Sweep Param.
Revised: Sweep Param.

Original: Mean [Table]
Revised: Mean Sequence { ¯ x k } k ∈ N

Original: 4.1 Validating Theory
Revised: 4.1 Validating Theory

Original: f ∈ [0 , [Table]
Revised: f ∈ [0 , 0 .

Original: ¯ x k = 0 .
Revised: ¯ x k = 0 .

Original: 5 sin(2 πfk ) , T = 1 /f
Revised: 5 sin(2 πfk ) , T = 1 /f

Original: 4.2 Ablating Periodicity
Revised: 4.2 Ablating Periodicity

Original: f ∈ [0 , 0 .
Revised: f ∈ [0 , 0 .

Original: ¯ x k = φ 1 ¯ x k − 1 + φ 2 ¯ x k − 2 + ξ k φ 1{+ϕ 
+} = 4 φ 2 ( φ 2 − 1) − 1 cos(2 πf )
Revised: ¯ x k = ϕ 1 ¯ x k − 1 + ϕ 2 ¯ x k − 2 + ξ k {+ϕ 
+} = 4 ϕ 2 ( ϕ 2 − 1) − 1 cos(2 πf )

Original: φ 2 , ξ k , ¯ x 1 , ¯ x 2 see Table
Revised: ϕ 2 , ξ k , ¯ x 1 , ¯ x 2 see Table

Original: 4.3 Ablating Expected Gradient
Revised: 4.3 Ablating Expected Gradient

Original: T ∈ 
Revised: T ∈ [0 , 120] ¯ x k = (cid:40) ξ [Table] ||

Original: [Table]
Revised: [Table]

Revised: || if

Original: [Table]
Revised: [Table]

Revised: mod

Original: =
Revised: [Table] | [Table] if

Original: [Table]
Revised: [Table]

Original: {+ξ ∼ N (0 , I d × d ) iid 4.
+}
Revised: {+ξ ∼ N (0 , I d × d ) iid 4.
+}

Original: 4.4 Ablating Periodicity Further
Revised: Ablating Periodicity Further

Original: T ∈ [0 , 50] ¯ x k = ξ i where i = (cid:22) k
Revised: T ∈ [0 , 50] ¯ x k = ξ i where i = (cid:22) k

Original: T (cid:23) ξ i ∼ N (0 , vI d × d ) iid 4.
Revised: T (cid:23) ξ i ∼ N (0 , vI d × d ) iid 4.

Original:  Ablating Optimizer Linearity
Revised: 5 Ablating Optimizer Linearity

Original: T ∈ [0 , 100] ¯ x k same as above.
Revised: T ∈ [0 , 100] ¯ x k same as above.

Original: 4.6 Ablating Model Linearity
Revised: Ablating Model Linearity

Original: T ∈ [0 , 100] ¯ x k same as above.
Revised: T ∈ [0 , 100] ¯ x k same as above.

Original: [Figure]
Revised: [Figure]

Original: 4. A BLATING P ERIODICITY
Revised: 4. A BLATING P ERIODICITY

Original: We repeat Experiment 4.1, but with the mean of X k varying stochastically instead of deterministically.
Revised: We repeat Experiment 4.1, but with the mean of X k varying stochastically instead of deterministically.

Original: The result in Theorem 1 assumes the expected gradient varies periodically over time given ﬁxed weights θ .
Revised: The result in Theorem 1 assumes the expected gradient varies periodically over time given fixed weights θ .

Original: But even with aperiodic and/or stochastic time variation, our LTV system equation 4 might be similarly susceptible to instability; the theory does not have periodicity as a necessary condition of divergence.
Revised: But even with aperiodic and/or stochastic time variation, our LTV system equation 4 might be similarly susceptible to instability; the theory does not have periodicity as a necessary condition of divergence.

Original: To test this, we replace our periodic covariate shift mean with a mean that moves according to an AR(2) process.
Revised: To test this, we replace our periodic covariate shift mean with a mean that moves according to an AR(2) process.

Original: The AR(2) process is tuned to have a frequency peak exactly at the frequency of the sinusoidal covariate shift of Experiment 4.1, to change only the nature of the shift—aperiodicity—rather than the frequency.
Revised: The AR(2) process is tuned to have a frequency peak exactly at the frequency of the sinusoidal covariate shift of Experiment 4.1, to change only the nature of the shift—aperiodicity—rather than the frequency.

Original: We can see from the heatmap in Figure 2b that under this setting we observe very similar resonance behaviour in the learning system, which aligns well with the identical predicted stability regions.
Revised: We can see from the heatmap in Figure 2b that under this setting we observe very similar resonance behaviour in the learning system, which aligns well with the identical predicted stability regions.

Original: Note that both theoretical and empirical results in Figures 2a and 2b suggest that sufﬁciently low momentum values µ mitigate the resonance phenomenon, which agrees with the role it plays in the
Revised: Note that both theoretical and empirical results in Figures 2a and 2b suggest that sufficiently low momentum values µ mitigate the resonance phenomenon, which agrees with the role it plays in the

Original: ODE system: decreasing µ increases damping.
Revised: ODE system: decreasing µ increases damping.

Original: We also observe that resonance regions shrink as step size η is decreased; see the Appendix for plots demonstrating this trend.
Revised: We also observe that resonance regions shrink as step size η is decreased; see the Appendix for plots demonstrating this trend.

Original: Analytically characterizing the bounds of stable µ, η in terms of system properties is an interesting future direction.
Revised: Analytically characterizing the bounds of stable µ, η in terms of system properties is an interesting future direction.

Original: 4. A BLATING E XPECTED G RADIENT
Revised: 4. A BLATING E XPECTED G RADIENT

Original: Here we perform linear regression with periodic mean covariate shift, and ablate the number of samples drawn from each X k to show the effect of increasing noise in the gradient signal.
Revised: Here we perform linear regression with periodic mean covariate shift, and ablate the number of samples drawn from each X k to show the effect of increasing noise in the gradient signal.

Original: Linear regression is performed mapping R 5 → R , with the weights learned via SGDm.
Revised: Linear regression is performed mapping R 5 → R , with the weights learned via SGDm.

Original: The mean sequence ¯ x k oscillates with strict periodicity between ± ¯ x , where ¯ x is a unit norm 5-vector randomly chosen for each run.
Revised: The mean sequence ¯ x k oscillates with strict periodicity between ± ¯ x , where ¯ x is a unit norm 5-vector randomly chosen for each run.

Original: i.e. the mean signal is a square wave in R 5 with period T .
Revised: i.e. the mean signal is a square wave in R 5 with period T .

Original: Linear regression implies a quadratic loss, and periodic covariate shift implies a loss with periodic time variation in expectation, so we are very near to the setting in which Theorem 1 is directly applicable.
Revised: Linear regression implies a quadratic loss, and periodic covariate shift implies a loss with periodic time variation in expectation, so we are very near to the setting in which Theorem 1 is directly applicable.

Original: However, as we ablate from 5 samples to 1 sample drawn from each X k , we move further away from the expected gradient, towards the fully stochastic gradient setting.
Revised: However, as we ablate from 5 samples to 1 sample drawn from each X k , we move further away from the expected gradient, towards the fully stochastic gradient setting.

Original: As we can see in Figure 3 resonance is dampened by stochasticity in the gradient signal, but nonetheless is still present.
Revised: As we can see in Figure 3 resonance is dampened by stochasticity in the gradient signal, but nonetheless is still present.

Revised: These findings are further supported in Appendix A.3, where Experiments 4.

Revised: and 4.2 are also ablated from expected to stochastic gradient signals.

Original: 4. A BLATING P ERIODICITY F URTHER
Revised: 4. A BLATING P ERIODICITY F URTHER

Original: We further depart from strictly periodic covariate shift by having piece-wise stationary means and nonsmooth changes: we randomly sample a new mean from a normal distribution N (0 , v 2 ) after every T update steps.
Revised: We further depart from strictly periodic covariate shift by having piece-wise stationary means and nonsmooth changes: we randomly sample a new mean from a normal distribution N (0 , v 2 ) after every T update steps.

Original: We further investigate the impact of this covariate shift under (a) increasing variance vand (b) increasing dimensionality of the inputs.
Revised: We further investigate the impact of this covariate shift under (a) increasing variance v 2 and (b) increasing dimensionality of the inputs.

Original: We observe divergence characteristic of the parametric resonance, with smaller T having a highly divergent response.
Revised: We observe divergence characteristic of the parametric resonance, with smaller T having a highly divergent response.

Original: In Figure 4a we can see that a higher variance results in much more instability, though even for smaller variance the distance to θ ∗ is still on the order of 10 3 .
Revised: In Figure 4a we can see that a higher variance results in much more instability, though even for smaller variance the distance to θ ∗ is still on the order of 10 3 .

Original: This results makes sense, as the variance is connected to the driving signal amplitude, on which parametric resonance has a strong dependence.
Revised: This results makes sense, as the variance is connected to the driving signal amplitude, on which parametric resonance has a strong dependence.

Original: We also observe a strong dependence on the number of input space dimensions, in Figure 4b.
Revised: We also observe a strong dependence on the number of input space dimensions, in Figure 4b.

Original: This aligns with the fact that the expected norm of samples drawn from a multivariate Gaussian increases with dimensionality, even with a ﬁxed covariance scale (of 0.1).
Revised: This aligns with the fact that the expected norm of samples drawn from a multivariate Gaussian increases with dimensionality, even with a fixed covariance scale (of 0.1).

Original: This result suggests that resonance phenomena may actually be exacerbated for higher-dimensional inputs, which is the setting we expect to see in practice.
Revised: This result suggests that resonance phenomena may actually be exacerbated for higher-dimensional inputs, which is the setting we expect to see in practice.

Original: 4. A BLATING O PTIMIZER L INEARITY
Revised: 4. A BLATING O PTIMIZER L INEARITY

Original: So far our experiments have ablated periodicity, input dimensionality, and gradient stochasticity.
Revised: So far our experiments have ablated periodicity, input dimensionality, and gradient stochasticity.

Original: As per the conditions in Section 3, each of these ablated systems still corresponds to a discretization of an LTV ODE.
Revised: As per the conditions in Section 3, each of these ablated systems still corresponds to a discretization of an LTV ODE.

Original: We now ask whether resonance can be observed in systems which do not correspond to a discretization of our ODE.
Revised: We now ask whether resonance can be observed in systems which do not correspond to a discretization of our ODE.

Original: To investigate this hypothesis, we replace SGDm with ADAM as an update rule for our learning algorithm.
Revised: To investigate this hypothesis, we replace SGDm with ADAM as an update rule for our learning algorithm.

Original: We no longer have the ODE representation of this learning system, but can empirically investigate the behaviour of the system under covariate shift.
Revised: We no longer have the ODE representation of this learning system, but can empirically investigate the behaviour of the system under covariate shift.

Original: We use the same stochastic mean switching problem as the previous experiment, where we regress from 5 dimensions to 1 with input samples.
Revised: We use the same stochastic mean switching problem as the previous experiment, where we regress from 5 dimensions to 1 with input samples.

Original: Again, we measure the distance || θ k − θ ∗ || , and we vary the ADAM parameter β 1 .
Revised: Again, we measure the distance || θ k − θ ∗ || , and we vary the ADAM parameter β 1 .

Original: Similar to the SGDm optimizer, we can see in Figure 4c that there is a band of mean switching intervals T for which convergence is worse.
Revised: Similar to the SGDm optimizer, we can see in Figure 4c that there is a band of mean switching intervals T for which convergence is worse.

Original: But unlike SGDm, there is no divergence.
Revised: Unlike SGDm, there is no divergence.

Original: Proper parametric resonance induces exponential divergence, which is why the previous results were presented with Euclidean distances between weights on the order of 10 19 .
Revised: Proper parametric resonance induces exponential divergence, which is why the previous results were presented with Euclidean distances between weights on the order of 10 19 .

Original: Note that in Figure 4c, the frequency response in weight space distance is instead measured on the order of 10 0 . 
Revised: Note that in Figure 4c, the frequency response in weight space distance is instead measured on the order of 10 0 . [Figure]

Original: 4. A BLATING M ODEL L INEARITY
Revised: 4. A BLATING M ODEL L INEARITY

Original: Our ﬁnal step away from ODE equation 3 abandons the quadratic loss surface.
Revised: Our final step away from ODE equation 3 abandons the quadratic loss surface.

Original: We ﬁt a nonlinear target function using a fully connected ReLU network, reusing the previous covariate shift.
Revised: We fit a nonlinear target function using a fully connected ReLU network, reusing the previous covariate shift.

Original: In previous experiments, we measured the distance || θ k − θ ∗ || .
Revised: In previous experiments we measured the distance || θ k − θ ∗ || .

Original: Now a unique target is not guaranteed,[Figure] so frequency response is in terms of loss.
Revised: Now a unique target is not guaranteed, so frequency response is in terms of loss.

Original: In order to amplify frequency response 10 samples were drawn from each X k to reduce gradient signal noise
Revised: In order to amplify frequency response, 10 samples were drawn from each X k to reduce gradient signal noise.

Original: Figure 5 shows a band of mean switching intervals T which damage convergence, which is characteristic of resonance.
Revised: Figure 5 shows a band of mean switching intervals T which damage convergence, which is characteristic of resonance.

Original: Though we do not see divergence, the impact on accuracy is problematic.
Revised: Though we do not see divergence, the impact on accuracy is problematic.

Original: In this case, models with loss > 0 .
Revised: In this case, models with loss > 0 .

Original: 3 ﬁt extremely poorly, but those with < 0 .
Revised: 3 fit extremely poorly, but those with < 0 .

Original: 05 perform well
Revised: 05 perform well.

Original:  L IMITATIONS AND P RACTICALITY
Revised:  D ISCUSSION

Original: This work is a ﬁrst step towards understanding parametric resonance due to covariate shift.
Revised: This work is a first step towards understanding frequency response due to covariate shift.

Original: Naturally, there are limitations and questions about broader applicability, which we discuss in this section.
Revised: Here we discuss future directions, limitations, and practicality.

Original: Limitations and Next Steps: In our theoretical analysis, we take expectation over observation noise and over the distribution of inputs X k , so the gradient coefﬁcient matrix B ( t ) in the ODE equationis deterministic.
Revised: Limitations and Next Steps: In our theoretical analysis, we take expectation over observation noise and over the distribution of inputs X k , so the gradient coefficient matrix B ( t ) in the ODE equation 3 is deterministic.

Original: This allows us to rigorously characterize when resonancewill occur for periodic B ( t ) .
Revised: This allows rigorous characterization of resonance for periodic B ( t ) .

Original: Future work might employ stochastic differential to avoid taking expectation. The case when B ( t ) is aperiodic and stochastic is of particular interest, since Experiments 4.2 and 4.4 demonstrate that resonance still drives divergence.
Revised: Experiments 4.2 and 4.4 demonstrate that resonance still drives divergence when B ( t ) is nondeterministic, so rigorous exploration in future work might employ stochastic differential equations.

Original: In particular, Experiment 4.2 aligns exactly with the theoretical predictions, suggesting that the resonance is in response to the frequency content of X t , not its periodicity.
Revised: In particular, Experiment 4.2 aligns exactly with the theoretical predictions, suggesting that the resonance is in response to the frequency content of X t , not its periodicity.

Original: For the sake of consistency across experiments, our non-iid sampling is always induced by Gaussian
Revised: For the sake of consistency across experiments, our non-iid sampling is always induced by Gaussian

Original: X k with ﬁxed diagonal covariance matrix.
Revised: X k with fixed diagonal covariance matrix.

Original: But Assumption 1 is far more relaxed. Even if one wishes to adhere to the strict periodicity requirements of Theorem 1, it is possible to induce time variation by varying other aspects of X k . Comparing the effects of these different time variations is an interesting future direction.
Revised: But Assumption 1 is far more relaxed, suggesting resonance is possible with time-varying covariance and non-Gaussian distributions, which may be explored in future empirical work

Original: When assessing a new phenomenon like  resonance, assessment on simple problems is a critical ﬁrst step, and we have endeavored to be comprehensive in that assessment.
Revised: When assessing a new phenomenon like covariate shift resonance, assessment on simple problems is a critical first step, and we have endeavored to be comprehensive in that assessment.

Original: But we only scratch the surface of more complexproblems, including more advanced optimizers and nonlinear models.
Revised: But we only scratch the surface of more complex optimizers and nonlinear models.

Original: Similarly, for the sake of experimental control and interpretability, we assess resonance on synthetic data instead of realworld data.
Revised: Similarly, for the sake of experimental control and interpretability, we assess resonance on synthetic data instead of real data.

Original: While it is obvious that many sources of real data have nontrivial frequency content (e.g. audio samples, machine sensors, etc.)
Revised: While it is obvious that many sources of real data have nontrivial frequency content (e.g. audio samples, machine sensors, etc.)

Original: it is not yet clear when realistic frequency content will resonate with the system trying to learn from it.
Revised: it is not yet clear when realistic frequency content will resonate with the system trying to learn from it.

Original: The two most exciting future directions for this work are to address these two gaps: to extend into the frequency response of more complex optimizers and models, and into the frequency content of real data.
Revised: Optimizers, models, and data all pose exciting future directions for this work: to extend into the frequency response of more complex optimizers and models, and into the frequency content of real data.

Original: Finally, the theory in this work is for linear models.
Revised: Finally, the theory in this work is for linear models.

Original: This is the right setting to understand ﬁrst, and is important in its own right.
Revised: This is the correct setting to understand first, and is of independent interest.

Original: Our theory applies to any linear models that are augmented with a nonlinear basis, such as a Fourier basis, polynomials or radial basis functions. Such bases are particularly useful in the online and streaming settings.
Revised: Our theory applies to any linear models that are augmented with a nonlinear basis (such as a Fourier basis, polynomials or radial basis functions) by examining the frequency content of the basis output signal.

Original: Further, a strategy in the continual learning setting is to slowly adjust the features, potentially produced by a neural network, and learn linear predictions that change more quickly on those slowly changing features.

Original: The features could be viewed as stationary relative to the linear prediction weights.

Original: Extensions to this setting are a natural direction to pursue next.

Original: Practical Implications: One potential conclusion from our results is that though resonance could potentially arise in our learning systems, it may not be problematic.
Revised: Practical Implications: One potential conclusion from our results is that though resonance could potentially arise in our learning systems, it may not be problematic.

Original: First, the heatmaps in Figures 2a and 2b indicate that SGDm converges much more often than it diverges, and that we can practically avoid resonance simply by reducing the momentum parameter µ .
Revised: First, the heatmaps in Figures 2a and 2b indicate that SGDm converges much more often than it diverges, and that we can practically avoid resonance simply by reducing the momentum parameter µ .

Original: As per Appendix A.2, we ﬁnd that sufﬁciently low µ does mitigate resonance.
Revised: As per Appendix A.2, we find that sufficiently low µ does mitigate resonance.

Original: Second, we may simply be able to avoid covariate shift in the ﬁrst place, by keeping buffers of data and shufﬂing samples.
Revised: Second, we may simply be able to avoid covariate shift in the first place, by keeping buffers of data and shuffling samples.

Original: However, the reality is not so simple.
Revised: However, the reality is not so simple.

Original: Higher levels of momentum are typically more effective in practice.
Revised: Higher levels of momentum are typically more effective in practice.

Original: There is a tension between the level of momentum and avoiding resonance, that cannot easily be addressed by simply picking low momentum by default.
Revised: There is a tension between the level of momentum and avoiding resonance, that cannot easily be addressed by simply picking low momentum by default.

Original: Instead, smarter approaches which recognize potential resonance and then reduce momentum may be a much more effective way to guard against this issue; the development of such methods can leverage the insights in this work.
Revised: Instead, smarter approaches which recognize potential resonance and then reduce momentum may be a much more effective way to guard against this issue; the development of such methods can leverage the insights in this work.

Original: In addition, relying on well-behaved or shufﬂed inputs may be impractical in certain settings.
Revised: In addition, relying on well-behaved or shuffled inputs may be impractical in certain settings.

Original: Algorithms in signal processing and the streaming literature often process data sequentially.
Revised: Algorithms in signal processing and the streaming literature often process data sequentially.

Original: Reinforcement learning approaches use sliding window buffers (in replay), where covariate shift may still persist across buffers.
Revised: Reinforcement learning approaches use sliding window buffers (in replay), where covariate shift may still persist across buffers.

Original: Further, it is even possible that adversarial attacks could be designed to exploit this weakness of SGDm, causing poor performance or even instability from attackers attempting to generate data that results in resonant behavior.
Revised: Further, it is even possible that adversarial attacks could be designed to exploit this weakness of SGDm, causing poor performance or even instability from attackers attempting to generate data that results in resonant behavior.

Original: Our work also suggests avenues towards the development of diagnostic tools.
Revised: Our work also suggests avenues towards the development of diagnostic tools.

Original: Indeed, our experimental approach to investigate this phenomena mimics those in the modal analysis literature (Avitabile, 2017).
Revised: Indeed, our experimental approach to investigate this phenomena mimics those in the modal analysis literature (Avitabile, 2017).

Original: We treat the input sampling process as an input signal which is noisy and/or stochastic, vary the signal’s frequency content across a wide band, and measure the learning system’s output as a response across input frequencies.
Revised: We treat the input sampling process as an input signal which is noisy and/or stochastic, vary the signal’s frequency content across a wide band, and measure the learning system’s output as a response across input frequencies.

Original: This is analogous to a mechanical engineer triggering an oscillatory vibration within a machine’s engine compartment, sweeping across frequencies, and measuring the resulting vibration amplitude in the machine’s housing.
Revised: This is analogous to a mechanical engineer triggering an oscillatory vibration within a machine’s engine compartment, sweeping across frequencies, and measuring the resulting vibration amplitude in the machine’s housing.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: To our knowledge, this work is the ﬁrst to investigate SGDm under covariate shift from the ODE perspective.
Revised: To our knowledge, this work is the first to investigate SGDm under covariate shift from the ODE perspective.

Original: This perspective reveals new insights, in particular the revelation that SGDm under covariate shift numerically integrates a parametric oscillator, rather than the simple undriven harmonic oscillator induced by iid sampling.
Revised: This perspective reveals new insights, in particular the revelation that SGDm under covariate shift numerically integrates a parametric oscillator, rather than the simple undriven harmonic oscillator induced by iid sampling.

Original: We leverage dynamical systems theory to analyze the stability of a learning system using SGDm under covariate shift, and are able to provide conditions for exponential divergence due entirely to the frequency content induced by this non-iid sampling.
Revised: We leverage dynamical systems theory to analyze the stability of a learning system using SGDm under covariate shift, and are able to provide conditions for exponential divergence due entirely to the frequency content induced by this non-iid sampling.

Original: Despite its age and establishment, understanding SGDm is an ongoing process.
Revised: Despite its age and establishment, understanding SGDm is an ongoing process.

Original: We contribute to this process in a way that provides physical intuition, rather than a merely algebraic proof.
Revised: We contribute to this process in a way that provides physical intuition, rather than a merely algebraic proof.

Original: We hope that the connection we have drawn to parametric oscillation in dynamical systems theory will help to further propel understanding of non-iid sampling and SGDm.
Revised: We hope that the connection we have drawn to parametric oscillation in dynamical systems theory will help to further propel understanding of non-iid sampling and SGDm.

Revised:  A CKNOWLEDGEMENTS

Revised: We would like to thank Yurij Salmaniw and Ryan Thiessen for their discussions regarding Floquet theory and helping us makes sense of our differential equations.

Revised: We would also like to thank Csaba

Revised: Szepesvari for pointing us to excellent resources for numerical integration, algorithmic convergence, stochastic processes, and various future directions for this work.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our work is theoretical and attempts to provide greater understanding of an optimization algorithm.
Revised: Our work is theoretical and attempts to provide greater understanding of an optimization algorithm.

Original: Due to the remoteness of this work from direct ethical concerns, it is difﬁcult to reason about any ethical implications of the work.
Revised: Due to the remoteness of this work from direct ethical concerns, it is difficult to reason about any ethical implications of the work.

Original: We hope that the insights into SGDm could help to make machine learning algorithm more robust, but beyond that we are not aware of any ethical concerns or implications of this work.
Revised: We hope that the insights into SGDm could help to make machine learning algorithm more robust, but beyond that we are not aware of any ethical concerns or implications of this work.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We have endeavored to be very clear and thorough in our explanation of the theoretical results, and all assumptions and proofs are explicitly stated and contained in either the main body of the paper or in the appendix.
Revised: We have endeavored to be very clear and thorough in our explanation of the theoretical results, and all assumptions and proofs are explicitly stated and contained in either the main body of the paper or in the appendix.

Original: The code for all of the experiments is included in the supplementary material and the instructions for reproduction are in the README ﬁle.
Revised: The code for all of the experiments is included in the supplementary material and the instructions for reproduction are in the README file.

Original: The data used is synthetic and code to generate the data is contained in the supplementary material.
Revised: The data used is synthetic and code to generate the data is contained in the supplementary material.

Original: All of the experimental details (e g. hyperparameter selection) are contained in tables in the paper and appendix and the supplementary material.
Revised: All of the experimental details (e g. hyperparameter selection) are contained in tables in the paper and appendix and the supplementary material.

Original: All of the experiments should be easily run on modest commodity hardware in the space of a few days, at most.
Revised: All of the experiments should be easily run on modest commodity hardware in the space of a few days, at most.

Original: A A PPENDIX A. A P ROCEDURE TO N UMERICALLY D ETERMINE S YSTEM S TABILITY
Revised: A A PPENDIX A. A P ROCEDURE TO N UMERICALLY D ETERMINE S YSTEM S TABILITY

Original: Here we expand on the example given at the end of Section 3.
Revised: Here we expand on the example given at the end of Section 3.

Original: As a reminder, we consider online linear least squares regression using SGDm with a ﬁxed η, µ under a periodic covariate shift.
Revised: As a reminder, we consider online linear least squares regression using SGDm with a fixed η, µ under a periodic covariate shift.

Original: Data is sampled from the sequence of random variables { X k , Y k } k ∈ N where X k is normally distributed and shifts periodically in expectation.
Revised: Data is sampled from the sequence of random variables { X k , Y k } k ∈ N where X k is normally distributed and shifts periodically in expectation.

Original: Y k is an unchanging linear function of X k with zero-mean observation noise.
Revised: Y k is an unchanging linear function of X k with zero-mean observation noise.

Original: For ease of exposition we consider the one dimensional regression problem with
Revised: For ease of exposition we consider the one dimensional regression problem with

Original: X k ∈ R and Y k ∈ R , however the theory applies to any learning system representable as an LTV
Revised: X k ∈ R and Y k ∈ R , however the theory applies to any learning system representable as an LTV

Original: ODE satisfying the assumptions required for Theorem 1.
Revised: ODE satisfying the assumptions required for Theorem 1.

Original: Under this covaiate shift, ¯ x k varies sinusoidally in [ − a, a ] with period T = 1 f .
Revised: Under this covaiate shift, ¯ x k varies sinusoidally in [ − a, a ] with period T = 1 f .

Original: Note that although
Revised: Note that although

Original: X k is periodic in expectation it admits a stationary distribution.
Revised: X k is periodic in expectation it admits a stationary distribution.

Original: The full speciﬁcation of the data generating process is as follows
Revised: The full specification of the data generating process is as follows

Original: [Equation]
Revised: [Equation]

Original: where θ ∗ =
Revised: where θ ∗ =

Original: [ θ ∗ 1 , θ ∗ 2 ] T are the target weights.
Revised: [ θ ∗ 1 , θ ∗ 2 ] T are the target weights.

Original: We consider a squared error loss function,
Revised: We consider a squared error loss function,

Original: [Equation]
Revised: [Equation]

Original: Hence, taking gradients at time k w.r.t.
Revised: Hence, taking gradients at time k w.r.t.

Original: θ k we have
Revised: θ k we have

Original: [Equation]
Revised: [Equation]

Original: Taking expected gradients,
Revised: Taking expected gradients,

Original: [Equation]
Revised: [Equation]

Original: In the notation of Proposition
Revised: In the notation of Proposition

Original: [Equation]
Revised: [Equation]

Original: we can write this as
Revised: we can write this as

Original: [Equation]
Revised: [Equation]

Original: ] T with

Original: [Equation]

Original: If we let ¯ x ( t ) =
Revised: If we let ¯ x ( t ) =

Original: a cos(2 πfη − 12 t ) , then we induce an X ( t ) from which { X k } is sampled, as per
Revised: a cos(2 πfη − 12 t ) , then we induce an X ( t ) from which { X k } is sampled, as per

Original: Section A.3.3.
Revised: Section A.4.3.

Original: This results in B k
Revised: This results in B k

Original: = B ( √ ηk with the matrix-valued function B ( t )
Revised: = B ( √ ηk with the matrix-valued function B ( t )

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: From Proposition 2, we have that our learning system numerically integrates an LTV of the form 
Revised: From Proposition 2, we have that our learning system numerically integrates an LTV of the form ¨

Original: [Equation]
Revised: [Equation]

Revised: with ( θ ( t ) − θ ∗ ) =

Revised: (cid:20) θ 1 ( t

Revised: [Equation]

Revised: − θ ∗ 1 ( t

Revised: [Equation]

Revised: θ 2 ( t ) − θ 2 ∗ ( t ) (cid:21) (5)

Original: Note that B ( t ) is piecewise continuous and periodic with period T = 1 f .
Revised: Note that B ( t ) is piecewise continuous and periodic with period T = 1 f .

Original: This second order ODE can be represented as a system of ﬁrst order ODEs with the transformation
Revised: This second order ODE can be represented as a system of first order ODEs with the transformation

Original: [Equation]
Revised: [Equation]

Original: so that the equation equation 5 can be written in standard ﬁrst order linear form ˙ ξ =
Revised: so that the equation equation 5 can be written in standard first order linear form ˙ ξ =

Original: A ( t ) ξ :
Revised: A ( t ) ξ :

Original: [Equation]
Revised: [Equation]

Original: The matrix B ( t ) is piecewise continuous and periodic, which implies the matrix A ( t ) also shares these properties.
Revised: The matrix B ( t ) is piecewise continuous and periodic, which implies the matrix A ( t ) also shares these properties.

Original: The ODE equation 5 satisﬁes all the assumptions needed to make use of Theoremto determine the stability of solution trajectories.
Revised: The ODE equation 5 satisfies all the assumptions needed to make use of Theoremto determine the stability of solution trajectories.

Original: We can obtain a fundamental solution matrix, ψ ( t ) , of equation 6 satisfying initial conditions ψ (0)
Revised: We can obtain a fundamental solution matrix, ψ ( t ) , of equation 6 satisfying initial conditions ψ (0)

Original: =
Revised: =

Original: I 4 × 4 by numerical computation through the use of an ODE solver.
Revised: I 4 × 4 by numerical computation through the use of an ODE solver.

Original: Once we have this matrix ψ ( t ) , we evaluate it at time T = 1 f to obtain the system’s monodromy matrix.
Revised: Once we have this matrix ψ ( t ) , we evaluate it at time T = 1 f to obtain the system’s monodromy matrix.

Original: Now we only need to determine the spectral radius, ρ , of ψ ( T ) to determine the stability of solution trajectories for the system.
Revised: Now we only need to determine the spectral radius, ρ , of ψ ( T ) to determine the stability of solution trajectories for the system.

Original: Details regarding why this is true are given in the proof of
Revised: Details regarding why this is true are given in the proof of

Original: Theorem 1.
Revised: Theorem 1.

Original: We repeat this process, sweeping over values of µ , η , and f to determine the stability of solution trajectories for systems to triples of ﬁxed values for these hyperparameters.
Revised: We repeat this process, sweeping over values of µ , η , and f to determine the stability of solution trajectories for systems to triples of fixed values for these hyperparameters.

Original: The results of this process are given in Figure 6 below.
Revised: The results of this process are given in Figure 6 below.

Original: A. R EDUCING R ESONANT R ESPONSES
Revised: A. R EDUCING R ESONANT R ESPONSES

Original: In experiments 4.1 and 4.2, we see that as the momentum parameter is reduced, the tendency to resonate is mitigated.
Revised: In experiments 4.1 and 4.2, we see that as the momentum parameter is reduced, the tendency to resonate is mitigated.

Original: This aligns with theoretical predictions, which (in the context of those particular experiments) suggest that a sufﬁciently low µ makes SGDm convergent across all frequencies in the band we evaluate.
Revised: This aligns with theoretical predictions, which (in the context of those particular experiments) suggest that a sufficiently low µ makes SGDm convergent across all frequencies in the band we evaluate.

Original: Intuitively, this aligns with the role µ plays in the ODE, as it appears in the following coefﬁcient on the ﬁrst order derivative of system equation 3 (i.e. the system’s ‘damping’ coefﬁcient), repeated below with the damping coefﬁcient α explicitly labelled:
Revised: Intuitively, this aligns with the role µ plays in the ODE, as it appears in the following coefficient on the first order derivative of system equation 3 (i.e. the system’s ‘damping’ coefficient), repeated below with the damping coefficient α explicitly labelled:

Original: [Equation]
Revised: [Equation]

Original: When α = 0 , the system has no damping (i.e. friction is zero, in the physical analogue) so resonant responses are maximized.
Revised: When α = 0 , the system has no damping (i.e. friction is zero, in the physical analogue) so resonant responses are maximized.

Original: As α increases, damping increases, and resonant responses are reduced.
Revised: As α increases, damping increases, and resonant responses are reduced.

Original: There are two ways to increase α : reduce µ or reduce η .
Revised: There are two ways to increase α : reduce µ or reduce η .

Original: A.2. R EDUCING THE MOMENTUM PARAMETER
Revised: A.2. R EDUCING THE MOMENTUM PARAMETER

Original: Figure 6b shows the theoretical heatmap across all possible momentum values, and across a much wider band of frequencies, which suggests a trend: setting µ to a sufﬁciently low value will completely mitigate resonance, though setting µ too low will worsen convergence rate.
Revised: Figure 6b shows the theoretical heatmap across all possible momentum values, and across a much wider band of frequencies, which suggests a trend: setting µ to a sufficiently low value will completely mitigate resonance, though setting µ too low will worsen convergence rate.

Original: For now, we will set aside the observation that µ too low worsens convergence rate, and we will now show that reducing µ will reliably dampen resonance across all other experiments.
Revised: For now, we will set aside the observation that µ too low worsens convergence rate, and we will now show that reducing µ will reliably dampen resonance across all other experiments.

Original: In 4.5, ADAM’s parameter β 1 was varied, and we see that reducing β 1 decreases the tendency to resonate.
Revised: In 4.5, ADAM’s parameter β 1 was varied, and we see that reducing β 1 decreases the tendency to resonate.

Original: Since β 1 is the nearest parameter to µ in SGDm, the desired trend has already been
Revised: Since β 1 is the nearest parameter to µ in SGDm, the desired trend has already been

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: demonstrated.
Revised: demonstrated.

Original: For the remainder of this section, we show the resonance-damping behaviour of µ in the remaining experiments: 4.3, 4.4, and 4.6.
Revised: For the remainder of this section, we show the resonance-damping behaviour of µ in the remaining experiments: 4.3, 4.4, and 4.6.

Original: In particular, from each experiment we choose the conﬁguration which had the highest tendency to resonate, and we modify the experiment by running them with several decreasing values of momentum µ .
Revised: In particular, from each experiment we choose the configuration which had the highest tendency to resonate, and we modify the experiment by running them with several decreasing values of momentum µ .

Original: See Figure 7 for results.
Revised: See Figure 7 for results.

Original: In Section 4, these experiments used momentum µ = 0 .
Revised: In Section 4, these experiments used momentum µ = 0 .

Original: 95 , and here we run with µ ∈ [0 . 85 , 0 .
Revised: 95 , and here we run with µ ∈ [0 . 85 , 0 .

Original: 95] , with all other experimental parameters identical.
Revised: 95] , with all other experimental parameters identical.

Original: A.2. R EDUCING THE STEP - SIZE
Revised: A.2. R EDUCING THE STEP - SIZE

Original: Another way to reduce tendency to resonate is suggested by the damping coefﬁcient equation 7: reducing step-size η .
Revised: Another way to reduce tendency to resonate is suggested by the damping coefficient equation 7: reducing step-size η .

Original: Here we repeat Experiments 4.1 and 4.2, including two smaller step-sizes η to empirically demonstrate that trend.
Revised: Here we repeat Experiments 4.1 and 4.2, including two smaller step-sizes η to empirically demonstrate that trend.

Original: Speciﬁcally, Figure 8 shows the resonance heatmap and spectral radius contour lines for regression in two weights with covariate shift, identically to Experiments 4.
Revised: Specifically, Figure 8 shows the resonance heatmap and spectral radius contour lines for regression in two weights with covariate shift, identically to Experiments 4.

Original: and 4.2.
Revised: and 4.2.

Original: The left column shows sinusoidal covariate shift, and the right column the AR(2) covariate shift.
Revised: The left column shows sinusoidal covariate shift, and the right column the AR(2) covariate shift.

Original: Each row corresponds to a ﬁxed step-size η ∈ { 0 . 01 , 0 .
Revised: Each row corresponds to a fixed step-size η ∈ { 0 . 01 , 0 .

Original: 001 } , and it is clear that resonant, diverging regions are signiﬁcantly reduced in size as step-size is decreased, with a narrower band of frequencies diverging, and the minimum momentum µ required for resonance increasing towards 1.
Revised: 001 } , and it is clear that resonant, diverging regions are significantly reduced in size as step-size is decreased, with a narrower band of frequencies diverging, and the minimum momentum µ required for resonance increasing towards 1.

Original: This trend is reﬂected in both the empirical heatmap results, as well as the theoretically predicted spectral radius contour lines.
Revised: This trend is reflected in both the empirical heatmap results, as well as the theoretically predicted spectral radius contour lines.

Revised: A. E XPERIMENTS 4.1 AND 4.2 WITH S TOCHASTIC G RADIENTS

Revised: Experiment 4.1 1 serves as a simple setting, as close to the theory of Section 3 as possible.

Revised: In particular, each descent step made at time k uses the gradients induced by 20 samples from each

Revised: X k so that the gradient signal is much closer to the expected gradient than the stochastic setting.

Revised: Experiment 4.2 uses the same technique.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: Here we repeat Experiments 4.1 and 4.2, but drawing only a single sample from each X k , so that the gradient signal is fully stochastic.

Revised: That is, we significantly increase gradient signal’s variance.

Revised: As

Revised: Figure 9 shows, the resonant response is significantly damped by increasing gradient signal variance.

Revised: We observe no other change in the empirical heatmap: regions of instability are neither introduced nor removed, and the existing regions do not change shape or location.

Revised: Only the size of instability regions is affected, which is most apparent under comparison to Figure 8, where the empirically observed instability regions are expanded to closer match the theoretical predictions derived from expected gradients.

Revised: Also the bright band of instability in all subfigures of Figure 9 are wider than those of 8.

Revised: This aligns with intuition from the conventional setting with iid data, where increasing gradient variance reduces optimizer stability.

Original: A. P ROOFS A.3. A DDITIONAL N OTATION
Revised: A.4. A DDITIONAL N OTATION

Original: Wherever we write the training input generating process { X k } , it is implicit that the last dimension is ﬁxed at 1 if one wishes to describe linear models with a bias term.
Revised: Wherever we write the training input generating process { X k } , it is implicit that the last dimension is fixed at 1 if one wishes to describe linear models with a bias term.

Original: In this way, the notation (cid:104) θ, X k (cid:105) accommodates linear models with or without a bias term.
Revised: In this way, the notation ⟨ θ, X k ⟩ accommodates linear models with or without a bias term.

Original: We use ξ to denote the phase space coordinates of weights θ , meaning ξ = (cid:20) θ ˙ θ (cid:21) , where vectors θ, ˙ θ ∈ R
Revised: We use ξ to denote the phase space coordinates of weights θ , meaning ξ = (cid:20) θ ˙ θ (cid:21) , where vectors θ, ˙ θ ∈ R

Original: d are stacked so that the resulting ξ ∈ R 2 d .
Revised: d are stacked so that the resulting ξ ∈ R 2 d .

Original: We denote the d -dimensional zero vector as 0 d .
Revised: We denote the d -dimensional zero vector as 0 d .

Original: For an ODE system ˙ ξ ( t ) =
Revised: For an ODE system ˙ ξ ( t ) =

Original: f ( ξ ( t ) , t ) with arbitrary solution trajectory ξ ( t ) approximated by a sequence of iterates { ξ k } , we denote the integration time step as h , and the numerical ﬂow as φ h,k , which is the map such that ξ k
Revised: f ( ξ ( t ) , t ) with arbitrary solution trajectory ξ ( t ) approximated by a sequence of iterates { ξ k } , we denote the integration time step as h , and the numerical flow as ϕ h,k , which is the map such that ξ k

Original: +

Revised: +

Original: =
Revised: =

Original: φ h,k ( ξ k ) .
Revised: ϕ h,k ( ξ k ) .

Original: We denote the k -th discrete timestep as t k , which we always use to refer to the k -th multiple of integration time step h , i.e. t k = hk .
Revised: We denote the k -th discrete timestep as t k , which we always use to refer to the k -th multiple of integration time step h , i.e. t k = hk .

Original: Similar to the main body, when a function of time θ ( t ) is approximated by a discrete time sequence { θ k } , they are denoted with the same symbol, and distinguished by parenthetical argument t and subscript k , respectively.
Revised: Similar to the main body, when a function of time θ ( t ) is approximated by a discrete time sequence { θ k } , they are denoted with the same symbol, and distinguished by parenthetical argument t and subscript k , respectively.

Original: At the risk of abusing notation, we will adhere to this convention and use { ˙ θ k } to be a discrete sequence approximating the function of time ˙ θ ( t ) , which refers to the time derivative of θ ( t ) .
Revised: At the risk of abusing notation, we will adhere to this convention and use { ˙ θ k } to be a discrete sequence approximating the function of time ˙ θ ( t ) , which refers to the time derivative of θ ( t ) .

Original: This means the symbol ˙
Revised: This means the symbol ˙

Original: θ k is the k -th iterate of a sequence approximating ˙
Revised: θ k is the k -th iterate of a sequence approximating ˙

Original: θ ( t ) .
Revised: θ ( t ) .

Original: We now restate Assumptions, Propositions, and the Theorem from Section 3, providing complete proofs.
Revised: We now restate Assumptions, Propositions, and the Theorem from Section 3, providing complete proofs.

Original: A.3. L INEAR T IME -V ARYING E XPECTED G RADIENT VIA C OVARIATE S HIFT
Revised: A.4. L INEAR T IME -V ARYING E XPECTED G RADIENT VIA C OVARIATE S HIFT

Original: Assumption 1.
Revised: Assumption 1.

Original: The covariate generating process { X k } k ∈ N is not identically distributed:
Revised: The covariate generating process { X k } k ∈ N is not identically distributed:

Original: Corr ( X k 1 , X k 1 ) (cid:54) =
Revised: [Equation]

Original: Corr ( X k 2 , X k 2 ) for some k 1 , k 2 ∈ N .
Revised: ) for some k 1 , k 2 ∈ N .

Original: Assumption 2.
Revised: Assumption 2.

Original: The targets Y k are a ﬁxed linear function of X k with iid zero-mean observation noise, (cid:15) k
Revised: The targets Y k are a fixed linear function of X k with iid zero-mean observation noise, ϵ k

Original: such that E [ (cid:15) k ] = 0 .
Revised: such that E [ ϵ k ] = 0 .

Original: That is, Y k
Revised: That is, Y k

Original: =(cid:104)
Revised: =

Original:  θ ∗ , X k (cid:105) + (cid:15) k where θ ∗ ∈ R d is ﬁxed for all k .
Revised: ⟨ θ ∗ , X k ⟩ + ϵ k where θ ∗ ∈ R d is fixed for all k .

Original: Proposition 1.
Revised: Proposition 1.

Original: Under Assumptions 1, 2, a linear model with weights θ k making predictions (cid:98) Y k
Revised: Under Assumptions 1, 2, a linear model with weights θ k making predictions (cid:98) Y k

Original: =(cid:104)
Revised: =

Original:  θ k , X k (cid:105) with a mean squared error (MSE) objective will induce time-varying linear expected loss gradients g k ( θ k )
Revised: ⟨ θ k , X k ⟩ with a mean squared error (MSE) objective will induce time-varying linear expected loss gradients g k ( θ k )

Original: =
Revised: =

Original: B k
Revised: B k

Original: ( θ k − θ ∗ ) for all k ∈ N , where B k ∈ R d × d and B k 1 (cid:54) = B k 2 for some k 1 , k 2 .
Revised: ( θ k − θ ∗ ) for all k ∈ N , where B k ∈ R d × d and B k 1 ̸ = B k 2 for some k 1 , k 2 .

Original: Proof.
Revised: Proof.

Original: We will show that the gradient of MSE loss ∇ θ k L ( Z ; θ k ) is a random variable whose expectation will take the linear form B k ( θ k − θ ∗ ) .
Revised: We will show that the gradient of MSE loss ∇ θ k L ( Z ; θ k ) is a random variable whose expectation will take the linear form B k ( θ k − θ ∗ ) .

Original: We start with gradient for arbitrary time step k 
Revised: We start with gradient for arbitrary time step k ∇ θ k L ( Z ; θ k ) =

Original: [Equation]
Revised: ∇ θ k ( (cid:98) Y k − Y k )

Original: MSE def’n [Equation]
Revised: MSE def’n = ∇ θ k

Original:  (cid:98) Y [Equation] from Prop.
Revised: [( ⟨ θ k , X k ⟩ − Y k ) 2 ] (cid:98) Y k def’n from Prop.= ∇ θ k

Original: [Equation]
Revised: [( ⟨ θ k , X k ⟩ − ⟨ θ ∗ , X k ⟩ + ϵ k ) 2 ]

Original: Y [Equation] from Ass.= [Equation]
Revised: Y k def’n from Ass.= ∇ θ k

Original: [Equation]{+[( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) 2 ] ⟨· , ·⟩ distributivity = 2( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) ∇ θ k [ ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ] chain rule = 2( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) X k = 2 ⟨ θ k − θ ∗ , X k ⟩ X k + 
+}
Revised: {+[( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) 2 ] ⟨· , ·⟩ distributivity = 2( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) ∇ θ k [ ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ] chain rule = 2( ⟨ θ k − θ ∗ , X k ⟩ + ϵ k ) X k = 2 ⟨ θ k − θ ∗ , X k ⟩ X k + 
+}

Original: ] (cid:104)· ,
Revised: ϵ k X k

Revised: Taking expectation with respect to distribution P k :

Original: [Equation]
Revised: [Equation]

Original: distributivity
Revised: g def’n

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] Taking expectation with respect to distribution P k :
Revised: by equation *

Original: [Equation]
Revised: [Equation]

Original: equation
Revised: linear [Equation] indep

Original: [Equation]
Revised: [Equation]

Original: scalar and vector commute
Revised: scalar and vector commute

Original: [Equation]
Revised: [Equation]

Original: commutative
Revised: commutative

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  matrix form
Revised: ·⟩ matrix form

Original: [Equation]
Revised: [Equation]

Original: matrix mult.
Revised: matrix mult.

Original: associative
Revised: associative

Original: [Equation]
Revised: [Equation]

Original: In the last step, we have deﬁned the matrix B k as twice the expected outer product of X k with itself, which is precisely the autocorrelation matrix, and is a function of X k ’s ﬁrst two moments as follows.
Revised: In the last step, we have defined the matrix B k as twice the expected outer product of X k with itself, which is precisely the autocorrelation matrix, and is a function of X k ’s first two moments as follows.

Original: [Equation]
Revised: [Equation]

Original: So if there exists k 1 , k 2 ∈ N such that the autocorrelation matrices differ
Revised: So if there exists k 1 , k 2 ∈ N such that the autocorrelation matrices differ

Original: [Equation]
Revised: [Equation]

Original: ( X

Original: [Equation]

Original: [Equation]

Original: X [Equation] ) then B k 1 (cid:54) = B k 2 .
Revised: then B k 1 ̸ = B k 2 .

Original: Assumption 1 provides the necessary k 1 , k 2 , so the proposition is proved.
Revised: Assumption 1 provides the necessary k 1 , k 2 , so the proposition is proved.

Original: A.3. ODE C ORRESPONDENCE
Revised: A.4. ODE C ORRESPONDENCE

Original: In Section 3 of the main body, we deﬁned B ( t ) , Lipschitz continuous in t , such that B k = B ( √ ηk ) .
Revised: In Section 3 of the main body, we defined B ( t ) , Lipschitz continuous in t , such that B k = B ( √ ηk ) .

Original: If we assume that { X k } are sampled at intervals of √ η from an underlying continuous-time stochastic process X ( t ) for which E P ( t ) [ X ( t )] and Cov ( X ( t ) , X ( t )) are Lipschitz continuous in t , this naturally induces a unique Lipschitz continuous B ( t ) .
Revised: If we assume that { X k } are sampled at intervals of √ η from an underlying continuous-time stochastic process X ( t ) for which E P ( t ) [ X ( t )] and Cov ( X ( t ) , X ( t )) are Lipschitz continuous in t , this naturally induces a unique Lipschitz continuous B ( t ) .

Original: Note P ( t ) is the distribution at time t , and we can extend equation 2 as follows.
Revised: Note P ( t ) is the distribution at time t , and we can extend equation 2 as follows.

Original: [Equation]
Revised: [Equation]

Original: This means that { B k } is sampled from a continuous B ( t ) , with subsequent B k spaced √ η apart.
Revised: This means that { B k } is sampled from a continuous B ( t ) , with subsequent B k spaced √ η apart.

Original: In essence, step-size η acts as a scaling constant, such that tne unit of t is ≈ 1 √ η discrete steps of k .
Revised: In essence, step-size η acts as a scaling constant, such that tne unit of t is ≈ 1 √ η discrete steps of k .

Original: (Exact when 1 √ η is an integer.)
Revised: (Exact when 1 √ η is an integer.)

Original: Proposition 2.
Revised: Proposition 2.

Original: The SGDm iterates { θ k } numerically integrate the ODE system equation 3 with integration step √ η and ﬁrst order consistency.
Revised: The SGDm iterates { θ k } numerically integrate the ODE system equation 3 with integration step √ η and first order consistency.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof proceeds two parts.
Revised: The proof proceeds two parts.

Original: First we derive a ﬁrst order operator-splitting integrator for the
Revised: First we derive a first order operator-splitting integrator for the

Original: ODE equation 3, then we show that it is equivalent to SGDm equation 1.
Revised: ODE equation 3, then we show that it is equivalent to SGDm equation 1.

Original: (Part 1: Operator-Splitting Integrator)
Revised: (Part 1: Operator-Splitting Integrator)

Original: Let ξ : R (cid:55)→ R 2 d be the vector valued function of time whose ﬁrst d elements are θ ( t ) , and last d elements are ˙ θ ( t ) : 
Revised: Let ξ : R (cid:55)→ R 2 d be the vector valued function of time whose first d elements are θ ( t ) , and last d elements are ˙ θ ( t ) : ξ ( t )

Original: [Equation]
Revised: = (cid:20) θ ( t ) − θ ∗ ˙

Original:  (cid:21) (9) i.e. ξ is a phase space transformation allowing us to rewrite ODE equation 3 in the form of equation 4, which can then be split into a sum of separate systems f
Revised: θ ( t ) (cid:21) (9) i.e. ξ is a phase space transformation allowing us to rewrite ODE equation 3 in the form of equation 4, which can then be split into a sum of separate systems f

Original: [1] , f [2] as follows
Revised: [1] , f [2] as follows

Original: [Equation]
Revised: [Equation]

Original: Let h > 0 be an integration time step, φ [1] h,k ( ξ k ) be the implicit Euler numerical ﬂow of f [1] ( ξ ( t ) , t ) :
Revised: Let h > 0 be an integration time step, ϕ [1] h,k ( ξ k ) be the implicit Euler numerical flow of f [1] ( ξ ( t ) , t ) :

Original: [Equation]
Revised: [Equation]

Original: Let φ [2] h,k ( ξ k ) be explicit Euler numerical ﬂow of f [2] ( ξ ( t ) , t ) : 
Revised: Let ϕ [2] h,k ( ξ k ) be explicit Euler numerical flow of f [2] ( ξ ( t ) , t ) : [Equation]

Original: [Equation]
Revised: [Equation]

Original: The composed ﬂow
Revised: The composed flow

Original: [Equation]
Revised: [Equation]

Original: (13) is a sequentially split operator, which has splitting error order 1 because equation 10 is time-varying linear system (Faragó et al., 2011).
Revised: 13) is a sequentially split operator, which has splitting error order 1 because equation 10 is time-varying linear system (Faragó et al., 2011).

Original: The operators being composed, implicit and explicit Euler, are both order 1 consistent with their respective systems (Iserles, 2008).
Revised: The operators being composed, implicit and explicit Euler, are both order 1 consistent with their respective systems (Iserles, 2008).

Original: The overall order of consistency of a split operator is the minimum of splitting error, and the orders of the composed ﬂows (Csomós &
Revised: The overall order of consistency of a split operator is the minimum of splitting error, and the orders of the composed flows (Csomós &

Original: Faragó, 2008).
Revised: Faragó, 2008).

Original: So we have that equation 13 approximates equation 10 with order 1 consistency.
Revised: So we have that equation 13 approximates equation 10 with order 1 consistency.

Original: (Part 
Revised: (Part 2: SGDm Equivalency)

Original: : SGDm Equivalency)

Original: We will now show that equation 13 is equivalent to SGDm when integration timestep h = √ η , where η is the SGDm step-size.
Revised: We will now show that equation 13 is equivalent to SGDm when integration timestep h = √ η , where η is the SGDm step-size.

Original: We start with the deﬁnition of ξ k +1 as the numerical ﬂow of ξ k : 
Revised: We start with the definition of ξ k +1 as the numerical flow of ξ k : [Equation] k

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Original: by equation
Revised: by equation

Original: [Equation]
Revised: [Equation]

Original: subst’n from equation
Revised: subst’n from equation

Original: [Equation]
Revised: [Equation]

Original: def’n of ξ k 
Revised: def’n of ξ k +

Original: [Equation]
Revised: [Equation]

Original: matrix mult.
Revised: matrix mult.

Original: [Equation]
Revised: [Equation]

Original: subst’n from equation
Revised: subst’n from equation

Original: [Equation]
Revised: [Equation]

Original: matrix mult.
Revised: matrix mult.

Original: [Equation]
Revised: [Equation]

Original: simpliﬁcation
Revised: simplification

Original: Recalling that ξ k = (cid:20) θ k − θ ∗ ˙ θ k (cid:21) , the above immediately provides the following two recurrence relations:
Revised: Recalling that ξ k = (cid:20) θ k − θ ∗ ˙ θ k (cid:21) [Equation] the above immediately provides the following two recurrence relations:

Original: [Equation]
Revised: [Equation]

Original: Now let integration time step h be the square root of SGDm step-size, h = √ η , and deﬁne v k = √
Revised: Now let integration time step h be the square root of SGDm step-size, h = √ η , and define v k = √

Original: η ˙ θ k .
Revised: η ˙ θ k .

Original: Substituting h, v k , we proceed via elementary algebra:
Revised: Substituting h, v k , we proceed via elementary algebra:

Original: [Equation]
Revised: [Equation]

Original: Note that B ( t k )
Revised: Note that B ( t k )

Original: =
Revised: =

Original: B k , and that B k deﬁnes the time-varying gradients induced by covariate shift and linear regression to a linear target (Proposition 1).
Revised: B k , and that B k defines the time-varying gradients induced by covariate shift and linear regression to a linear target (Proposition 1).

Original: Hence, under those conditions, we have arrived at
Revised: Hence, under those conditions, we have arrived at

Original: SGDm equation 1.
Revised: SGDm equation 1.

Original: The proof is complete.
Revised: The proof is complete.

Original: The reader may observe that, given a solution ξ ( t ) to the system ˙ ξ ( t ) =
Revised: The reader may observe that, given a solution ξ ( t ) to the system ˙ ξ ( t ) =

Original: A ( t ) ξ ( t ) , the proof above shows that the SGDm iterates { θ k } are precisely a ﬁrst order numerical approximation of the ﬁrst d dimensions of ξ ( t )
Revised: A ( t ) ξ ( t ) , the proof above shows that the SGDm iterates { θ k } are precisely a first order numerical approximation of the first d dimensions of ξ ( t )

Original: = (cid:20) θ ( t ) − θ ∗ ˙
Revised: = (cid:20) θ ( t ) − θ ∗ ˙

Original: θ ( t ) (cid:21) , but the remaining d dimensions are approximated only up to a scale factor √ η by iterates { v k } .
Revised: θ ( t ) (cid:21) , but the remaining d dimensions are approximated only up to a scale factor √ η by iterates { v k } .

Original: However, this is not an issue.
Revised: However, this is not an issue.

Original: Solutions θ ( t ) to the system equation 3 are embedded within the ﬁrst d dimensions of ξ ( t ) , so the remaining d dimensions and iterates { v k } do not affect the result.
Revised: Solutions θ ( t ) to the system equation 3 are embedded within the first d dimensions of ξ ( t ) , so the remaining d dimensions and iterates { v k } do not affect the result.

Original: A.3. P ARAMETRIC R ESONANCE FOR ODE C ONVERGENCE AND D IVERGENCE
Revised: A.4. P ARAMETRIC R ESONANCE FOR ODE C ONVERGENCE AND D IVERGENCE

Original: Theorem 1.
Revised: Theorem 1.

Original: When B ( t ) is periodic such that B ( t )
Revised: When B ( t ) is periodic such that B ( t )

Original: =
Revised: =

Original: B ( t + T ) , the spectral radius ρ of ψ ( T ) characterizes the stability of solution trajectories of equation 3 as follows:
Revised: B ( t + T ) , the spectral radius ρ of ψ ( T ) characterizes the stability of solution trajectories of equation 3 as follows:

Original: • ρ > 1 = ⇒ trivial solution θ ( t )
Revised: • ρ > 1 = ⇒ trivial solution θ ( t )

Original: =
Revised: =

Original: θ ∗ is unstable.
Revised: θ ∗ is unstable.

Original: All other solutions diverge as θ ( t ) → ∞ exponentially with rate ρ .
Revised: All other solutions diverge as θ ( t ) → ∞ exponentially with rate ρ .

Original: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as θ ( t ) → θ ∗ exponentially with rate ρ .
Revised: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as θ ( t ) → θ ∗ exponentially with rate ρ .

Original: Proof.
Revised: Proof.

Original: The proof of this theorem relies heavily on the well-established mathematics of Floquet theory and the stability result is contained in Theorem 1.9 and Theorem 1.10 of (Halanay, 1966).
Revised: The proof of this theorem relies heavily on the well-established mathematics of Floquet theory and the stability result is contained in Theorem 1.9 and Theorem 1.10 of (Halanay, 1966).

Original: Theorem 1.10 states that for a system of differential equations of the form
Revised: Theorem 1.10 states that for a system of differential equations of the form

Original: [Equation]
Revised: [Equation]

Original: where A ( t ) is piecewise continuous, the stability of the trivial solution ξ ( t ) ≡ 0 is determined by the spectral radius of the system’s monodromy matrix M deﬁned below
Revised: where A ( t ) is piecewise continuous, the stability of the trivial solution ξ ( t ) ≡ 0 is determined by the spectral radius of the system’s monodromy matrix M defined below

Original: [Equation]
Revised: [Equation]

Original: where ˙ ψ ( t ) =
Revised: where ˙ ψ ( t ) =

Original: A ( t ) ψ ( t ) (Monodromy Matrix)
Revised: A ( t ) ψ ( t ) (Monodromy Matrix)

Original: Matrix-valued functions ψ ( t ) are the system’s fundamental solution matrix , and elementary existence results for linear systems allow one to choose a ψ ( t ) such that ψ (0)
Revised: Matrix-valued functions ψ ( t ) are the system’s fundamental solution matrix , and elementary existence results for linear systems allow one to choose a ψ ( t ) such that ψ (0)

Original: =
Revised: =

Original: I so that the monodromy matrix simpliﬁes as
Revised: I so that the monodromy matrix simplifies as

Original: [Equation]
Revised: [Equation]

Original: Below we denote the spectral radius of ψ ( T ) (and hence of M ) as ρ .
Revised: Below we denote the spectral radius of ψ ( T ) (and hence of M ) as ρ .

Original: (First Order Linear Form, Stability via Floquet)
Revised: (First Order Linear Form, Stability via Floquet)

Original: Below we show that equation 3 can be transformed to the form equation 14 with A ( t ) continuous such that trivial solution stability of equation 14 is equivalent to stability of the solution θ ( t )
Revised: Below we show that equation 3 can be transformed to the form equation 14 with A ( t ) continuous such that trivial solution stability of equation 14 is equivalent to stability of the solution θ ( t )

Original: = θ ∗ .
Revised: = θ ∗ .

Original: Let ξ ( t ) be the phase space transformation of θ ( t ) , similarly to the proof of Proposition
Revised: Let ξ ( t ) be the phase space transformation of θ ( t ) , similarly to the proof of Proposition

Original: [Equation]
Revised: [Equation]

Original: so that for each t , θ ( t ) ∈ R d and ξ ( t ) ∈ R 2 d .
Revised: so that for each t , θ ( t ) ∈ R d and ξ ( t ) ∈ R 2 d .

Original: This means equation 3 (restated below)
Revised: This means equation 3 (restated below)

Original: [Equation]
Revised: [Equation]

Original: is equivalent to the following ﬁrst order linear form
Revised: is equivalent to the following first order linear form

Original: [Equation]
Revised: [Equation]

Original: Since B ( t ) is periodic and continuous, which is stronger than the requisite piecewise continuity.
Revised: Since B ( t ) is periodic and continuous, which is stronger than the requisite piecewise continuity.

Original: Since all other submatrices of A ( t ) are constant, we have that A ( t ) is also periodic and (piecewise) continuous.
Revised: Since all other submatrices of A ( t ) are constant, we have that A ( t ) is also periodic and (piecewise) continuous.

Original: Now Theorem 1.10 in (Halanay, 1966) immediately provides the following
Revised: Now Theorem 1.10 in (Halanay, 1966) immediately provides the following

Original: • ρ > 1 = ⇒ trivial solution ξ ( t ) = 0 is unstable.
Revised: • ρ > 1 = ⇒ trivial solution ξ ( t ) = 0 is unstable.

Original: All other solutions diverge as ξ ( t ) → ∞ exponentially with rate ρ .
Revised: All other solutions diverge as ξ ( t ) → ∞ exponentially with rate ρ .

Original: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as ξ ( t ) →exponentially with rate ρ .
Revised: • ρ < 1 = ⇒ trivial solution is asymptotically stable, all other solutions converge as ξ ( t ) →exponentially with rate ρ .

Original: Since ξ ( t ) =
Revised: Since ξ ( t ) =

Original: [Equation]
Revised: [Equation]

Original: • The trivial solution ξ ( t ) = 0 is equivalent to θ ( t )
Revised: • The trivial solution ξ ( t ) = 0 is equivalent to θ ( t )

Original: = θ ∗ • ξ ( t ) → 0 is equivalent to θ ( t ) → θ ∗ • ξ ( t ) → ∞ is equivalent to θ ( t ) → ∞
Revised: = θ ∗ • ξ ( t ) → 0 is equivalent to θ ( t ) → θ ∗ • ξ ( t ) → ∞ is equivalent to θ ( t ) → ∞

Original: The theorem is proved.
Revised: The theorem is proved.

Original: A. D ETAILED DESCRIPTIONS OF EXPERIMENTS
Revised: A. D ETAILED DESCRIPTIONS OF EXPERIMENTS

Original: Below we provide details of the experiments from section 4 in the main body.
Revised: Below we provide details of the experiments from section 4 in the main body.

Original: The tables below contain details about the data generating process, learning algorithm and optimizer, and hyperparameter choices.
Revised: The tables below contain details about the data generating process, learning algorithm and optimizer, and hyperparameter choices.

Original: [Table]
Revised: Experiments 4.1-4.4 introduce new data generating processes { X k } via their mean sequences { ¯ x k } . For those experiments, a sample trajectory is visualized for several values of the process’ frequency tuning parameter, either f or T . In addition to the sample trajectory, the frequency content of each process is shown in two ways: the power spectral density (PSD) of { X k } estimated via a long sample trajectory, as well as the PSD of the underlying mean sequence { ¯ x k } .

Original: [Equation] [Table] [Equation] [Table]
Revised: As one would expect from the sampling relationship between X k and ¯ x k , the PSD of { X k } is simply the PSD of { ¯ x k } with a noise floor.

Original: [Equation] [Table] [Equation] [Table]
Revised: Experiments 4.5, 4.6 reuse the process from experiment 4.4, so the visual depiction is elided.

Original: [Equation]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Figure]

Original: [Equation]
Revised: for three values of frequency tuning parameter T .

Original: [Table] [Equation] [Table] [Equation]
Revised: The signal is a square wave in a single dimension for the figure, but is a square wave randomly oriented in higher dimensions for the actual experiment.

Original: [Table] [Equation]
Revised: There are multiple frequency peaks, with the frequency domain contracted by changing T .

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation]
Revised: d

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: iid

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Table] if (cid:98) 2 kT (cid:99) ≡ 0 mod− ξ 2 || ξ || if (cid:98) 2 kT (cid:99) ≡ 1 modξ ∼ N (0 , I d  × d  ) iid
Revised: θ ∗ ∼ N (0 , cI d +1 × d +1 ) where c

Original: Input Sampling X k ∼ N (¯ x k , 0 .I d × d ) Target Function (ﬁxed for all k )
Revised: = 0 .

Original: Y k = (cid:104) θ ∗ [1: d ] , X k (cid:105) + θ ∗ d +1 + (cid:15) k θ ∗ ∼ N (0 , cI d +1 × d +1 ) where c = 0 .(cid:15)
Revised: [Equation]

Original: k ∼ N (0 , 0 . 1)
Revised: ∼ N (0 , 0 . 1)

Original: Model and Optimizer (cid:99) 
Revised: Model and Optimizer (cid:99) Y

Original: Y k = (cid:104) θ [1: d ] , Xk (cid:105) + θ
Revised: [Equation] = ⟨ θ [1: d ] , X

Original: d
Revised: [Equation]

Original:  + θ 0 ∼ N (0 , cI d +1 × d +1 ) where c = 0 .
Revised: ⟩ + θ d + θ 0 ∼ N (0 , cI d +1 × d +1 ) where c = 0 .

Original: ( θ k ) k ∈ N ← SGDm ( η, µ ) η = 0 .
Revised: ( θ k ) k ∈ N ← SGDm ( η, µ ) η = 0 .

Original: 01 , µ = 0 .k ∈
Revised: 01 , µ = 0 .k ∈

Original: [Table]
Revised: [Table]

Original: v = 0 .

Original: 25 for Figure 4b, v ∈ [0 , 0 . 4] for Figure 4a

Original: [Table]
Revised: [Figure]

Original: d ∈ [1 , 9] for Figure 4b, d = 5 for Figure 4a [Table] k = [Table]
Revised: A.5. E XPERIMENT 4.5 D ETAILS

Original: i [Table] i = (cid:22) k T (cid:23)
Revised: For visual depiction, see Figure 13.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: to Table

Original: Y k = cos( π || X k || )

Original: + (cid:15) [Table] (cid:15)
Revised: [Table] A.5. E XPERIMENT 4.6 D ETAILS

Revised: For visual depiction, see Figure 13.

Original: [Table]
Revised: [Table]

Revised: N ← SGDm (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]