Original: L EARNING IN TEMPORALLY STRUCTURED ENVIRONMENTS
Revised: L EARNING IN TEMPORALLY STRUCTURED ENVIRONMENTS

Original: A BSTRACT
Revised: A BSTRACT

Original: Natural environments have temporal structure at multiple timescales.
Revised: Natural environments have temporal structure at multiple timescales.

Original: This property is reﬂected in biological learning and memory but typically not in machine learning systems.
Revised: This property is reﬂected in biological learning and memory but typically not in machine learning systems.

Original: We advance a multiscale learning method in which each weight in a neural network is decomposed as a sum of subweights with different learning and decay rates.
Revised: We advance a multiscale learning method in which each weight in a neural network is decomposed as a sum of subweights with different learning and decay rates.

Original: Thus knowledge becomes distributed across different timescales, enabling rapid adaptation to task changes while avoiding catastrophic interference.
Revised: Thus knowledge becomes distributed across different timescales, enabling rapid adaptation to task changes while avoiding catastrophic interference.

Original: First, we prove previous models that learn at multiple timescales, but with complex coupling between timescales, are equivalent to multiscale learning via a reparameterization that eliminates this coupling.
Revised: First, we prove previous models that learn at multiple timescales, but with complex coupling between timescales, are equivalent to multiscale learning via a reparameterization that eliminates this coupling.

Original: The same analysis yields a new characterization of momentum learning, as a fast weight with a negative learning rate.
Revised: The same analysis yields a new characterization of momentum learning, as a fast weight with a negative learning rate.

Original: Second, we derive a model of Bayesian inference over 1 /f noise, a common temporal pattern in many online learning domains that involves long-range (power law) autocorrelations.
Revised: Second, we derive a model of Bayesian inference over 1 /f noise, a common temporal pattern in many online learning domains that involves long-range (power law) autocorrelations.

Original: The generative side of the model expresses 1 /f noise as a sum of diffusion processes at different timescales, and the inferential side tracks these latent processes using a Kalman ﬁlter.
Revised: The generative side of the model expresses 1 /f noise as a sum of diffusion processes at different timescales, and the inferential side tracks these latent processes using a Kalman ﬁlter.

Original: We then derive a variational approximation to the Bayesian model and show how it is an extension of the multiscale learner.
Revised: We then derive a variational approximation to the Bayesian model and show how it is an extension of the multiscale learner.

Original: The result is an optimizer that can be used as a drop-in replacement in an arbitrary neural network architecture.
Revised: The result is an optimizer that can be used as a drop-in replacement in an arbitrary neural network architecture.

Original: Third, we evaluate the ability of these methods to handle nonstationarity by testing them in online prediction tasks characterized by 1 /f noise in the latent parameters.
Revised: Third, we evaluate the ability of these methods to handle nonstationarity by testing them in online prediction tasks characterized by 1 /f noise in the latent parameters.

Original: We ﬁnd that the Bayesian model signiﬁcantly outperforms online stochastic gradient descent and two batch 
Revised: We ﬁnd that the Bayesian model signiﬁcantly outperforms online stochastic gradient descent and two batch heuristics that rely preferentially or

Original: heuristics that rely preferentially or exclusively on more recent data.
Revised: exclusively on more recent data.

Revised: Moreover,

Revised: the

Original: Moreover, the variational approximation performs nearly as well as the full Bayesian model, and with memory requirements that are linear in the size of the network
Revised: variational approximation performs nearly as well as the full Bayesian model, and with memory requirements that are linear in the size of the network

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many online tasks facing both biological and artiﬁcial intelligence systems involve changes in data distribution over time.
Revised: Many online tasks facing both biological and artiﬁcial intelligence systems involve changes in data distribution over time.

Original: Natural environments exhibit correlations at a wide range of timescales, a pattern variously referred to as self-similarity, power-law correlations, and 1 /f noise (Keshner, 1982).
Revised: Natural environments exhibit correlations at a wide range of timescales, a pattern variously referred to as self-similarity, power-law correlations, and 1 /f noise (Keshner, 1982).

Original: This is in stark contrast with the iid environments assumed by many machine learning (ML) methods, and with diffusion or random-walk environments that exhibit only short-range correlations.
Revised: This is in stark contrast with the iid environments assumed by many machine learning (ML) methods, and with diffusion or random-walk environments that exhibit only short-range correlations.

Original: Moreover, biological learning systems are well-tuned to the temporal statistics of natural environments, as seen in phenomena of human cognition including power laws in learning (Anderson, 1982), power-law forgetting (Wixted & Ebbesen, 1997), long-range sequential effects (Wilder et al., 2013), and spacing effects (Anderson & Schooler, 1991; Cepeda et al., 2008).
Revised: Moreover, biological learning systems are well-tuned to the temporal statistics of natural environments, as seen in phenomena of human cognition including power laws in learning (Anderson, 1982), power-law forgetting (Wixted & Ebbesen, 1997), long-range sequential effects (Wilder et al., 2013), and spacing effects (Anderson & Schooler, 1991; Cepeda et al., 2008).

Original: An important goal is to incorporate similar inductive biases into ML systems for online or continual learning.
Revised: An important goal is to incorporate similar inductive biases into ML systems for online or continual learning.

Original: This paper analyzes a framework for learning in temporally structured environments, multiscale learning , which for neural networks (NNs) can be implemented as a new kind of optimizer.
Revised: This paper analyzes a framework for learning in temporally structured environments, multiscale learning , which for neural networks (NNs) can be implemented as a new kind of optimizer.

Original: A common explanation for self-similar temporal structure in nature is that it arises from a mixture of events at various timescales.
Revised: A common explanation for self-similar temporal structure in nature is that it arises from a mixture of events at various timescales.

Original: Indeed, many generative models of 1 /f noise involve summing independent stochastic processes with varying time constants (Eliazar & Klafter, 2009).
Revised: Indeed, many generative models of 1 /f noise involve summing independent stochastic processes with varying time constants (Eliazar & Klafter, 2009).

Original: Accordingly, the multiscale optimizer comprises multiple learning processes operating in parallel at different timescales.
Revised: Accordingly, the multiscale optimizer comprises multiple learning processes operating in parallel at different timescales.

Original: In a NN, every weight w j is replaced by a family of subweights ! ij , each with its own learning rate and decay rate, that sum to determine the weight as a whole.
Revised: In a NN, every weight w j is replaced by a family of subweights ω ij , each with its own learning rate and decay rate, that sum to determine the weight as a whole.

Original: Learning at multiple timescales is a key idea in several theories in neuroscience, including conditioning (Staddon et al., 2002), learning (Benna & Fusi, 2016), memory (Howard & Kahana, 2002; Mozer et al., 2009), and motor control (Kording et al., 2007), and has also been exploited in ML (Hinton & Plaut, 1987; Rusch et al., 2022).
Revised: Learning at multiple timescales is a key idea in several theories in neuroscience, including conditioning (Staddon et al., 2002), learning (Benna & Fusi, 2016), memory (Howard & Kahana, 2002; Mozer et al., 2009), and motor control (Kording et al., 2007), and has also been exploited in ML (Hinton & Plaut, 1987; Rusch et al., 2022).

Original: The multiscale learner isolates and simpliﬁes this idea, by assuming knowledge at different timescales evolves independently and that credit assignment follows gradient descent.
Revised: The multiscale learner isolates and simpliﬁes this idea, by assuming knowledge at different timescales evolves independently and that credit assignment follows gradient descent.

Original: The ﬁrst part of this paper (Sections 2 and 3) proves three other models are formally equivalent to instances of the multiscale optimizer: a new variant of fast weights (cf.
Revised: The ﬁrst part of this paper (Sections 2 and 3) proves three other models are formally equivalent to instances of the multiscale optimizer: a new variant of fast weights (cf.

Original: Ba et al., 2016; Hinton &
Revised: Ba et al., 2016; Hinton &

Original: Plaut, 1987), the model synapse of Benna & Fusi (2016), and momentum learning (Rumelhart et al., 1986; Qian, 1999).
Revised: Plaut, 1987), the model synapse of Benna & Fusi (2016), and momentum learning (Rumelhart et al., 1986; Qian, 1999).

Original: The insight behind these proofs is that each of these models can be written in terms of a linear update rule with diagonalizable transition matrix.
Revised: The insight behind these proofs is that each of these models can be written in terms of a linear update rule with diagonalizable transition matrix.

Original: Thus the eigenvectors of this matrix correspond to states that evolve independently.
Revised: Thus the eigenvectors of this matrix correspond to states that evolve independently.

Original: By writing the state of the model as a mixture of eigenvectors, we effect a coordinate transformation that exactly yields the multiscale optimizer.
Revised: By writing the state of the model as a mixture of eigenvectors, we effect a coordinate transformation that exactly yields the multiscale optimizer.

Original: These results imply that the complicated coupling among timescales assumed by some models can be superﬂuous.
Revised: These results imply that the complicated coupling among timescales assumed by some models can be superﬂuous.

Original: They also provide a new perspective on momentum learning, with implications for how and when it is beneﬁcial and how it interacts with nonstationarity in the task environment.
Revised: They also provide a new perspective on momentum learning, with implications for how and when it is beneﬁcial and how it interacts with nonstationarity in the task environment.

Original: In Section 4, we provide a normative grounding for multiscale learning in terms of Bayesian inference over 1 /f noise.
Revised: In Section 4, we provide a normative grounding for multiscale learning in terms of Bayesian inference over 1 /f noise.

Original: Our starting point is a generative model of 1 /f noise as a sum of diffusion processes at different timescales.
Revised: Our starting point is a generative model of 1 /f noise as a sum of diffusion processes at different timescales.

Original: Exact Bayesian inference with respect to this generative process is possible using a Kalman ﬁlter (KF) that tracks the component processes jointly (Kording et al., 2007).
Revised: Exact Bayesian inference with respect to this generative process is possible using a Kalman ﬁlter (KF) that tracks the component processes jointly (Kording et al., 2007).

Original: When learning a single environmental parameter ✓ , such as mean reward for some action in a bandit task, this amounts to modeling ✓ ( t )=
Revised: When learning a single environmental parameter θ , such as mean reward for some action in a bandit task, this amounts to modeling θ ( t )

Original: P ni =

Revised: = (cid:80) ni =

Original: z i ( t ) , where each z i is a diffusion process with a different characteristic timescale ⌧ i , and doing joint inference over Z = ( z 1 , . . . , z n ) .
Revised: z i ( t ) , where each z i is a diffusion process with a different characteristic timescale τ i , and doing joint inference over Z = ( z 1 , . . . , z n ) .

Original: We then generalize this approach to an arbitrary statistical model, h ( x , ✓ ) , where x is the input and ✓ 2 R m is a parameter vector to be estimated.
Revised: We then generalize this approach to an arbitrary statistical model, h ( x , θ ) , where x is the input and θ ∈ R m is a parameter vector to be estimated.

Original: For instance, h might be a NN with parameters ✓ .
Revised: For instance, h might be a NN with parameters θ .

Original: Our
Revised: Our

Original: Bayesian model places a 1 /f prior on ✓ (as a stochastic process), by assuming ✓ ( t )=
Revised: Bayesian model places a 1 /f prior on θ (as a stochastic process), by assuming θ ( t )

Original: P ni =

Revised: = (cid:80) ni =

Original: z i ( t ) for diffusion processes z i 2 R m with characteristic timescales ⌧ i .
Revised: z i ( t ) for diffusion processes z i ∈ R m with characteristic timescales τ i .

Original: We then do approximate inference over the joint state Z
Revised: We then do approximate inference over the joint state Z

Original: = ( z 1 , . . .
Revised: = ( z 1 , . . .

Original: , z n ) , using an extended Kalman ﬁlter (EKF) that linearizes h by calculating its Jacobian at each step (Singhal & Wu, 1989; Puskorius & Feldkamp, 2003).
Revised: , z n ) , using an extended Kalman ﬁlter (EKF) that linearizes h by calculating its Jacobian at each step (Singhal & Wu, 1989; Puskorius & Feldkamp, 2003).

Original: Next, we derive a variational approximation to the EKF that constrains the covariance matrix to be diagonal, and show how it extends the multiscale optimizer.
Revised: Next, we derive a variational approximation to the EKF that constrains the covariance matrix to be diagonal, and show how it extends the multiscale optimizer.

Original: Speciﬁcally, writing w j and ! ij as the current mean estimates of ✓ j and z ij (for weight j and time scale i ), the variational update to each ! ij follows that of the multiscale optimizer, with additional machinery for determining decay rates based on ⌧ i and adapting learning rates based on the current prior variance s 2 ij ( t ) .
Revised: Speciﬁcally, writing w j and ω ij as the current mean estimates of θ j and z ij (for weight j and time scale i ), the variational update to each ω ij follows that of the multiscale optimizer, with additional machinery for determining decay rates based on τ i and adapting learning rates based on the current prior variance s 2 ij ( t ) .

Original: In Section 5, we test our methods in online prediction and classiﬁcation tasks with nonstationary distributions.
Revised: In Section 5, we test our methods in online prediction and classiﬁcation tasks with nonstationary distributions.

Original: In online learning, nonstationarity often manifests as poorer generalization performance on future data versus held-out data from within the training interval.
Revised: In online learning, nonstationarity often manifests as poorer generalization performance on future data versus held-out data from within the training interval.

Original: Common solutions are to train on a window of ﬁxed length (to exclude “stale” data) or to use stochastic gradient descent (SGD) with ﬁxed learning rate and weight decay, which leads older observations to have less inﬂuence (Ditzler et al., 2015).
Revised: Common solutions are to train on a window of ﬁxed length (to exclude “stale” data) or to use stochastic gradient descent (SGD) with ﬁxed learning rate and weight decay, which leads older observations to have less inﬂuence (Ditzler et al., 2015).

Original: Here, we demonstrate that performance can be signiﬁcantly improved by retaining all data and using a learning model that accounts for the temporal structure of the environment.
Revised: Here, we demonstrate that performance can be signiﬁcantly improved by retaining all data and using a learning model that accounts for the temporal structure of the environment.

Original: We introduce nonstationarity in our simulations by varying the latent data-generating parameters according to 1 /f noise.
Revised: We introduce nonstationarity in our simulations by varying the latent data-generating parameters according to 1 /f noise.

Original: Thus an important caveat is the task domains are matched to the Bayesian model.
Revised: Thus an important caveat is the task domains are matched to the Bayesian model.

Original: Notwithstanding, we test robustness by using a different set of timescales for task generation versus learning (Section 5.1), a generative process that mismatches the NN architecture (Section 5.2), and a construction of 1 /f noise that differs from the sum-of-diffusion process the model assumes (Section 5.3).
Revised: Notwithstanding, we test robustness by using a different set of timescales for task generation versus learning (Section 5.1), a generative process that mismatches the NN architecture (Section 5.2), and a construction of 1 /f noise that differs from the sum-of-diffusion process the model assumes (Section 5.3).

Original: Results show the Bayesian methods (KF and EKF) outperform windowing and online SGD, as well as a novel heuristic of training the network on all past data with gradients weighted by recency.
Revised: Results show the Bayesian methods (KF and EKF) outperform windowing and online SGD, as well as a novel heuristic of training the network on all past data with gradients weighted by recency.

Original: We also ﬁnd the variational approximation performs nearly as well as the full model (Section 5.1) and scales well to a multilayer NN trained on real data (Section 5.3).
Revised: We also ﬁnd the variational approximation performs nearly as well as the full model (Section 5.1) and scales well to a multilayer NN trained on real data (Section 5.3).

Original:  M ULTISCALE OPTIMIZER
Revised:  M ULTISCALE OPTIMIZER

Original: Assume a statistical model ˆ y ( t )
Revised: Assume a statistical model ˆ y ( t )

Original: =
Revised: =

Original: h ( x ( t ) , w ( t )) and loss function L ( y , ˆ y ) , where x ( t ) is the input on step t , w ( t ) is the parameter estimate, ˆ y ( t ) is the model output, and y ( t ) is the target output.
Revised: h ( x ( t ) , w ( t )) and loss function L ( y , ˆ y ) , where x ( t ) is the input on step t , w ( t ) is the parameter estimate, ˆ y ( t ) is the model output, and y ( t ) is the target output.

Original: In a
Revised: In a

Original: [Figure]
Revised: [Figure]

Original: NN, w ( t ) is the vector of current weights.
Revised: NN, w ( t ) is the vector of current weights.

Original: (Under the Bayesian framing in Section 4, w is the mean estimate of the optimal parameters ✓ .)
Revised: (Under the Bayesian framing in Section 4, w is the mean estimate of the optimal parameters θ .)

Original: For exposition, we assume the weights are updated by SGD,
Revised: For exposition, we assume the weights are updated by SGD,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and we henceforth abbreviate the gradient as @ w ( t ) L .
Revised: and we henceforth abbreviate the gradient as ∂ w ( t ) L .

Original: However, the following approach can be naturally composed with other optimizers, such as extensions of SGD or Hebbian learning, by replacing  ↵@ w ( t ) L with the appropriate update term.
Revised: However, the following approach can be naturally composed with other optimizers, such as extensions of SGD or Hebbian learning, by replacing − α∂ w ( t ) L with the appropriate update term.

Original: The multiscale optimizer is motivated by the assumption that, in online learning tasks, the true or optimal parameters change over time, on multiple timescales.
Revised: The multiscale optimizer is motivated by the assumption that, in online learning tasks, the true or optimal parameters change over time, on multiple timescales.

Original: Accordingly, it expands each weight into a sum of subweights, w j = P ! ij , each with a different learning rate ↵ i and decay rate  i .
Revised: Accordingly, it expands each weight into a sum of subweights, w j = (cid:80) ω ij , each with a different learning rate α i and decay rate γ i .

Original: Here j indexes weights in w , and i indexes timescales.
Revised: Here j indexes weights in w , and i indexes timescales.

Original: The subweights evolve according to:
Revised: The subweights evolve according to:

Original: [Equation]
Revised: [Equation]

Original: Each ! ij has characteristic timescale⌧ i :=
Revised: Each ω ij has characteristic timescale

Original:  (  log  i )  1 .
Revised: τ i := ( − log γ i ) − 1 .

Original: Note that @ w j ( t ) L = @ ! ij ( t ) L , so one can think of the gradient for w j being apportioned among the subweights (with total learning rate ↵ = P ↵ i ), or equivalently of each subweight following its own gradient.
Revised: Note that ∂ w j ( t ) L = ∂ ω ij ( t ) L , so one can think of the gradient for w j being apportioned among the subweights (with total learning rate α = (cid:80) α i ), or equivalently of each subweight following its own gradient.

Original: 2. F AST WEIGHTS
Revised: 2. F AST WEIGHTS

Original: A potentially important special case of multiscale learning arises with two timescales, w = !
Revised: A potentially important special case of multiscale learning arises with two timescales, w = ω slow

Original: slow + ! fast .
Revised: + ω fast .

Original: We assume  slow = 1 (no decay) and ↵ fast > ↵ slow .
Revised: We assume γ slow = 1 (no decay) and α fast > α slow .

Original: Thus each ! slow ,j can be thought of as the original weight, which is augmented by ! fast ,j , a second channel between the same neurons that both learns and decays rapidly.
Revised: Thus each ω slow ,j can be thought of as the original weight, which is augmented by ω fast ,j , a second channel between the same neurons that both learns and decays rapidly.

Original: The fast weight enables the system to adapt quickly to distribution shifts while resisting catastrophic forgetting (Figure 1).
Revised: The fast weight enables the system to adapt quickly to distribution shifts while resisting catastrophic forgetting (Figure 1).

Original: This model is conceptually similar to the fast weights approach of Ba et al.
Revised: This model is conceptually similar to the fast weights approach of Ba et al.

Original: (2016) and Hinton &
Revised: (2016) and Hinton &

Original: Plaut (1987).
Revised: Plaut (1987).

Original: In that work, the weights are updated by a different mechanism (Hebbian learning) than the primary weights, and they act as a memory of recent hidden states in a recurrent network.
Revised: In that work, the weights are updated by a different mechanism (Hebbian learning) than the primary weights, and they act as a memory of recent hidden states in a recurrent network.

Original: In the present conception, fast weights optimize the same loss as the primary weights, only with different temporal properties, and they act as a memory for recent learning signals (e.g., loss gradients).
Revised: In the present conception, fast weights optimize the same loss as the primary weights, only with different temporal properties, and they act as a memory for recent learning signals (e.g., loss gradients).

Original: Thus they are perhaps better suited for handling distribution shifts of the sort considered here.
Revised: Thus they are perhaps better suited for handling distribution shifts of the sort considered here.

Original:  E QUIVALENCE RESULTS 3. B ENNA -F USI SYNAPSE
Revised:  E QUIVALENCE RESULTS 3. B ENNA -F USI SYNAPSE

Original: Benna & Fusi’s (2016) model synapse is designed to capture how biochemical mechanisms in real synapses implement a cascading hierarchy of timescales, and has been adopted in ML for continual reinforcement learning (Kaplanis et al., 2018; 2019).
Revised: Benna & Fusi’s (2016) model synapse is designed to capture how biochemical mechanisms in real synapses implement a cascading hierarchy of timescales, and has been adopted in ML for continual reinforcement learning (Kaplanis et al., 2018; 2019).

Original: We consider a single weight w in a network, suppressing the index j .
Revised: We consider a single weight w in a network, suppressing the index j .

Original: The Benna-Fusi model assumes that the information in w is maintained in a 1d hierarchy of variables u 1 , . . .
Revised: The Benna-Fusi model assumes that the information in w is maintained in a 1d hierarchy of variables u 1 , . . .

Original: , u n , each dynamically coupled to its immediate neighbors:
Revised: , u n , each dynamically coupled to its immediate neighbors:

Original: [Equation]
Revised: [Equation]

Original: for 2  k  n , with g n = 0 .
Revised: for 2 ≤ k ≤ n , with g n = 0 .

Original: The external behavior of the synapse comes from u 1 alone (i.e., w = u 1 ), while u 2: n act as stores with progressively longer timescales.
Revised: The external behavior of the synapse comes from u 1 alone (i.e., w = u 1 ), while u 2: n act as stores with progressively longer timescales.

Original: This update rule can be rewritten as
Revised: This update rule can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: with transition matrix T determined by the coefﬁcients in Equations 3 and 4, and external signal d ( t ) deﬁned by d 1 ( t )
Revised: with transition matrix T determined by the coefﬁcients in Equations 3 and 4, and external signal d ( t ) deﬁned by d 1 ( t )

Original: = 1 C 1 @ w ( t ) L and d 2: n ⌘ 0 .
Revised: = 1 C 1 ∂ w ( t ) L and d 2: n ≡ 0 .

Original: It can be shown that the transition matrix is diagonalizable, T = V ⇤ V  1 , with eigenvalues ⇤ ii
Revised: It can be shown that the transition matrix is diagonalizable, T = V Λ V − 1 , with eigenvalues Λ ii

Original: = 
Revised: = λ

Original:  i < 1 (see Appendix A).
Revised: i < 1 (see Appendix A).

Original: We can further enforce V 1 · = 1 , for a purpose explained below.
Revised: We can further enforce V 1 · = 1 , for a purpose explained below.

Original: We refer to the eigenvectors (columns V · i ) as modes of the system, because they are preserved over time up to a scalar.
Revised: We refer to the eigenvectors (columns V · i ) as modes of the system, because they are preserved over time up to a scalar.

Original: That is, if the initial state is proportional to mode i , then in the absence of external signal ( d ⌘ 0 ), the system will remain in that mode, decaying exponentially with rate factor  i :
Revised: That is, if the initial state is proportional to mode i , then in the absence of external signal ( d ≡ 0 ), the system will remain in that mode, decaying exponentially with rate factor λ i :

Original: [Equation]
Revised: [Equation]

Original: In general, any state can be written uniquely as a linear combination of modes, u = 
Revised: In general, any state can be written uniquely as a linear combination of modes, u = (cid:80) ω

Original: [Equation]
Revised: i V · i = V ω .

Original: Therefore, reparameterizing the model as ! :
Revised: Therefore, reparameterizing the model as ω :

Original: = V  1 u yields the simpliﬁed update equation:
Revised: = V − 1 u yields the simpliﬁed update equation:

Original: [Equation]
Revised: [Equation]

Original: where V  1 d ( t ) = 1 C 1 [ V  1 ] · 1 @ w ( t ) L .
Revised: where V − 1 d ( t ) = 1 C 1 [ V − 1 ] · 1 ∂ w ( t ) L .

Original: Because ⇤ is diagonal, there is no cross-talk between the modes, unlike in the original dynamics.
Revised: Because Λ is diagonal, there is no cross-talk between the modes, unlike in the original dynamics.

Original: Thus we have derived an instance of the multiscale optimizer, with subweights !
Revised: Thus we have derived an instance of the multiscale optimizer, with subweights ω

Original: i ( t ) , decay rates  i , and learning rates 1 C 1 [ V  1 ] i 1 .
Revised: i ( t ) , decay rates λ i , and learning rates 1 C 1 [ V − 1 ] i 1 .

Original: The assumption above,
Revised: The assumption above,

Original: V 1 · = 1 , implies w = u 1 = P !
Revised: V 1 · = 1 , implies w = u 1 = (cid:80) ω

Original: i , so the models agree on the external behavior of the weight as a whole.
Revised: i , so the models agree on the external behavior of the weight as a whole.

Original: Figure 2 illustrates the translation between the two models.
Revised: Figure 2 illustrates the translation between the two models.

Original: 3. M OMENTUM LEARNING
Revised: 3. M OMENTUM LEARNING

Original: The standard rationale for momentum learning is to smooth updates over time, so that oscillations along directions of high curvature cancel out while progress can be made in directions with consistent gradients (Rumelhart et al., 1986).
Revised: The standard rationale for momentum learning is to smooth updates over time, so that oscillations along directions of high curvature cancel out while progress can be made in directions with consistent gradients (Rumelhart et al., 1986).

Original: To simplify notation, we again focus on a single weight w in the network, suppressing the index j .
Revised: To simplify notation, we again focus on a single weight w in the network, suppressing the index j .

Original: The momentum g is deﬁned as an exponentially ﬁltered running average of gradients, with weight update determined by current momentum:
Revised: The momentum g is deﬁned as an exponentially ﬁltered running average of gradients, with weight update determined by current momentum:

Original: [Equation]
Revised: [Equation]

Original: This formulation is equivalent to one in which the update  w ( t ) =
Revised: This formulation is equivalent to one in which the update ∆ w ( t ) =

Original: w ( t + 1)  w ( t ) includes a portion of the previous update:  w ( t ) =
Revised: w ( t + 1) − w ( t ) includes a portion of the previous update: ∆ [Equation]

Original:  ↵@
Revised: [Equation]

Original: w ( t ) L +   w ( t  1) , with ↵ = ⌘ (1   ) .
Revised: t ) [Equation] w ( t − 1) , with α = η (1 − β ) .

Original: Paralleling the analysis in Section 3.1, we write the state of the momentum optimizer as [ w, g ] > and use Equations 8 and 9 to obtain the update rule:
Revised: Paralleling the analysis in Section 3.1, we write the state of the momentum optimizer as [ w, g ] (cid:62) and use Equations 8 and 9 to obtain the update rule:

Original: [Equation]
Revised: [Equation]

Original: The transition matrix has eigenvectors [1 , 0] > with eigenvalue 1 , and [1 , 1   ⌘ ] > with eigenvalue  .
Revised: The transition matrix has eigenvectors [1 , 0] (cid:62) with eigenvalue 1 , and [1 , 1 − β ηβ ] (cid:62) with eigenvalue β .

Original: Now use this eigenbasis to deﬁne a reparameterization:
Revised: Now use this eigenbasis to deﬁne a reparameterization:

Original: [Equation]
Revised: [Equation]

Original: Substitution into Equation 10 yields the reparameterized update rule:
Revised: Substitution into Equation 10 yields the reparameterized update rule:

Original: [Equation]
Revised: [Equation]

Original: Thus we recover the fast-weight optimizer, with decay  fast =
Revised: Thus we recover the fast-weight optimizer, with decay γ fast =

Original:  and learning rates ↵ slow = ⌘ and ↵ fast =  ⌘ .
Revised: β and learning rates α slow = η and α fast = − ηβ .

Original: The negative fast learning rate is perhaps surprising but can be understood as follows:
Revised: The negative fast learning rate is perhaps surprising but can be understood as follows:

Original: When " fast < 0 , the subweights learn in opposite directions, with the latent knowledge in ! slow overshooting the observable knowledge inw = !
Revised: When ε fast < 0 , the subweights learn in opposite directions, with the latent knowledge in ω slow overshooting the observable knowledge in

Original:  slow + ! fast .
Revised: w = ω slow + ω fast .

Original: As ! fast decays toward 0 , w catches up to ! slow , so that the model appears to continue learning from past input, just as it would with momentum.
Revised: As ω fast decays toward 0 , w catches up to ω slow , so that the model appears to continue learning from past input, just as it would with momentum.

Original: This analysis highlights the contrasting rationales of these two methods: Learning at multiple timescales is motivated by an expectation of positive autocorrelation in the environment, whereas momentum is effective at smoothing out negative autocorrelation in the gradient signal.
Revised: This analysis highlights the contrasting rationales of these two methods: Learning at multiple timescales is motivated by an expectation of positive autocorrelation in the environment, whereas momentum is effective at smoothing out negative autocorrelation in the gradient signal.

Original: A
Revised: A

Original: B
Revised: B

Original: C
Revised: C

Original: F
Revised: F

Original: E
Revised: E

Original: D       80.
Revised: D       80.

Original: weighted sum of modes.
Revised: weighted sum of modes.

Original: Learning rates for the corresponding subweights,  ! i , can be read off as the values of the curves at k = 1 (because V 1 i = 1 ).
Revised: Learning rates for the corresponding subweights, ∆ ω i , can be read off as the values of the curves at k = 1 (because V 1 i = 1 ).

Original:  B AYESIAN MULTISCALE OPTIMIZER
Revised:  B AYESIAN MULTISCALE OPTIMIZER

Original: We turn now to a normative analysis of learning at multiple timescales, based on Bayesian inference over 1 /f noise.
Revised: We turn now to a normative analysis of learning at multiple timescales, based on Bayesian inference over 1 /f noise.

Original: The Bayesian model introduced here assumes that the latent parameters ✓ governing the observed data in some learning task vary over time according to 1 /f noise.
Revised: The Bayesian model introduced here assumes that the latent parameters θ governing the observed data in some learning task vary over time according to 1 /f noise.

Original: When the statistical model h ( x , ✓ ) is linear in ✓ , exact Bayesian inference is possible with a KF that maintains a posterior over an expanded representation of ✓ .
Revised: When the statistical model h ( x , θ ) is linear in θ , exact Bayesian inference is possible with a KF that maintains a posterior over an expanded representation of θ .

Original: When the model is nonlinear, approximate Bayesian inference is achieved by an EKF that uses a linear approximation of h .
Revised: When the model is nonlinear, approximate Bayesian inference is achieved by an EKF that uses a linear approximation of h .

Original: We then show that a variational approximation of the KF or EKF, in which the posterior covariance matrix is constrained to be diagonal, yields an extension of the multiscale optimizer that adapts its learning rates online by tracking uncertainty.
Revised: We then show that a variational approximation of the KF or EKF, in which the posterior covariance matrix is constrained to be diagonal, yields an extension of the multiscale optimizer that adapts its learning rates online by tracking uncertainty.

Original: 4. G ENERATIVE MODEL FOR 1 /f NOISE
Revised: 4. G ENERATIVE MODEL FOR 1 /f NOISE

Original: Let z i ( t ) be an Ornstein-Uhlenbeck process (i.e., diffusion with decay), with timescale or inverse decay rate ⌧ i and diffusion rate  2 i , deﬁned by the following stochastic differential equation:
Revised: Let z i ( t ) be an Ornstein-Uhlenbeck process (i.e., diffusion with decay), with timescale or inverse decay rate τ i and diffusion rate σ 2 i , deﬁned by the following stochastic differential equation:

Original: [Equation]
Revised: [Equation]

Original: Here W ( t ) is a standard Wiener process (Brownian motion).
Revised: Here W ( t ) is a standard Wiener process (Brownian motion).

Original: As a Gaussian process, z i has kernel
Revised: As a Gaussian process, z i has kernel

Original: E [ z i ( t ) z i ( t + s )]/
Revised: E [ z i ( t ) z i ( t + s )]

Original:  e  | s | / ⌧ i , implying exponentially decaying autocorrelations.
Revised: ∝ e −| s | /τ i , implying exponentially decaying autocorrelations.

Original: However, a superposition of such processes at different timescales can have qualitatively different properties (Eliazar
Revised: However, a superposition of such processes at different timescales can have qualitatively different properties (Eliazar

Original: Klafter, 2009).
Revised: Klafter, 2009).

Original: In particular, consider
Revised: In particular, consider

Original: [Equation]
Revised: [Equation]

Original: 14) where ⌧ i = ⌫
Revised: 14) where τ i = ν

Original: i and  i = ⌫  i/ 2 for a chosen ⌫ > 1 , and n is an integer such that ⌧ n is very large.
Revised: i and σ i = ν − i/ 2 for a chosen ν > 1 , and n is an integer such that τ n is very large.

Original: We show in Appendix B that ⇠ has power-law (i.e., long range) autocorrelations, E 
Revised: We show in Appendix B that ξ has power-law (i.e., long range) autocorrelations, E [ ξ ( t ) ξ ( t + s )]

Original: [ ⇠ ( t ) ⇠ ( t + s )] / | s | for s ⌧ ⌧ n , and accordingly a power spectrum that is well-approximated by 1 /f for frequencies f  ⌧  1 n .
Revised: ∝ | s | −for s (cid:28) τ n , and accordingly a power spectrum that is well-approximated by 1 /f for frequencies f (cid:29) τ − 1 n .

Original: Moreover, m independent copies of this process constitute m -dimensional 1 /f noise, due to the rotational invariance of multidimensional Ornstein-Uhlenbeck processes.
Revised: Moreover, m independent copies of this process constitute m -dimensional 1 /f noise, due to the rotational invariance of multidimensional Ornstein-Uhlenbeck processes.

Original: This construction affords a ﬂexible generative model of nonstationarity in a variety of online learning domains, by applying it to the latent parameters governing the relationships among observable variables.
Revised: This construction affords a ﬂexible generative model of nonstationarity in a variety of online learning domains, by applying it to the latent parameters governing the relationships among observable variables.

Original: Assume we receive observations x ( t ) , y ( t ) that we wish to model with a statistical model h that is parameterized by ✓ 2 R m :
Revised: Assume we receive observations x ( t ) , y ( t ) that we wish to model with a statistical model h that is parameterized by θ ∈ R m :

Original: [Equation]
Revised: [Equation]

Original: For example, h may be a NN with weights ✓ , input x , and target output y .
Revised: For example, h may be a NN with weights θ , input x , and target output y .

Original: The generative side of our
Revised: The generative side of our

Original: Bayesian model posits latent variables
Revised: Bayesian model posits latent variables

Original: z i ( i = 1 , . .
Revised: z i ( i = 1 , . .

Original: . , n )
Revised: . , n )

Original: such that each z i is an Ornstein-Uhlenbeck process in R m with timescale ⌧ i , and these processes sum to determine the original parameters:
Revised: such that each z i is an Ornstein-Uhlenbeck process in R m with timescale τ i , and these processes sum to determine the original parameters:

Original: [Equation]
Revised: [Equation]

Original: These assumptions imply that ✓ follows a 1 /f process, and they entail an expanded state representation, Z = ( z 1 , . . .
Revised: These assumptions imply that θ follows a 1 /f process, and they entail an expanded state representation, Z = ( z 1 , . . .

Original: , z n ) 2 R nm , that enables efﬁcient inference as described in Section 4.2.
Revised: , z n ) ∈ R nm , that enables efﬁcient inference as described in Section 4.2.

Original: 4. I NFERENCE OVER 1 /f NOISE VIA EXTENDED K ALMAN FILTER
Revised: 4. I NFERENCE OVER 1 /f NOISE VIA EXTENDED K ALMAN FILTER

Original: We consider Bayesian methods that adopt the construction in Section 4.1 as a generative model to account for nonstationarity.
Revised: We consider Bayesian methods that adopt the construction in Section 4.1 as a generative model to account for nonstationarity.

Original: Equations 13 and 14 describe a linear dynamic system with state
Revised: Equations 13 and 14 describe a linear dynamic system with state

Original: Z = ( z 1 , . . .
Revised: Z = ( z 1 , . . .

Original: , z n ) 2 R n .
Revised: , z n ) ∈ R n .

Original: If ⇠ is directly observed at discrete intervals, then optimal Bayesian online prediction of each ⇠ ( t ) based on all preceding observations can be implemented by a KF over Z (Kording et al., 2007)
Revised: If ξ is directly observed at discrete intervals, then optimal Bayesian online prediction of each ξ ( t ) based on all preceding observations can be implemented by a KF over Z (Kording et al., 2007)

Original: (see Appendix D).
Revised: (see Appendix D).

Original: We extend this approach to arbitrary statistical models with nonstationarity in their latent parameters, as in Equations 15 and 16.
Revised: We extend this approach to arbitrary statistical models with nonstationarity in their latent parameters, as in Equations 15 and 16.

Original: When h is linear in ✓ (and hence in Z ), such as in the regression task and 1-layer perceptron model in Section 5.1, exact inference is possible with a standard KF (Appendix D).
Revised: When h is linear in θ (and hence in Z ), such as in the regression task and 1-layer perceptron model in Section 5.1, exact inference is possible with a standard KF (Appendix D).

Original: For a general h , such as a multilayer NN, we use an EKF.
Revised: For a general h , such as a multilayer NN, we use an EKF.

Original: The EKF makes a local linear approximation of h based on its Jacobian, the matrix of gradients of predictions ˆ y with respect to ✓ (Appendix E).
Revised: The EKF makes a local linear approximation of h based on its Jacobian, the matrix of gradients of predictions ˆ y with respect to θ (Appendix E).

Original: We use Ollivier’s (2018) generalization of the EKF that replaces Gaussian observation noise with any exponential family p ( y | ˆ y ) , which is better suited for modeling discrete outcomes such as the classiﬁcation tasks of Sections 5.2 and 5.3.
Revised: We use Ollivier’s (2018) generalization of the EKF that replaces Gaussian observation noise with any exponential family p ( y | ˆ y ) , which is better suited for modeling discrete outcomes such as the classiﬁcation tasks of Sections 5.2 and 5.3.

Original: 4. V ARIATIONAL APPROXIMATION
Revised: 4. V ARIATIONAL APPROXIMATION

Original: Finally, we derive a variational approximation of the EKF that extends the multiscale optimizer and affords efﬁcient implementation in large NNs (Appendix F).
Revised: Finally, we derive a variational approximation of the EKF that extends the multiscale optimizer and affords efﬁcient implementation in large NNs (Appendix F).

Original: As is standard, the EKF maintains an iterative prior over the latent state based on all previous observations:
Revised: As is standard, the EKF maintains an iterative prior over the latent state based on all previous observations:

Original: [Equation]
Revised: [Equation]

Original: The mean, ! ( t ) , is the vector of current subweight estimates in the network, while S ( t ) captures their joint uncertainty and hence determines updates (as a preconditioner on the gradient).
Revised: The mean, ω ( t ) , is the vector of current subweight estimates in the network, while S ( t ) captures their joint uncertainty and hence determines updates (as a preconditioner on the gradient).

Original: We use variational inference to approximate the distribution in Equation 17 by one in which S ( t ) ⇡ ˜
Revised: We use variational inference to approximate the distribution in Equation 17 by one in which S ( t ) ≈ ˜

Original: S ( t ) , where ˜ S ( t ) is constrained to be diagonal, written as ˜ S ( t ) =
Revised: S ( t ) , where ˜ S ( t ) is constrained to be diagonal, written as ˜ S ( t ) =

Original: diag( s 2 ( t )) .
Revised: diag( s 2 ( t )) .

Original: This reduces the complexity from O ( m 2 n 2 ) to O ( mnk ) (the size of the Jacobian, where k is the size of the output layer).
Revised: This reduces the complexity from O ( m 2 n 2 ) to O ( mnk ) (the size of the Jacobian, where k is the size of the output layer).

Original: The simplest case is a KF that tracks a single 1 /f variable, with no inputs or latent variables.
Revised: The simplest case is a KF that tracks a single 1 /f variable, with no inputs or latent variables.

Original: That is, y ( t ) =
Revised: That is, y ( t ) =

Original: ⇠ ( t ) 2 R 1 as in Equation 14.
Revised: ξ ( t ) ∈ R 1 as in Equation 14.

Original: Appendix F.1 derives the variational update rule as
Revised: Appendix F.1 derives the variational update rule as

Original: [Equation]
Revised: [Equation]

Original: where y i ( t )  ˆ y i ( t ) =  @ w L (i.e., square loss), and the learning rates are given by
Revised: where y i ( t ) − ˆ y i ( t ) = − ∂ w L (i.e., square loss), and the learning rates are given by

Original: [Equation]
Revised: [Equation]

Original: For the EKF with a general nonlinear model h ( x , ✓ ) , a slight extension of the variational approximation, derived in Appendix F.3, provides the following update:
Revised: For the EKF with a general nonlinear model h ( x , θ ) , a slight extension of the variational approximation, derived in Appendix F.3, provides the following update:

Original: [Equation]
Revised: [Equation]

Original: Here diag( s 0 2 ) is the diagonal variational approximation of the posterior variance after observing y ( t ) , and ˜ L is the negative loglikelihood in the EKF’s Gaussian approximate output distribution.
Revised: Here diag( s (cid:48) 2 ) is the diagonal variational approximation of the posterior variance after observing y ( t ) , and ˜ L is the negative loglikelihood in the EKF’s Gaussian approximate output distribution.

Original: Importantly, the update rule for the variance uses the diagonal of the precision matrix but can be calculated without matrix inversion, which is relevant to scaling up to large networks.
Revised: Importantly, the update rule for the variance uses the diagonal of the precision matrix but can be calculated without matrix inversion, which is relevant to scaling up to large networks.

Original: Thus the variational method amounts to decomposing every weight in the network as a sum of subweights, w j = P i ! ij , that learn independently according to their individual gradients, with decay rates ⌧ i and learning rates coming from ˜ S ( t ) .
Revised: Thus the variational method amounts to decomposing every weight in the network as a sum of subweights, w j = (cid:80) i ω ij , that learn independently according to their individual gradients, with decay rates τ i and learning rates coming from ˜ S ( t ) .

Original: This is a special case within the family of multiscale optimizers, with additional machinery to adapt the learning rates based on current uncertainty.
Revised: This is a special case within the family of multiscale optimizers, with additional machinery to adapt the learning rates based on current uncertainty.

Original:  S IMULATION TESTS 5. R EGRESSION TASK
Revised:  S IMULATION TESTS 5. R EGRESSION TASK

Original: As a simple demonstration, we created an online linear regression task with 10 features (including a bias term), in which the true weights  varied over time according to 1 /f noise using Equation 16.
Revised: As a simple demonstration, we created an online linear regression task with 10 features (including a bias term), in which the true weights β varied over time according to 1 /f noise using Equation 16.

Original: The outcome was generated as y = x >  (no noise term was needed because  is inherently noisy).
Revised: The outcome was generated as y = x (cid:62) β (no noise term was needed because β is inherently noisy).

Original: The corresponding predictive model is a one-layer perceptron, which we write as ˆ y = x > w to distinguish weight estimates ( w ) from true parameters (  ).
Revised: The corresponding predictive model is a one-layer perceptron, which we write as ˆ y = x (cid:62) w to distinguish weight estimates ( w ) from true parameters ( β ).

Original: We model the data using the perceptron and compare methods for optimization, using square loss, L = 12 ( y  ˆ y ) 2 .
Revised: We model the data using the perceptron and compare methods for optimization, using square loss, L = 12 ( y − ˆ y ) 2 .

Original: We tested two baseline training methods, representing common heuristi practices with nonstationary data (Ditzler et al., 2015; Parisi et al., 2019).
Revised: We tested two baseline training methods, representing common heuristi practices with nonstationary data (Ditzler et al., 2015; Parisi et al., 2019).

Original: First, we tested a batch learning method that uses a ﬁxed memory horizon H .
Revised: First, we tested a batch learning method that uses a ﬁxed memory horizon H .

Original: To produce a prediction on step t , the batch learner ﬁts the perceptron to trials t  H through t  1 .
Revised: To produce a prediction on step t , the batch learner ﬁts the perceptron to trials t − H through t − 1 .

Original: Figure 3A shows performance is U-shaped: accuracy suffers with short horizons because of sampling error, but it also suffers from longer horizons because older observations are less valid.
Revised: Figure 3A shows performance is U-shaped: accuracy suffers with short horizons because of sampling error, but it also suffers from longer horizons because older observations are less valid.

Original: Second, we tested SGD, in which the weights are updated once after each step t , based on x ( t ) and y ( t ) .
Revised: Second, we tested SGD, in which the weights are updated once after each step t , based on x ( t ) and y ( t ) .

Original: Figure 3B shows performance is best with an intermediate learning rate, which roughly corresponds to assuming the environment changes on a single characteristic timescale (see
Revised: Figure 3B shows performance is best with an intermediate learning rate, which roughly corresponds to assuming the environment changes on a single characteristic timescale (see

Original: Appendix C).
Revised: Appendix C).

Original: As applied to the perceptron, the Bayesian 1 /f model described in Section 4.2 decomposes the weight for each feature j into subweights, w j = P i ! ij , and tracks the ! ij jointly with a KF (generalizing Dayan & Kakade, 2000).
Revised: As applied to the perceptron, the Bayesian 1 /f model described in Section 4.2 decomposes the weight for each feature j into subweights, w j = (cid:80) i ω ij , and tracks the ω ij jointly with a KF (generalizing Dayan & Kakade, 2000).

Original: The subweights combine to predict the outcome on each step: ˆ y ( t ) =
Revised: The subweights combine to predict the outcome on each step: ˆ y ( t ) =

Original: [Equation] t [Equation] ( t ) .
Revised: (cid:80) ij x i ( t ) ω ij ( t ) .

Original: Relative to blind prediction (guessing the overall mean of y on
Revised: Relative to blind prediction (guessing the overall mean of y on

Original: [Figure]
Revised: [Figure]

Original: every trial), this exact Bayesian solution explains 54.0% more variance in the outcome than the best batch learner, and 25.7% more than the best parameterization of SGD (Figure 3C).
Revised: every trial), this exact Bayesian solution explains 54.0% more variance in the outcome than the best batch learner, and 25.7% more than the best parameterization of SGD (Figure 3C).

Original: Moreover, the variational model that constrains the KF covariance matrix to be diagonal (see Appendix F.2) performs nearly (96.9%) as well as the full Bayesian model.
Revised: Moreover, the variational model that constrains the KF covariance matrix to be diagonal (see Appendix F.2) performs nearly (96.9%) as well as the full Bayesian model.

Original: Finally, we tested a discounting method, similar to windowed batch optimization except all past trials were used for training, discounted by a function of lag.
Revised: Finally, we tested a discounting method, similar to windowed batch optimization except all past trials were used for training, discounted by a function of lag.

Original: Because the 1 /f environment has powerlaw correlations, we weighted each observation t  k by k  a and optimized a .
Revised: Because the 1 /f environment has powerlaw correlations, we weighted each observation t − k by k − a and optimized a .

Original: This method also signiﬁcantly outperforms windowed batch and SGD, showing that accounting for an environment’s autocorrelation function can achieve much of the advantage of the Bayesian approach.
Revised: This method also signiﬁcantly outperforms windowed batch and SGD, showing that accounting for an environment’s autocorrelation function can achieve much of the advantage of the Bayesian approach.

Original: Nevertheless, the full KF and variational methods still outperform discounting, by 5.5% and 2.2% respectively.
Revised: Nevertheless, the full KF and variational methods still outperform discounting, by 5.5% and 2.2% respectively.

Original: 5. L INEAR CLASSIFICATION TASK
Revised: 5. L INEAR CLASSIFICATION TASK

Original: Next, we investigated an online 10-way classiﬁcation task with 10 features (including a bias term).
Revised: Next, we investigated an online 10-way classiﬁcation task with 10 features (including a bias term).

Original: The data were generated by ﬁrst sampling the class, y ( t ) ⇠ softmax( e ( t )) , and then sampling the feature vector, x ( t ) | y ( t ) ⇠ N ( µ y ( t ) , I ) .
Revised: The data were generated by ﬁrst sampling the class, y ( t ) ∼ softmax( e ( t )) , and then sampling the feature vector, x ( t ) | y ( t ) ∼ N ( µ y ( t ) , I ) .

Original: The logits e j and the feature-class means µ ij were independently sampled from 1 /f processes (using Equation 16), so that there was nonstationarity in both p ( y ) and p ( x | y ) .
Revised: The logits e j and the feature-class means µ ij were independently sampled from 1 /f processes (using Equation 16), so that there was nonstationarity in both p ( y ) and p ( x | y ) .

Original: Scaling of e and µ was chosen to equate to maximum possible performance based on perfect knowledge of either one alone (both yielding average loss ¯ L ⇡ 1 ).
Revised: Scaling of e and µ was chosen to equate to maximum possible performance based on perfect knowledge of either one alone (both yielding average loss ¯ L ≈ 1 ).

Original: For the predictive model, we used a one-layer perceptron with a softmax output layer, ˆ y = softmax( x > W ) , where W is a matrix of learnable feature-class weights.
Revised: For the predictive model, we used a one-layer perceptron with a softmax output layer, ˆ y = softmax( x (cid:62) W ) , where W is a matrix of learnable feature-class weights.

Original: We assumed cross-entropy loss, L ( y, ˆ y )=
Revised: We assumed cross-entropy loss, L ( y, ˆ y )

Original:  log ˆ y y .
Revised: = − log ˆ y y .

Original: The batch method trained the network on trials t  H through t  1 until convergence before predicting y ( t ) .
Revised: The batch method trained the network on trials t − H through t − 1 until convergence before predicting y ( t ) .

Original: Weight decay was included for regularization, optimized for each value of H .
Revised: Weight decay was included for regularization, optimized for each value of H .

Original: Figure 4A shows a U-shaped pattern of performance, reﬂecting the tradeoff between sampling error and stale data.
Revised: Figure 4A shows a U-shaped pattern of performance, reﬂecting the tradeoff between sampling error and stale data.

Original: We also used weight decay with SGD, optimized for each learning rate.
Revised: We also used weight decay with SGD, optimized for each learning rate.

Original: Figure 4B again shows a U-shaped pattern of performance.
Revised: Figure 4B again shows a U-shaped pattern of performance.

Original: The variational EKF for this task and model is derived in
Revised: The variational EKF for this task and model is derived in

Original: Appendix F.3.
Revised: Appendix F.3.

Original: We applied ` 2 regularization to the prior on each time step, on par with the SGD and batch optimization methods.
Revised: We applied (cid:96) 2 regularization to the prior on each time step, on par with the SGD and batch optimization methods.

Original: Figure 4C shows the variational method outperforms the other two.
Revised: Figure 4C shows the variational method outperforms the other two.

Original: 5. MNIST CLASSIFICATION
Revised: 5. MNIST CLASSIFICATION

Original: Finally, we tested our methods on classifying a stream of handwritten MNIST digits (LeCun et al., 2010).
Revised: Finally, we tested our methods on classifying a stream of handwritten MNIST digits (LeCun et al., 2010).

Original: We created a nonstationary online learning task by sampling an example from the 10-way
Revised: We created a nonstationary online learning task by sampling an example from the 10-way

Original: MNIST training set on each time step according to class logits that followed 1 /f noise (Figure 5A).
Revised: MNIST training set on each time step according to class logits that followed 1 /f noise (Figure 5A).

Original: For a predictive model, we used a convolutional neural network (CNN) with two convolution layers followed by two dense layers, with 824458 parameters.
Revised: For a predictive model, we used a convolutional neural network (CNN) with two convolution layers followed by two dense layers, with 824458 parameters.

Original: This experiment provides a more stringent test of the present methods in several ways.
Revised: This experiment provides a more stringent test of the present methods in several ways.

Original: First, it tests whether the EKF’s linear approximation and the variational diagonal-variance approximation perform well on a multilayer NN, and whether the algorithm is efﬁcient with a moderately large number of parameters.
Revised: First, it tests whether the EKF’s linear approximation and the variational diagonal-variance approximation perform well on a multilayer NN, and whether the algorithm is efﬁcient with a moderately large number of parameters.

Original: Second, the predictive model the optimizer is doing inference over is unrelated to how the images and labels were generated.
Revised: Second, the predictive model the optimizer is doing inference over is unrelated to how the images and labels were generated.

Original: Third, the sampling procedure for 1 /f noise did not accord with the additive generative process
Revised: Third, the sampling procedure for 1 /f noise did not accord with the additive generative process

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: assumed by the Bayesian model (Section 4.1), but instead used a spectral procedure described in
Revised: assumed by the Bayesian model (Section 4.1), but instead used a spectral procedure described in

Original: Appendix G.
Revised: Appendix G.

Original: The variational EKF was compared to SGD with momentum, in both 1 /f and standard iid environments.
Revised: The variational EKF was compared to SGD with momentum, in both 1 /f and standard iid environments.

Original: Hyperparameters for both methods (noise variance for EKF, learning and momentum rates for SGD) were optimized separately for the two environments.
Revised: Hyperparameters for both methods (noise variance for EKF, learning and momentum rates for SGD) were optimized separately for the two environments.

Original: Because the models learn quickly, we evaluated them (within each replication) on a sequence comprising only a random subset of the
Revised: Because the models learn quickly, we evaluated them (within each replication) on a sequence comprising only a random subset of the

Original: MNIST training set.
Revised: MNIST training set.

Original: Performance was measured by top-1 error rate.
Revised: Performance was measured by top-1 error rate.

Original: This should be interpreted as generalization (i.e., test) performance, because each item was observed only once.
Revised: This should be interpreted as generalization (i.e., test) performance, because each item was observed only once.

Original: The variational
Revised: The variational

Original: EKF outperforms SGD by 1.4% in the iid environment, because its tracking of uncertainty enables more effective gradient steps.
Revised: EKF outperforms SGD by 1.4% in the iid environment, because its tracking of uncertainty enables more effective gradient steps.

Original: However, its advantage jumps to 2.4% in the 1 /f environment, showing once again that it is better able to leverage dynamics at multiple scales.
Revised: However, its advantage jumps to 2.4% in the 1 /f environment, showing once again that it is better able to leverage dynamics at multiple scales.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: Our analytic and simulation results demonstrate how online learning performance in nonstationary environments can be improved by incorporating a model of temporal structure.
Revised: Our analytic and simulation results demonstrate how online learning performance in nonstationary environments can be improved by incorporating a model of temporal structure.

Original: The Bayesian 1 /f model amounts to distributing knowledge across multiple timescales, and the variational EKF enables approximate implementation in a neural network using subweights with different learning and decay rates.
Revised: The Bayesian 1 /f model amounts to distributing knowledge across multiple timescales, and the variational EKF enables approximate implementation in a neural network using subweights with different learning and decay rates.

Original: The variational EKF extends the multiscale optimizer, which is closely related to previous models in both neuroscience and ML, and in some cases is equivalent to them despite being simpler in having no coupling between timescales.
Revised: The variational EKF extends the multiscale optimizer, which is closely related to previous models in both neuroscience and ML, and in some cases is equivalent to them despite being simpler in having no coupling between timescales.

Original: We have implemented the variational EKF optimizer in JAX in a format compatible with Optax ( link to be added ).
Revised: We have implemented the variational EKF optimizer in JAX in a format compatible with Optax.

Original: In the MNIST simulations of Section 5.3, we ﬁnd our optimizer code (with 8 timescales) is actually 1.6% faster than Optax’s off-the-shelf SGD, in compute time per example.
Revised: In the MNIST simulations of Section 5.3, we ﬁnd our optimizer code (with 8 timescales) is actually 1.6% faster than Optax’s off-the-shelf SGD, in compute time per example.

Original: Note also that the multiplexing of subweights is not expensive relative to current optimizers (e.g., Adam; Kingma & Ba, 2015), which also store multiple variables for each weight.
Revised: Note also that the multiplexing of subweights is not expensive relative to current optimizers (e.g., Adam; Kingma & Ba, 2015), which also store multiple variables for each weight.

Original: In sum, the multiscale optimizer and variational EKF enjoy a combination of normative, heuristic, and biological justiﬁcation, good performance, and computational efﬁciency.
Revised: In sum, the multiscale optimizer and variational EKF enjoy a combination of normative, heuristic, and biological justiﬁcation, good performance, and computational efﬁciency.

Original: Our ongoing work aims to extend the theory in several ways.
Revised: Our ongoing work aims to extend the theory in several ways.

Original: Anonymous (2022) compares the present variational method to the fully-decoupled EKF of Puskorius & Feldkamp (2003).
Revised: Anonymous (2022) compares the present variational method to the fully-decoupled EKF of Puskorius & Feldkamp (2003).

Original: Another possible variational method is to assume a block-diagonal matrix that maintains covariance information only between subweights (timescales) within each weight, so that computational complexity still scales linearly with network size.
Revised: Another possible variational method is to assume a block-diagonal matrix that maintains covariance information only between subweights (timescales) within each weight, so that computational complexity still scales linearly with network size.

Original: Finally, the present method is not limited to 1 /f noise but generalizes to other power laws (e.g., 1 /f  ) by appropriate choice of the timescales ⌧ i and noise variances  i in the generative model (see Appendix B).
Revised: Finally, the present method is not limited to 1 /f noise but generalizes to other power laws (e.g., 1 /f β ) by appropriate choice of the timescales τ i and noise variances σ i in the generative model (see Appendix B).

Original: If, for example, data or theory is available bearing on the power spectrum of the dynamics in a given domain, the optimizer could be tuned accordingly.
Revised: If, for example, data or theory is available bearing on the power spectrum of the dynamics in a given domain, the optimizer could be tuned accordingly.

Original: A D IAGONALIZATION OF B ENNA -F USI TRANSITION MATRIX
Revised: A D IAGONALIZATION OF B ENNA -F USI TRANSITION MATRIX

Original: This section provides details on diagonalizing the transition matrix of the update rule for the Benna & Fusi (2016) model, and the corresponding reparameterization in terms of the eigenbasis.
Revised: This section provides details on diagonalizing the transition matrix of the update rule for the Benna & Fusi (2016) model, and the corresponding reparameterization in terms of the eigenbasis.

Original: First, reparameterize the Benna-Fusi model so that its transition matrix is symmetric, as follows.
Revised: First, reparameterize the Benna-Fusi model so that its transition matrix is symmetric, as follows.

Original: Recursively deﬁne
Revised: Recursively deﬁne

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and write the state of the Benna-Fusi synapse as
Revised: and write the state of the Benna-Fusi synapse as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The update becomes
Revised: The update becomes

Original: [Equation]
Revised: [Equation]

Original: 23) with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: Symmetry of  implies it has an orthonormal eigenbasis, {  1 , . . . ,  n } , with corresponding eigenvalues  1 , . . .
Revised: Symmetry of Γ implies it has an orthonormal eigenbasis, { ψ 1 , . . . , ψ n } , with corresponding eigenvalues λ 1 , . . .

Original: ,  n .
Revised: , λ n .

Original: Because the scaling of eigenvectors is arbitrary, we can enforce  i, 1 = 1 for all i .
Revised: Because the scaling of eigenvectors is arbitrary, we can enforce ψ i, 1 = 1 for all i .

Original: To translate the eigenbasis back to u , deﬁne B = diag( p b ) so that  = Bu and T =
Revised: To translate the eigenbasis back to u , deﬁne B = diag( √ b ) so that φ = Bu and T =

Original: B  1  B .
Revised: B − 1 Γ B .

Original: It is then easily veriﬁed that
Revised: It is then easily veriﬁed that

Original: [Equation]
Revised: [Equation]

Original: (26) is an eigenvector of T with eigenvalue  i .
Revised: (26) is an eigenvector of T with eigenvalue λ i .

Original: Therefore T = V ⇤ V  1 , as claimed in the main text.
Revised: Therefore T = V Λ V − 1 , as claimed in the main text.

Original: Note that the choice  i, 1 = 1 entails V 1 ,i = 1 (because b 1 = 1 ).
Revised: Note that the choice ψ i, 1 = 1 entails V 1 ,i = 1 (because b 1 = 1 ).

Original: Finally,  = [  1 , . . . ,  n ] is invertible because it comprises a basis, and therefore so is V = B  1  .
Revised: Finally, Ψ = [ ψ 1 , . . . , ψ n ] is invertible because it comprises a basis, and therefore so is V = B − 1 Ψ .

Original: Therefore the reparameterization u 7! !
Revised: Therefore the reparameterization u (cid:55)→ ω

Original: = V  1 u is well-deﬁned.
Revised: = V − 1 u is well-deﬁned.

Original: B G ENERATIVE MODEL FOR 1 /f NOISE
Revised: B G ENERATIVE MODEL FOR 1 /f NOISE

Original: Consider a single Ornstein-Uhlenbeck (OU) process z ( t ) described by Equation 13 with  = 1 .
Revised: Consider a single Ornstein-Uhlenbeck (OU) process z ( t ) described by Equation 13 with σ = 1 .

Original: The covariance function of z is given by
Revised: The covariance function of z is given by

Original: [Equation]
Revised: [Equation]

Original: Note that this expression decays exponentially with the lag s , yielding short-range autocorrelations.
Revised: Note that this expression decays exponentially with the lag s , yielding short-range autocorrelations.

Original: The power spectrum of z , as a function of frequency f , is the Fourier transform of the covariance function:
Revised: The power spectrum of z , as a function of frequency f , is the Fourier transform of the covariance function:

Original: [Equation]
Revised: [Equation]

Original: where 2 ⇡ f is angular frequency.
Revised: where 2 πf is angular frequency.

Original: To deﬁne a generative model of 1 /f noise, we deﬁne a continuous mixture model over a family of
Revised: To deﬁne a generative model of 1 /f noise, we deﬁne a continuous mixture model over a family of

Original: OU processes ( z ⌧ ) 0 < ⌧  T for some large T :
Revised: OU processes ( z τ ) 0 <τ ≤ T for some large T :

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: The power spectrum of ⇠ is then a mixture over the component spectra P z ⌧ :
Revised: The power spectrum of ξ is then a mixture over the component spectra P z τ :

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which is approximately 1 /f for f  1 /T , i.e. for all but very low frequencies.
Revised: which is approximately 1 /f for f (cid:29) 1 /T , i.e. for all but very low frequencies.

Original: We next deﬁne a discrete approximation of the continuum mixture model in Equation 31,
Revised: We next deﬁne a discrete approximation of the continuum mixture model in Equation 31,

Original: [Equation]
Revised: [Equation]

Original: where z i has timescale ⌧ i and scaling parameter  i .
Revised: where z i has timescale τ i and scaling parameter σ i .

Original: To approximate a 1 /f spectrum,  2 i ( ⌧ i +1  ⌧ i ) should scale as 4 ⌧  2 i , the squared weight density in Equation 31 (because power is additive and proportional to  2 ).
Revised: To approximate a 1 /f spectrum, σ 2 i ( τ i +1 − τ i ) −should scale as 4 τ − 2 i , the squared weight density in Equation 31 (because power is additive and proportional to σ 2 ).

Original: For example, the ⌧ i could be arithmetically spaced with  i / ⌧  1 .
Revised: For example, the τ i could be arithmetically spaced with σ i ∝ τ − 1 .

Original: Instead, we assume geometric spacing, with n components deﬁned by ⌧ i = ⌫
Revised: Instead, we assume geometric spacing, with n components deﬁned by τ i = ν

Original: i and  i = 2 ⇢⌧  1 / 2 i , where ⇢ 2 is a hyperparameter determining overall steady-state variance.
Revised: i and σ i = 2 ρτ − 1 / 2 i , where ρ 2 is a hyperparameter determining overall steady-state variance.

Original: Figure 6 illustrates this construction and exempliﬁes the accuracy of the discrete approximation.
Revised: Figure 6 illustrates this construction and exempliﬁes the accuracy of the discrete approximation.

Original: To deﬁne a generative model of 1 /f noise in R m , we assume an independent copy of this process for each dimension.
Revised: To deﬁne a generative model of 1 /f noise in R m , we assume an independent copy of this process for each dimension.

Original: The resulting joint process is rotationally symmetric, a property inherited from
Revised: The resulting joint process is rotationally symmetric, a property inherited from

Original: OU processes.
Revised: OU processes.

Original: That is, any linear combination [Equation]
Revised: That is, any linear combination (cid:80)

Original:  process.
Revised: j c j ξ j is a 1d 1 /f process.

Original: C K ALMAN FILTER FOR A SINGLE TIMESCALE
Revised: C K ALMAN FILTER FOR A SINGLE TIMESCALE

Original: Start as in Appendix B with a single OU process z , with  = 1 , and assume it is observed at unit intervals ( y t ) t 2 N with Gaussian observation noise of variance ⌘ 2 .
Revised: Start as in Appendix B with a single OU process z , with σ = 1 , and assume it is observed at unit intervals ( y t ) t ∈ N with Gaussian observation noise of variance η 2 .

Original: Assume a conjugate iterative prior:
Revised: Assume a conjugate iterative prior:

Original: [Equation]
Revised: [Equation]

Original: The posterior after each observation is
Revised: The posterior after each observation is

Original: [Equation]
Revised: [Equation]

Original: Evolving the process to time t + 1 amounts to decay by e  1 / ⌧ and accumulation of new noise.
Revised: Evolving the process to time t + 1 amounts to decay by e − 1 /τ and accumulation of new noise.

Original: At each time t 0 2 [ t, t + 1] , variance from the noise appearing at time t 0 (i.e., from d W ( t 0 ) in Equation 13) decays by a factor e  2( t +1  t 0 )/
Revised: At each time t (cid:48) ∈ [ t, t + 1] , variance from the noise appearing at time t (cid:48) (i.e., from d W ( t (cid:48) ) in Equation 13) decays by a factor e − 2( t +1 − t (cid:48) )

Original: ⌧ by time t + 1 .
Revised: /τ by time t + 1 .

Original: Therefore the total accumulated variance is:
Revised: Therefore the total accumulated variance is:

Original: [Equation]
Revised: [Equation]

Original: Therefore the prior for the next observation is
Revised: Therefore the prior for the next observation is

Original: [Equation]
Revised: [Equation]

Original: We thus obtain the recursion
Revised: We thus obtain the recursion

Original: [Equation]
Revised: [Equation]

Original: This is a temporal-difference learning rule, or gradient descent on squared error 12 ( y ( t )  w ( t )) , with learning rate ↵ ( t ) =
Revised: This is a temporal-difference learning rule, or gradient descent on squared error 12 ( y ( t ) − w ( t )) , with learning rate α ( t ) =

Original: s 2 ( t ) s 2 ( t )+ ⌘ 2 .
Revised: s 2 ( t ) s 2 ( t )+ η 2 .

Original: This connection between temporal-difference learning and
Revised: This connection between temporal-difference learning and

Original: the
Revised: the

Original: Kalman ﬁlter is well known (Sutton, 1992).
Revised: Kalman ﬁlter is well known (Sutton, 1992).

Original: The steady state for s 2 is given by the solution to s 2 ( t ) =
Revised: The steady state for s 2 is given by the solution to s 2 ( t ) =

Original: s 2 ( t + 1) , a quadratic with one positive root.
Revised: s 2 ( t + 1) , a quadratic with one positive root.

Original: If s 2 (0) is initialized to this value, then ↵ ( t ) will be constant.
Revised: If s 2 (0) is initialized to this value, then α ( t ) will be constant.

Original: Note that this algorithm (and the extension to 1 /f noise in Appendix D) generalizes to irregularly spaced observations, in which case one can derive how the optimal learning rate should vary on each step.
Revised: Note that this algorithm (and the extension to 1 /f noise in Appendix D) generalizes to irregularly spaced observations, in which case one can derive how the optimal learning rate should vary on each step.

Original: D K ALMAN FILTER OVER 1 /f NOISE
Revised: D K ALMAN FILTER OVER 1 /f NOISE

Original: Assume we receive a sequence of observations at unit time intervals, y ( t ) for t 2 N , and we want to do online prediction under the assumption that y is a 1 /f noise process.
Revised: Assume we receive a sequence of observations at unit time intervals, y ( t ) for t ∈ N , and we want to do online prediction under the assumption that y is a 1 /f noise process.

Original: Then we can make the generative assumption y ( t ) =
Revised: Then we can make the generative assumption y ( t ) =

Original: ⇠ ( t ) with ⇠ deﬁned as at the end of Appendix B. For simplicity and in contrast to Appendix C, we assume no observation noise, because the shortest timescales ( z 1 , etc.) already play this role.
Revised: ξ ( t ) with ξ deﬁned as at the end of Appendix B. For simplicity and in contrast to Appendix C, we assume no observation noise, because the shortest timescales ( z 1 , etc.) already play this role.

Original: Because y ( t ) is conditionally independent of the history given (and indeed is fully determined by) the joint state Z ( t )
Revised: Because y ( t ) is conditionally independent of the history given (and indeed is fully determined by) the joint state Z ( t )

Original: = ( z 1 ( t ) , . .
Revised: = ( z 1 ( t ) , . .

Original: , z n ( t )) , it sufﬁces to compute the posterior for the latter.
Revised: , z n ( t )) , it sufﬁces to compute the posterior for the latter.

Original: Write the iterative prior for Z as
Revised: Write the iterative prior for Z as

Original: [Equation]
Revised: [Equation]

Original:  implying an optimal (maximum-likelihood or least-squares) prediction of ˆ y ( t ) =
Revised: 43) implying an optimal (maximum-likelihood or least-squares) prediction of ˆ y ( t ) =

Original: P i ! i ( t ) .
Revised: (cid:80) i ω i ( t ) .

Original: The posterior after observing y ( t ) is the intersection of the prior with the hyperplane 
Revised: The posterior after observing y ( t ) is the intersection of the prior with the hyperplane (cid:80)

Original: P
Revised: i z i ( t ) = y ( t ) : [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here, 1 is the vector with all elements equal to 1 , and P =
Revised: Here, 1 is the vector with all elements equal to 1 , and P =

Original: I  1 n 11 > is the orthogonal projector.
Revised: I − 1 n 11 (cid:62) is the orthogonal projector.

Original: The prior for the next time step is then obtained by applying decay and adding variance from the noise accumulated over the intervening interval.
Revised: The prior for the next time step is then obtained by applying decay and adding variance from the noise accumulated over the intervening interval.

Original: Deﬁne D as the diagonal matrix of decay factors, e  1 / ⌧ i , and N as the diagonal matrix of added noise 2 ⇢ 2 (1  e  2 / ⌧ i ) , obtained by multiplying the
Revised: Deﬁne D as the diagonal matrix of decay factors, e − 1 /τ i , and N as the diagonal matrix of added noise 2 ρ 2 (1 − e − 2 /τ i ) , obtained by multiplying the

Original: RHS of Equation 38 by  2 i = 4 ⇢ 2 ⌧  1 i (from Appendix B).
Revised: RHS of Equation 38 by σ 2 i = 4 ρ 2 τ − 1 i (from Appendix B).

Original: Then we have the update equations for the exact Bayesian model:
Revised: Then we have the update equations for the exact Bayesian model:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Next, consider the perceptron in Section 5.1, where y = x > ✓ .
Revised: Next, consider the perceptron in Section 5.1, where y = x (cid:62) θ .

Original: The 1 /f generative model assumes ✓ j = P i z ij for each j , where i indexes timescales and j indexes features, and z ij has timescale ⌧ i and scaling parameter  i deﬁned as above.
Revised: The 1 /f generative model assumes θ j = (cid:80) i z ij for each j , where i indexes timescales and j indexes features, and z ij has timescale τ i and scaling parameter σ i deﬁned as above.

Original: The latent state is described by Z = ( z ij ) ij .
Revised: The latent state is described by Z = ( z ij ) ij .

Original: We treat ij as a single composite index, so that Z is a vector.
Revised: We treat ij as a single composite index, so that Z is a vector.

Original: As above, write the iterative prior as
Revised: As above, write the iterative prior as

Original: [Equation]
Revised: [Equation]

Original: Let X be a multiplexed copy of the input x , so that X ij = x j .
Revised: Let X be a multiplexed copy of the input x , so that X ij = x j .

Original: Assuming square loss, the optimal prediction for y ( t ) is ˆ y ( t ) =
Revised: Assuming square loss, the optimal prediction for y ( t ) is ˆ y ( t ) =

Original: X ( t ) > ! ( t ) .
Revised: X ( t ) (cid:62) ω ( t ) .

Original: The posterior after observing y ( t ) is the intersection of the prior with the hyperplane X ( t ) > Z ( t ) =
Revised: The posterior after observing y ( t ) is the intersection of the prior with the hyperplane X ( t ) (cid:62) Z ( t ) =

Original: y ( t ) :
Revised: y ( t ) :

Original: [Equation]
Revised: [Equation]

Original:  where P X =I
Revised: ) where P X =

Original:  XX > / X > X is the projector orthogonal to X .
Revised: I − XX (cid:62) / X (cid:62) X is the projector orthogonal to X .

Original: Generalizing the deﬁnitions above, let D and N be the diagonal matrices of decay factors and noise accumulation, D ij,ij = e  1 / ⌧ i and N ij,ij = 2 ⇢ 2 (1  e  2 / ⌧ i ) .
Revised: Generalizing the deﬁnitions above, let D and N be the diagonal matrices of decay factors and noise accumulation, D ij,ij = e − 1 /τ i and N ij,ij = 2 ρ 2 (1 − e − 2 /τ i ) .

Original: Applying these to Equation 48 to obtain the prior for the next time step yields the update equations for the exact Bayesian model of the regression task:
Revised: Applying these to Equation 48 to obtain the prior for the next time step yields the update equations for the exact Bayesian model of the regression task:

Original: [Equation]
Revised: [Equation]

Original: Equation 49 exempliﬁes how the variance matrix, S ( t ) , can be thought of as deﬁning a preconditioner of the gradient, X ( t )( y ( t )  ˆ y ( t )) .
Revised: Equation 49 exempliﬁes how the variance matrix, S ( t ) , can be thought of as deﬁning a preconditioner of the gradient, X ( t )( y ( t ) − ˆ y ( t )) .

Original: E E XTENDED K ALMAN FILTER
Revised: E E XTENDED K ALMAN FILTER

Original: Given a nonlinear model h ( x , ✓ ) , the 1 /f EKF posits that the optimal parameters follow 1 /f dynamics according to Equation 16, with expanded latent state Z
Revised: Given a nonlinear model h ( x , θ ) , the 1 /f EKF posits that the optimal parameters follow 1 /f dynamics according to Equation 16, with expanded latent state Z

Original: =
Revised: =

Original: ( z 1 , . . .
Revised: ( z 1 , . . .

Original: , z n ) > .
Revised: , z n ) (cid:62) .

Original: It is convenient to introduce the expanded model ˜ h deﬁned by ˜ 
Revised: It is convenient to introduce the expanded model ˜ h deﬁned by ˜ h

Original: h ( x , Z ) = h ( x , 
Revised: ( x , Z ) = h ( x , (cid:80) i

Original: [Equation]
Revised: z i ) .

Original: The 1 /f EKF maintains an iterative prior over Z as in Equation 47 and updates that prior by linearizing ˜ h about Z = ! ( t ) :
Revised: The 1 /f EKF maintains an iterative prior over Z as in Equation 47 and updates that prior by linearizing ˜ h about Z = ω ( t ) :

Original: [Equation]
Revised: [Equation]

Original: Here, J ˜ h
Revised: Here, J ˜ h

Original: = @ ˆ y @
Revised: = ∂ ˆ y ∂

Original: Z is the Jacobian matrix of ˜ h , evaluated at Z = ! ( t ) .
Revised: Z is the Jacobian matrix of ˜ h , evaluated at Z = ω ( t ) .

Original: Note that J ˜ h is just n copies of
Revised: Note that J ˜ h is just n copies of

Original: J h .
Revised: J h .

Original: Following Ollivier (2018), we assume the observation y is governed by some exponential family
Revised: Following Ollivier (2018), we assume the observation y is governed by some exponential family

Original: P ( y | ⌘ (ˆ y )) , with vector of sufﬁcient statistics T ( y ) .
Revised: P ( y | η (ˆ y )) , with vector of sufﬁcient statistics T ( y ) .

Original: The model’s output
Revised: The model’s output

Original: ˆ y =
Revised: ˆ y =

Original: ˜
Revised: ˜

Original: h
Revised: h

Original: ( x , ! ) is taken to encode the predicted mean parameter of that family: ˆ y =
Revised: ( x , ω ) is taken to encode the predicted mean parameter of that family: ˆ y =

Original: E y ⇠ P ( ·| ⌘ (ˆ y ))
Revised: E y ∼ P ( ·| η (ˆ y ))

Original: [ T ( y )]
Revised: [ T ( y )]

Original: (this can be read as a deﬁnition of the mapping ˆ y7!
Revised: (this can be read as a deﬁnition of the mapping ˆ y

Revised: (cid:55)→ η

Original: ⌘ (ˆ y ) ).
Revised: (ˆ y ) ).

Original: It then approximates the conditional distribution of the sufﬁcient statistics as a Gaussian,
Revised: It then approximates the conditional distribution of the sufﬁcient statistics as a Gaussian,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where R ˆ y =
Revised: where R ˆ y =

Original: Var ( T ( y ) | ˆ y ) is the conditional variance.
Revised: Var ( T ( y ) | ˆ y ) is the conditional variance.

Original: For example, when h is a classiﬁcation model as in Section 5.2 or 5.3, the output of the network is a vector ˆ y of class probabilities, and the sufﬁcient statistics T ( y ) are a one-hot vector.
Revised: For example, when h is a classiﬁcation model as in Section 5.2 or 5.3, the output of the network is a vector ˆ y of class probabilities, and the sufﬁcient statistics T ( y ) are a one-hot vector.

Original: For numerical stability, we exclude the ﬁnal element of ˆ y and T ( y ) , which are determined by the other elements.
Revised: For numerical stability, we exclude the ﬁnal element of ˆ y and T ( y ) , which are determined by the other elements.

Original: The conditional outcome variance is given by
Revised: The conditional outcome variance is given by

Original: [Equation]
Revised: [Equation]

Original: Under the approximations of Equations 51 and 52, the posterior is given by the standard KF formula:
Revised: Under the approximations of Equations 51 and 52, the posterior is given by the standard KF formula:

Original: [Equation]
Revised: [Equation]

Original: Applying decay ( D ) and accumulated noise ( N ) as in Appendix D to obtain the prior for the next time step yields the update equations for the 1 /f EKF:
Revised: Applying decay ( D ) and accumulated noise ( N ) as in Appendix D to obtain the prior for the next time step yields the update equations for the 1 /f EKF:

Original: [Equation]
Revised: [Equation]

Original: Although we apply the EKF to feedforward NNs in this paper [Equation] we note that the approach naturally generalizes to other probabilistic causal models relating the observed variables.
Revised: Although we apply the EKF to feedforward NNs in this paper, we note that the approach naturally generalizes to other probabilistic causal models relating the observed variables.

Original: Thus it offers a means to model nonstationarity that covers all of the traditional forms of distribution shift, following the causal framework of Sch¨olkopf et al.
Revised: Thus it offers a means to model nonstationarity that covers all of the traditional forms of distribution shift, following the causal framework of Sch¨olkopf et al.

Original: Consider a classiﬁcation task, with input features x ( t ) 2 R m and class labels y ( t ) 2 Z k .
Revised: Consider a classiﬁcation task, with input features x ( t ) ∈ R m and class labels y ( t ) ∈

Original:  Under a generative causal model where x depends on y , label shift can be modeled by 1 /f noise in the distribution p ( y ) , for example y ⇠ softmax ( ` ) with ` given by Equation 16.
Revised: Z k . Under a generative causal model where x depends on y , label shift can be modeled by 1 /f noise in the distribution p ( y ) , for example y ∼ softmax ( (cid:96) ) with (cid:96) given by Equation 16.

Original: Manifestation shift can be modeled by 1 /f noise in the distributions p ( x | y ) , for example x ( t ) | y ( t ) ⇠ N  µ y ( t ) , ⌃  with µ y given by Equation 16 for each y .
Revised: Manifestation shift can be modeled by 1 /f noise in the distributions p ( x | y ) , for example x ( t ) | y ( t ) ∼ N (cid:0) µ y ( t ) , Σ (cid:1) with µ y given by Equation 16 for each y .

Original: Likewise, under a discriminative causal model where y depends on x , covariate shift can be modeled by 1 /f noise in p ( x ) , and concept shift can be modeled by 1 /f noise in p ( y | x ) .
Revised: Likewise, under a discriminative causal model where y depends on x , covariate shift can be modeled by 1 /f noise in p ( x ) , and concept shift can be modeled by 1 /f noise in p ( y | x ) .

Original: F V ARIATIONAL APPROXIMATION
Revised: F V ARIATIONAL APPROXIMATION

Original: This section derives variational approximations of the KF and EKF models, with S ( t ) ⇡ ˜ S ( t ) where ˜ S ( t ) :=
Revised: This section derives variational approximations of the KF and EKF models, with S ( t ) ≈ ˜ S ( t ) where ˜ S ( t ) :

Original:  diag( s 2 ( t )) .
Revised: = diag( s 2 ( t )) .

Original: Thus the resulting algorithms need only track the individual variance terms in s 2 ( t ) rather than the full covariance matrix.
Revised: Thus the resulting algorithms need only track the individual variance terms in s 2 ( t ) rather than the full covariance matrix.

Original: Moreover, the update equations (63,64,68,69,74,77) all avoid matrix inversion—even though this might initially seem necessary from Equation 57—a property that may be relevant for efﬁcient scaling.
Revised: Moreover, the update equations (63,64,68,69,74,77) all avoid matrix inversion—even though this might initially seem necessary from Equation 57—a property that may be relevant for efﬁcient scaling.

Original: F. U NCUED INFERENCE
Revised: F. U NCUED INFERENCE

Original: We begin with the simple Bayesian 1 /f model in Equations 45 and 46, where there are no predictors and the model merely tracks an observable y ( t ) .
Revised: We begin with the simple Bayesian 1 /f model in Equations 45 and 46, where there are no predictors and the model merely tracks an observable y ( t ) .

Original: Given an arbitrary Gaussian distribution, the variational approximation (in the sense of minimizing Kullback-Leibler divergence) by another Gaussian with diagonal covariance matrix is obtained by taking the diagonal of the original distribution’s precision matrix.
Revised: Given an arbitrary Gaussian distribution, the variational approximation (in the sense of minimizing Kullback-Leibler divergence) by another Gaussian with diagonal covariance matrix is obtained by taking the diagonal of the original distribution’s precision matrix.

Original: Thus in the present case we have
Revised: Thus in the present case we have

Original: [Equation]
Revised: [Equation]

Original:  where S ( t +1) is given by Equation 46 with S  1 ( t ) replaced by ˜ S  1 ( t ) (the inductive assumption).
Revised: (57) where S ( t +1) is given by Equation 46 with S − 1 ( t ) replaced by ˜ S − 1 ( t ) (the inductive assumption).

Original: To calculate S  1 ( t + 1) under this assumption, we ﬁrst observe the identity
Revised: To calculate S − 1 ( t + 1) under this assumption, we ﬁrst observe the identity

Original: [Equation]
Revised: [Equation]

Original: Therefore Equation 46 becomes
Revised: Therefore Equation 46 becomes

Original: [Equation]
Revised: [Equation]

Original: where diag( a ) =
Revised: where diag( a ) =

Original: D ˜ S
Revised: D ˜ S

Original: ( t ) D + N , b
Revised: ( t ) D + N , b

Original: = Ds 2 ( t ) , and c =
Revised: = Ds 2 ( t ) [Equation]

Original: P i s 2 i ( t ) .
Revised: and c = (cid:80) i s 2 i ( t ) .

Original: We next use the identity✓
Revised: We next use the identity

Original: [Equation]
Revised: [Equation]

Original: implying the diagonal elements are
Revised: implying the diagonal elements are

Original: [Equation]
Revised: [Equation]

Original: Combining Equations 57, 60, and 62 gives the ﬁnal form of the variational update:
Revised: Combining Equations 57, [Equation] and 62 gives the ﬁnal form of the variational update:

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: This update of the variance converges exponentially to a unique ﬁxed point.
Revised: This update of the variance converges exponentially to a unique ﬁxed point.

Original: Numerical simulations conﬁrm that, in this limit, s 

Revised: Numerical simulations conﬁrm that, in this limit, s

Original: i is larger for smaller ⌧ i , meaning faster learning rates for shorter timescales.
Revised: i is larger for smaller τ i , meaning faster learning rates for shorter timescales.

Original: If the prior variances are initialized at the ﬁxed point then they are constant throughout learning.
Revised: If the prior variances are initialized at the ﬁxed point then they are constant throughout learning.

Original: By substituting the diagonal matrix ˜ S ( t ) for S ( t ) , the update for the mean in Equation 45 simpliﬁes to
Revised: By substituting the diagonal matrix ˜ S ( t ) for S ( t ) [Equation] the update for the mean in Equation 45 simpliﬁes to

Original: [Equation]
Revised: [Equation]

Original: 65) where ˆ y ( t )  y ( t ) is the loss gradient (assuming square loss), and the learning rates are given by
Revised: (65) where ˆ y ( t ) − y ( t ) is the loss gradient (assuming square loss) [Equation] and the learning rates are given by

Original: [Equation]
Revised: [Equation]

Original: That is, the subweights learn independently according to their gradients, with different decay rates and learning rates.
Revised: That is, the subweights learn independently according to their gradients, with different decay rates and learning rates.

Original: Thus we have recovered an extension of the multiscale optimizer, with an additional mechanism that adapts the learning rates on each time step (via s ).
Revised: Thus we have recovered an extension of the multiscale optimizer, with an additional mechanism that adapts the learning rates on each time step (via s ).

Original: F . K ALMAN FILTER
Revised: F. K ALMAN FILTER

Original: To derive the variational KF for the regression task, we apply the analysis of Section F.1 to the KF update in Equations 49 and 50.
Revised: To derive the variational KF for the regression task, we apply the analysis of Section F.1 to the KF update in Equations 49 and 50.

Original: Paralleling the derivation of Equation 59, the variance update in
Revised: Paralleling the derivation of Equation 59, the variance update in

Original: Equation 50 (i.e., before applying the diagonal variational approximation) can be written as
Revised: Equation 50 (i.e., before applying the diagonal variational approximation) can be written as

Original: [Equation]
Revised: [Equation]

Original: Paralleling the derivation of Equations 63 and 64, the variational update comes out to be
Revised: Paralleling the derivation of Equations 63 and 64, the variational update comes out to be

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  with
Revised: (68) with

Original: [Equation]
Revised: [Equation]

Original: By substituting the diagonal matrix ˜ S ( t ) for S ( t ) , the mean update in Equation 49 simpliﬁes to
Revised: By substituting the diagonal matrix ˜ S ( t ) for S ( t ) , the mean update in Equation 49 simpliﬁes to

Original: [Equation]
Revised: [Equation]

Original: (70) where x j ( t ) (ˆ y ( t )  y ( t )) is the loss gradient for ! ij , and the learning rates are given by
Revised: (70) where x j ( t ) (ˆ y ( t ) − y ( t )) is the loss gradient for ω ij , and the learning rates are given by

Original: [Equation]
Revised: [Equation]

Original: Thus we have recovered an extension of the multiscale optimizer, with an additional mechanism that adapts the learning rates on each time step (via x and s ).
Revised: Thus we have recovered an extension of the multiscale optimizer, with an additional mechanism that adapts the learning rates on each time step (via x and s ).

Original: F. E XTENDED K ALMAN FILTER
Revised: F. E XTENDED K ALMAN FILTER

Original: To derive a closed-form variational approximation for the general EKF, such as for the classiﬁcation models in Sections 5.2 and 5.3, it turns out that we need to apply the variational approximation to the posterior in Equation 54, rather than to the iterative prior in Equation 56.
Revised: To derive a closed-form variational approximation for the general EKF, such as for the classiﬁcation models in Sections 5.2 and 5. [Equation] it turns out that we need to apply the variational approximation to the posterior in Equation 54, rather than to the iterative prior in Equation 56.

Original: Using Woodbury’s identity, the posterior variance can be rewritten as
Revised: Using Woodbury’s identity [Equation] the posterior variance can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: The form on the RHS is convenient because it is in terms of precision, allowing us to read off the diagonal entries directly.
Revised: The form on the RHS is convenient because it is in terms of precision, allowing us to read off the diagonal entries directly.

Original: That is, the variational approximation for the posterior variance is diag( s 0 2 ( t )) , with
Revised: That is, the variational approximation for the posterior variance is diag( s (cid:48) 2 ( t )) , with

Original: [Equation]
Revised: [Equation]

Original: Here was have used the inductive assumption S ( t ) ⇡ diag( s 2 ( t )) .
Revised: Here was have used the inductive assumption S ( t ) ≈ diag( s 2 ( t )) .

Original: Applying the transition from posterior on step t to prior on step t + [Equation] we obtain the variance update for the variational model, replacing Equation 56:
Revised: Applying the transition from posterior on step t to prior on step t + 1 , we obtain the variance update for the variational model, replacing Equation 56:

Original: [Equation]
Revised: [Equation]

Original: Woodbury’s identity also enables the mean update from Equation 55 to be rewritten, as
Revised: Woodbury’s identity also enables the mean update from Equation 55 to be rewritten, as

Original: [Equation]
Revised: [Equation]

Original: Substituting the gradient of the EKF’s approximate likelihood (denoted ˜ L ) from Equation 52,
Revised: Substituting the gradient of the EKF’s approximate likelihood (denoted ˜ L ) from Equation 52,

Original: [Equation]
Revised: [Equation]

Original: ) yields
Revised: yields

Original: [Equation]
Revised: [Equation]

Original: The preconditioner on the gradient here is the posterior variance (see Equation 72), which we have approximated as diag( s 0 2 ( t )) .
Revised: The preconditioner on the gradient here is the posterior variance (see Equation 72), which we have approximated as diag( s (cid:48) 2 ( t )) .

Original: Although not necessarily entailed by the variational approximation, we can consider applying the same approximation in updating the mean.
Revised: Although not necessarily entailed by the variational approximation, we can consider applying the same approximation in updating the mean.

Original: This yields
Revised: This yields

Original: [Equation]
Revised: [Equation]

Original: which once again extends the multiscale optimizer by adapting its learning rate to current uncertainty.
Revised: which once again extends the multiscale optimizer by adapting its learning rate to current uncertainty.

Original: G I MPLEMENTATION DETAILS
Revised: G I MPLEMENTATION DETAILS

Original: Latent parameters for the synthetic tasks in Sections 5.1 and 5.2 were sampled using the generative model in Appendix B.
Revised: Latent parameters for the synthetic tasks in Sections 5.1 and 5.2 were sampled using the generative model in Appendix B.

Original: That is, the data-generating process matched the generative assumptions of the Bayesian model in both of these cases.
Revised: That is, the data-generating process matched the generative assumptions of the Bayesian model in both of these cases.

Original: We used 20 timescales, geometrically spaced from ⌧ 1 =to ⌧ 20 = 1000 , as illustrated in Figure 6A.
Revised: We used 20 timescales, geometrically spaced from τ 1 =to τ 20 = 1000 , as illustrated in Figure 6A.

Original: Each component OU process was run for 10 ⌧ i burn-in steps to ensure stationarity.
Revised: Each component OU process was run for 10 τ i burn-in steps to ensure stationarity.

Original: The regression task was run for 10k trials, and the linear classiﬁcation task for 1000 trials.
Revised: The regression task was run for 10k trials, and the linear classiﬁcation task for 1000 trials.

Original: The 1 /f power spectrum was conﬁrmed with a log-log plot (see Figure 6B).
Revised: The 1 /f power spectrum was conﬁrmed with a log-log plot (see Figure 6B).

Original: In the regression task of Section 5.1, the ﬁrst feature was a constant bias term, x 1 ⌘ 1 .
Revised: In the regression task of Section 5.1, the ﬁrst feature was a constant bias term, x 1 ≡ 1 .

Original: The otherfeatures were sampled independently as N ( 0 , I ) on each time step.
Revised: The otherfeatures were sampled independently as N ( 0 , I ) on each time step.

Original: For the linear classiﬁcation task of Section 5.2, class logits were sampled from 1 /f processes and then multiplied by 0 .
Revised: For the linear classiﬁcation task of Section 5.2, class logits were sampled from 1 /f processes and then multiplied by 0 .

Original: 886 before entering into softmax to determine class probabilities.
Revised: 886 before entering into softmax to determine class probabilities.

Original: Featureclass means were ﬁxed at 1 for the ﬁrst feature (i.e., bias term) and were sampled from mutually independent 1 /f processes for features 2-10, multiplied by 0 .
Revised: Featureclass means were ﬁxed at 1 for the ﬁrst feature (i.e., bias term) and were sampled from mutually independent 1 /f processes for features 2-10, multiplied by 0 .

Original: These scaling factors were chosen so that perfect knowledge of either the prior probabilities or the feature-class means on every trial would yield ideal-observer performance of ¯ L ⇡ 1 .
Revised: These scaling factors were chosen so that perfect knowledge of either the prior probabilities or the feature-class means on every trial would yield ideal-observer performance of ¯ L ≈ 1 .

Original: Perfect knowledge of both would yield ¯ L ⇡ 0 . 
Revised: Perfect knowledge of both would yield ¯ L ≈ 0 . 35 .

Original: These were merely guidelines for equating prior and likelihood information, as perfect knowledge of either source of information is not possible even with an optimal model of the dynamics.
Revised: These were merely guidelines for equating prior and likelihood information, as perfect knowledge of either source of information is not possible even with an optimal model of the dynamics.

Original: In both Sections 5.1 and 5.2, all Bayesian and variational models assumed 10 timescales, geometrically spaced from ⌧ 1 = 2 to ⌧ 10 = 800 .
Revised: In both Sections 5.1 and 5.2, all Bayesian and variational models assumed 10 timescales, geometrically spaced from τ 1 = 2 to τ 10 = 800 .

Original: This deliberate deviation from the data-generating process (see the 20 timescales listed above) provided a mild test of robustness, speciﬁcally the hypothesis that it is the aggregate 1 /f character of the environment and of the model that matters, not the choice of component timescales used to approximate that character.
Revised: This deliberate deviation from the data-generating process (see the 20 timescales listed above) provided a mild test of robustness, speciﬁcally the hypothesis that it is the aggregate 1 /f character of the environment and of the model that matters, not the choice of component timescales used to approximate that character.

Original: For the MNIST classiﬁcation task in Section 5.3, the class on each time step was sampled by softmax applied to class logits that varied across steps according to 1 /f noise.
Revised: For the MNIST classiﬁcation task in Section 5.3, the class on each time step was sampled by softmax applied to class logits that varied across steps according to 1 /f noise.

Original: The item on that time step was than sampled without replacement from all members of that class in the MNIST training set.
Revised: The item on that time step was than sampled without replacement from all members of that class in the MNIST training set.

Original: The logit sequences were generated as follows.
Revised: The logit sequences were generated as follows.

Original: First, for each class, we sampled a Standard Gaussian random vector of length equal to the total number of time steps (10k).
Revised: First, for each class, we sampled a Standard Gaussian random vector of length equal to the total number of time steps (10k).

Original: Then we applied a discrete
Revised: Then we applied a discrete

Original: Fourier transform, multiplied the result by f  1 / 2 , and ﬁnally applied the inverse Fourier transform.
Revised: Fourier transform, multiplied the result by f − 1 / 2 , and ﬁnally applied the inverse Fourier transform.

Original: Thus the resulting sequence had 1 /f power spectrum.
Revised: Thus the resulting sequence had 1 /f power spectrum.

Original: For the iid environment, the logits were sampled as white noise (constant power spectrum), by sampling a Standard Gaussian vector as above and using it unaltered.
Revised: For the iid environment, the logits were sampled as white noise (constant power spectrum), by sampling a Standard Gaussian vector as above and using it unaltered.