Original: L EARNING TO O PTIMIZE Q UASI -N EWTON M ETHODS
Revised: L EARNING TO O PTIMIZE Q UASI -N EWTON M ETHODS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce a novel machine learning optimizer called LODO, which online meta-learns an implicit inverse Hessian of the loss as a subroutine of quasi-Newton optimization.
Revised: We introduce a novel machine learning optimizer called LODO, which online meta-learns an implicit inverse Hessian of the loss as a subroutine of quasi-Newton optimization.

Original: Our optimizer merges Learning to Optimize (L2O) techniques with quasi-Newton methods to learn neural representations of symmetric matrix vector products, which are more flexible than those in other quasi-Newton methods.
Revised: Our optimizer merges Learning to Optimize (L2O) techniques with quasi-Newton methods to learn neural representations of symmetric matrix vector products, which are more flexible than those in other quasi-Newton methods.

Original: Unlike other L2O methods, ours does not require any meta-training on a training task distribution, and instead learns to optimize on the fly while optimizing on the test task, adapting to the local characteristics of the loss landscape while traversing it.
Revised: Unlike other L2O methods, ours does not require any meta-training on a training task distribution, and instead learns to optimize on the fly while optimizing on the test task, adapting to the local characteristics of the loss landscape while traversing it.

Original: Theoretically, we show that our optimizer approximates the inverse Hessian in noisy loss landscapes and is capable of representing a wide range of inverse
Revised: Theoretically, we show that our optimizer approximates the inverse Hessian in noisy loss landscapes and is capable of representing a wide range of inverse

Original: Hessians.
Revised: Hessians.

Original: We experimentally verify our algorithm’s performance in the presence of noise, and show that simpler alternatives for representing the inverse Hessians worsen performance.
Revised: We experimentally verify our algorithm’s performance in the presence of noise, and show that simpler alternatives for representing the inverse Hessians worsen performance.

Original: Lastly, we use our optimizer to train a semi-realistic deep neural network with 95k parameters, and obtain competitive results against standard neural network optimizers.
Revised: Lastly, we use our optimizer to train a semi-realistic deep neural network with 95k parameters, and obtain competitive results against standard neural network optimizers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many optimization algorithms like stochastic gradient descent (SGD) (Rosenblatt, 1958), and
Revised: Many optimization algorithms like stochastic gradient descent (SGD) (Rosenblatt, 1958), and

Original: Adam (Kingma & Ba, 2014) have been widespread and successful in the rapid training of deep neural networks.
Revised: Adam (Kingma & Ba, 2014) have been widespread and successful in the rapid training of deep neural networks.

Original: (Sun et al., 2019)
Revised: (Sun et al., 2019)

Original: Fundamentally, this is a problem of minimizing a loss which is a function of a large vector containing the weights of the network.
Revised: Fundamentally, this is a problem of minimizing a loss which is a function of a large vector containing the weights of the network.

Original: The time it takes to optimize a neural network is a bottleneck in machine learning, so the more quickly a network can be trained, the more computational resources are saved, and therefore researchers have devoted great effort into creating new, faster optimizers.
Revised: The time it takes to optimize a neural network is a bottleneck in machine learning, so the more quickly a network can be trained, the more computational resources are saved, and therefore researchers have devoted great effort into creating new, faster optimizers.

Original: (Jain & Kar, 2017) (Metz et al., 2020)
Revised: (Jain & Kar, 2017; Metz et al., 2020;

Original: (Bernstein et al., 2020) (Martens & Grosse, 2015a)
Revised: Bernstein et al., 2020; Martens & Grosse, 2015a)

Original: We present a novel algorithm drawing from the field of learning to optimize (L2O) spearheaded by (Li & Malik, 2016) and (Andrychowicz et al., 2016).
Revised: We present a novel algorithm drawing from the field of learning to optimize (L2O) spearheaded by (Li & Malik, 2016) and (Andrychowicz et al., 2016).

Original: Namely, we use a meta-optimizer to online learn an implicit representation of the local inverse Hessian, which is used in a quasi-Newton method, without any L2O meta training time on a training task distribution.
Revised: Namely, we use a meta-optimizer to online learn an implicit representation of the local inverse Hessian, which is used in a quasi-Newton method, without any L2O meta training time on a training task distribution.

Original: Unlike other L2O algorithms which learn to optimize before optimization (Chen et al., 2021), our algorithm Learns to
Revised: Unlike other L2O algorithms which learn to optimize before optimization (Chen et al., 2021), our algorithm Learns to

Original: Optimize During Optimization (LODO).
Revised: Optimize During Optimization (LODO).

Original: We intend for LODO to be trained from scratch for each use case and then discarded.
Revised: We intend for LODO to be trained from scratch for each use case and then discarded.

Original: This way, LODO learns local features of the loss landscape at a specific point in training for a specific task, instead of only characteristics shared throughout training trajectories for a set of training tasks.
Revised: This way, LODO learns local features of the loss landscape at a specific point in training for a specific task, instead of only characteristics shared throughout training trajectories for a set of training tasks.

Original: Our work targets the Hessian, which varies with both the task and the point along the trajectory.
Revised: Our work targets the Hessian, which varies with both the task and the point along the trajectory.

Original: Our use of linear neural networks is what imports the efficiency of the
Revised: Our use of linear neural networks is what imports the efficiency of the

Original: Newton method to our algorithm, while our use of a meta-optimizer like in L2O is what allows us to learn more powerful and general parameterizations of optimizers.
Revised: Newton method to our algorithm, while our use of a meta-optimizer like in L2O is what allows us to learn more powerful and general parameterizations of optimizers.

Original: Our contributions are as follows.
Revised: Our contributions are as follows.

Original: We show theoretically and experimentally that a simplified version of LODO correctly learns the inverse Hessian in a stochastic convex setting.
Revised: We show theoretically and experimentally that a simplified version of LODO correctly learns the inverse Hessian in a stochastic convex setting.

Original: We show theoretically that LODO’s inverse Hessian representation is highly expressive, and experimentally that simpler alternatives perform worse.
Revised: We show theoretically that LODO’s inverse Hessian representation is highly expressive, and experimentally that simpler alternatives perform worse.

Original: We finally demonstrate the use of LODO in a semi-realistic vision task.
Revised: We finally demonstrate the use of LODO in a semi-realistic vision task.

Original: This paper serves as a stepping stone in the development of meta-training-free online L2O.
Revised: This paper serves as a stepping stone in the development of meta-training-free online L2O.

Original: The remainder of this paper is structured as follows.
Revised: The remainder of this paper is structured as follows.

Original: Section 2 discusses relevant background and contributions in optimization and L2O. Section 3 shows how LODO works.
Revised: Section 2 discusses relevant background and contributions in optimization and L2O. Section 3 shows how LODO works.

Original: Section 4 provides theoretical justification for our design of LODO.
Revised: Section 4 provides theoretical justification for our design of LODO.

Original: Sections 5 and 6 show experiments which explore what makes LODO work and why.
Revised: Sections 5 and 5.2 show experiments which explore what makes LODO work and why.

Original: Section 7 discusses and summarize our findings.
Revised: Section 6 discusses and summarize our findings.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Research into the construction of faster optimizers has mostly fallen under two branches of work.
Revised: Research into the construction of faster optimizers has mostly fallen under two branches of work.

Original: The older branch attempts to endow SGD with adaptive capabilities, often through modifications involving calculation of the first and/or second moments of the gradient (mean and variance) using exponential moving averages (EMAs).
Revised: The older branch attempts to endow SGD with adaptive capabilities, often through modifications involving calculation of the first and/or second moments of the gradient (mean and variance) using exponential moving averages (EMAs).

Original: RMSprop (Hinton et al., 2012) and Adam use the variance to normalize the step size and the mean to induce momentum.
Revised: RMSprop (Hinton et al., 2012) and Adam use the variance to normalize the step size and the mean to induce momentum.

Original: LARS (You et al., 2017) and Yogi (Zaheer et al., 2018) modify both modify the variance calculation, but for different reasons: to normalize layer-wise, and to control increases in effective learning rate in a slower manner, respectively.
Revised: LARS (You et al., 2017) and Yogi (Zaheer et al., 2018) modify both modify the variance calculation, but for different reasons: to normalize layer-wise, and to control increases in effective learning rate in a slower manner, respectively.

Original: Some of these methods such as the Newton method and natural gradient descent (Martens & Grosse, 2015b) (George et al., 2018) precondition the step with adaptive estimates of the inverse Hessian and the inverse Fisher information matrices, respectively.
Revised: Some of these methods such as the Newton method and natural gradient descent (Martens & Grosse, 2015b; George et al., 2018) precondition the step with adaptive estimates of the inverse Hessian and the inverse Fisher information matrices, respectively.

Original: The Newton method converges quickly but is vulnerable to gradient noise and impractical to implement due to the resources spent in calculating and/or inverting the high dimensional Hessian.
Revised: The Newton method converges quickly but is vulnerable to gradient noise and impractical to implement due to the resources spent in calculating and/or inverting the high dimensional Hessian.

Original: Many researchers have developed approximations— called quasi-Newton methods—which reduce the Newton method’s time and memory complexity, such as L-BFGS (Nocedal & Wright, 1999) and variants (Schraudolph et al., 2007) (Parker-Holder et al., 2020) better suited to the stochasticity  present in machine learning.
Revised: Many researchers have developed approximations— called quasi-Newton methods—which reduce the Newton method’s time and memory complexity, such as L-BFGS (Nocedal & Wright, 1999) and variants (Schraudolph et al., 2007; Parker-Holder et al., 2020; Goldfarb et al., 2020; Park & Oliva, 2019) better suited to the stochasticity and structure present in machine learning.

Original: The most related methods to our work are hypergradient methods, which online learn low-rank (Moskovitz et al., 2019), diagonal (Amid et al., 2022) (Baydin et al., 2017), or Kronecker-factorized (Bae et al., 2022) preconditioner matrices to transform the gradient when choosing the step.
Revised: The most related methods to our work are hypergradient methods, which online learn low-rank (Moskovitz et al., 2019), diagonal (Amid et al., 2022; Baydin et al., 2017), or Kronecker-factorized (Bae et al., 2022) preconditioner matrices to transform the gradient when choosing the step.

Original: We improve on these methods by using a more expressive class of preconditioners.
Revised: We improve on these methods by using a more expressive class of preconditioners.

Original: More recently, a subfield of meta-learning known as learning to optimize (L2O) has shown that deep networks can themselves be trained to perform optimization, at a speed which exceeds that of popular traditional optimizers.
Revised: More recently, a subfield of meta-learning known as learning to optimize (L2O) has shown that deep networks can themselves be trained to perform optimization, at a speed which exceeds that of popular traditional optimizers.

Original: The aim of this effort is to leverage deep neural networks to learn faster optimizers, in hopes of further accelerating training procedures for other deep neural networks.
Revised: The aim of this effort is to leverage deep neural networks to learn faster optimizers, in hopes of further accelerating training procedures for other deep neural networks.

Original: Li &
Revised: Li &

Original: Malik (2016; 2017) and Andrychowicz et al.
Revised: Malik (2016; 2017) and Andrychowicz et al.

Original: (2016) were among the first to successfully use backpropagation to train neural networks to map gradients to steps.
Revised: (2016) were among the first to successfully use backpropagation to train neural networks to map gradients to steps.

Original: Since then, many other variations of this idea have successfully produced optimizers exceeding the speed of common optimizers for narrow ranges of machine learning models (Metz et al., 2018), though theoretical analysis of these learned optimizers tends to be difficult and scarce.
Revised: Since then, many other variations of this idea have successfully produced optimizers exceeding the speed of common optimizers for narrow ranges of machine learning models (Metz et al., 2018), though theoretical analysis of these learned optimizers tends to be difficult and scarce.

Original: A major goal of L2O research is to learn a single optimizer which can generalize to be able to train a wide variety of machine learning models with speed.
Revised: A major goal of L2O research is to learn a single optimizer which can generalize to be able to train a wide variety of machine learning models with speed.

Original: (Lv et al., 2017)
Revised: (Lv et al., 2017)

Original: Two issues also prevent L2O optimizers from being rapidly developed experimentally.
Revised: Two issues also prevent L2O optimizers from being rapidly developed experimentally.

Original: Firstly, a carefully chosen “task distribution” for the optimizer to practice on is required for the meta-learning of the L2O optimizer, playing the role analogous to the “dataset”.
Revised: Firstly, a carefully chosen “task distribution” for the optimizer to practice on is required for the meta-learning of the L2O optimizer, playing the role analogous to the “dataset”.

Original: These tasks are difficult to curate because the issue of generalization error applies; we want the test task to be similar to the task distribution.
Revised: These tasks are difficult to curate because the issue of generalization error applies; we want the test task to be similar to the task distribution.

Original: Secondly, this meta-learning of the L2O optimizer is prohibitively costly, in that it involves nested training loops, where the inner loop takes a large amount of time and memory to evaluate and backpropagate through (Metz et al., 2019).
Revised: Secondly, this meta-learning of the L2O optimizer is prohibitively costly, in that it involves nested training loops, where the inner loop takes a large amount of time and memory to evaluate and backpropagate through (Metz et al., 2019).

Original: Altogether, the choice of task distribution and lengthy meta-training has been a necessary burden in L2O, and we overcome these with LODO.
Revised: Altogether, the choice of task distribution and lengthy meta-training has been a necessary burden in L2O, and we overcome these with LODO.

Original:  H OW LODO W ORKS
Revised:  H OW LODO W ORKS

Original: In a quasi-Newton method, the approximate solution x t ∈ R n is refined by x t +1 = x
Revised: In a quasi-Newton method, the approximate solution x t ∈ R n is refined by x t +1 = x

Original: t − ˜ α G t g t for some learning rate ˜ α > 0 , where G t ≈ ( ∇
Revised: t − ˜ α G t g t for some learning rate ˜ α > 0 , where G t ≈ ( ∇

Original: 2 x t f ( x t )) − 1 ∈ R n × n is some approximation of the inverse Hessian and g t = ∇ x t f ( x t ) ∈
Revised: 2 x t f ( x t )) − 1 ∈ R n × n is some approximation of the inverse Hessian and g t = ∇ x t f ( x t ) ∈

Original: R n is the gradient computed by backpropagation through the task f .
Revised: R n is the gradient computed by backpropagation through the task f .

Original: ˜
Revised: ˜

Original: α
Revised: α

Original: = 1 produces the exact solution if f is quadratic, so we set ˜ α
Revised: = 1 produces the exact solution if f is quadratic, so we set ˜ α

Original: = 1 .
Revised: = 1 .

Original: Our algorithm approximates the inverse Hessian using a matrix G ( θ t ) ∈ R n × n parameterized by a vector θ t of weights learned over time t , described later in this section.
Revised: Our algorithm approximates the inverse Hessian using a matrix G ( θ t ) ∈ R n × n parameterized by a vector θ t of weights learned over time t , described later in this section.

Original: After every step t ← t + 1 using the formula x t +1 = x
Revised: After every step t ← t + 1 using the formula x t +1 = x

Original: t − G ( θ t ) g t , the loss f ( x t +1 ) is computed.
Revised: t − G ( θ t ) g t , the loss f ( x t +1 ) is computed.

Original: Then the new gradient ∇ x t +1 f
Revised: Then the new gradient ∇ x t +1 f

Original: ( x t +1 ) in x t +1 is computed through backpropagation as usual, but we continue backpropagation into the step-choosing process until we find the “hypergradient” ∇ θ t f
Revised: ( x t +1 ) in x t +1 is computed through backpropagation as usual, but we continue backpropagation into the step-choosing process until we find the “hypergradient” ∇ θ t f

Original: ( x t +1 ) in the optimizer weights θ t ,
Revised: ( x t +1 ) in the optimizer weights θ t ,

Original: Require: f : R n → R , x 0 ∈ R n : Function to minimize, initialization.
Revised: Require: f : R n → R , x 0 ∈ R n : Function to minimize, initialization.

Original: Require: α ∈ R : Meta-learning rate (default 0.001),
Revised: Require: α ∈ R : Meta-learning rate (default 0.001),

Original: Require: α 0 ∈ R : Initial learning rate (default 1.0),
Revised: Require: α 0 ∈ R : Initial learning rate (default 1.0),

Original: Require: 0 ≤ β < 1 : Momentum (default 0.9), t ←▷ Start time θ 0 ← random initialization ▷ Initialization for neural network.
Revised: Require: 0 ≤ β < 1 : Momentum (default 0.9), t ←▷ Start time θ 0 ← random initialization ▷ Initialization for neural network.

Original: m 

Revised: m

Original: ←▷ Initialize EMAs while not converged do x t +1 ← x t − G ( θ t , α 0 )
Revised: ←▷ Initialize EMAs while not converged do x t +1 ← x t − G ( θ t , α 0 )

Original: m t ▷ Pick step with inverse Hessian model G from Equation (1) ℓ t +1 ← f ( x t +1 )
Revised: m t ▷ Pick step with inverse Hessian model G from Equation (1) ℓ t +1 ← f ( x t +1 )

Original: ▷ Compute loss after step m t +1 ← β m t + (1 − β ) ∇ x t +1 ℓ t + ▷ Update gradient EMAs update θ t → θ t +1 using Adam with ∇ θ t ℓ t + ▷ Tune the model to choose a better step t ← t +▷ Increment time end while return θ t
Revised: ▷ Compute loss after step m t +1 ← β m t + (1 − β ) ∇ x t +1 ℓ t + ▷ Update gradient EMAs update θ t → θ t +1 using Adam with ∇ θ t ℓ t + ▷ Tune the model to choose a better step t ← t +▷ Increment time end while return θ t

Original: [Figure]
Revised: [Figure]

Original: which allows us to update them as well.
Revised: which allows us to update them as well.

Original: Thus, θ is trained such that the quasi-Newton method tries to minimize the loss upon a
Revised: Thus, θ is trained such that the quasi-Newton method tries to minimize the loss upon a

Original: In the L2O interpretation, this would be equivalent to unrolling the inner loop optimization for only one step, causing severe truncation bias; the optimizer learns to greedily optimize within too short of a time horizon, thus suffering in the long term (Metz et al., 2019).
Revised: In the L2O interpretation, this would be equivalent to unrolling the inner loop optimization for only one step, causing severe truncation bias; the optimizer learns to greedily optimize within too short of a time horizon, thus suffering in the long term (Metz et al., 2019).

Original: As a countermeasure, we take inspiration from the Momentum modification to SGD, and replace g t by (1 − β )
Revised: As a countermeasure, we take inspiration from the Momentum modification to SGD, and replace g t by (1 − β )

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: τ =0 β t − τ g τ before input into the step generation formula.
Revised: τ =0 β t − τ g τ before input into the step generation formula.

Original: This changes LODO’s step generation formula to x t +

Revised: This changes LODO’s step generation formula to x t +

Original: = x t − (1 − β ) G ( θ t ) (cid:80) ∞
Revised: = x t − (1 − β ) G ( θ t ) (cid:80) ∞

Original: τ =0 β t − τ g τ instead, for some decay parameter β .
Revised: τ =0 β t − τ g τ instead, for some decay parameter β .

Original: This results in
Revised: This results in

Original: LODO, summarized in Algorithm 1.
Revised: LODO, summarized in Algorithm 1.

Original: Our parameterization of G ( θ ) separates our work from other hypergradient methods; ours is inspired by the FFT-style Efficient Unitary Neural Network (EUNN)
Revised: Our parameterization of G ( θ ) separates our work from other hypergradient methods; ours is inspired by the FFT-style Efficient Unitary Neural Network (EUNN)

Original: (Jing et al., 2017) designed for parameter-efficient representations of unitary matrices.
Revised: (Jing et al., 2017) designed for parameter-efficient representations of unitary matrices.

Original: We replace the FFT matrices in the EUNN with fixed randomly chosen sparse matrices, and write G ( θ ) to be a large product of matrices:
Revised: We replace the FFT matrices in the EUNN with fixed randomly chosen sparse matrices, and write G ( θ ) to be a large product of matrices:

Original: [Equation]
Revised: [Equation]

Original: where α 0 is a fixed initial learning rate, P i are randomly selected permutation matrices, B
Revised: where α 0 is a fixed initial learning rate, P i are randomly selected permutation matrices, B

Original: ( θ ( i ) ) are block-diagonal matrices whose contents are listed by θ ( i )
Revised: ( θ ( i ) ) are block-diagonal matrices whose contents are listed by θ ( i )

Original: where θ =
Revised: where θ =

Original: ( θ (1) , . . .
Revised: ( θ (1) , . . .

Original: θ ( N ) ) , as illustrated in Figure 1.
Revised: θ ( N ) ) , as illustrated in Figure 1.

Original: Every block is size k × k for some chosen k (we use 4 for our setup), where the blocks’ contents are listed by θ .
Revised: Every block is size k × k for some chosen k (we use 4 for our setup), where the blocks’ contents are listed by θ .

Original: The ( I 0) and ( I 0)
Revised: The ( I 0) and ( I 0)

Original: T matrices are n × ˜ n and ˜ n × n respectively, and all other matrices are ˜ n × ˜ n , where ˜ n is some chosen integer larger than n .
Revised: T matrices are n × ˜ n and ˜ n × n respectively, and all other matrices are ˜ n × ˜ n , where ˜ n is some chosen integer larger than n .

Original: For our setup, we choose ˜ n =
Revised: For our setup, we choose ˜ n =

Original: ⌊ 2 n/k ⌋ k .
Revised: ⌊ 2 n/k ⌋ k .

Original: Though G depends on both α 0 and θ , we omit the α 0 in writing for brevity.
Revised: Though G depends on both α 0 and θ , we omit the α 0 in writing for brevity.

Original: By initializing each block matrix in B ( θ ( i ) ) to a random orthogonal matrix, we initialize G ( θ ) to a multiple of the identity, despite the input information diffusing and mixing with itself inside the layers of the network.
Revised: By initializing each block matrix in B ( θ ( i ) ) to a random orthogonal matrix, we initialize G ( θ ) to a multiple of the identity, despite the input information diffusing and mixing with itself inside the layers of the network.

Original: The initialization to the identity also allows us to make the depth N large without worrying about issues with gradient magnitudes, and for our setup we choose N = 16 .
Revised: The initialization to the identity also allows us to make the depth N large without worrying about issues with gradient magnitudes, and for our setup we choose N = 16 .

Original: The product G ( θ ) intentionally resembles the operation performed by a deep neural network with N layers, millions of hidden neurons arranged in inverted bottleneck style, very sparse connections, and no activation functions.
Revised: The product G ( θ ) intentionally resembles the operation performed by a deep neural network with N layers, millions of hidden neurons arranged in inverted bottleneck style, very sparse connections, and no activation functions.

Original: The random connections between neurons are intended to bring expander graph properties to the computation graph of the neural network, such that signals can diffuse and self-mix quickly without travelling long distances within the computation graph.
Revised: The random connections between neurons are intended to bring expander graph properties to the computation graph of the neural network, such that signals can diffuse and self-mix quickly without travelling long distances within the computation graph.

Original: Since the true inverse Hessian is guaranteed to be symmetric positive-definite, we chose G ( θ ) to have a form which guarantees it to be a positive-semidefinite symmetric matrix.
Revised: Since we wanted LODO to be repelled by saddle points, we chose G ( θ ) to produce a step which has guaranteed positive dot product with the gradient, so we parameterized G ( θ ) in a way which guarantees it to be a positive-semidefinite symmetric matrix.

Original: Furthermore, since any positivesemidefinite symmetric matrix can be eigendecomposed into UDU T =
Revised: Furthermore, since any positive-semidefinite symmetric matrix can be eigendecomposed into UDU T =

Original: ( UD 1 / 2 )( UD 1 / 2 ) T for unitary U and positive-semidefinite diagonal D , we chose G ( θ ) to represent the product of a matrix with its transpose.
Revised: ( UD 1 / 2 )( UD 1 / 2 ) T for unitary U and positivesemidefinite diagonal D , we chose G ( θ ) to represent the product of a matrix with its transpose.

Original: Since we expect the receptive field of output nodes to grow exponentially with depth, it takes
Revised: Since we expect the receptive field of output nodes to grow exponentially with depth, it takes

Original: O (log n ) layers for all of the n inputs to interact with each other, so we intend the depth N to be some multiple of log n .
Revised: O (log n ) layers for all of the n inputs to interact with each other, so we intend the depth N to be some multiple of log n .

Original: Applying permutations and block diagonal matrices both cost O ( n ) time, so the inverse Hessian estimate G ( θ ) costs O ( n log n ) time.
Revised: Applying permutations and block diagonal matrices both cost O ( n ) time, so the inverse Hessian estimate G ( θ ) costs O ( n log n ) time.

Original: This cost is small if the task is to train an architecture with much more parameter reuse than log n , such a CNN or transformer.
Revised: This cost is small if the task is to train an architecture with much more parameter reuse than log n , such a CNN or transformer.

Original: The memory LODO needs is O ( n log n ) since we have O ( n log n ) learnable parameters, though other
Revised: The memory LODO needs is O ( n log n ) since we have O ( n log n ) learnable parameters, though other

Original: L2O optimizers which cost O ( n ) may easily outstrip this cost via a large enough constant factor, for example by storing and processing a long history of past gradients.
Revised: L2O optimizers which cost O ( n ) may easily outstrip this cost via a large enough constant factor, for example by storing and processing a long history of past gradients.

Original:  T HEORETICAL P ROPERTIES OF LODO
Revised:  T HEORETICAL P ROPERTIES OF LODO

Original: In this section, we illustrate LODO’s behavior through its theoretical properties.
Revised: In this section, we illustrate LODO’s behavior through its theoretical properties.

Original: To make our analysis easier to show, this section only refers to LODO that is modified in the following ways: ( i ) We update θ t using SGD of learning rate α instead of Adam; and ( ii ) we feed the gradients directly into the G network rather than accumulating them into EMAs first.
Revised: To make our analysis easier to show, this section only refers to LODO that is modified in the following ways: ( i ) We update θ t using SGD of learning rate α instead of Adam; and ( ii ) we use no momentum, β = 0 .

Original: We present two results about desirable inverse Hessian learning dynamics and expressiveness, and in Appendix C, some further support from the literature for desirable properties of LODO’s inverse Hessian approximation error.
Revised: We present two results about desirable inverse Hessian learning dynamics and expressiveness, and in Appendix D, some further support from the literature for desirable properties of LODO’s inverse Hessian approximation error.

Original: 4. H ESSIAN L EARNING D YNAMICS (LODO C AN L EARN THE I NVERSE H ESSIAN )
Revised: 4. H ESSIAN L EARNING D YNAMICS (LODO C AN L EARN THE I NVERSE H ESSIAN )

Original: We first show that under certain conditions, LODO learns the  Hessian over time.
Revised: In this section, we show that with certain assumptions and approximations, G ( θ t ) converges to the inverse Hessian in a stochastic setting. We restrict our analysis to a version of LODO simplified from Algorithm 1, where the parameterization of G ( θ t ) ∈ R n × n is instead a rearrangement of the parameter vector θ t ∈ R n 2 and is not necessarily symmetric, though the experiment of Section 5. supports similar findings for the original version of LODO as well.

Original: We assume that the  problem setup is of a  quadratic bowl of fixed Hessian H and minimum x ∗ t which is perturbed by noise s t .
Revised: We also work in the limit of small learning rate α . The problem setup is of a noisily moving quadratic bowl: s t ∈ [Equation]

Original: s t is i.i.d.
Revised: n , [Equation]

Original: with respect to the step t , and  isfrom some light tailed distribution of zero mean, for example a multivariate normal distribution N ( s t ; 0 , Σ ) .
Revised: where H is a positive-definite symmetric Hessian matrix and S is some light tailed distribution of zero mean, for example a multivariate normal distribution N ( s t ; 0 , Σ ) .

Original: For this section, we restrict our analysis to a simplified version of LODO, though the experiment of Section 5.1 supports similar findings for the full version of LODO as well. In our simplified version, the inverse Hessian is approximated by a full dense matrix G ( θ t ) which is learned, so this result is relevant to many hypergradient methods and not just LODO.
Revised: Lastly, we assume that || I − G ( θ t ) H || 2 < 1 for all t , that G ( θ t ) never travels too far from H − 1 .

Original: Under these conditions, the gradient of loss with respect to both x t and θ t can be solved for so LODO turns out to follow the training dynamics
Revised: Under these conditions, the gradient of loss with respect to both x t and θ t can be solved for so LODO turns out to follow the training dynamics

Original: [Equation]
Revised: [Equation]

Original: where we define A t = I − G ( θ t ) H and b t
Revised: where we define A t = I − G ( θ t ) H and b t

Original: = x t − x ∗ t .
Revised: = x t − x ∗ t to be the errors in estimates of the inverse Hessian and the minimum, respectively.

Original: A full derivation is in Appendix A.1.
Revised: A full derivation is in Appendix A.1.

Original: Assuming that A t begins with spectral norm less than 1, then with small learning rate α , the b t  exponentially decays to a fixed distribution, and the long term dynamics of A t should not be greatly affected by the quick compensating movement of b t ’s distribution.
Revised: b t then exponentially decays to a fixed distribution, and the long term dynamics of A t should not be greatly affected by the quick compensating movement of b t ’s distribution.

Original: Thus, to determine the movement of A t , we can approximate the dynamics along a short trajectory by fixing A t to its value at t = tto study the settled distribution of b t .
Revised: Thus, to determine the movement of A t , we can approximate the dynamics along a short trajectory by fixing A t to its value at t = t 0 to study the settled distribution of b t .

Original: 1 Appendix A.2 gives more rigorous justification for this approximation—we keep the A t +1 update but rewrite the b t +1 update as
Revised: 1 Appendix A.2 gives more rigorous justification for this approximation— we keep the A t +1 update but rewrite the b t +1 update as

Original: [Equation]
Revised: [Equation]

Original: The total contribution of all s τ for t 0 ≤ τ ≤ t to the movement of A t in the infinite time horizon is then
Revised: The total contribution of all s τ for t 0 ≤ τ ≤ t to the movement of A t in the infinite time horizon is then

Original: [Equation]
Revised: [Equation]

Original: with some extra terms multiplied with pairwise products between s i and s j ( i ̸ = j ), and other terms with b t 0 .
Revised: with some extra terms multiplied with pairwise products between s i and s j ( i ̸ = j ), and other terms with b t 0 .

Original: In the long term, the average contribution of each s τ to the total then converges to
Revised: In the long term, the average contribution of each s τ to the total then converges to

Original: [Equation]
Revised: [Equation]

Original: by law of large numbers, where any pairwise products of s i and s j for i ̸
Revised: by law of large numbers, where any pairwise products of s i and s j for i ̸

Original: = j have no effect due to independence and the mean of s t being zero, and the effect of b t 0 vanishes due to small α .
Revised: = j have no effect due to independence and the mean of s t being zero, and the effect of b t 0 vanishes due to small α .

Original: The expected error for this convergence decays to zero like O (( t − t 0 ) − 1 / 2 ) due to the light tailedness of s τ (eg.
Revised: The expected error for this convergence decays to zero like O (( t − t 0 ) − 1 / 2 ) due to the light tailedness of s τ (eg.

Original: when s τ iid ∼ N ( s τ ; 0 , Σ ) ) leading to the central limit theorem.
Revised: when s τ iid ∼ N ( s τ ; 0 , Σ ) ) leading to the central limit theorem.

Original: Expression 5 is then the step direction for the Euler method for approximating the solution to the differential equation
Revised: Expression 6 is then the step direction for the Euler method for approximating the solution to the differential equation

Original: [Equation]
Revised: [Equation]

Original: which can be shown to cause A to flow towards zero such that G ( θ t ) converges to H − 1 .
Revised: which can be shown to cause A to flow towards zero such that G ( θ t ) converges to H − 1 .

Original: Therefore, it is reasonable to believe that LODO learns the inverse Hessian over time, given small enough learning rate.
Revised: Therefore, it is reasonable to believe that LODO learns the inverse Hessian over time, given small enough learning rate.

Original: The Frobenius norm of the error decays faster when magnitudes/norms of H and E (cid:2) ss T (cid:3) are higher, indicating that both curvature of the Hessian and noisy movement of the quadratic bowl’s minimum are good for learning the Hessian for fixed α , as long as the approximations mentioned stay accurate.
Revised: The Frobenius norm of the error decays faster when magnitudes/norms of H and E (cid:2) ss T (cid:3) are higher, indicating that both curvature of the Hessian and noisy movement of the quadratic bowl’s minimum are good for learning the Hessian for fixed α , as long as the approximations mentioned stay accurate.

Original: One interpretation of this is that when the noise s is zero in a certain direction, the error b in that direction quickly decays to zero so LODO no longer receives any data along that direction in order to learn that direction’s curvature.
Revised: One interpretation of this is that when the noise s is zero in a certain direction, the error b in that direction quickly decays to zero so LODO no longer receives any data along that direction in order to learn that direction’s curvature.

Original: Furthermore if the Hessian H does not penalize deviation in some direction, then the training signal for that direction vanishes.
Revised: Furthermore if the Hessian H does not penalize deviation in some direction, then the training signal for that direction vanishes.

Original: 4. R EPRESENTABILITY (LODO CAN BE E XPRESSIVE ) 
Revised: We may also produce another analysis for when H has a direction of negative curvature, to show that LODO repels saddle points. Here, we instead choose to parameterize G ( θ t ) as a positivedefinite symmetric matrix, and take the limit of small learning rate α such that G ( θ t ) is effectively fixed. Then, G ( θ t ) H has some negative eigenvalue λ with eigenvector y (see Appendix B for proof), such that A

Original: Here we motivate that for any linear operator H − 1 = ˜ F T ˜ F where ˜ F ∈ R ˜ n × n can be computed by a sufficiently small linear neural network (which we call the “reference” network), the linear neural network architecture of ˜ G ( θ  ) in LODO is capable of representing ˜ F with high probability simply by being O (log ˜ n ) times deeper than the reference network.
Revised: = I − G ( θ t ) H has an eigenvalue of norm greater than 1, meaning that the error vector b blows up exponentially in the y direction. Since G ( θ t ) Hy = λ y , left multiplying by y T G ( θ t ) − 1 further shows that y T Hy < 0 , implying that the direction of saddle point repulsion is one of negative curvature, which decreases the loss.

Revised: 4. R EPRESENTABILITY (LODO CAN BE E XPRESSIVE )

Original: It is thus capable of representing a large class ofpotential inverse Hessians from other hypergradient methods, including (Baydin et al., 2017) and (Amid et al., 2022) with  only O (log ˜ n ) times more parameters, and (Moskovitz et al., 2019) with only O (log 2 ˜ n ) times more parameters.
Revised: In this section, we seek to show that our parameterization of G ( θ ) is highly expressive; LODO may represent a wider range of inverse Hessians than many other methods. In fact, we show that with [Equation] an increase in parameter count by only a logarithmic factor, our fixed parameterization of G ( θ ) can reproduce any possible parameterization G ( θ ) =

Original: By sufficiently small, we mean that the reference neural network has depth d and at most ˜ n nonzero weights between consecutive layers.
Revised: ˜ F T ˜ F where vector multiplication with ˜ F is computable with some sparse linear neural network.

Original: Our specific contribution is  to state Theorem 1 following from Definition 1.
Revised: Our parameterization is thus capable of representing inverse Hessians from (Baydin et al., 2017) and (Amid et al., 2022) with only O (log ˜ n ) times more parameters, and (Moskovitz et al., 2019) with only O (log 2 ˜ n ) times more parameters. Definition 1 creates a function ϵ to characterize the mixing rate of random transpositions when trying to shuffle lists, while Theorem 1 uses this ϵ function to lower bound the probability that the ˜ G ( θ ) network in LODO can represent other linear neural networks.

Original: Definition 1.
Revised: Definition 1.

Original: We must first define a function ϵ : N × N → R . We uniformly sample an infinite sequence of  transpositions of two  of ˜ n elements,  with the condition that every successive block of ˜ n/ 2∈ N transpositions commutes internally (transpositions can be rearranged within a block).
Revised: Uniformly sample a sequence of N ˜ n/ 2 transpositions of two out of ˜ n elements, for integers ˜ n/ 2 ∈ N and N ∈ N , with the condition that every successive block of ˜ n/ 2 transpositions commutes internally (transpositions can be rearranged within a block).

Original: We replace each transposition with the identity operation with probability 1 / 2 , and then compose the first N ˜ n/ 2 transpositions/identities to form a permutation.
Revised: We replace each transposition with the identity operation with probability 1 / 2 , and then compose the sequence of transpositions/identities to form a permutation.

Original: Let the expected entropy of this permutation, given the original sequence but not given the indices which were replaced with identities, be log ˜ n !− ϵ ( N ˜ n/ 2 , ˜ n ) for some function ϵ of N ˜ n/ 2 and ˜ n . [Equation]
Revised: Then, we define the function ϵ ( N ˜ n/ 2 , ˜ n ) such that the expected entropy of this permutation, given the original sequence but not given the locations where identities were placed, is log ˜ n !

Original: [Equation]
Revised: − ϵ ( N ˜ n/ 2 , ˜ n ) .

Original: Theorem 1.
Revised: Theorem 1.

Revised: Uniformly sample permutations P i and create block-diagonal matrices B ( θ ( i ) ) where every block is 2 × 2 , and whose block contents are listed by the parameters θ ( i ) .

Original: If we randomly sample a ˜ G ( θ )∈ R ˜ n × ˜ n network asdefined in Section 3 with  depth 
Revised: Use these to construct the LODO subnetwork ˜ G ( θ ) as in Equation 1 with some depth N and hidden dimension ˜ n .

Original: N = 2 Md ( ⌈ log 2 k ⌉ +1) for fixed constants k , d , and M , ˜ n neurons per layer, and block size f = 2 ,  there is a probability of at least
Revised: Construct any linear neural network ˜ F with input dimension, output dimension, and number of weights per layer at most ˜ n , at most k incoming and at most k outgoing weights for every neuron, depth d , and any arrangement of weights. Then, there is a probability of at least

Original: [Equation]
Revised: [Equation]

Original: that  ˜ G ( θ ) can represent every linear neural network ˜ Fof input and output dimension ≤ ˜ n , depth d , ≤ ˜ n nonzero weights per layer, and ≤ k incoming and ≤ k outgoing weights for every neuron.
Revised: that we can make ˜ G ( θ ) = ˜ F for some θ .

Original: The proof is in Appendix B.
Revised: The proof is in Appendix C.

Original: Since the Cayley graph over the symmetric group generated by all transpositions has good expansion properties (Konstantinova & Kravchuk, 2022), we believe it is reasonable that with large M and ˜ n , we have ϵ ( M ˜ n/ 2 , ˜ n )
Revised: We believe that random transpositions in the style of Definition 1 are a quick way to shuffle, since the Cayley graph over the symmetric group generated by all transpositions has good expansion properties (Konstantinova & Kravchuk, 2022). In other words, we hypothesize that for large N and ˜ n , we have ϵ ( N ˜ n/ 2 , ˜ n )

Original: ≈ c 1 ˜ ne − c 2 M ˜ n/ 2 for some positive constants c 1 and c 2 .
Revised: ≈ c 1 ˜ ne − c 2 N ˜ n/ 2 for some positive constants c 1 and c 2 .

Original: This would imply that the probability that ˜ G ( θ ) can represent all possible ˜ F is at least approximately
Revised: This would imply that the probability that ˜ G ( θ ) can represent all possible ˜ F is at least approximately

Original: [Equation]
Revised: [Equation]

Original: which can be made to be above 1 − ( n !)
Revised: which can be made to be above 1 − ( n !)

Original: c for any constant c by using sufficient depth N which goes like N ∝ d (log 2 k ) log ˜ n , due to Stirling’s approximation.
Revised: c for any constant c by using sufficient depth N which goes like N ∝ d (log 2 k ) log ˜ n , due to Stirling’s approximation.

Original: Thus we believe that by only being
Revised: Thus we believe that by only being

Original: O ((log 2 k ) log ˜ n ) times deeper thanthe reference neural network ˜ F , we can make it very likely that our model ˜ G ( θ ) can represent all possible reference networks, in the sense of Equation (11).
Revised: O ((log 2 k ) log ˜ n ) times deeper than ˜ F , we can make it very likely that our model ˜ G ( θ ) can represent all possible ˜ F .

Original:  E XPERIMENTS WITH LODO
Revised:  E XPERIMENTS WITH LODO

Original: We present here a number of tasks which provide experimental evidence for the theoretical results we claim, though we test a variety of optimizers on these tasks.
Revised: We present here a number of tasks which provide experimental evidence for the theoretical results we claim, though we test a variety of optimizers on these tasks.

Revised: Appendix G presents an additional experiment on the Rosenbrock function.

Original: Appendix D explains how we tuned each optimizer’s hyperparameters for each task.
Revised: Appendix E explains how we tuned each optimizer’s hyperparameters for each task.

Original: Optimizers were used 8 times for every experiment to ensure reproducibility of results, unless otherwise stated.
Revised: Optimizers were used 8 times for every experiment to ensure reproducibility of results, unless otherwise stated.

Original: 5. N OISY Q UADRATIC B OWL
Revised: 5. N OISY Q UADRATIC B OWL

Original: We use various optimizers to track the minimum of a quadratic bowl of fixed true Hessian H as its minimum is perturbed by noise at every step, to demonstrate that LODO correctly learns its inverse
Revised: We use various optimizers to track the minimum of a quadratic bowl of fixed true Hessian H as its minimum is perturbed by noise at every step, to demonstrate that LODO correctly learns its inverse

Original: Hessian representation as claimed in Section 4.1.
Revised: Hessian representation as claimed in Section 4.1.

Original: The setup of the noisy quadratic bowl is fully described in Appendix E.1.
Revised: The setup of the noisy quadratic bowl is the same as in Section 4.1 and details are provided in Appendix F.1.

Original: We interpret an optimizer to be better if it can maintain a lower loss in the infinite step limit, since the error builds at a constant rate over time and the optimizer’s role is to react and correct it as quickly as possible.
Revised: We interpret an optimizer to be better if it can maintain a lower loss in the infinite step limit, since the error builds at a constant rate over time and the optimizer’s role is to react and correct it as quickly as possible.

Original: We tested each optimizer by using it to track the minimum of the moving quadratic bowl over 100k steps.
Revised: We tested each optimizer by using it to track the minimum of the moving quadratic bowl over 100k steps.

Original: Learning curves in Figure 3 and losses in Table 1 show that LODO tracks the minimum more accurately than all other optimizers, and that an estimator of the inverse Hessian approximation error ||
Revised: Learning curves in Figure 2 and losses in Table 5 of Appendix F.1 show that LODO tracks the minimum more accurately than all other optimizers, and that an estimator of the inverse Hessian approximation error ||

Original: I − G ( θ t )
Revised: I − G ( θ t )

Original: H
Revised: H

Original: || 2 F /n decays over time.
Revised: || 2 F /n decays over time.

Original: 5. R OSENBROCK F UNCTION

Original: [Figure]

Original: We probe the behavior of LODO with a small test task of finding the minimum of a rescaled Rosenbrock function f ( x, y )

Original: =

Original: [-01( x − 1) 
 + ( x 2 − y ) 2 , which has no local minima-] and one global minimum at ( x, y ) = We initialized the optimizers at ( x, y ) = ( − 0 .
Revised: [-01( x − 1) 
 + ( x 2 − y ) 2 , which has no local minima-]Other optimizers underperform because their preconditioners are less expressive, being either diagonal or low-rank and thus unable to capture the off-diagonal elements or non-top-few singular values of the true inverse Hessian, leading to a higher loss.

Original: and gave them 200 steps to run.

Original: The trajectory taken by LODO, shown in Figure 2, is similar to the short timescale dynamics of other optimizers using momentum, in that it tends to overshoot and then correct itself in an oscillatory manner.

Original: Learning curves in Figure 9 and losses in Tableof Appendix F shows the performance of all the optimizers on this task.

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original:  I MAGE G ENERATION
Revised: 5. I MAGE G ENERATION

Original: We design a challenge for LODO—an autoregressive image generator for MNIST.
Revised: We design a challenge for LODO—an autoregressive image generator for MNIST.

Original: We intend to demonstrate the effectiveness of LODO at scale when used to train a semi-realistic neural network.
Revised: We intend to demonstrate the effectiveness of LODO at scale when used to train a semi-realistic neural network.

Original: The task is similar to training a PixelCNN (Oord et al., 2016) to generate MNIST images (Lecun et al., 1998), and is fully described in Appendix E.2.
Revised: The task is similar to training a PixelCNN (Oord et al., 2016) to generate MNIST images (Lecun et al., 1998), and is fully described in Appendix F.2.

Original: We tested LODO alongside a variety optimizers for 300k steps, though we could not get any quasi-Newton methods to converge on this task.
Revised: We tested LODO alongside a variety optimizers for 300k steps, though we could not get any quasi-Newton methods to converge on this task.

Original: Learning curves in Figures 4 and losses in Table 2 show that LODO trains at a speed that is competitive against other optimizers.
Revised: Learning curves in Figures 3 and losses in Table 1 show that LODO trains at a speed that is competitive against other optimizers.

Original: Figure 8 in Appendix E.2 shows some MNIST digits we generated.
Revised: Figure 7 in Appendix F.2 shows some MNIST digits we generated [Table] [Figure] [Table]

Original: D ECONSTRUCTING LODO (A BLATIONS ON I MAGE G ENERATION T ASK ) 6. E IGENDECOMPOSED A LTERNATE P ARAMETERIZATION
Revised: We study the contributions of various components of LODO to its performance by replacing components to observe a performance change, as detailed below. For example, we replace the Adam meta-optimizer with SGD to form a new version of LODO (which we call “LODO-SGD”); its performance is shown in Table 2. Other modifications are listed below. 5.2. E IGENDECOMPOSED A LTERNATE P ARAMETERIZATION

Original: We would like to show that architecture LODO uses to choose the step is flexible.
Revised: We would like to show that architecture LODO uses to choose the step is flexible.

Original: We modify the matrix decomposition in Section 3 to G ( θ ) =
Revised: We modify the matrix decomposition in Section 3 to G ( θ ) =

Original: α 

Revised: α

Original: ( I 0)
Revised: ( I 0)

Original: ˜ G ( θ ) T D ˜ G
Revised: ˜ G ( θ ) T D ˜ G

Original: ( θ ) ( I 0) T by adding learned diagonal matrix D in the middle and changing ˜ G ( θ ) to a product of many weighted permutations [Table]
Revised: ( θ ) ( I 0) T by adding learned diagonal matrix D in the middle and changing ˜ G ( θ ) to a product of many weighted permutations each added to the identity matrix.

Original: [Figure] each added to identity ( θ ) connections
Revised: The neural network which ˜ G ( θ ) represents now has residual connections, and the initialization is modified accordingly.

Original: Losses  Table 3 show  this version (which we call “LODO-Residuals”) performs only slightly worse than LODO for the [Table]
Revised: Losses in Table 2 show that this version (which we call “LODO-Residuals”) performs only slightly worse than LODO for the same number of steps, reflecting the flexibility in the design of LODO.

Original: 6. S IMPLER A PPROXIMATE H ESSIANS
Revised: 5.2. S IMPLER A PPROXIMATE H ESSIANS

Original: We presumed that the representability result of Section 4.2 is only useful because LODO’s strength comes from the flexibility that G ( θ ) gives in configuring pairwise interactions between parameters.
Revised: We presumed that the representability result of Section 4.2 is only useful because LODO’s strength comes from the flexibility that G ( θ ) gives in configuring pairwise interactions between parameters.

Original: We therefore expect that using a simpler form of Hessian should hurt the performance of LODO.
Revised: We therefore expect that using a simpler form of Hessian should hurt the performance of LODO.

Original: We test two simpler forms of G ( θ ) : G ( θ )
Revised: We test two simpler forms of G ( θ ) : G ( θ )

Original: =
Revised: =

Original: α 0 diag ( θ ) (which we call “LODO-Diagonal”) for θ ∈ R n initialized to a vector of ones, similar to (Amid et al., 2022)—and the even simpler G ( θ ) =
Revised: α 0 diag ( θ ) (which we call “LODO-Diagonal”) for θ ∈ R n initialized to a vector of ones, similar to (Amid et al., 2022)—and the even simpler G ( θ ) =

Original: α 0 θ
Revised: α 0 θ

Original: I (which we call “LODO-Global”) for θ ∈ R initialized to 1, as in (Baydin et al., 2017).
Revised: I (which we call “LODO-Global”) for θ ∈ R initialized to 1, as in (Baydin et al., 2017).

Original: Losses in
Revised: Losses in

Original: Table 3 show that the original version of LODO performs the best, verifying our hypothesis.
Revised: Table 2 show that the original version of LODO performs the best, verifying our hypothesis.

Original: 6. E FFECTS OF U SING EMA S OF G RADIENTS
Revised: 5.2. E FFECTS OF U SING EMA S OF G RADIENTS

Original: [Figure]

Original: Similarly to how momentum works for SGD, LODO’s input gradients are preproccessed by accumulation into EMAs.
Revised: Similarly to how momentum works for SGD, LODO’s input gradients are preproccessed by accumulation into EMAs.

Original: To test our claim in Section 3 that momentum benefits LODO, we try 8 separate momentum decay rates in a logarithmic grid from no momentum to the optimal amount of momentum found ( β = 0 . 9343 ), and test each decay rate once.
Revised: To test our claim in Section 3 that momentum benefits LODO, we tryseparate momentum decay rates in a logarithmic grid from no momentum to the optimal amount of momentum found ( β = 0 . 9343 ), and test each decay rate once.

Original: Figure 5 shows a clear trend that at least up to the optimal decay rate, increasing the effect of momentum improves LODO.
Revised: Figure 4 shows a clear trend that at least up to the optimal decay rate, increasing the effect of momentum improves LODO.

Revised: We also try removing momentum completely (we call the modified version “LODO-No-Momentum”); results are shown in Table 2.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Revised: [Figure]

Revised: tum improves LODO

Revised: Error

Original:  LODO is a middle point between L2O methods and quasi-Newton methods, retaining significant advantages of both classes of optimizers.
Revised: LODO’s uncertainty Table LODO is a middle point between L2O methods and quasi-Newton methods, retaining significant advantages of both classes of optimizers.

Original: Via LODO, we bring ideas from both classes of optimization methods to offer potential solutions to problems common on the other side.
Revised: Via LODO, we bring ideas from both classes of optimization methods to offer potential solutions to problems common on the other side.

Original: Relative to quasi-Newton methods, LODO offers advantages associated with the use of a metaoptimizer on a neural optimizer.
Revised: Relative to quasi-Newton methods, LODO offers advantages associated with the use of a meta-optimizer on a neural optimizer.

Original: Crucially, LODO determines its inverse Hessian estimate using all past gradients, whereas most other quasi-Newton methods use a finite history of them.
Revised: Crucially, LODO determines its inverse Hessian estimate using all past gradients, whereas most other quasi-Newton methods use a finite history of them.

Original: This allows LODO to retain information about the inverse Hessian for much longer than other methods.
Revised: This allows LODO to retain information about the inverse Hessian for much longer than other methods.

Original: This is useful if the gradients contain enough noise that useful signals can only be obtained by accumulating information from many gradient samples.
Revised: This is useful if the gradients contain enough noise that useful signals can only be obtained by accumulating information from many gradient samples.

Original: Our theory further shows that the linear neural network in LODO is optimal to a certain extent: it can probably represent all linear neural networks smaller by a logarithmic factor—allowing a huge class of inverse Hessians.
Revised: Our theory further shows that the linear neural network in LODO is optimal to a certain extent: it can probably represent all linear neural networks smaller by a logarithmic factor— allowing a huge class of inverse Hessians.

Original: Our image generation task demonstrates that LODO succeeds in a semi-realistic stochastic nonconvex task where other quasiNewton optimizers diverge.
Revised: Our image generation task demonstrates that LODO succeeds in a semi-realistic stochastic nonconvex task where other quasi-Newton optimizers diverge.

Original: Due to our use of L2O, LODO also has flexibility in the design of its linear neural network, which makes it amenable to further research and refinement.
Revised: Due to our use of L2O, LODO also has flexibility in the design of its linear neural network, which makes it amenable to further research and refinement.

Original: Relative to L2O, LODO offers advantages associated with the restructuring of the outer and inner loop into a single loop.
Revised: Relative to L2O, LODO offers advantages associated with the restructuring of the outer and inner loop into a single loop.

Original: Most importantly, our modification to L2O alleviates the requirement for meta-training time and the training task distribution.
Revised: Most importantly, our modification to L2O alleviates the requirement for meta-training time and the training task distribution.

Original: This is at the cost of increased inner loop unrolling truncation bias, but it takes advantage of this sacrifice in resolving the need to compute second-order gradients.
Revised: This is at the cost of increased inner loop unrolling truncation bias, but it takes advantage of this sacrifice in resolving the need to compute second-order gradients.

Original: LODO still inherits issues of high memory usage and slow step computation from L2O methodology though.
Revised: LODO still inherits issues of high memory usage and slow step computation from L2O methodology though.

Original: Our theory offers some understanding of how LODO learns to optimize, which is rare for L2O methods: the Hessian approximation error decays as learning progresses.
Revised: Our theory offers some understanding of how LODO learns to optimize, which is rare for L2O methods: the Hessian approximation error decays as learning progresses.

Original: We import the idea from quasi-Newton methods that the gradient of one parameter can affect the step for another, which comes from the presence of off-diagonal elements in the Hessian.
Revised: We import the idea from quasi-Newton methods that the gradient of one parameter can affect the step for another, which comes from the presence of off-diagonal elements in the Hessian.

Original: As shown in Section 4.2, LODO presents an efficient way of approximating subsets of the O ( n 2 ) possible pairwise parameter interactions in O ( n log n ) time.
Revised: As shown in Section 4.2, LODO presents an efficient way of approximating subsets of the O ( n 2 ) possible pairwise parameter interactions in O ( n log n ) time.

Original: Such interactions are commonly ignored in the design of L2O and more mainstream optimizers, yet our image generation task demonstrates their importance, as evidenced by the improved performance of LODO over ablated versions as well as SGD.
Revised: Such interactions are commonly ignored in the design of L2O and more mainstream optimizers, yet our image generation task demonstrates their importance, as evidenced by the improved performance of LODO over ablated versions as well as SGD.

Original: Conclusion Through LODO, we provide a new way of using L2O methods online without any meta-training to perform quasi-Newton optimization.
Revised: Conclusion Through LODO, we provide a new way of using L2O methods online without any meta-training to perform quasi-Newton optimization.

Original: We introduce the strengths and advantages of quasi-Newton methods and L2O to each other and combine them in a harmonious manner.
Revised: We introduce the strengths and advantages of quasi-Newton methods and L2O to each other and combine them in a harmonious manner.

Original: LODO’s abilities showcase the applicability of online L2O methods with nested optimizers to the training of modern neural networks.
Revised: LODO’s abilities showcase the applicability of online L2O methods with nested optimizers to the training of modern neural networks.

Original: Our unique methodology serves as a stepping stone for the further development and use of L2O in quasi-Newton optimization and vice versa.
Revised: Our unique methodology serves as a stepping stone for the further development and use of L2O in quasi-Newton optimization and vice versa.

Original:  R EPRODUCIBILITY OF R ESULTS
Revised:  R EPRODUCIBILITY OF R ESULTS

Original: We have described all the details of the construction of our optimizer in Section 3, and all its modified versions in Section 6.
Revised: We have described all the details of the construction of our optimizer in Section 3, and all its modified versions in Section 5.2.

Original: All the tasks used for experiments are introduced in Section 5 and are fully described in Appendices E.1 and E.2.
Revised: All the tasks used for experiments are introduced in Section 5 and are fully described in Appendices F.1 and F.2.

Original: To further ensure reproducibility, we have also listed all the hyperparameters found for each optimizer and task in Appendix D. All our training runs were repeated 8 times to verify that each measurement of empirical behavior is consistent, with low standard deviation.
Revised: To further ensure reproducibility, we have also listed all the hyperparameters found for each optimizer and task in Appendix E. All our training runs were repeated 8 times to verify that each measurement of empirical behavior is consistent, with low standard deviation.

Original: Furthermore, we will release the code for all our optimizers and experiments on
Revised: Furthermore, we will release the code for all our optimizers and experiments on

Original: Github for others to see.
Revised: Github for others to see.

Original: A E LABORATION ON H ESSIAN L EARNING D YNAMICS A. D ERIVATION OF T RAINING D YNAMICS
Revised: A E LABORATION ON H ESSIAN L EARNING D YNAMICS A. D ERIVATION OF T RAINING D YNAMICS

Original: This section gives a derivation of the result that under the problem setup of Section 4.1, LODO follows the Hessian learning dynamics
Revised: This section gives a derivation of the result that under the problem setup of Section 4.1, LODO follows the Hessian learning dynamics

Original: [Equation]
Revised: [Equation]

Original: where A t =
Revised: where A t =

Original: I − G ( θ t ) H and b t
Revised: I − G ( θ t ) H and b t

Original: = x t − x ∗ t as long as G ( θ t ) is parameterized as a dense matrix filled with elements of θ t , and no momentum is used.
Revised: = x t − x ∗ t as long as G ( θ t ) is parameterized as a dense matrix filled with elements of θ t , and no momentum is used.

Original: We first let b t be x t − x ∗ t .
Revised: We first let b t be x t − x ∗ t .

Original: The loss at time t is then
Revised: The loss at time t is then

Original: [Equation]
Revised: [Equation]

Original: The gradient is then computed to be
Revised: The gradient is then computed to be

Original: [Equation]
Revised: [Equation]

Original: The step taken then produces the next parameters:
Revised: The step taken then produces the next parameters:

Original: [Equation]
Revised: [Equation]

Original: Subtracting x ∗ t +1 = x ∗ t + s t , we get the recurrence for b t ,
Revised: Subtracting x ∗ t +1 = x ∗ t + s t , we get the recurrence for b t ,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The loss at time t + 1 is computed to be
Revised: The loss at time t + 1 is computed to be

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: LODO also computes a step of θ t using the loss on the next step.
Revised: LODO also computes a step of θ t using the loss on the next step.

Original: Since the elements of θ t are just a rearrangement of the elements of G ( θ t ) in our derivation, an update of θ t can be treated instead like an update of G ( θ t ) .
Revised: Since the elements of θ t are just a rearrangement of the elements of G ( θ t ) in our derivation, an update of θ t can be treated instead like an update of G ( θ t ) .

Original: The gradient of ℓ t +1 with respect to G ( θ t ) is then computed to be
Revised: The gradient of ℓ t +1 with respect to G ( θ t ) is then computed to be

Original: [Equation]
Revised: [Equation]

Original: and the step of G ( θ t ) is
Revised: and the step of G ( θ t ) is

Original: [Equation]
Revised: [Equation]

Original: resulting in the recurrence for A t :
Revised: resulting in the recurrence for A t :

Original: [Equation]
Revised: [Equation]

Original: A. V ALIDITY OF A PPROXIMATION A RGUMENT
Revised: A. V ALIDITY OF A PPROXIMATION A RGUMENT

Original: This section gives justification for the approximation in Section 4.1 of the long term trajectory of the recurrence
Revised: This section gives justification for the approximation in Section 4.1 of the long term trajectory of the recurrence

Original: [Equation]
Revised: [Equation]

Original: by replacing with
Revised: by replacing with

Original: [Equation]
Revised: [Equation]

Original: when α is small and the initial conditions at t 0 are the same: A t 

Revised: when α is small and the initial conditions at t 0 are the same: A t

Original: =
Revised: =

Original: A ′ t 0 and b t 0 = b ′ t 0 = 0 .
Revised: A ′ t 0 and b t 0 = b ′ t 0 = 0 .

Original: We will work in the bounded noise case ||
Revised: We will work in the bounded noise case ||

Original: s t || 2 < ∞ , where || b ′ t
Revised: s t || 2 < ∞ , where || b ′ t

Original: || 2 is upper bounded by some ||
Revised: || 2 is upper bounded by some ||

Original: A ′ t 0 ||dependent constant b max due to exponential decay in Equation 30.
Revised: A ′ t 0 ||dependent constant b max due to exponential decay in Equation 31.

Original: In the case where the noise is not bounded, a probabilistic analysis can be done instead, though we do not provide one.
Revised: In the case where the noise is not bounded, a probabilistic analysis can be done instead, though we do not provide one.

Original: To justify this approximation, we prove that the spectral norm of long term deviation corrected for learning rate is small over short distances r , in the following theorem:
Revised: To justify this approximation, we prove that the spectral norm of long term deviation corrected for learning rate is small over short distances r , in the following theorem:

Original: Theorem 2.
Revised: Theorem 2.

Original: [Equation]
Revised: [Equation]

Original: In other words, the local movement of A rescaled for learning rate is unaffected by our approximation when the learning rate α is small.
Revised: In other words, the local movement of A rescaled for learning rate is unaffected by our approximation when the learning rate α is small.

Original: Proof.
Revised: Proof.

Original: Our proof strategy is as follows:
Revised: Our proof strategy is as follows:

Original: We will first define variables to denote bounds on the movement of A and the approximation error in b .
Revised: We will first define variables to denote bounds on the movement of A and the approximation error in b .

Original: We will show that these variables bound each other, and then we will combine these bounds to create a single recursive bound on the movement of A . 3.
Revised: We will show that these variables bound each other, and then we will combine these bounds to create a single recursive bound on the movement of A . 3.

Original: We will characterize the bound’s growth and it will turn out that A has a maximum movement speed along any trajectory of sufficiently short length.
Revised: We will characterize the bound’s growth and it will turn out that A has a maximum movement speed along any trajectory of sufficiently short length.

Original: Due to the slow movement of A , we can deduce that the approximation error in b increases at a bounded rate.
Revised: Due to the slow movement of A , we can deduce that the approximation error in b increases at a bounded rate.

Original: Since approximation errors in A are an accumulation of errors in b , we will show that deviation between the true and approximate A trajectories is quadratic in the distance along the trajectory.
Revised: Since approximation errors in A are an accumulation of errors in b , we will show that deviation between the true and approximate A trajectories is quadratic in the distance along the trajectory.

Original: We conclude that the approximation error vanishes for short trajectories and small learning rates.
Revised: We conclude that the approximation error vanishes for short trajectories and small learning rates.

Original: First part.
Revised: First part.

Original: We first define the maximum drift in A
Revised: We first define the maximum drift in A

Original: [Equation]
Revised: [Equation]

Original: up to time difference ∆ t for 0 ≤ ∆ t ≤ R/α for some chosen small constant trajectory length
Revised: up to time difference ∆ t for 0 ≤ ∆ t ≤ R/α for some chosen small constant trajectory length

Original: R > 0 .
Revised: R > 0 .

Original: We will pick R later.
Revised: We will pick R later.

Original: We will also define the maximum error in b
Revised: We will also define the maximum error in b

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: up to the same time.
Revised: up to the same time.

Original: Second part.
Revised: Second part.

Original: For the bound in one direction, we have that for all τ such that t 0 ≤ τ ≤ t
Revised: For the bound in one direction, we have that for all τ such that t 0 ≤ τ ≤ t

Original: [Equation]
Revised: [Equation]

Original: ∆ t ,
Revised: ∆ t ,

Original: [Equation]
Revised: [Equation]

Original: using the triangle inequality and sub-multiplicativity for the spectral norm ||·|| 2 .
Revised: using the triangle inequality and sub-multiplicativity for the spectral norm ||·|| 2 .

Original: This is a recurrence in || b τ − b ′ τ || 2 ; by induction we have that for t 0 ≤ τ ≤ t
Revised: This is a recurrence in || b τ − b ′ τ || 2 ; by induction we have that for t 0 ≤ τ ≤ t

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: such that we produce the bound
Revised: such that we produce the bound

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we show a reverse bound: for all τ such that t 0 ≤ τ ≤ t 0 + ∆ t , we have
Revised: we show a reverse bound: for all τ such that t 0 ≤ τ ≤ t 0 + ∆ t , we have

Original: [Equation]
Revised: [Equation]

Original: By induction we have for t 0 ≤ τ ≤ t 0 + ∆ t + 1 ,
Revised: By induction we have for t 0 ≤ τ ≤ t 0 + ∆ t + 1 ,

Original: [Equation]
Revised: [Equation]

Original:  such that we produce the reverse bound
Revised: 50) such that we produce the reverse bound

Original: [Equation]
Revised: [Equation]

Original: Third part.
Revised: Third part.

Original: Substituting the bound in Equation 44 into the bound in Equation 50, we produce the recurrence
Revised: Substituting the bound in Equation 45 into the bound in Equation 51, we produce the recurrence

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: To bound the movement of A , we must use the fact
Revised: To bound the movement of A , we must use the fact

Original: that when
Revised: that when

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: the function f maps the interval
Revised: the function f maps the interval

Original: [Equation]
Revised: [Equation]

Original: to a subset of itself.
Revised: to a subset of itself.

Original: Since at ∆ t = 

Revised: Since at ∆ t =

Original: we have ϵ A ,t 0 +∆ t = 0 ∈
Revised: we have ϵ A ,t 0 +∆ t = 0 ∈

Original: I ∆ t , and we also have I ∆ t ⊆
Revised: I ∆ t , and we also have I ∆ t ⊆

Original: I ∆ t +1 , we may deduce by induction on ∆ t that ϵ A ,t 0 +∆ t ∈
Revised: I ∆ t +1 , we may deduce by induction on ∆ t that ϵ A ,t 0 +∆ t ∈

Original: I ∆ t as long as Equation 55 holds, and thus there is a bound
Revised: I ∆ t as long as Equation 56 holds, and thus there is a bound

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: on the movement speed of A as long as Equation 55 holds.
Revised: on the movement speed of A as long as Equation 56 holds.

Original: Fourth part.
Revised: Fourth part.

Original: Note that we have assumed that 0 ≤ ∆ t ≤ R/α for some constant R which we have not yet picked.
Revised: Note that we have assumed that 0 ≤ ∆ t ≤ R/α for some constant R which we have not yet picked.

Original: By choosing
Revised: By choosing

Original: [Equation]
Revised: [Equation]

Original: we may always guarantee Equation 55, which implies Equation 57.
Revised: we may always guarantee Equation 56, which implies Equation 58.

Original: Then when Equation 57 is substituted into Equation 44, we create a small bound on the approximation error in b which begins at zero and increases with time,
Revised: Then when Equation 58 is substituted into Equation 45, we create a small bound on the approximation error in b which begins at zero and increases with time,

Original: [Equation]
Revised: [Equation]

Original: for 0 ≤ ∆ t ≤ R/α .
Revised: for 0 ≤ ∆ t ≤ R/α .

Original: This also holds trivially for ∆ t = − 1 = ⇒ ϵ b ,t 

Revised: This also holds trivially for ∆ t = − 1 = ⇒ ϵ b ,t

Original: +∆ t +1 = 0 , so we may re-index to have
Revised: +∆ t +1 = 0 , so we may re-index to have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: for 0 ≤ ∆ t ≤ R/α + 1 .
Revised: for 0 ≤ ∆ t ≤ R/α + 1 .

Original: Since the right side of Equation 61 is convex in ∆ t over ∆ t ∈ [0 , R/α ] , we may bound by a linear function with the same endpoints
Revised: Since the right side of Equation 62 is convex in ∆ t over ∆ t ∈ [0 , R/α ] , we may bound by a linear function with the same endpoints

Original: [Equation]
Revised: [Equation]

Original: for 0 ≤ ∆ t ≤ R/α .
Revised: for 0 ≤ ∆ t ≤ R/α .

Original: Fifth part.
Revised: Fifth part.

Original: Finally, we use this bound on approximation error in b to bound approximation error in A .
Revised: Finally [Equation] we use this bound on approximation error in b to bound approximation error in A .

Original: [Equation]
Revised: [Equation]

Original: By induction, we find that the approximation error of A is quadratic in time for short times 0 ≤ ∆ t ≤ R/α ,
Revised: By induction, we find that the approximation error of A is quadratic in time for short times 0 ≤ ∆ t ≤ R/α ,

Original: [Equation]
Revised: [Equation]

Original: Sixth part.
Revised: Sixth part.

Original: Now take ∆ t =
Revised: Now take ∆ t =

Original: ⌊ r/α ⌋ , which for r → 0 is eventually ≤ R/α as required.
Revised: ⌊ r/α ⌋ , which for r → 0 is eventually ≤ R/α as required.

Original: As we sought to prove, the learning rate rescaled approximation error of the local drift direction and speed goes to zero:
Revised: As we sought to prove [Equation] the learning rate rescaled approximation error of the local drift direction and speed goes to zero:

Original: [Equation]
Revised: [Equation]

Revised: B P ROOF THAT G ( θ t ) H HAS A N EGATIVE E IGENVALUE

Revised: This is true, because we can substitute A = G ( θ t ) and B = H in following lemma:

Revised: Lemma 1.

Revised: Let us A and B be symmetric, full-rank n × n matrices.

Revised: Let A be positive-definite and B have at least one negative eigenvalue.

Revised: Then, the product AB has at least one negative eigenvalue.

Revised: Proof.

Revised: Let x be an eigenvector of B with negative eigenvalue λ .

Revised: Then, we have

Revised: [Equation]

Revised: [Equation]

Revised: meaning that the positive-definite symmetric matrix A 1 / 2 BA 1 / 2 must have at least one negative eigenvalue λ ′ with eigenvector x ′ .

Revised: Then, we have the eigenvalue equation

Revised: [Equation]

Revised: which shows that AB has a negative eigenvalue λ ′ .

Original: B P ROOF OF R EPRESENTABILITY T HEOREM
Revised: C P ROOF OF R EPRESENTABILITY T HEOREM

Original: This section gives a proof of the representability theorem stated in Section 4.2:
Revised: This section gives a proof of the representability theorem stated in Section 4.2:

Original: Theorem 1.
Revised: Theorem 1.

Revised: Uniformly sample permutations P i and create block-diagonal matrices B ( θ ( i ) ) where every block is 2 × 2 , and whose block contents are listed by the parameters θ ( i ) .

Original: If we randomly sample a ˜ G ( θ )∈ R ˜ n × ˜ n network asdefined in Section 3 with  depth 
Revised: Use these to construct the LODO subnetwork ˜ G ( θ ) as in Equation 1 with some depth N and hidden dimension ˜ n .

Original: N = 2 Md ( ⌈ log 2 k ⌉ +1) for fixed constants k , d , and M , ˜ n neurons per layer, and block size f = 2 ,  there is a probability of at least[Equation]
Revised: Construct any linear neural network ˜ F with input dimension, output dimension, and number of weights per layer at most ˜ n , at most k incoming and at most k outgoing weights for every neuron, depth d , and any arrangement of weights. Then, there is a probability of at least

Original: [Equation]
Revised: [Equation]

Original: that  ˜ G ( θ ) can represent every linear neural network ˜ Fof input and output dimension ≤ ˜ n , depth d , ≤ ˜ n nonzero weights per layer, and ≤ k incoming and ≤ k outgoing weights for every neuron.
Revised: that we can make ˜ G ( θ ) = ˜ F for some θ .

Original: Proof.
Revised: Proof.

Original: Our result comes in two parts: the first shows that ˜ G ( θ ) can represent arbitrary permutations, and the second shows that we can pick these ˜ G ( θ ) permutations and interleave them with block diagonal matrices to create a deeper ˜ G ( θ ) network which manifests any desired neural network.
Revised: Our result comes in two parts: the first shows that ˜ G ( θ ) can represent arbitrary permutations, and the second shows that we can pick these ˜ G ( θ ) permutations and interleave them with block diagonal matrices to create a deeper ˜ G ( θ ) network which manifests any desired neural network.

Original: We use the terms fanin and fanout to mean the number of incoming and outgoing weights into and out of a neuron, respectively.
Revised: We use the terms fanin and fanout to mean the number of incoming and outgoing weights into and out of a neuron, respectively.

Original: First part.
Revised: First part.

Original: Assume we would like to apply a given target permutation to ˜ n elements using ˜ G ( θ )
Revised: Assume we would like to apply a given target permutation to ˜ n elements using ˜ G ( θ )

Original: = (cid:81) Ni =1 B ( θ ( i ) )
Revised: = (cid:81) N i =1 B ( θ ( i ) )

Original: P i consisting of N layers with randomly chosen permutations
Revised: P i consisting of N layers with randomly chosen permutations

Original: P i .
Revised: P i .

Original: Our goal is to perform the target permutation given P i by controlling the block diagonal matrices B ( θ ( i ) ) .
Revised: Our goal is to perform the target permutation given P i by controlling the block diagonal matrices B ( θ ( i ) ) .

Original: The form of ˜ G ( θ ) from Section 3 is
Revised: The form of ˜ G ( θ ) from Section 3 is

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which can be rewritten as
Revised: which can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: with random independent uniform permutations Q i instead.
Revised: with random independent uniform permutations Q i instead.

Original: For each block in the matrix B ( θ ( i ) ) , we may restrict ourselves to two options: to swap or to not swap the pair of indices.
Revised: For each block in the matrix B ( θ ( i ) ) , we may restrict ourselves to two options: to swap or to not swap the pair of indices.

Original: The conjugation by random permutation Q i then shuffles these pairings, such that applying ˜ G ( θ ) is equivalent to repeatedly randomly pairing up indices for optional transposition instead, and then applying a final permutation Q 1 .
Revised: The conjugation by random permutation Q i then shuffles these pairings, such that applying ˜ G ( θ ) is equivalent to repeatedly randomly pairing up indices for optional transposition instead, and then applying a final permutation Q 1 .

Original: We will work under this new equivalent formulation, since it is more amenable to analysis.
Revised: We will work under this new equivalent formulation, since it is more amenable to analysis.

Original: Let us choose to apply each transposition with probability 1 / 2 .
Revised: Let us choose to apply each transposition with probability 1 / 2 .

Original: Then, the expected entropy of ˜
Revised: Then, the expected entropy of ˜

Original: G ( θ ) given the whole sequence of pairings is at least log ˜ n !
Revised: G ( θ ) given the whole sequence of pairings is at least log ˜ n !

Original: − ϵ ( N ˜ n/ 2 , ˜ n ) under Definition 1.
Revised: − ϵ ( N ˜ n/ 2 , ˜ n ) under Definition 1.

Original: In other words, the expected KL divergence of this distribution of ˜ G ( θ ) from the uniform is at most ϵ ( N ˜ n/ 2 , ˜ n ) .
Revised: In other words, the expected KL divergence of this distribution of ˜ G ( θ ) from the uniform is at most ϵ ( N ˜ n/ 2 , ˜ n ) .

Original: Then by Pinsker’s inequality, the expected total variation distance from the uniform is then at most
Revised: Then by Pinsker’s inequality, the expected total variation distance from the uniform is then at most

Original: [Equation]
Revised: [Equation]

Original: This guarantees that at most
Revised: This guarantees that at most

Original: [Equation]
Revised: [Equation]

Original: possible target permutations have a probability density of zero, in expectation, which is then an upper bound on the number of inaccessible target permutations.
Revised: possible target permutations have a probability density of zero, in expectation, which is then an upper bound on the number of inaccessible target permutations.

Original: This means the probability that all target permutations are accessible is then at least
Revised: This means the probability that all target permutations are accessible is then at least

Original: [Equation]
Revised: [Equation]

Original: Note that the leftmost Q 1 in Equation 73 merely introduces a bijection between target permutations, and so does not change how many are accessible.
Revised: Note that the leftmost Q 1 in Equation 77 merely introduces a bijection between target permutations, and so does not change how many are accessible.

Original: Second part.
Revised: Second part.

Original: Suppose we would like to represent p target permutations using p independently generated ˜ G ( θ ) networks each of depth M .
Revised: Suppose we would like to represent p target permutations using p independently generated ˜ G ( θ ) networks each of depth M .

Original: Equation 76 lower bounds the probability that each network can represent its respective permutation.
Revised: Equation 80 lower bounds the probability that each network can represent its respective permutation.

Original: Then the probability that all of the p target permutations are accessible by their respective copies of ˜ G ( θ ) is union bounded by
Revised: Then the probability that all of the p target permutations are accessible by their respective copies of ˜ G ( θ ) is union bounded by

Original: [Equation]
Revised: [Equation]

Original: where the union is over the probability that each copy fails to represent its target permutation.
Revised: where the union is over the probability that each copy fails to represent its target permutation.

Original: Given that this succeeds, we now need to chain these permutations together with block diagonal matrices to represent arbitrary neural networks ˜ F , with Equation 77 lower bounding the probability of failure.
Revised: Given that this succeeds, we now need to chain these permutations together with block diagonal matrices to represent arbitrary neural networks ˜ F , with Equation 81 lower bounding the probability of failure.

Original: Suppose we are successful in representing any combination of p target permutations using p distinct independently generated ˜ G ( θ ) networks of depth M .
Revised: Suppose we are successful in representing any combination of p target permutations using p distinct independently generated ˜ G ( θ ) networks of depth M .

Original: Then, since each ˜ G ( θ ) network’s final (leftmost) operation is a block diagonal matrix, applying an additional block diagonal matrix afterward does not affect the operations that ˜ G
Revised: Then, since each ˜ G ( θ ) network’s final (leftmost) operation is a block diagonal matrix, applying an additional block diagonal matrix afterward does not affect the operations that ˜ G

Original: ( θ ) can represent, since the set of block diagonal matrices we use is closed under matrix multiplication.
Revised: ( θ ) can represent, since the set of block diagonal matrices we use is closed under matrix multiplication.

Original: Importantly, each block matrix can be used to copy a value from one index into two or to add two values together to leave one, creating fanin or fanout in the network.
Revised: Importantly, each block matrix can be used to copy a value from one index into two or to add two values together to leave one, creating fanin or fanout in the network.

Original: Then, interleaving p + 1 chained copies of ˜ G ( θ ) with block diagonal matrices leftmultiplied for a total depth of ( p + 1) M therefore creates an aggregate operation which can still be represented by a single ˜ G ( θ ) network of depth ( p + 1) M .
Revised: Then, interleaving p + 1 chained copies of ˜ G ( θ ) with block diagonal matrices leftmultiplied for a total depth of ( p + 1) M therefore creates an aggregate operation which can still be represented by a single ˜ G ( θ ) network of depth ( p + 1) M .

Original: This aggregate operation has up to ˜ n arbitrary connections from input nodes to output nodes, with either all fanin at most 1 and all fanout at most 2 p , or all fanout at most 1 and all fanin at most 2 p .
Revised: This aggregate operation has up to ˜ n arbitrary connections from input nodes to output nodes, with either all fanin at most 1 and all fanout at most 2 p , or all fanout at most 1 and all fanin at most 2 p .

Original: This is done by building a forest of fanin/fanout trees like illustrated on the right side of Figure 6.
Revised: This is done by building a forest of fanin/fanout trees like illustrated on the right side of Figure 5.

Original: If we compose such a ˜ G ( θ ) network of depth ( p + 1) M with fanout up to 2 p together with a ˜ G ( θ ) network of depth ( p + 1) M with fanin up to 2 p , then we may represent any sparse matrix structure with at most ˜ n nonzero weights and maximum fanin and fanout at most 2 p , using a ˜ G ( θ ) network of depth 2( p + 1) M .
Revised: If we compose such a ˜ G ( θ ) network of depth ( p + 1) M with fanout up to 2 p together with a ˜ G ( θ ) network of depth ( p + 1) M with fanin up to 2 p , then we may represent any sparse matrix structure with at most ˜ n nonzero weights and maximum fanin and fanout at most 2 p , using a ˜ G ( θ ) network of depth 2( p + 1) M .

Original: This construction is illustrated on the left side of Figure 6.
Revised: This construction is illustrated on the left side of Figure 5.

Original: We may adjust the final (leftmost) block diagonal matrix on the fanout side to change the values of the ˜ n arbitrarily positioned weights in the desired sparse matrix.
Revised: We may adjust the final (leftmost) block diagonal matrix on the fanout side to change the values of the ˜ n arbitrarily positioned weights in the desired sparse matrix.

Original: Therefore, any sparse matrix with at most ˜ n weights and max node indegree and outdegree at most k can be represented by a ˜ G ( θ ) of depth 2( ⌈ log 2 k
Revised: Therefore, any sparse matrix with at most ˜ n weights and max node indegree and outdegree at most k can be represented by a ˜ G ( θ ) of depth 2( ⌈ log 2 k

Original: + 1) M .
Revised: + 1) M .

Original: Then, any linear neural network of depth at most d , at most ˜ n weights per layer, and maximum fanin and fanout of at most k can be represented by a ˜ G ( θ ) network of depth 2 Md ( ⌈ log 2 k
Revised: Then, any linear neural network of depth at most d , at most ˜ n weights per layer, and maximum fanin and fanout of at most k can be represented by a ˜ G ( θ ) network of depth 2 Md ( ⌈ log 2 k

Original: ⌉ + 1) , by composition of sparse matrices.
Revised: ⌉ + 1) , by composition of sparse matrices.

Original: The probability that all of this is successful is merely the probability that all the p = 

Revised: The probability that all of this is successful is merely the probability that all the p =

Original: Md ( ⌈ log 2 k ⌉ + 1) permutations can be represented, which by Equation 77 is at least
Revised: Md ( ⌈ log 2 k ⌉ + 1) permutations can be represented, which by Equation 81 is at least

Original: [Equation]
Revised: [Equation]

Original: Thus in summary, if we randomly generate a ˜ G ( θ ) network of depth N = 2 Md ( ⌈ log 2 k ⌉ + 1) for fixed constants k , d , and M , ˜ n neurons per layer, and block size f = 2 , then there is a probability of at least
Revised: Thus in summary, if we randomly generate a ˜ G ( θ ) network of depth N = 2 Md ( ⌈ log 2 k ⌉ + 1) for fixed constants k , d , and M , ˜ n neurons per layer, and block size f = 2 , then there is a probability of at least

Original: [Equation]
Revised: [Equation]

Original: that ˜ G ( θ ) can represent every possible linear neural network of input and output dimension ≤ ˜ n , depth d , at most ˜ n nonzero weights per layer, and max fanin/fanout of at most k .
Revised: that ˜ G ( θ ) can represent every possible linear neural network of input and output dimension ≤ ˜ n , depth d , at most ˜ n nonzero weights per layer, and max fanin/fanout of at most k .

Original: C H ESSIAN L EARNING L OCAL M INIMA (LODO CAN G ENERALIZE )
Revised: D H ESSIAN L EARNING L OCAL M INIMA (LODO CAN G ENERALIZE )

Original: Laurent & von Brecht (2018) gives a theorem showing that for dense linear neural networks on convex losses where each layer is at least as wide as the input or output layer, all local minima in the neural network weights are also global minima.
Revised: Laurent & von Brecht (2018) gives a theorem showing that for dense linear neural networks on convex losses where each layer is at least as wide as the input or output layer, all local minima in the neural network weights are also global minima.

Original: If we simplify LODO by approximating the inverse
Revised: If we simplify LODO by approximating the inverse

Original: Hessian with a full dense matrix G ( θ t ) , then this theorem applies to the Hessian approximation error ||
Revised: Hessian with a full dense matrix G ( θ t ) , then this theorem applies to the Hessian approximation error ||

Original: I − G ( θ t )
Revised: I − G ( θ t )

Original: H || 2 F , which the rescaled error ||
Revised: H || 2 F , which the rescaled error ||

Original: BD − 1 || 2 F used in Section 4.1 is a proxy for.
Revised: BD − 1 || 2 F used in Section 4.1 is a proxy for.

Original: Thus we may expect that any inverse Hessian approximation which LODO could converge to is of similar high quality to the best possible inverse Hessian approximation.
Revised: Thus we may expect that any inverse Hessian approximation which LODO could converge to is of similar high quality to the best possible inverse Hessian approximation.

Original: D H YPERPARAMETERS
Revised: E H YPERPARAMETERS

Original: In every experiment, we tuned the hyperparameters of each optimizer using a genetic algorithm of 10 generations and 32 individuals per generation.
Revised: In every experiment, we tuned the hyperparameters of each optimizer using a genetic algorithm of 10 generations and 32 individuals per generation.

Original: Each hyperparamter was rescaled using x (cid:55)→ ln x if the hyperparameter was a learning rate and x (cid:55)→ 1 − ln(1 − x ) if the hyperparameter was a decay parameter, so that the genetic algorithm would operate in a more well-behaved hyperparameter space.
Revised: Each hyperparamter was rescaled using x (cid:55)→ ln x if the hyperparameter was a learning rate and x (cid:55)→ 1 − ln(1 − x ) if the hyperparameter was a decay parameter, so that the genetic algorithm would operate in a more well-behaved hyperparameter space.

Original: Starting from the default hyperparameters, each generation’s mean hyperparameters were added to some Gaussian noise to create mutated hyperparameters for each individual, where the standard deviation of the noise was generation-dependent and followed a specific schedule.
Revised: Starting from the default hyperparameters, each generation’s mean hyperparameters were added to some Gaussian noise to create mutated hyperparameters for each individual, where the standard deviation of the noise was generation-dependent and followed a specific schedule.

Original: Each individual performed a generation-dependent number of steps of optimization, also according to a schedule.
Revised: Each individual performed a generation-dependent number of steps of optimization, also according to a schedule.

Original: The next generation’s mean hyperparameters were chosen to be the mean hyperparameters of the better performing half of the previous generation, as judged by average training loss during the last 10% of training.
Revised: The next generation’s mean hyperparameters were chosen to be the mean hyperparameters of the better performing half of the previous generation, as judged by average training loss during the last 10% of training.

Original: We also halved all the learning rates (equiv.
Revised: We also halved all the learning rates (equiv.

Original: initial learning rates for
Revised: initial learning rates for

Original: LODO versions) after tuning for the image generation task because tests showed the loss to diverge if training time horizons were longer than 8k steps.
Revised: LODO versions) after tuning for the image generation task because tests showed the loss to diverge if training time horizons were longer than 8k steps.

Original: Since LODO is a random optimizer, we used a different randomization seed for every individual.
Revised: Since LODO is a random optimizer, we used a different randomization seed for every individual.

Original: Table 4 lists the parameters of the tuning schedule for every task.
Revised: Table 3 lists the parameters of the tuning schedule for every task.

Original: The tuned hyperparameters can be found in Table 5. [Table]
Revised: The tuned hyperparameters can be found in Table 4.

Original: E T ASK S ETUP D ETAILS E. N OISY Q UADRATIC B OWL T ASK D ETAILS
Revised: F T ASK S ETUP D ETAILS F. N OISY Q UADRATIC B OWL T ASK D ETAILS

Original: This section fully explains the details of the setup of the noisy quadratic bowl task of Section 5.1.
Revised: This section fully explains the details of the setup of the noisy quadratic bowl task of Section 5.1.

Original: This 100 parameter task consists of a quadratic bowl for its loss landscape.
Revised: This 100 parameter task consists of a quadratic bowl for its loss landscape.

Original: Using a random uniform orthogonal matrix U and a diagonal matrix D consisting of a geometric sequence of 100 values starting at 0.001 and ending at 1, the Hessian of the quadratic bowl is set to H = UDU T , and the center is set to the origin.
Revised: Using a random uniform orthogonal matrix U and a diagonal matrix D consisting of a geometric sequence of 100 values starting at 0.001 and ending at 1, the Hessian of the quadratic bowl is set to H = UDU T , and the center is set to the origin.

Original: However, whenever the loss and gradient are evaluated, the center of the quadratic bowl is perturbed by an i.i.d.
Revised: However, whenever the loss and gradient are evaluated, the center of the quadratic bowl is perturbed by an i.i.d.

Original: random standard normal offset in each dimension.
Revised: random standard normal offset in each dimension.

Original: The initialization for this task is set to the origin.
Revised: The initialization for this task is set to the origin.

Original: Due to the random wandering of the center, the expected loss rises linearly over time—unless the optimizer acts to prevent this, driving the error towards a steady state distribution.
Revised: Due to the random wandering of the center, the expected loss rises linearly over time—unless the optimizer acts to prevent this, driving the error towards a steady state distribution.

Original: The expected loss after infinitely many steps can then be taken as a measure of the quality of an optimizer.
Revised: The expected loss after infinitely many steps can then be taken as a measure of the quality of an optimizer.

Original: The optimal solution for this task is to select the current  minimum of the quadratic bowl at every timestep (which is what the Newton method would do).
Revised: The optimal solution for this task is to select the current [Table] minimum of the quadratic bowl at every timestep (which is what the Newton method would do).

Original: Due to the movement of the minimum between steps and loss evaluations, we should still expect this strategy to a achieve nonzero loss which can be analytically calculated to be 7.412.
Revised: Due to the movement of the minimum between steps and loss evaluations [Table] we should still expect this strategy to a achieve nonzero loss which can be analytically calculated to be 7.412.

Revised: Tableshows various optimizers’ performances after having taken many steps.

Original: E. CNN I MAGE G ENERATION T ASK D ETAILS
Revised: F. CNN I MAGE G ENERATION T ASK D ETAILS

Original: This section explains the CNN image generation task of Section 5.3, which is similar to the task of training a PixelCNN (Oord et al., 2016).
Revised: This section explains the CNN image generation task of Section 5.2, which is similar to the task of training a PixelCNN (Oord et al., 2016).

Original: Like PixelCNN, our CNN generates pixels row by row and column by column, and classifies the brightness of each pixel into 256 classes with crossentropy loss.
Revised: Like PixelCNN, our CNN generates pixels row by row and column by column, and classifies the brightness of each pixel into 256 classes with crossentropy loss.

Original: Our data preprocessing is as follows.
Revised: Our data preprocessing is as follows.

Original: An MNIST image randomly selected, and one pixel location is chosen uniformly at random and blackened.
Revised: An MNIST image randomly selected, and one pixel location is chosen uniformly at random and blackened.

Original: All pixels below, or to the right of and in the same row as the selected pixel are blackened.
Revised: All pixels below, or to the right of and in the same row as the selected pixel are blackened.

Original: The input into the CNN consists of this partially[Table] masked/blackened image (divided by 256 for normalization), an image indicating which pixels are specifically masked/blackened (indicated by − 1 and 1), another image indicating which pixels are left of the selected pixel (indicated by − 1 and 1), an image of a linear gradient from − 1 to 1 in the x direction, and the same for the y direction.
Revised: The input into the CNN consists of this partially masked/blackened image (divided by 256 for normalization), an image indicating which pixels are specifically masked/blackened (indicated by − 1 and 1), another image indicating which pixels are left of the selected pixel (indicated by − 1 and 1), an image of a linear gradient from − 1 to 1 in the x direction, and the same for the y direction.

Original: The last three images are present purely to break horizontal and vertical translation symmetries in the data, which has been shown to be helpful in vision tasks involving collection of information from specific locations of an image specified by the data (Liu et al., 2018).
Revised: The last three images are present purely to break horizontal and vertical translation symmetries in the data, which has been shown to be helpful in vision tasks involving collection of information from specific locations of an image specified by the data (Liu et al., 2018).

Original: The preprocessed data is visualized in Figure 7.
Revised: The preprocessed data is visualized in Figure 6.

Original: The CNN architecture we use for autoregression consists of:
Revised: The CNN architecture we use for autoregression consists of:

Original: • 5 input channels as previously described.
Revised: • 5 input channels as previously described.

Original: • Residual connection to Point A. – One pixel of zero padding, and convolution with 3 by 3 filters to 20 channels, with no activation.
Revised: • Residual connection to Point A. – One pixel of zero padding, and convolution with 3 by 3 filters to 20 channels, with no activation.

Original: • Point A. •
Revised: • Point A. •

Original: The following indented items are repeated 5 times.
Revised: The following indented items are repeated 5 times.

Original: – The following indented items are repeated 4 times.
Revised: – The following indented items are repeated 4 times.

Original: * Residual connection to Point B. · Convolution with 1 by 1 filters to 40 channels, with arctangent activation.
Revised: * Residual connection to Point B. · Convolution with 1 by 1 filters to 40 channels, with arctangent activation.

Original: We use the arctangent to mitigate gradient norm issues.
Revised: We use the arctangent to mitigate gradient norm issues.

Original: · One pixel of zero padding, and three different depthwise convolutions concatenated together, with 3 by 3 filters to 120 channels, with arctangent activation.
Revised: · One pixel of zero padding, and three different depthwise convolutions concatenated together, with 3 by 3 filters to 120 channels, with arctangent activation.

Original: · Convolution with 1 by 1 filters to 20 channels, with no activation.
Revised: · Convolution with 1 by 1 filters to 20 channels, with no activation.

Original: * Point B.  – Average pooling of 2 by 2 blocks to halve the image size, with a pixel of zero padding beforehand if the image size is odd.
Revised: Point B. [Table] – Average pooling of 2 by 2 blocks to halve the image size, with a pixel of zero padding beforehand if the image size is odd.

Original: • Convolution with 1 by 1 filters to 256 channels, with softmax activation.
Revised: • Convolution with 1 by 1 filters to 256 channels, with softmax activation.

Original: The loss is then the average crossentropy between true brightness of the query pixel and the distribution given by the softmax output of this CNN, over a batch size of 256 images.
Revised: The loss is then the average crossentropy between true brightness of the query pixel and the distribution given by the softmax output of this CNN, over a batch size of 256 images.

Original: The whole CNN has about 94696 parameters, which are initialized with LeCun normal initialization (Klambauer et al., 2017).
Revised: The whole CNN has about 94696 parameters, which are initialized with LeCun normal initialization (Klambauer et al., 2017).

Original: The task for the optimizer is to train these parameters to minimize this loss.
Revised: The task for the optimizer is to train these parameters to minimize this loss.

Original: Figure 8 shows some imitation MNIST images generated using this CNN by sampling pixels one by one. 
Revised: Figure 7 shows some imitation MNIST images generated using this CNN by sampling pixels one by one. [Table]

Original: F R OSENBROCK F UNCTION M INIMIZATION T ASK P ERFORMANCE
Revised: G R OSENBROCK F UNCTION M INIMIZATION E XPERIMENT

Original: Refer to final losses in Table 6 and learning curves in Figure 9 for the performance of each optimizer on the  Rosenbrock function minimization task.{+f ( x, y ) = 01( x − 1) 
 + ( x 2 − y ) 2 , which has no local minima and one global minimum at ( x, y ) =+}
Revised: We probe the behavior of LODO with a small test task of finding the minimum of a rescaled Rosenbrock function {+f ( x, y ) = 01( x − 1) 
 + ( x 2 − y ) 2 , which has no local minima and one global minimum at ( x, y ) =+}

Revised: We initialized the optimizers at ( x, y ) =

Original: [Figure]
Revised: ( − 0 .

Original: [Table]
Revised: and gave themsteps to run. The trajectory taken by LODO, shown in Figure 9, is similar to the short timescale dynamics of other optimizers using momentum, in that it tends to overshoot and then correct itself in an oscillatory manner. Learning curves in Figure 8 and losses in Table 6 show the performance of all the optimizers on this task.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Table] [Figure] [Table]

Original: [Table] [Figure] [Table] [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: G T RAINING L OOP T IMING D ETAILS
Revised: H T RAINING L OOP T IMING D ETAILS

Original:  This section describes how we timed each optimizer to report performance at specified times and step per second training speeds.
Revised: [Figure] This section describes how we timed each optimizer to report performance at specified times and step per second training speeds.

Original: We performed all optimization runs in TensorFlow 2, each with 40 Intel Xeon Gold 6248 CPUs and 2 Nvidia Volta V10 GPUs.
Revised: We performed all optimization runs in TensorFlow 2, each with 40 Intel Xeon Gold 6248 CPUs and 2 Nvidia Volta V10 GPUs.

Original: Time reported includes all training time (forward and backward propagation, optimizer computation, data loading, preprocessing, etc.)
Revised: Time reported includes all training time (forward and backward propagation, optimizer computation, data loading, preprocessing, etc.)

Original: except it does not include time taken to evaluate metrics such as the Hessian approximation error and the validation and test losses and accuracies.
Revised: except it does not include time taken to evaluate metrics such as the Hessian approximation error and the validation and test losses and accuracies.