Original: H ALF -I NVERSE G RADIENTS FOR P HYSICAL D EEP L EARNING
Revised: H ALF -I NVERSE G RADIENTS FOR P HYSICAL D EEP L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent works in deep learning have shown that integrating differentiable physics simulators into the training process can greatly improve the quality of results.
Revised: Recent works in deep learning have shown that integrating differentiable physics simulators into the training process can greatly improve the quality of results.

Original: Although this combination represents a more complex optimization task than supervised neural network training, the same gradient-based optimizers are typically employed to minimize the loss function.
Revised: Although this combination represents a more complex optimization task than supervised neural network training, the same gradient-based optimizers are typically employed to minimize the loss function.

Original: However, the integrated physics solvers have a profound effect on the gradient ﬂow as manipulating scales in magnitude and direction is an inherent property of many physical processes.
Revised: However, the integrated physics solvers have a profound effect on the gradient ﬂow as manipulating scales in magnitude and direction is an inherent property of many physical processes.

Original: Consequently, the gradient ﬂow is often highly unbalanced and creates an environment in which existing gradient-based optimizers perform poorly.
Revised: Consequently, the gradient ﬂow is often highly unbalanced and creates an environment in which existing gradient-based optimizers perform poorly.

Original: In this work, we analyze the characteristics of both physical and neural network optimizations to derive a new method that does not suffer from this phenomenon.
Revised: In this work, we analyze the characteristics of both physical and neural network optimizations to derive a new method that does not suffer from this phenomenon.

Original: Our method is based on a halfinversion of the Jacobian and combines principles of both classical network and physics optimizers to solve the combined optimization task.
Revised: Our method is based on a halfinversion of the Jacobian and combines principles of both classical network and physics optimizers to solve the combined optimization task.

Original: Compared to state-ofthe-art neural network optimizers, our method converges more quickly and yields better solutions, which we demonstrate on three complex learning problems involving nonlinear oscillators, the Schr ¨ odinger equation and the Poisson problem.
Revised: Compared to state-ofthe-art neural network optimizers, our method converges more quickly and yields better solutions, which we demonstrate on three complex learning problems involving nonlinear oscillators, the Schr ¨ odinger equation and the Poisson problem.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The groundbreaking successes of deep learning (Krizhevsky et al., 2012; Sutskever et al., 2014;
Revised: The groundbreaking successes of deep learning (Krizhevsky et al., 2012; Sutskever et al., 2014;

Original: Silver et al., 2017) have led to ongoing efforts to study the capabilities of neural networks across all scientiﬁc disciplines.
Revised: Silver et al., 2017) have led to ongoing efforts to study the capabilities of neural networks across all scientiﬁc disciplines.

Original: In the area of physical simulation, neural networks have been used in various ways, such as creating accurate reduced-order models (Morton et al., 2018), inferring improved discretization stencils (Bar-Sinai et al., 2019), or suppressing numerical errors (Um et al., 2020).
Revised: In the area of physical simulation, neural networks have been used in various ways, such as creating accurate reduced-order models (Morton et al., 2018), inferring improved discretization stencils (Bar-Sinai et al., 2019), or suppressing numerical errors (Um et al., 2020).

Original: The long-term goal of these methods is to exceed classical simulations in terms of accuracy and speed, which has been achieved, e.g., for rigid bodies (de Avila Belbute-Peres et al., 2018), physical inverse problems (Holl et al., 2020), and two-dimensional turbulence (Kochkov et al., 2021).
Revised: The long-term goal of these methods is to exceed classical simulations in terms of accuracy and speed, which has been achieved, e.g., for rigid bodies (de Avila Belbute-Peres et al., 2018), physical inverse problems (Holl et al., 2020), and two-dimensional turbulence (Kochkov et al., 2021).

Original: The successful application of deep learning to physical systems naturally hinges on the training setup.
Revised: The successful application of deep learning to physical systems naturally hinges on the training setup.

Original: In recent years, the use of physical loss functions has proven beneﬁcial for the training procedure, yielding substantial improvements over purely supervised training approaches (Tompson et al., 2017; Wu & Tegmark, 2019; Greydanus et al., 2019).
Revised: In recent years, the use of physical loss functions has proven beneﬁcial for the training procedure, yielding substantial improvements over purely supervised training approaches (Tompson et al., 2017; Wu & Tegmark, 2019; Greydanus et al., 2019).

Original: These improvements were shown to stem from three aspects (Battaglia et al., 2016;
Revised: These improvements were shown to stem from three aspects (Battaglia et al., 2016;

Original: Holl et al., 2020): (i)
Revised: Holl et al., 2020): (i)

Original: Incorporating prior knowledge from physical principles facilitates the learning process , (ii) the ambiguities of multimodal cases are resolved naturally, and (iii) simulating the physics at training time can provide more realistic data distributions than pre-computed data sets.
Revised: Incorporating prior knowledge from physical principles facilitates the learning process , (ii) the ambiguities of multimodal cases are resolved naturally, and (iii) simulating the physics at training time can provide more realistic data distributions than pre-computed data sets.

Original: Approaches for training with physical losses can be divided into two categories.
Revised: Approaches for training with physical losses can be divided into two categories.

Original: On the one hand, equation-focused approaches that introduce physical residuals (Tompson et al., 2017; Raissi et al., 2019), and on the other hand, solver-focused approaches that additionally integrate well-established numerical procedures into the training (Um et al., 2020; Kochkov et al., 2021).
Revised: On the one hand, equation-focused approaches that introduce physical residuals (Tompson et al., 2017; Raissi et al., 2019), and on the other hand, solver-focused approaches that additionally integrate well-established numerical procedures into the training (Um et al., 2020; Kochkov et al., 2021).

Original: From a mathematical point of view, training a neural network with a physical loss function bears the difﬁculties of both network training and physics optimization.
Revised: From a mathematical point of view, training a neural network with a physical loss function bears the difﬁculties of both network training and physics optimization.

Original: In order to obtain satisfying results, it is vital to treat ﬂat regions of the optimization landscapes effectively.
Revised: In order to obtain satisfying results, it is vital to treat ﬂat regions of the optimization landscapes effectively.

Original: In learning, the challenging loss landscapes are addressed using gradient-based optimizers with data-based normalizing schemes such as Adam (Kingma & Ba, 2015), whereas in physics, the optimizers of choice are higher-order techniques such as Newton’s method (Gill & Murray, 1978) that inherently make use of inversion processes.
Revised: In learning, the challenging loss landscapes are addressed using gradient-based optimizers with data-based normalizing schemes such as Adam (Kingma & Ba, 2015), whereas in physics, the optimizers of choice are higher-order techniques such as Newton’s method (Gill & Murray, 1978) that inherently make use of inversion processes.

Original: However, Holl et al.
Revised: However, Holl et al.

Original: (2021) found that these approaches can not effectively handle the joint optimization of network and physics.
Revised: (2021) found that these approaches can not effectively handle the joint optimization of network and physics.

Original: Gradient-descent-based optimizers suffer from vanishing or exploding gradients, preventing effective convergence, while higher-order methods do not generally scale to the high-dimensional parameter spaces required by deep learning (Goodfellow et al., 2016).
Revised: Gradient-descent-based optimizers suffer from vanishing or exploding gradients, preventing effective convergence, while higher-order methods do not generally scale to the high-dimensional parameter spaces required by deep learning (Goodfellow et al., 2016).

Original: We likewise focus on inversion-based approach but propose a new method for joint physics and network optimization which we refer to as half-inverse gradients .
Revised: We likewise focus on inversion-based approach but propose a new method for joint physics and network optimization which we refer to as half-inverse gradients .

Original: At its core lies a partial matrix inversion, which we derive from the interaction between network and physics both formally and geometrically.
Revised: At its core lies a partial matrix inversion, which we derive from the interaction between network and physics both formally and geometrically.

Original: An important property of our method is that its runtime scales linearly in the number of network parameters.
Revised: An important property of our method is that its runtime scales linearly in the number of network parameters.

Original: To demonstrate the wide-ranging and practical applicability of our method, we show that it yields signiﬁcant improvements in terms of convergence speed and ﬁnal loss values over existing methods.
Revised: To demonstrate the wide-ranging and practical applicability of our method, we show that it yields signiﬁcant improvements in terms of convergence speed and ﬁnal loss values over existing methods.

Original: These improvements are measured both in terms of absolute accuracy as well as wall-clock time.
Revised: These improvements are measured both in terms of absolute accuracy as well as wall-clock time.

Original: We evaluate a diverse set of physical systems, such as the Schr¨odinger equation, a nonlinear chain system and the Poisson problem.
Revised: We evaluate a diverse set of physical systems, such as the Schr¨odinger equation, a nonlinear chain system and the Poisson problem.

Original:  G RADIENTS BASED ON H ALF -I NVERSE J ACOBIANS
Revised:  G RADIENTS BASED ON H ALF -I NVERSE J ACOBIANS

Original: Optimization on continuous spaces can be effectively performed with derivative-based methods, the simplest of which is gradient descent.
Revised: Optimization on continuous spaces can be effectively performed with derivative-based methods, the simplest of which is gradient descent.

Original: For a target function L ( θ ) to be minimized  and learning rate η , gradient descent proceeds by repeatedly applying updates
Revised: For a target function L ( θ ) to be minimized of several variables θ , using bold symbols for vector-valued quantitites in this section, and learning rate η , gradient descent proceeds by repeatedly applying updates

Original: [Equation]
Revised: [Equation]

Original: For quadratic objectives, this algorithm convergences linearly with the rate of convergence depending on the condition number λ of the Hessian matrix (Lax, 2014).
Revised: For quadratic objectives, this algorithm convergences linearly with the rate of convergence depending on the condition number λ of the Hessian matrix (Lax, 2014).

Original: In the ill-conditioned case λ (cid:29) 1 , ﬂat regions in the optimization landscape can signiﬁcantly slow down the optimization progress.
Revised: In the ill-conditioned case λ (cid:29) 1 , ﬂat regions in the optimization landscape can signiﬁcantly slow down the optimization progress.

Original: This is a ubiquitous problem in non-convex optimization tasks of the generic form:
Revised: This is a ubiquitous problem in non-convex optimization tasks of the generic form:

Original: [Equation]
Revised: [Equation]

Original: Here ( x i , ˆ y i ) denotes the i th data points from a chosen set of measurements, f is a function parametrized by θ to be optimized to model the relationship between the data points y i ( θ ) =
Revised: Here ( x i , ˆ y i ) denotes the i th data points from a chosen set of measurements, f is a function parametrized by θ to be optimized to model the relationship between the data points y i ( θ ) =

Original: f ( x i ; θ ) , and l denotes a loss function measuring the optimization progress.
Revised: f ( x i ; θ ) , and l denotes a loss function measuring the optimization progress.

Original: In the following, we assume the most common case of l ( y i , ˆ y i ) =
Revised: In the following, we assume the most common case of l ( y i , ˆ y i ) =

Original: 12 ||
Revised: 12 ||

Original: y
Revised: y

Original: i − ˆ
Revised: i − ˆ

Original: y
Revised: y

Original: i ||
Revised: i ||

Original: 22 being a quadratic loss.
Revised: 22 being the squared l2-loss

Original: Physics Optimization.
Revised: Physics Optimization.

Revised: Simulating a physical system consists of two steps: (i) mathematically modeling the system by a differential equation, and (ii) discretizing the occuring differential operators to receive a solver usable on a computer.

Revised: If we then wish to manipulate a physical system by an external force to reach a given conﬁguration, we have to solve an inverse problem of the form 2.

Original: In physical optimization problems, the sumin equation 2 reduces to a single data point ( x, ˆ y ) with x being the initial and ˆ y the target state and  f representinga time evolutionmapping y ( θ ) =
Revised: In such a control task, the sum reduces to a single data point ( x , ˆ y ) with x being the initial state, ˆ y the target state and θ the external force we want to ﬁnd. The physical solver corresponds to the function f representing time evolution y ( θ ) =

Original: f ( x ; θ ) .
Revised: f ( x ; θ ) .

Revised: This single data point sum still includes squared summation over vector components of y − ˆ y in the l2-loss.

Revised: Sensitive behavior of the physical system connected to its high-frequency modes is present in the physical solver f , and produces small singular values in its linear approximation.

Original: Flat regions in the optimization landscape can be traversed quickly by using a method that incorporates more information than only the gradient.
Revised: This leads to an ill-conditioned Jacobian and ﬂat regions in the optimization landscape when minimizing 2. This is addressed by using methods that incorporate more information than only the gradient.

Original: Prominent examples are Newton’s method or the Gauss-Newton’s algorithm
Revised: Prominent examples are Newton’s method or the Gauss-Newton’s algorithm

Original: (Gill & Murray, 1978); the latter one is based on the Jacobian of f and the loss gradient:
Revised: (Gill & Murray, 1978); the latter one is based on the Jacobian of f and the loss gradient:

Original: [Equation]
Revised: [Equation]

Revised: Here, the inversion of the Jacobian is done with the pseudoinverse.

Original: The Gauss-Newton update maps the steepest descent direction in y space to the parameter space θ .
Revised: The Gauss-Newton update maps the steepest descent direction in y space to the parameter space θ .

Original: Therefore, to ﬁrst order, the resulting update approximates gradient descent steps in y space, further details of which are given in the appendix.
Revised: Therefore, to ﬁrst order, the resulting update approximates gradient descent steps in y space, further details of which are given in the appendix.

Original: An advantage of this formulation is that the update steps in y space are invariant under arbitrary rescaling of the parameters θ , which cancels inherent scales in f and ensures quick progress in the optimization landscape. 
Revised: An advantage of this formulation is that the update steps in y space are invariant under arbitrary rescaling of the parameters θ , which cancels inherent scales in f and ensures quick progress in the optimization landscape. [Figure] W [Figure]

Revised: ond iti on e d b)

Revised: I ll

Original:  Neural Network Training.
Revised: [Figure] c ond iti on e d [Figure] Neural Network Training.

Original: When f represents a neural network in equation 2, the optimization matches the typical supervised learning task.
Revised: When f represents a neural network in equation 2, the optimization matches the typical supervised learning task.

Original: In this context, the problem of ﬂat regions in the optimization landscape is also referred to as pathological curvature (Martens, 2010).
Revised: In this context, the problem of ﬂat regions in the optimization landscape is also referred to as pathological curvature (Martens, 2010).

Original: Solving this problem with higher-order methods is considered to be too expensive given the large number of parameters θ .
Revised: Solving this problem with higher-order methods is considered to be too expensive given the large number of parameters θ .

Original: For learning tasks, widely used optimizers such as Adam instead use gradient information from earlier update steps, for instance in the form of momentum or adaptive learning [Figure] W [Figure]
Revised: For learning tasks, widely used optimizers such as Adam instead use gradient information from earlier update steps, for instance in the form of momentum or adaptive learning rate terms, thereby achieving better optimization progress at little additional computational cost.

Original: ll

Original: [Figure]

Original: c ond iti on e d b)

Original: I ll

Original: [Figure]

Original: c ond iti on e d

Original: [Figure]

Original: rate terms, thereby achieving better optimization progress at little additional computational cost.

Original: Additionally, the updates are computed on mini-batches instead of the full data set, which saves computational resources and beneﬁts generalization.
Revised: Additionally, the updates are computed on mini-batches instead of the full data set, which saves computational resources and beneﬁts generalization.

Original: Neural Network Training with Physics Objectives.
Revised: Neural Network Training with Physics Objectives.

Original: For the remainder of the paper, we consider joint optimization problems, where f denotes a composition of a neural network parameterized by θ and a physics solver.
Revised: For the remainder of the paper, we consider joint optimization problems, where f denotes a composition of a neural network parameterized by θ and a physics solver.

Original: Using classical network optimizers for minimizing equation 2 is inefﬁcient in this case since data normalization in the network output space is not possible and the classical initialization schemes cannot normalize the effects of the physics solver.
Revised: Using classical network optimizers for minimizing equation 2 is inefﬁcient in this case since data normalization in the network output space is not possible and the classical initialization schemes cannot normalize the effects of the physics solver.

Original: As such, they are unsuited to capture the strong coupling between optimization parameters typically encountered in physics applications.
Revised: As such, they are unsuited to capture the strong coupling between optimization parameters typically encountered in physics applications.

Original: While Gauss-Newton seems promising for these cases, the involved Jacobian inversion tends to result in large overshoots in the updates when the involved physics solver is ill-conditioned.
Revised: While Gauss-Newton seems promising for these cases, the involved Jacobian inversion tends to result in large overshoots in the updates when the involved physics solver is ill-conditioned.

Original: As we will demonstrate, this leads to oversaturation of neurons, hampering the learning capability of the neural network.
Revised: As we will demonstrate, this leads to oversaturation of neurons, hampering the learning capability of the neural network.

Original: 2. A N I LL -C ONDITIONED T OY E XAMPLE
Revised: 2. A N I LL -C ONDITIONED T OY E XAMPLE

Original: To illustrate the argumentation so far, we consider a data set sampled from ˆ y ( x )=(sin(6 x ) , cos(9 x )) for x ∈ [ − 1 , 1] : We train a neural network to describe this data set by using the loss function l ( y, ˆ y ; λ ) =
Revised: To illustrate the argumentation so far, we consider a data set sampled from ˆ y ( x )=(sin(6 x ) , cos(9 x )) for x ∈ [ − 1 , 1] : We train a neural network to describe this data set by using the loss function l ( y , ˆ y ; γ ) =

Original: 12 (cid:0) y 1 − ˆ y 1 (cid:1) 2 + 12 (cid:0) λ · y 2 − ˆ y 2 (cid:1) 2 , where y 1 and y 2 denote the ﬁrst and second component of the y -vectors.
Revised: 12 (cid:0) y 1 − ˆ y 1 (cid:1) 2 + 12 (cid:0) λ · y 2 − ˆ y 2 (cid:1) 2 , where y 1 and y 2 denote the ﬁrst and second component of the y -vectors.

Original: For a scale factor of λ = 1 , we receive the well-conditioned mean squared error loss.
Revised: For a scale factor of γ = 1 , we receive the well-conditioned mean squared error loss.

Original: However, l becomes increasingly ill-conditioned as λ is decreased, which mimics the effects of a physics solver.
Revised: However, l becomes increasingly ill-conditioned as γ is decreased, which mimics the effects of a physics solver.

Original: For real-world physics solvers the situation would typically be even more pronounced, as these scales usually vary strongly in direction and magnitude across different data points and optimization steps.
Revised: For real-world physics solvers the situation would typically be even more pronounced, as these scales usually vary strongly in direction and magnitude across different data points and optimization steps.

Original: We use a small neural network with a single hidden layer with 7 neurons and a tanh activation.
Revised: We use a small neural network with a single hidden layer with 7 neurons and a tanh activation.

Original: We then compare training with the well-conditioned λ = 1 loss against an ill-conditioned λ = 0 .
Revised: We then compare training with the well-conditioned γ = 1 loss against an ill-conditioned γ = 0 .

Original: 01 loss.
Revised: 01 loss.

Original: In both cases, we train the network using both Adam and
Revised: In both cases, we train the network using both Adam and

Original: Gauss-Newton as representatives of gradient-based and higher-order optimizers, respectively.
Revised: Gauss-Newton as representatives of gradient-based and higher-order optimizers, respectively.

Original: The results are shown in ﬁgure 1.
Revised: The results are shown in ﬁgure 1.

Original: [Table]

Original: performs Steepest Descent:

Original: Norm ( θ -space)

Original: Norm ( y -space)

Original: [Table]

Original: in Parameter Space || · || θ = || · || J − 1 y

Original: [Table]

Original: in Physics Space || · ||

Original: Jθ = || · || y

Original: Ours in Intermediate Space || · || J 1 / 2 θ

Original: = || · || J − 1 / 2 y In the well-conditioned case, Adam and Gauss-Newton behave similarly, decreasing the loss by about three orders of magnitude.
Revised: In the well-conditioned case, Adam and Gauss-Newton behave similarly, decreasing the loss by about three orders of magnitude.

Original: However, in the ill-conditioned case, both optimizers fail to minimize the objective beyond a certain point.
Revised: However, in the ill-conditioned case, both optimizers fail to minimize the objective beyond a certain point.

Original: To explain this observation, we ﬁrst illustrate the behavior from the physics viewpoint by considering the trajectory of the network output f ( x ) for a single value x during training (ﬁgure 1, right).
Revised: To explain this observation, we ﬁrst illustrate the behavior from the physics viewpoint by considering the trajectory of the network output f ( x ) for a single value x during training (ﬁgure 1, right).

Original: For γ = 1 , Adam optimizes the network to accurately predict ˆ y ( x ) while for γ = 0 .
Revised: For γ = 1 , Adam optimizes the network to accurately predict ˆ y ( x ) while for γ = 0 .

Original: 01 , the updates neglect the second component preventing Adam to move efﬁciently along the small-scale coordinate (blue curve in ﬁgure 1b, right).
Revised: 01 , the updates neglect the second component preventing Adam to move efﬁciently along the small-scale coordinate (blue curve in ﬁgure 1b, right).

Original: To illustrate the situation from the viewpoint of the network, we consider the variance in the outputs of speciﬁc neurons over different x (ﬁgure 1, middle).
Revised: To illustrate the situation from the viewpoint of the network, we consider the variance in the outputs of speciﬁc neurons over different x (ﬁgure 1, middle).

Original: When γ = 1 , all neurons process information by producing different outcomes for different x .
Revised: When γ = 1 , all neurons process information by producing different outcomes for different x .

Original: However, for γ = 0 .
Revised: However, for γ = 0 .

Original: 01 , Gauss-Newton’s inversion of the small-scale component y 2 results in large updates, leading to an oversaturation of neurons (red curve in ﬁgure 1b, middle).
Revised: 01 , Gauss-Newton’s inversion of the small-scale component y 2 results in large updates, leading to an oversaturation of neurons (red curve in ﬁgure 1b, middle).

Original: These neurons stop processing information, reducing the effective capacity of the network and preventing the network from accurately ﬁtting ˆ y .
Revised: These neurons stop processing information, reducing the effective capacity of the network and preventing the network from accurately ﬁtting ˆ y .

Original: Facing these problems, a natural questions arises: Is it possible to construct an algorithm that can successfully process the inherently different scales of a physics solver while training a neural network at the same time?
Revised: Facing these problems, a natural questions arises: Is it possible to construct an algorithm that can successfully process the inherently different scales of a physics solver while training a neural network at the same time?

Original: 2. U PDATES BASED ON H ALF -I NVERSE J ACOBIANS
Revised: 2. U PDATES BASED ON H ALF -I NVERSE J ACOBIANS

Original: We propose a novel method for optimizing neural networks with physics objectives.
Revised: We propose a novel method for optimizing neural networks with physics objectives.

Original: Since pure physics or neural network optimization can be thought of as special cases of the joint optimization, we analogously look for a potential method in the continuum of optimization methods between gradient descent and Gauss-Newton.
Revised: Since pure physics or neural network optimization can be thought of as special cases of the joint optimization, we analogously look for a potential method in the continuum of optimization methods between gradient descent and Gauss-Newton.

Original: We consider both of them to be the most elementary algorithms representing network and physics optimizers, respectively.
Revised: We consider both of them to be the most elementary algorithms representing network and physics optimizers, respectively.

Original: The following equation describes an update that lies in between the two.
Revised: The following equation describes an update that lies in between the two.

Original: [Equation]
Revised: [Equation]

Original: Here, the exponent < β, κ > of the Jacobian denotes the following procedure deﬁned with the aid of the singular value decomposition J = U Λ V (cid:62) as:
Revised: Here, the exponent < β, κ > of the Jacobian denotes the following procedure deﬁned with the aid of the singular value decomposition J = U Λ V (cid:62) as:

Original: [Equation]
Revised: [Equation]

Original: When β = 0 and κ = 1 , equation 4 reduces to the well-known form of gradient descent.
Revised: When β = 0 and κ = 1 , equation 4 reduces to the well-known form of gradient descent.

Original: Likewise, the case β = 0 and κ = − 1 yields Gauss-Newton since the result of the Jacobian exponentiation then gives the pseudoinverse of the Jacobian.
Revised: Likewise, the case β = 0 and κ = − 1 yields Gauss-Newton since the result of the Jacobian exponentiation then gives the pseudoinverse of the Jacobian.

Original: Unlike other possible interpolations between gradient descent and Gauss-Newton, exponentiation by κ as in equation 4 signiﬁcantly affects the scales inherent in the Jacobian.
Revised: Unlike other possible interpolations between gradient descent and Gauss-Newton, exponentiation by κ as in equation 4 signiﬁcantly affects the scales inherent in the Jacobian.

Original: This is highly important to appropriately inﬂuence physics and neural network scales.
Revised: This is highly important to appropriately inﬂuence physics and neural network scales.

Original: To determine κ , we recall our goal to perform update steps which are normalized in both θ - and y -space.
Revised: To determine κ , we recall our goal to perform update steps which are normalized in both θ - and y -space.

Original: However, since any update ∆ θ and its corresponding effect on the network output ∆ y are connected by the inherent scales encoded in the Jacobian, no single κ exists that normalizes both at the same time.
Revised: However, since any update ∆ θ and its corresponding effect on the network output ∆ y are connected by the inherent scales encoded in the Jacobian, no single κ exists that normalizes both at the same time.

Original: Instead, we distribute the burden equally between network and physics by choosing κ = − 1 / 2 .
Revised: Instead, we distribute the burden equally between network and physics by choosing κ = − 1 / 2 .

Original: From a geometric viewpoint, the resulting update can be regarded as a steepest descent step measured by the norm given in table 1.
Revised: From a geometric viewpoint, the resulting update can be regarded as a steepest descent step measured by the norm given in table 1.

Original: In addition to the exponentiation, the scalar prefactor in equation 5 enables us to obtain a weaker form of scale invariance: Setting β = − 1 − κ yields algorithms that rescale the largest singular value to 1 , which ensures that the resulting updates cannot produce arbitrarily large updates in y -space.
Revised: In addition to the exponentiation, the scalar prefactor in equation 5 enables us to obtain a weaker form of scale invariance: Setting β = − 1 − κ yields algorithms that rescale the largest singular value to 1 , which ensures that the resulting updates cannot produce arbitrarily large updates in y -space.

Original: For a batch of size b , we deﬁne an update with a learning rate η by stacking network-solver Jacobians ∂y i ∂θ (cid:12)(cid:12)
Revised: For a batch of size b , we deﬁne an update with a learning rate η by stacking network-solver Jacobians ∂ y i ∂ θ (cid:12)(cid:12)

Original: x
Revised: x

Original: i and loss gradients ∂L ∂y i (cid:12)(cid:12) x i , ˆ y i of different data points ( x i , ˆ y i ) to form the following update step 
Revised: i and loss gradients ∂L∂ y i (cid:12)(cid:12) x i , ˆ y i of different data points ( x i , ˆ y i ) to form the following update step for our method:

Original: [Equation]
Revised: [Table]

Revised: performs Steepest Descent:

Revised: Norm ( θ -space)

Revised: Norm ( y -space)

Original: 
Revised: [Table] in Parameter Space || · || θ = || · || [Equation] − 1 y

Original: [Equation]
Revised: [Table]

Original:  
Revised: in Physics Space || · || J θ = || · || y Ours in Intermediate Space |

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: ·   ∂L∂y
Revised: − 1 /

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)
Revised: 

Original: [Equation]
Revised: [Equation]

Original: y∂L∂y
Revised:  

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)
Revised: ·  

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: b   (6)
Revised:   (6)

Original: Besides batch size b and learning rate η , we specify a truncation parameter τ as an additional hyperparameter enabling us to suppress numerical noise during the half-inversion process in equation 5.
Revised: Besides batch size b and learning rate η , we specify a truncation parameter τ as an additional hyperparameter enabling us to suppress numerical noise during the half-inversion process in equation 5.

Original: As with the computation of the pseudoinverse via SVD, we set the result of the − 12 -exponentiation of every singular value less than τ to 0 .
Revised: As with the computation of the pseudoinverse via SVD, we set the result of the − 12 -exponentiation of every singular value less than τ to 0 .

Original: The use of a half inversion – instead of a full inversion – helps to prevent exploding updates of network parameters while still guaranteeing substantial progress in directions of low curvature.
Revised: The use of a half inversion – instead of a full inversion – helps to prevent exploding updates of network parameters while still guaranteeing substantial progress in directions of low curvature.

Original: With the procedure outlined above, we arrived at a balanced method that combines the advantages of optimization methods from deep learning and physics.
Revised: With the procedure outlined above, we arrived at a balanced method that combines the advantages of optimization methods from deep learning and physics.

Original: As our method uses gradients based on halfinverse Jacobians, we refer to them in short as half-inverse gradients (HIGs) in the following.
Revised: As our method uses gradients based on halfinverse Jacobians, we refer to them in short as half-inverse gradients (HIGs) in the following.

Original: Half-inverse Gradients in the Toy Example.
Revised: Half-inverse Gradients in the Toy Example.

Original: With the deﬁnition of HIGs from equation 6, we optimize the toy example introduced in section 2.1.
Revised: With the deﬁnition of HIGs from equation 6, we optimize the toy example introduced in section 2.1.

Original: The results in ﬁgure 1 show that for γ = 1 ,
Revised: The results in ﬁgure 1 show that for γ = 1 ,

Original: HIGs minimize the objective as well as Adam and Gauss-Newton’s method.
Revised: HIGs minimize the objective as well as Adam and Gauss-Newton’s method.

Original: More interestingly,
Revised: More interestingly,

Original: HIGs achieve a better result than the other two methods for γ = 0 .
Revised: HIGs achieve a better result than the other two methods for γ = 0 .

Original: On the one hand, the physics trajectory (ﬁgure 1b, right) highlights that HIGs can process information along the smallscale component of y 2 well and successfully progress along this direction.
Revised: On the one hand, the physics trajectory (ﬁgure 1b, right) highlights that HIGs can process information along the smallscale component of y 2 well and successfully progress along this direction.

Original: On the other hand, by checking neuron saturation (ﬁgure 1b, middle), we see that HIGs – in contrast to Gauss Newton – avoid oversaturating neurons.
Revised: On the other hand, by checking neuron saturation (ﬁgure 1b, middle), we see that HIGs – in contrast to Gauss Newton – avoid oversaturating neurons.

Original: 2. P RACTICAL C ONSIDERATIONS
Revised: 2. P RACTICAL C ONSIDERATIONS

Original: Computational Cost.
Revised: Computational Cost.

Original: A HIG update step consists of constructing the stacked Jacobian and computing the half-inversion.
Revised: A HIG update step consists of constructing the stacked Jacobian and computing the half-inversion.

Original: The ﬁrst step is handled via a vectorized Jacobian computation, and therefore induces a  cost comparable to regular backpropagation.
Revised: The ﬁrst step is handled via a vectorized Jacobian computation, and therefore induces a runtime cost comparable to regular backpropagation at the expense of higher memory requirements.

Original: In our experiments, this contribution is negligible compared to the half-inversion,  at the heart of which lies a divide and conquer algorithm for the singular value decomposition (Trefethen & Bau, 1997).Hence, the scaling of the method is
Revised: The latter is not too problematic since our method achieves its optimal performance at lower batch sizes than the established gradient methods. In situations where the length of the HIG step is dominated by the half-inversion, memory requirements can be further reduced by parallelizing the Jacobian computation only partially. Next, the half-inversion is performed, at the heart of which lies a divide and conquer algorithm for the singular value decomposition (Trefethen & Bau, 1997).

Original:  O ( | θ | · ( b · | y | ) 2 ) , i.e., itis linear in the number of network parameters | θ | , and quadratic in the batch size b and the dimension of the physical state | y | .
Revised: Hence, the length of a HIG step scales as O ( | θ | · ( b · | y | ) 2 ) , i.e. it linear in the number of network parameters | θ | , and quadratic in the batch size b and the dimension of the physical state | y | .

Revised: Concrete numbers for memory requirements and duration of a HIG step are listed in the appendix.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: Our method requires choosing three hyperparameters: (i)
Revised: Our method requires choosing several hyperparameters.

Original: The truncation threshold τ .

Original: Here it is important not to truncate an overly large part of the small-scale modes.

Original: After all, preserving these modes for the network update is one of the central beneﬁts of HIGs.

Original: In contrast, truncating too little can amplify numerical noise.

Original: In our experiments, the precise value of τ had little impact on the achievable accuracy. As opposed to physics optimization, where amplifying numerical noise leads to an immediate loss of accuracy, in neural network optimization, small amounts of noise can have a regularizing effect. (ii) Learning rate η . Our method requires a suitable choice of the learning rate.
Revised: First, we need a suitable choice of the learning rate.

Original: In contrast to commonly used gradient descent variants, the normalizing effects ofthe HIGs allow for performing update steps on the order of the resulting gradient.
Revised: The normalizing effects of HIGs allow for larger learning rates than commonly used gradient descent variants.

Original: Hence, we use η = 1 for many of our experiments.
Revised: We are able to use η = 1 for many of our experiments.

Original: (iii) Batch size b. This parameter affects the number of data points included in the half-inversion process.
Revised: Second, the batch size b affects the number of data points included in the half-inversion process.

Original:  Instead of thetypical averaging procedure of individual gradients of a mini batch, our approach constructs an update that is optimized for the complete batch.
Revised: It should be noted that the way the feedback of individual data points is processed is fundamentally different from the standard gradient optimizers: Instead of the averaging procedure of individual gradients of a mini batch, our approach constructs an update that is optimized for the complete batch.

Original: Consequently, the quality of updates increases with higher batch size.
Revised: Consequently, the quality of updates increases with higher batch size.

Original: However, overly large batch sizes can cause the Jacobian to become increasingly ill-conditioned and destabilize the learning progress (Trefethen & Bau, 1997).
Revised: However, overly large batch sizes can cause the Jacobian to become increasingly ill-conditioned and destabilize the learning progress (Trefethen & Bau, 1997).

Original: Examples of how the batch size inﬂuences convergence are provided in section 3 and  in the appendix.
Revised: In the appendix C, we discuss the remaining parameters τ , β , and κ with several ablation experiments to illustrate their effects in detail. [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method on three physical systems and tasks: controlling a nonlinear chain system, the Poisson problem, and a dipole problem from quantum mechanics.
Revised: We evaluate our method on three physical systems and tasks: controlling a nonlinear oscillator system, the Poisson problem, and a dipole problem from quantum mechanics.

Original: Details of the numerical[Figure] setups are given in the appendix, together with results for a broad range of hyperparameter choices.
Revised: Details of the numerical setups are given in the appendix, together with results for a broad range of hyperparameter choices.

Original: For a fair comparison, we show results with the best set of hyperparameters for each of the methods below and plot the loss against wall-clock time.
Revised: For a fair comparison, we show results with the best set of hyperparameters for each of the methods below and plot the loss against wall-clock time.

Original: All learning curves are recorded on a previously unseen data set.
Revised: All learning curves are recorded on a previously unseen data set.

Original: 3. C ONTROL OF N ONLINEAR O SCILLATORS
Revised: 3. C ONTROL OF N ONLINEAR O SCILLATORS

Original: First, we consider a control task for a system of coupled oscillators with a nonlinear interaction term.
Revised: First, we consider a control task for a system of coupled oscillators with a nonlinear interaction term.

Original: This system is of practical importance in many areas of physics, such as solid state physics (Ibach & L ¨ uth, 2003).
Revised: This system is of practical importance in many areas of physics, such as solid state physics (Ibach & L ¨ uth, 2003).

Original: Its equations of motions are governed by the Hamiltonian
Revised: Its equations of motions are governed by the Hamiltonian

Original: [Equation]
Revised: [Equation]

Original: where x i and p i denote the Hamiltonian conjugate variables of oscillator i , α the interaction strength.
Revised: where x i and p i denote the Hamiltonian conjugate variables of oscillator i , α the interaction strength.

Original: The constant vector c speciﬁes how the control is applied.
Revised: The constant vector c speciﬁes how the control is applied.

Original: u ( t ) denotes the scalar-valued control signal to be learned by the neural network.
Revised: u ( t ) denotes the scalar-valued control signal to be learned by the neural network.

Original: For the integration of the 96 time steps, we use a 4th order Runge-Kutta scheme.
Revised: For the integration of the 96 time steps, we use a 4th order Runge-Kutta scheme.

Original: The control task is to learn the signal u ( t ) that transforms a given initial state into a given target state.
Revised: The control task is to learn the signal u ( t ) that transforms a given initial state into a given target state.

Original: We use a dense neural network with three hidden layers totalling 2956 trainable parameters and ReLU activations.
Revised: We use a dense neural network with three hidden layers totalling 2956 trainable parameters and ReLU activations.

Original: The Mean-Squared-Error loss is used to quantify differences between predicted and target state.
Revised: The Mean-Squared-Error loss is used to quantify differences between predicted and target state.

Original: A visualization of this control task is shown in ﬁgure 2a.
Revised: A visualization of this control task is shown in ﬁgure 2a.

Original: Optimizer comparison.
Revised: Optimizer comparison.

Original: The goal of our ﬁrst experiments is to give a broad comparison of the proposed HIGs with commonly used optimizers.
Revised: The goal of our ﬁrst experiments is to give a broad comparison of the proposed HIGs with commonly used optimizers.

Original: This includes stochastic gradient descent (SGD), Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), RMSprop (Hinton et al., 2012), Adam (Kingma & Ba, 2015), and Gauss-Newton (GN) applied to mini batches.
Revised: This includes stochastic gradient descent (SGD), Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), RMSprop (Hinton et al., 2012), Adam (Kingma & Ba, 2015), and Gauss-Newton (GN) applied to mini batches.

Original: The results are shown in ﬁgure 2b where all curves show the best runs for each optimizer with suitable hyperparameters independently selected, as explained in the appendix.
Revised: The results are shown in ﬁgure 2b where all curves show the best runs for each optimizer with suitable hyperparameters independently selected, as explained in the appendix.

Original: We ﬁnd that the state-of-the-art optimizers stagnate early, with Adam achieving the best result oscillating around a ﬁnal loss value of 10 − 4 .
Revised: We ﬁnd that the state-of-the-art optimizers stagnate early, with Adam achieving the best result oscillating around a ﬁnal loss value of 10 − 4 .

Original: In comparison, our method and GN converge faster, exceeding Adam’s accuracy after about three minutes.
Revised: In comparison, our method and GN converge faster, exceeding Adam’s accuracy after about three minutes.

Original: While
Revised: While

Original: GN exhibits stability problems, the best stable run from our hyperparameter search reaches a loss value of 10 − 6 .
Revised: GN exhibits stability problems, the best stable run from our hyperparameter search reaches a loss value of 10 − 6 .

Original: HIGs, on the other hand, yield the best result with a loss value of 10 − 7 .
Revised: HIGs, on the other hand, yield the best result with a loss value of 10 − 7 .

Original: These results clearly show the potential of our method to process different scales of the physics solver more accurately and robustly.
Revised: These results clearly show the potential of our method to process different scales of the physics solver more accurately and robustly.

Original: They also make clear that the poor result of the widely-used network optimizers cannot be attributed to simple numerical issues as HIG converges to better levels of accuracy with an otherwise identical setup.
Revised: They also make clear that the poor result of the widely-used network optimizers cannot be attributed to simple numerical issues as HIG converges to better levels of accuracy with an otherwise identical setup.

Original: To provide further insights into the workings of HIGs, we focus on detailed comparisons with Adam as the most widely used, modern GD variant in the following.

Original: Role of the batch size.
Revised: Role of the batch size.

Original: We conduct multiple experiments using different values for the batch size b as a central parameter of our method.
Revised: We conduct multiple experiments using different values for the batch size b as a central parameter of our method.

Original: The results are shown in ﬁgure 2c.
Revised: The results are shown in ﬁgure 2c.

Original: We observe that, forAdam, all runs converge about equally quickly while HIGs show notable improvements from larger batch
Revised: We observe that for

Original: [Figure]
Revised: [Figure]

Original:  sizes.
Revised: Adam, all runs converge about equally quickly while HIGs, and GN, show notable improvements from larger batch sizes.

Original: Thisis illustrates an important difference between Adam and HIG: Adam uses an average of point-wise gradients in the mini batch.
Revised: This illustrates an important difference between Adam and HIG: Adam uses an average of point-wise gradients in the mini batch.

Original: As soon as the batch size is large enough, this average becomes representative of the expectation.
Revised: As soon as the batch size is large enough, this average becomes representative of the expectation.

Original: Further increasing the batch size has little inﬂuence on the updates.
Revised: Further increasing the batch size has little inﬂuence on the updates.

Original: Our method, on the other hand, includes the individual data point gradients without averaging.
Revised: Our method, on the other hand, includes the individual data point gradients without averaging.

Original: As shown in equation 6, we construct updates that are optimized for the whole batch by solving a linear system.
Revised: As shown in equation 6, we construct updates that are optimized for the whole batch by solving a linear system.

Original: This gives our method the ability to hit target states very accurately with increasing batch size.
Revised: This gives our method the ability to hit target states very accurately with increasing batch size.

Revised: To provide further insights into the workings of HIGs, we focus on detailed comparisons with Adam as the most widely used gradient descent variant.

Original: 3. P OISSON P ROBLEM
Revised: 3. P OISSON P ROBLEM

Original: Next we consider Poisson’s equation to illustrate advantages and current limitations of HIGs.
Revised: Next we consider Poisson’s equation to illustrate advantages and current limitations of HIGs.

Original: Poisson problems play an important role in electrostatics, Newtonian gravity, and ﬂuid dynamics (Ames, 2014).
Revised: Poisson problems play an important role in electrostatics, Newtonian gravity, and ﬂuid dynamics (Ames, 2014).

Original: For a source distribution ρ ( x ) , the goal is to ﬁnd the corresponding potential ﬁeld φ ( x ) fulﬁlling the following differential equation:
Revised: For a source distribution ρ ( x ) , the goal is to ﬁnd the corresponding potential ﬁeld φ ( x ) fulﬁlling the following differential equation:

Original: [Equation]
Revised: [Equation]

Original: Classically, Poisson problems are solved by solving the corresponding system of linear equations on the chosen grid resolution.
Revised: Classically, Poisson problems are solved by solving the corresponding system of linear equations on the chosen grid resolution.

Original: Instead, we train a dense neural network with three hidden layers and 41408 trainable parameters to solve the Poisson problem for a given right hand side ρ .
Revised: Instead, we train a dense neural network with three hidden layers and 41408 trainable parameters to solve the Poisson problem for a given right hand side ρ .

Original: We consider a two-dimensional system with a spatial discretization of 8 × 8 degrees of freedom.
Revised: We consider a two-dimensional system with a spatial discretization of 8 × 8 degrees of freedom.

Original: An example distribution and solution for the potential ﬁeld are shown in ﬁgure 3a.
Revised: An example distribution and solution for the potential ﬁeld are shown in ﬁgure 3a.

Original: Convergence and Runtime.
Revised: Convergence and Runtime.

Original: Figure 3b shows the learning curves when training the network with
Revised: Figure 3b shows the learning curves when training the network with

Original: Adam and HIGs for different learning rates.
Revised: Adam and HIGs for different learning rates.

Original: As we consider a two-dimensional system, this optimization task is challenging for both methods and requires longer training runs.
Revised: As we consider a two-dimensional system, this optimization task is challenging for both methods and requires longer training runs.

Original: We ﬁnd that both
Revised: We ﬁnd that both

Original: Adam and HIGs are able to minimize the loss by up to three orders of magnitude.
Revised: Adam and HIGs are able to minimize the loss by up to three orders of magnitude.

Original: The performance of Adam varies, and its two runs with larger η quickly slow down.
Revised: The performance of Adam varies, and its two runs with larger η quickly slow down.

Original: In terms of absolute convergence per time, the Adam curve with the smallest η (in light blue) shows advantages in this scenario.
Revised: In terms of absolute convergence per time, the Adam curve with the smallest η (in light blue) shows advantages in this scenario.

Original: However, choosing a log-scale for the time axis reveals that both methods have not fully converged.
Revised: However, choosing a log-scale for the time axis reveals that both methods have not fully converged.

Original: In particular, while the Adam curve begins to ﬂatten at the end, the slope of the HIG curve remains constant and decreases with a steeper slope than Adam.
Revised: In particular, while the Adam curve begins to ﬂatten at the end, the slope of the HIG curve remains constant and decreases with a steeper slope than Adam.

Original: The performance of Adam in this example can be explained by two reasons.
Revised: The performance of Adam in this example can be explained by two reasons.

Original: First, the time to compute a single Adam update is much smaller than for HIGs, which requires the SVD solve from equation 5.
Revised: First, the time to compute a single Adam update is much smaller than for HIGs, which requires the SVD solve from equation 5.

Original: While these could potentially be sped up with appropriate methods (Foster et al., 2011; Allen-Zhu & Li, 2016), the absolute convergence per iteration, shown in the appendix in ﬁgure 7, shows how much each HIG update improves over
Revised: While these could potentially be sped up with appropriate methods (Foster et al., 2011; Allen-Zhu & Li, 2016), the absolute convergence per iteration, shown in the appendix in ﬁgure 7, shows how much each HIG update improves over

Original: Adam.
Revised: Adam.

Original: Second, compared to the other examples, the Poisson problem is relatively simple, requiring only a single matrix inversion.
Revised: Second, compared to the other examples, the Poisson problem is relatively simple, requiring only a single matrix inversion.

Original: This represents a level of difﬁculty which Adam is still able to handle relatively well. 
Revised: This represents a level of difﬁculty which Adam is still able to handle relatively well. [Figure]

Original: HIGs with Adam Pretraining.
Revised: HIGs with Adam Pretraining.

Original: To further investigate the potential of HIGs, we repeat the training, this time using a state of the Adam model from ﬁgure 3b for the network initialization.
Revised: To further investigate the potential of HIGs, we repeat the training, this time using a state of the Adam model from ﬁgure 3b for the network initialization.

Original: While
Revised: While

Original: [Figure] Adam progresses slowly, the HIG training is able to quickly improve the state of the neural network, resulting in a signiﬁcant drop of the loss values, followed by a faster descent than Adam.
Revised: Adam progresses slowly, the HIG training is able to quickly improve the state of the neural network, resulting in a signiﬁcant drop of the loss values, followed by a faster descent than Adam.

Original: Interestingly, this experiment indicates that the HIG updates are able to improve aspects of the solution which Adam is agnostic to.
Revised: Interestingly, this experiment indicates that the HIG updates are able to improve aspects of the solution which Adam is agnostic to.

Original: Despite outlining the potential gains from faster SVD calculations, this example also highlights the quality of the HIG updates for simpler, ﬁeld-based PDEs.
Revised: Despite outlining the potential gains from faster SVD calculations, this example also highlights the quality of the HIG updates for simpler, ﬁeld-based PDEs.

Original: 3. Q UANTUM D IPOLE
Revised: 3. Q UANTUM D IPOLE

Original: As a ﬁnal example we target the quantum dipole problem, which is a standard control task formulated on the Schr ¨ odinger equation and is highly relevant in quantum physics (Von Neumann, 2018).
Revised: As a ﬁnal example, we target the quantum dipole problem, which is a standard control task formulated on the Schr ¨ odinger equation and is highly relevant in quantum physics (Von Neumann, 2018).

Original: Given an initial and a target state we train a neural network to compute the temporal transition function u ( t ) in an inﬁnite-well potential V according the evolution equation of the physical state Ψ :
Revised: Given an initial and a target state, we train a neural network to compute the temporal transition function u ( t ) in an inﬁnite-well potential V according the evolution equation of the physical state Ψ :

Original: [Equation]
Revised: [Equation]

Original: We employ a modiﬁed Crank-Nicolson scheme (Winckel et al., 2009) for the discretization of spatial and temporal derivatives.
Revised: We employ a modiﬁed Crank-Nicolson scheme (Winckel et al., 2009) for the discretization of spatial and temporal derivatives.

Original: Thus, each training iteration consists of multiple implicit time integration steps – 384 in our setup – for the forward as well as the backward pass of each mini-batch.
Revised: Thus, each training iteration consists of multiple implicit time integration steps – 384 in our setup – for the forward as well as the backward pass of each mini-batch.

Original: The control task consists of inferring a signal that converts the ground state to a given randomized linear combination of the ﬁrst and the second excited state.
Revised: The control task consists of inferring a signal that converts the ground state to a given randomized linear combination of the ﬁrst and the second excited state.

Original: We use a dense neural network with three hidden layers, 9484 trainable parameters and tanh activations.
Revised: We use a dense neural network with three hidden layers, 9484 trainable parameters and tanh activations.

Original: As loss function, we use the inner product of quantum mechanics L (Ψ a , Ψ b ) = 1 − |(cid:104) Ψ a , Ψ b (cid:105)| 2 .A visualization of this control task is shown in ﬁgure 4a.
Revised: As loss function, we use the inner product of quantum mechanics L (Ψ a , Ψ b ) = 1 − |(cid:104) Ψ a , Ψ b (cid:105)| 2 . A visualization of this control task is shown in ﬁgure 4a.

Original: Speed and Accuracy.
Revised: Speed and Accuracy.

Original: We observe that HIGs minimize the loss faster and reach a better ﬁnal level of accuracy than Adam (ﬁgure 4b).
Revised: We observe that HIGs minimize the loss faster and reach a better ﬁnal level of accuracy than Adam (ﬁgure 4b).

Original: While the Adam run with the largest learning rate drops faster initially, its ﬁnal performance is worse than all other runs.
Revised: While the Adam run with the largest learning rate drops faster initially, its ﬁnal performance is worse than all other runs.

Original: In this example, the difference between the ﬁnal loss values is not as large as for the previous experiments.
Revised: In this example, the difference between the ﬁnal loss values is not as large as for the previous experiments.

Original: This is due to the numerical accuracy achievable by a pure physics optimization, which for our choice of parameters is around 10 − 6 .
Revised: This is due to the numerical accuracy achievable by a pure physics optimization, which for our choice of parameters is around 10 − 6 .

Original: Hence, we can not expect to improve upon this lower bound for derived learning problems.
Revised: Hence, we can not expect to improve upon this lower bound for derived learning problems.

Original: Our results indicate that the partial inversion of the Jacobian successfully leads to the observed improvements in convergence speed and accuracy.
Revised: Our results indicate that the partial inversion of the Jacobian successfully leads to the observed improvements in convergence speed and accuracy.

Original: Low and High Energy Components.
Revised: Low and High Energy Components.

Original: The quantum control problem also serves to highlight the weakness of gradient-based optimizers in appropriately processing different scales of the solutions.
Revised: The quantum control problem also serves to highlight the weakness of gradient-based optimizers in appropriately processing different scales of the solutions.

Original: In the initial training stage, the Adam curves stagnate at a loss value of 0 . 5 .
Revised: In the initial training stage, the Adam curves stagnate at a loss value of 0 . 5 .

Original: This is most pronounced for η = 10 − 4 in dark blue.
Revised: This is most pronounced for η = 10 − 4 in dark blue.

Original: To explain this effect, we recall that our learning objective targets transitions to combinations of the 1st and 2nd excited quantum states, and both states appear on average with equal weight in the training data.
Revised: To explain this effect, we recall that our learning objective targets transitions to combinations of the 1st and 2nd excited quantum states, and both states appear on average with equal weight in the training data.

Original: The transition to the energetically higher state is more difﬁcult and connected to smaller scales in the physics solver causing Adam to ﬁt the lower-energetic component
Revised: The transition to the energetically higher state is more difﬁcult and connected to smaller scales in the physics solver causing Adam to ﬁt the lower-energetic component

Original: ﬁrst.
Revised: ﬁrst.

Original: In contrast, our method is constructed to consider all scales via the partial inversion of the Jacobian and can therefore process the small scales more efﬁciently.
Revised: In contrast, our method is constructed to consider all scales via the partial inversion of the Jacobian and can therefore process the small scales more efﬁciently.

Original: As a consequence, it does not stagnate at 0 .
Revised: As a consequence, it does not stagnate at 0 .

Original: We support this explanation by explicitly plotting separate loss curves in ﬁgure 4c quantifying how well the low and high energy component of the target state was learned.
Revised: We support this explanation by explicitly plotting separate loss curves in ﬁgure 4c quantifying how well the low and high energy component of the target state was learned.

Original: Not only does Adam prefer to minimize the low-energy loss, it also increases the same loss again before it is able to minimize the high-energy loss.
Revised: Not only does Adam prefer to minimize the low-energy loss, it also increases the same loss again before it is able to minimize the high-energy loss.

Original: In contrast, we observe that the HIG minimizes both losses uniformly.
Revised: In contrast, we observe that the HIG minimizes both losses uniformly.

Original: This is another indication that the theory outlined above carries over to an even processing of different gradient magnitudes in joint physics and neural network objectives.
Revised: This is another indication that the theory outlined above carries over to an even processing of different gradient magnitudes in joint physics and neural network objectives.

Original:  R ELATED W ORKOptimization algorithms.
Revised:  R ELATED W ORK

Revised: Optimization algorithms.

Original: Optimization on continuous spaces is a huge ﬁeld that offers a vast range of techniques (Ye et al., 2019).
Revised: Optimization on continuous spaces is a huge ﬁeld that offers a vast range of techniques (Ye et al., 2019).

Original: Famous examples are gradient descent (Curry, 1944), GaussNewton’s method (Gill & Murray, 1978), Conjugate Gradient (Hestenes et al., 1952), or the limitedmemory BFGS algorithm (Liu & Nocedal, 1989).
Revised: Famous examples are gradient descent (Curry, 1944), GaussNewton’s method (Gill & Murray, 1978), Conjugate Gradient (Hestenes et al., 1952), or the limitedmemory BFGS algorithm (Liu & Nocedal, 1989).

Original: In deep learning, the preferred methods instead rely on ﬁrst order information in the form of the gradient, such as SGD (Bottou, 2010) and RMSProp (Hinton et al., 2012).
Revised: In deep learning, the preferred methods instead rely on ﬁrst order information in the form of the gradient, such as SGD (Bottou, 2010) and RMSProp (Hinton et al., 2012).

Original: Several methods approximate the diagonal of the Hessian to improve scaling behavior, such as Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), and most prominently, Adam (Kingma & Ba, 2015).
Revised: Several methods approximate the diagonal of the Hessian to improve scaling behavior, such as Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), and most prominently, Adam (Kingma & Ba, 2015).

Original: However, due to neglecting all inter-dependencies of the parameters, these methods are limited in their capabilities to handle uneven learning objectives.
Revised: However, due to neglecting all inter-dependencies of the parameters, these methods are limited in their capabilities to handle uneven learning objectives.

Original: Despite the computational cost, higher-order methods have also been studied in deep learning (Pascanu & Bengio, 2013).
Revised: Despite the computational cost, higher-order methods have also been studied in deep learning (Pascanu & Bengio, 2013) .

Original: Practical methods have been suggested bystochastic LBFGS variants (Schraudolph et al., 2007), by using a Kroenecker-factorization of the
Revised: Practical methods have been suggested by using a Kroenecker-factorization of the

Original: Fisher matrix (Martens & Grosse, 2015), via iterative linear solvers in (Martens, 2010), or by recursive approximations of the Hessian (Botev et al., 2017).
Revised: Fisher matrix (Martens & Grosse, 2015) via iterative linear solvers in (Martens, 2010), or by recursive approximations of the Hessian (Botev et al., 2017).

Original: Likewise, optimization algorithms were proposed as layers in neural network architectures Amos & Kolter (2017).

Original: To the best of our knowledge, the only other technique speciﬁcally targeting optimization of neural networks with physics objectives are ”physical gradients” (Holl et al., 2021).
Revised: To the best of our knowledge, the only other technique speciﬁcally targeting optimization of neural networks with physics objectives are physical gradients (PGs) (Holl et al., 2021).

Original: Despite targeting inversions, they fundamentally differ from our approach as they are based on inverse physics solvers, while we address the problem byinverting the Jacobian and treating network and solver as an entity.
Revised: Despite targeting inversions, they fundamentally differ from our approach as PGs are based on inverse physics solvers, while we address the problem by treating network and solver as an entity and half-inverting its Jacobian.

Revised: Thus, we work on the level of linear approximations while PGs are able to use higher-order information provided that an higher-order inverse solver exists.

Revised: Additionally, PGs compute their update by averaging gradients over different data points, in line with typical gradient-based neural network optimizers.

Revised: HIGs instead process the feedback of different data points via collective inversion.

Original: Incorporating physics.
Revised: Incorporating physics.

Original: Many works involve differentiable formulations of physical models, e.g., for robotics (Toussaint et al., 2018), to enable deep architectures (Chen et al., 2018), as a means for scene understanding (Battaglia et al., 2013; Santoro et al., 2017), or the control of rigid body environments de Avila Belbute-Peres et al.
Revised: Many works involve differentiable formulations of physical models, e.g., for robotics (Toussaint et al., 2018), to enable deep architectures (Chen et al., 2018), as a means for scene understanding (Battaglia et al., 2013; Santoro et al., 2017), or the control of rigid body environments de Avila Belbute-Peres et al.

Original: Additional works have shown the advantages of physical loss formulations Greydanus et al.
Revised: Additional works have shown the advantages of physical loss formulations Greydanus et al.

Original: ; Cranmer et al.
Revised: ; Cranmer et al.

Original: Differentiable simulation methods were proposed for a variety of phenomena, e.g. for ﬂuids (Schenck & Fox, 2018), PDE discretizations (Bar-Sinai et al., 2019), molecular dynamics (Wang et al., 2020), reducing numerical errors (Um et al., 2020), and cloth (Liang et al., 2019; Rasheed et al., 2020).
Revised: Differentiable simulation methods were proposed for a variety of phenomena, e.g. for ﬂuids (Schenck & Fox, 2018), PDE discretizations (Bar-Sinai et al., 2019), molecular dynamics (Wang et al., 2020), reducing numerical errors (Um et al., 2020), and cloth (Liang et al., 2019; Rasheed et al., 2020).

Original: It is worth noting that none of these works question the use standard deep learning optimizers, such as Adam.
Revised: It is worth noting that none of these works question the use standard deep learning optimizers, such as Adam.

Original: In addition, by now a variety of specialized software frameworks are available to realize efﬁcient implementations (Hu et al., 2020; Schoenholz & Cubuk, 2019; Holl et al., 2020).
Revised: In addition, by now a variety of specialized software frameworks are available to realize efﬁcient implementations (Hu et al., 2020; Schoenholz & Cubuk, 2019; Holl et al., 2020).

Original:  D ISCUSSION AND O UTLOOK
Revised:  D ISCUSSION AND O UTLOOK

Original: We investigated the training of neural networks for physical simulations.
Revised: We investigated the training of neural networks for physical simulations.

Original: As including a physics solver greatly affects the gradient ﬂow, we questioned the current practice of using the standard gradient-based network optimizers for training.
Revised: As including a physics solver greatly affects the gradient ﬂow, we questioned the current practice of using the standard gradient-based network optimizers for training.

Original: By considering ways to smoothly transition from gradient descent to the Gauss-Newton algorithm, we identiﬁed a procedure to better address the joint optimization problem.
Revised: By considering ways to smoothly transition from gradient descent to the Gauss-Newton algorithm, we identiﬁed a procedure to better address the joint optimization problem.

Original: Compared to existing optimizers, our method can achieve a faster and more accurate minimization of the learning objectives.
Revised: Compared to existing optimizers, our method can achieve a faster and more accurate minimization of the learning objectives.

Original: We believe that our work can provide a starting point for further research into improved learning methods for physical problems.
Revised: We believe that our work can provide a starting point for further research into improved learning methods for physical problems.

Original: In the context of HIGs, highly interesting avenues for future work are efﬁcient methods for the inversion of the Jacobian matrix, or including unstructured solvers.
Revised: In the context of HIGs, highly interesting avenues for future work are efﬁcient methods for the inversion of the Jacobian matrix, or including unstructured solvers.

Original: The combination of ﬁnite elements (Zienkiewicz et al., 1977) with graph-neural networks (Battaglia et al., 2013; Ummenhofer et al., 2020; Pfaff et al., 2020) provides a sound foundation that is especially interesting for future applications of HIGs.
Revised: The combination of ﬁnite elements (Zienkiewicz et al., 1977) with graph-neural networks (Battaglia et al., 2013; Ummenhofer et al., 2020; Pfaff et al., 2020) provides a sound foundation that is especially interesting for future applications of HIGs.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To ensure reproducibility of our work we will publish all source code that is necessary to reproduce the results of our paper upon acceptance.
Revised: To ensure reproducibility of our work we will publish all source code that is necessary to reproduce the results of our paper upon acceptance.

Original: This includes data generation, neural network training and evaluation.
Revised: This includes data generation, neural network training and evaluation.

Original: Hyperparameters are also provided in the appendix for completeness
Revised: Hyperparameters are also provided in the appendix for completeness

Original: A PPENDIX A G AUSS -N EWTON ’ S M ETHOD IN THE C ONTEXT OF P HYSICAL D EEP L EARNING
Revised: A PPENDIX A G AUSS -N EWTON ’ S M ETHOD IN THE C ONTEXT OF P HYSICAL D EEP L EARNING

Original: Our work considers optimization algorithms for functions of the form f ( x ; θ ) =
Revised: Our work considers optimization algorithms for functions of the form f ( x ; θ ) =

Original: y with θ, ∆ θ ∈ R t , denoting weight vector and weight update vector, respectively, while x ∈ R n and y ∈ R m denote input and output.
Revised: y with θ , ∆ θ ∈ R t , denoting weight vector and weight update vector, respectively, while x ∈ R n and y ∈ R m denote input and output.

Original: The learning process solves the minimization problem argmin θ L ( f ( x ; θ ) , ˆ y ) via a sequence θ k +

Revised: The learning process solves the minimization problem argmin θ L ( f ( x ; θ ) , ˆ y ) via a sequence θ k +

Original: =
Revised: =

Original: θ k + η ∆ θ .
Revised: θ k + η ∆ θ .

Original: Here, ˆ y are the reference solutions, and we target losses of the form L ( x, ˆ y ; θ ) = (cid:80)i
Revised: Here, ˆ y are the reference solutions, and we target losses of the form L ( x , ˆ y ; θ ) = (cid:80)

Original:  l (cid:0) f
Revised: i l (cid:0) f

Original: ( x i ; θ ) , ˆ y i (cid:1) with i being an index for multiple data points (i.e., observations).
Revised: ( x i ; θ ) , ˆ y i (cid:1) with i being an index for multiple data points (i.e., observations).

Original: l denotes a quadratic loss (cid:80)
Revised: l denotes the l2-loss (cid:80)

Original: j || x
Revised: j || x

Original: j − ˆ y j || 22 with j referencing the entries of a mini batch of size b .
Revised: j − ˆ y j || 22 with j referencing the entries of a mini batch of size b .

Original: Update Step.
Revised: Update Step.

Original: With this notation, the update step of the Gauss-Newton algorithm (Adby, 2013) for η = 1 is given by:
Revised: With this notation, the update step of the Gauss-Newton algorithm (Adby, 2013) for η = 1 is given by:

Original: [Equation]
Revised: [Equation]

Original: The size of the Jacobian matrix is given by y-space and parameter dimensions.
Revised: The size of the Jacobian matrix is given by y -space and parameter dimensions.

Original: Hence, for a full-rank
Revised: For a full-rank

Original: Jacobian corresponding toa non-constrained optimization, the Gauss-Newton update is equivalent to:
Revised: Jacobian corresponding to non-constrained optimization, the Gauss-Newton update is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: This shortened form of the update step is given in equation 3, and is the basis for our discussion in the main text.
Revised: This shortened form of the update step is given in equation 3, and is the basis for our discussion in the main text.

Original: Geometric Interpretation.
Revised: Geometric Interpretation.

Original: These updates can be regarded as a gradient descent in the y -variables up to ﬁrst order in the update step as can be seen with the following equation:
Revised: These updates can be regarded as a gradient descent in the y -variables up to ﬁrst order in the update step as can be seen with the following equation:

Original: [Equation]
Revised: [Equation]

Original: In a constrained setting, we can reparametrize the coordinates to obtain an unconstrained optimization problem on the accessible manifold.
Revised: In a constrained setting, we can reparametrize the coordinates to obtain an unconstrained optimization problem on the accessible manifold.

Original: We can then repeat the argument above.
Revised: We can then repeat the argument above.

Original: Hence, even in the constrained setting the Gauss-Newton algorithm can be seen as gradient descent on the y manifold.
Revised: Hence, even in the constrained setting the Gauss-Newton algorithm can be seen as gradient descent on the y -manifold.

Original: In ﬁgure 1 of the main paper this behavior is visible in both the well- and ill-conditioned case in terms of the straight line that the GN curve takes towards the target even in the presence of a neural network.
Revised: In ﬁgure 1 of the main paper this behavior is visible in both the well- and ill-conditioned case in terms of the straight line that the GN curve takes towards the target even in the presence of a neural network.

Original: The ﬁrst few, seemingly erratic, movements in y -space which are observable for both GN and HIG stem from the inversion operations.
Revised: The ﬁrst few, seemingly erratic, movements in y -space which are observable for both GN and HIG stem from the inversion operations.

Original: The behavior of GN to perform steepest descent in y -space stands in contrast to gradient descent methods, which instead perform steepest descent on the θ manifold.
Revised: The behavior of GN to perform steepest descent in y -space stands in contrast to gradient descent methods, which instead perform steepest descent on the θ -manifold.

Original: In combination, this geometric viewpoint, as summarized in table 1 above motivates our construction of the HIGs
Revised: In combination, this geometric viewpoint, as summarized in table 1 above motivates our construction of the HIGs

Original: Stability of Inversions in the Context of Physical Deep Learning.
Revised: Stability of Inversions in the Context of Physical Deep Learning.

Original: In the following, we illustrate how the full inversion of GN can lead to instabilities at training time.
Revised: In the following, we illustrate how the full inversion of GN can lead to instabilities at training time.

Original: Interestingly, physical solvers are not the only cause of small singular values in the Jacobian.
Revised: Interestingly, physical solvers are not the only cause of small singular values in the Jacobian.

Original: They can also occur when applying equation 11 to a mini batch to train a neural network and are not caused by numerical issues.
Revised: They can also occur when applying equation 11 to a mini batch to train a neural network and are not caused by numerical issues.

Original: Consider the simple case of two data points ( x 1 , ˆ y 1 ) and ( x 2 , ˆ y 2 ) and a one-dimensional output.
Revised: Consider the simple case of two data points ( x 1 , ˆ y 1 ) and ( x 2 , ˆ y 2 ) and a one-dimensional output.

Original: Let f be the neural network and J the Jacobian, which is in this case the gradient of the network output.
Revised: Let f be the neural network and J the Jacobian, which is in this case the gradient of the network output.

Original: Then equation 11 yields:
Revised: Then equation 11 yields:

Original: [Equation]
Revised: [Equation]

Original: Next, we linearly approximate the second row by using the Hessian H by assuming the function to be learned is ˆ f , i.e. ˆ f ( x 1 ) =
Revised: Next, we linearly approximate the second row by using the Hessian H by assuming the function to be learned is ˆ f , i.e. ˆ f ( x 1 ) =

Original: y 1 and ˆ f ( x 2 )
Revised: y 1 and ˆ f ( x 2 )

Original: =
Revised: =

Original: y 2 .
Revised: y 2 .

Original: Neglecting terms beyond the linear approximation, we receive:
Revised: Neglecting terms beyond the linear approximation, we receive:

Original: [Equation]
Revised: [Equation]

Original: Considering the case of two nearby data points, i.e. x 2 − x 1 being small, the two row vectors in the stacked Jacobian on the left-hand side are similar, i.e. the angle between them is small.
Revised: Considering the case of two nearby data points, i.e. x 2 − x 1 being small, the two row vectors in the stacked Jacobian on the left-hand side are similar, i.e. the angle between them is small.

Original: This leads to a small singular value of the stacked Jacobian.
Revised: This leads to a small singular value of the stacked Jacobian.

Original: In the limit of x 2 = x 1 both row vectors are linearly dependant and hence, one singular value becomes zero.
Revised: In the limit of x 2 = x 1 both row vectors are linearly dependant and hence, one singular value becomes zero.

Original: Moreover, even if x 2 is not close to x 1 , small singular values can occur if the batch size increases: for a growing number of row vectors it becomes more and more likely that the Jacobian contains similar or linearly dependent vectors.
Revised: Moreover, even if x 2 is not close to x 1 , small singular values can occur if the batch size increases: for a growing number of row vectors it becomes more and more likely that the Jacobian contains similar or linearly dependent vectors.

Original: After inversion, a small singular value becomes large.
Revised: After inversion, a small singular value becomes large.

Original: This leads to a large update ∆ θ GN when the right-hand side of equation 14 overlaps with the corresponding singular vector.
Revised: This leads to a large update ∆ θ GN when the right-hand side of equation 14 overlaps with the corresponding singular vector.

Original: This can easily happen if the linear approximation of the right-hand side is poor, for instance when ˆ f is a solution to an inverse physics problem.
Revised: This can easily happen if the linear approximation of the right-hand side is poor, for instance when ˆ f is a solution to an inverse physics problem.

Original: Then ˆ f can have multiple modes and can, even within a mode, exhibit highly sensitive or even singular behavior.
Revised: Then ˆ f can have multiple modes and can, even within a mode, exhibit highly sensitive or even singular behavior.

Original: In turn, applying large updates to the network weights naturally can lead to the oversaturation of neurons, as illustrated above, and diverging training runs in general.
Revised: In turn, applying large updates to the network weights naturally can lead to the oversaturation of neurons, as illustrated above, and diverging training runs in general.

Original: As illustrated in the main paper, these inherent problems of GN are alleviated by the partial inversion of the HIG.
Revised: As illustrated in the main paper, these inherent problems of GN are alleviated by the partial inversion of the HIG.

Original: It yields a fundamentally different order of scaling via its square-root inversion, which likewise does not guarantee that small singular values lead to overshoots (hence the truncation), but in general strongly stabilizes the training process.
Revised: It yields a fundamentally different order of scaling via its square-root inversion, which likewise does not guarantee that small singular values lead to overshoots (hence the truncation), but in general strongly stabilizes the training process.

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: In the following, we provide details of the physical simulations used for our experiments in the main paper.
Revised: In the following, we provide details of the physical simulations used for our experiments in the main paper.

Original: For the different methods, we use the following abbreviations: half-inverse gradients (HIG),
Revised: For the different methods, we use the following abbreviations: half-inverse gradients (HIG),

Original: Gauss-Newton’s method (GN), and stochastic gradient descent (GD).
Revised: Gauss-Newton’s method (GN), and stochastic gradient descent (GD).

Original: Learning rates are denoted by η , batch sizes by b , and truncation parameters for HIG and GN by τ .
Revised: Learning rates are denoted by η , batch sizes by b , and truncation parameters for HIG and GN by τ .

Original: All loss results are given for the average loss over a test set with samples distinct from the training data set.
Revised: All loss results are given for the average loss over a test set with samples distinct from the training data set.

Original: For each method, we run a hyperparameter search for every experiment, varying the learning rate by several orders of magnitude, and the batch size in factors of two.
Revised: For each method, we run a hyperparameter search for every experiment, varying the learning rate by several orders of magnitude, and the batch size in factors of two.

Original: Unless noted otherwise, the best runs in terms of ﬁnal test loss were selected and shown in the main text.
Revised: Unless noted otherwise, the best runs in terms of ﬁnal test loss were selected and shown in the main text.

Original: The following sections contain several examples from the hyperparameter search to illustrate how the different methods react to the changed settings.
Revised: The following sections contain several examples from the hyperparameter search to illustrate how the different methods react to the changed settings.

Original: Runtime Measurements Runtimes for the non-linear chain and quantum dipole were measured on a machine with Intel Xeon 6240 CPUs and NVIDIA GeForce RTX 2080 Ti GPUs.
Revised: Runtime Measurements Runtimes for the non-linear chain and quantum dipole were measured on a machine with Intel Xeon 6240 CPUs and NVIDIA GeForce RTX 2080 Ti GPUs.

Original: The Poisson experimentsadditionally used an Intel Xeon W-2235 CPU with NVIDIA Quadro RTX 8000 GPU, and an Intel i9-9900X CPU with a NVIDIA TITAN RTX GPU. We experimentally veriﬁed that these platforms yield an on-par performance for our implementation.
Revised: The Poisson experiments used an Intel Xeon W-2235 CPU with NVIDIA Quadro RTX 8000 GPU.We experimentally veriﬁed that these platforms yield an on-par performance for our implementation.

Original: As deep learning API we used TensorFlow version 2.5.
Revised: As deep learning API we used TensorFlow version 2.5.

Original: If not stated otherwise, each experiment retained the default settings.
Revised: If not stated otherwise, each experiment retained the default settings.

Original: All runtime graphs in the main paper and appendix contain wall-clock measurements that include all steps of a learning run, such as initialization, in addition to the evaluation time of each epoch.
Revised: All runtime graphs in the main paper and appendix contain wall-clock measurements that include all steps of a learning run, such as initialization, in addition to the evaluation time of each epoch.

Original: However, the evaluations of the test sets to determine the performance in terms of loss are not included.
Revised: However, the evaluations of the test sets to determine the performance in terms of loss are not included.

Original: As optimizers such as Adam typically performs a larger number of update steps including these evaluations would have put these optimizers at an unnecessary disadvantage.
Revised: As optimizers such as Adam typically performs a larger number of update steps including these evaluations would have put these optimizers at an unnecessary disadvantage.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: T OY E XAMPLE ( SECTION 2.1)
Revised: T OY E XAMPLE ( SECTION 2.1)

Original: For the toy example, the target function is given by ˆ f ( x ) =
Revised: For the toy example, the target function is given by ˆ f ( x ) =

Original: (sin(6 x ) , cos(9 x )) .
Revised: (sin(6 x ) , cos(9 x )) .

Original: We used a dense neural network consisting of one hidden layer with 7 neurons and tanh activation, and an output layer with 2 neurons and linear activation.
Revised: We used a dense neural network consisting of one hidden layer with 7 neurons and tanh activation, and an output layer with 2 neurons and linear activation.

Original: For training, we use 1024 data points uniformly sampled from the [ − 1 , 1] interval, and a batch size of 256.
Revised: For training, we use 1024 data points uniformly sampled from the [ − 1 , 1] interval, and a batch size of 256.

Original: The test set consists of 1024 new samples.
Revised: The test set consists of 1024 new samples.

Original: For the optimizers, the following hyperparameters were used for both the well-conditioned loss and the ill-conditioned loss: Adam η = 0 .
Revised: For the optimizers, the following hyperparameters were used for both the well-conditioned loss and the ill-conditioned loss: Adam η = 0 .

Original: 3 ; has no learning rate (equivalent to η = 1 ), τ = 10 − 4 ; HIG η = 1 . 0 , τ = 10 − 6 .
Revised: 3 ; has no learning rate (equivalent to η = 1 ), τ = 10 − 4 ; HIG η = 1 . 0 , τ = 10 − 6 .

Original: B. C ONTROL OF N ONLINEAR O SCILLATORS ( SECTION 3.1)
Revised: B. C ONTROL OF N ONLINEAR O SCILLATORS ( SECTION 3.1)

Revised: The Hamiltonian function given in equation 7 leads to the following equations of motions:

Original:  The simulations of the nonlinear oscillators were performed for two mass points and a time interval of 12 units with a time step ∆ t = 0 .
Revised: ¨ [Equation] The simulations of the nonlinear oscillators were performed for two mass points and a time interval of 12 units with a time step ∆ t = 0 .

Original: This results in 96 time steps via 4th order Runge-Kutta per learning iteration.
Revised: This results in 96 time steps via 4th order Runge-Kutta per learning iteration.

Original: We generated 4096 data points for a control vector c = (0 . 0 , 3 . 0) , and an interaction strength α = 1 . 0 with randomized conjugate variables x and p .
Revised: We generated 4096 data points for a control vector c = (0 . 0 , 3 . 0) , and an interaction strength α = 1 . 0 with randomized conjugate variables x and p .

Original: The test set consists of 4096 new data points.
Revised: The test set consists of 4096 new data points.

Original: For the neural network, we set up a fully-connected network with ReLU activations passing inputs through three hidden layers with 20 neurons in each layer before being mapped to a 96 output layer with linear activation.
Revised: For the neural network, we set up a fully-connected network with ReLU activations passing inputs through three hidden layers with 20 neurons in each layer before being mapped to a 96 output layer with linear activation.

Original: For the comparison with other optimizers (ﬁgure 2b) we performed a broad hyperparameter search for each method, as outlined above, to determine suitable settings.
Revised: For the comparison with other optimizers (ﬁgure 2b) we performed a broad hyperparameter search for each method, as outlined above, to determine suitable settings.

Original: The parameters for Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), RMSprop (Hinton et al., 2012), Gauss-Newton (Gill & Murray, 1978), HIGs, and stochastic gradient descent (Curry, 1944) are summarized in table 2.
Revised: The parameters for Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), RMSprop (Hinton et al., 2012), Gauss-Newton (Gill & Murray, 1978), HIGs, and stochastic gradient descent (Curry, 1944) are summarized in table 2.

Original: Instead, for ﬁgure 2c the following hyperparameters were used: η = 3 · 10 − 4 for Adam, and η = 1 . 0 , τ = 10 − 6 for HIG.
Revised: For ﬁgure 2c the following hyperparameters were used: η = 3 · 10 − 4 for Adam, and η = 1 . 0 , τ = 10 − 6 for HIG.

Original: Further Experiments.
Revised: Further Experiments.

Original: Figure 5 and ﬁgure 6 contain additional runs with different hyperparameters for the method comparison of ﬁgure 2b in the main paper.
Revised: Figure 5 and ﬁgure 6 contain additional runs with different hyperparameters for the method comparison of ﬁgure 2b in the main paper.

Original: The graphs illustrate that all ﬁve method do not change their behavior signiﬁcantly for the different batch sizes in each plot, but become noticeably unstable for larger learning rates η (plots on the right sides of each section).
Revised: The graphs illustrate that all ﬁve method do not change their behavior signiﬁcantly for the different batch sizes in each plot, but become noticeably unstable for larger learning rates η (plots on the right sides of each section).

Revised: Details on the memory footprint and update durations can be found in table 3.

Revised: Since our simulations were not limited by memory, we used an implementation for the Jacobian computation of HIGs, which scales quadratically in the batch size.

Revised: Should this become a bottleneck, this scaling could potentially be made linear by exploiting that the Jacobian of the physical solver for multiple data points is blockdiagonal.

Revised: [Figure]

Revised: [Figure]

Original: B. P OISSON P ROBLEM ( SECTION 3.2)
Revised: B. P OISSON P ROBLEM ( SECTION 3.2)

Original: We discretize Poisson’s equation on a regular grid for a two-dimensional domain Ω = [0 , 8] × [0 , 8] with a grid spacing of ∆ x = 1 .
Revised: We discretize Poisson’s equation on a regular grid for a two-dimensional domain Ω = [0 , 8] × [0 , 8] with a grid spacing of ∆ x = 1 .

Original: Dirichlet boundary conditions of φ = 0 are imposed on all four sides of Ω .
Revised: Dirichlet boundary conditions of φ = 0 are imposed on all four sides of Ω .

Original: The Laplace operator is discretized with a ﬁnite difference stencil (Ames, 2014).
Revised: The Laplace operator is discretized with a ﬁnite difference stencil (Ames, 2014).

Original: For the neural network, we set up a fully-connected network with tanh activation functions.
Revised: For the neural network, we set up a fully-connected network with tanh activation functions.

Original: The 8x8 inputs pass through three hidden layers with 64, 256 and 64 neurons, respectively, before being mapped to 8x8 in the output layer.
Revised: The 8x8 inputs pass through three hidden layers with 64, 256 and 64 neurons, respectively, before being mapped to 8x8 in the output layer.

Original: For training, source distributions ρ are sampled from random frequencies in Fourier space, and transformed to real space via the inverse Fourier transform.
Revised: For training, source distributions ρ are sampled from random frequencies in Fourier space, and transformed to real space via the inverse Fourier transform.

Original: The mean value is normalized to zero.
Revised: The mean value is normalized to zero.

Original: We sample data on-the-ﬂy, resulting in an effectively inﬁnite data set.
Revised: We sample data on-the-ﬂy, resulting in an effectively inﬁnite data set.

Original: This makes a separate test set redundant as all training data is previously unseen. 
Revised: This makes a separate test set redundant as all training data is previously unseen. [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Original: Further Experiments.
Revised: Further Experiments.

Original: Figure 7 shows Adam and HIG runs from ﬁgure 3b over epochs.
Revised: Figure 7a shows Adam and HIG runs from ﬁgure 3b over epochs.

Original: The HIG runs convergesigniﬁcantly faster per iteration, which indicates that HIGs perform qualitatively better updates.
Revised: The HIG runs converge faster per iteration, which indicates that HIGs perform qualitatively better updates.

Original: Additionally, we use the pretrained HIG run from ﬁgure 3c as a starting point for further Adam training.
Revised: Additionally, we use the pretrained HIG run from ﬁgure 3c as a starting point for further Adam training.

Original: The results are shown in 8.
Revised: The results are shown in 7b.

Original: We observe that the network quickly looses the progress the HIGs have made, and continues with a loss value similar to the orginal Adam run.
Revised: We observe that the network quickly looses the progress the HIGs have made and continues with a loss value similar to the orginal Adam run.

Original: This again[Figure] [Figure] supports our intuition that Adam, in contrast to HIGs, cannot harness the full potential of the physics solver.
Revised: This again supports our intuition that Adam, in contrast to HIGs, cannot harness the full potential of the physics solver.

Revised: Details on the memory footprint and update durations can be found in table

Original: B. Q UANTUM D IPOLE ( SECTION 3.3)
Revised: B. Q UANTUM D IPOLE ( SECTION 3.3)

Original: For the quantum dipole problem, we discretize the Schr ¨ odinger equation on a spatial domain Ω = [0 , 2] with a spacing of ∆ x = 0 .
Revised: For the quantum dipole problem, we discretize the Schr¨odinger equation on a spatial domain Ω = [0 , 2] with a spacing of ∆ x = 0 .

Original: 133 resulting in 16 discretization points.
Revised: 133 resulting in 16 discretization points.

Original: In time, we simulate up to a time of 19 . 2 with a time step of ∆ t = 0 .
Revised: In time, we simulate up to a time of 19 . 2 with a time step of ∆ t = 0 .

Original: 05 , which yields 384 time steps.
Revised: 05 , which yields 384 time steps.

Original: Spatial and temporal discretization use a modiﬁed Crank-Nicolson scheme (Winckel et al., 2009) which is tailored to quantum simulations.
Revised: Spatial and temporal discretization use a modiﬁed Crank-Nicolson scheme (Winckel et al., 2009) which is tailored to quantum simulations.

Original: The training data set consists of 1024 randomized superpositions of the ﬁrst and second excited state, while the test set contains a new set of 1024 randomized superpositions.
Revised: The training data set consists of 1024 randomized superpositions of the ﬁrst and second excited state, while the test set contains a new set of 1024 randomized superpositions.

Original: For the neural network, we set up a fully-connected network with tanh activations passing the inputs[Figure] [Figure] through three hidden layers with 20 neurons in each layer before being mapped to a 384 output layer with linear activation.
Revised: For the neural network, we set up a fully-connected network with tanh activations passing the inputs through three hidden layers with 20 neurons in each layer before being mapped to a 384 output layer with linear activation.

Original: Overall, the network contains 9484 trainable parameters.
Revised: Overall, the network contains 9484 trainable parameters.

Original: Experimental details.
Revised: Experimental details.

Original: For the training runs in ﬁgure 4b, Adam used b = 16 , while for HIG b = 16 , and τ = 10 − 5 were used.
Revised: For the training runs in ﬁgure 4b, Adam used b = 16 , while for HIG b = 16 , and τ = 10 − 5 were used.

Original: For the training runs in ﬁgure 4c, Adam used b = 16 , η = 0 .
Revised: For the training runs in ﬁgure 4c, Adam used b = 16 , η = 0 .

Original: 0001 , while
Revised: 0001 , while

Original: HIGs used b = 16 , τ = 10 − 5 , and η = 0 .
Revised: HIGs used b = 16 , τ = 10 − 5 , and η = 0 .

Revised: Details on the memory footprint and update durations can be found in table

Original:  Figure 9 and ﬁgure 10 show the performance of both methods for a broader range of τ settings for
Revised: [Figure] [Figure] Figure 8 and ﬁgure 9 show the performance of both methods for a broader range of τ settings for

Original: HIGs, and η for Adam.
Revised: HIGs, and η for Adam.

Original: For Adam, a trade-off between slow convergence and oscillating updates exists.
Revised: For Adam, a trade-off between slow convergence and oscillating updates exists.

Original: The HIGs yield high accuracy in training across a wide range of values for τ , ranging from 10 − 5 to 10 − 3 .
Revised: The HIGs yield high accuracy in training across a wide range of values for τ , ranging from 10 − 5 to 10 − 3 .

Original: This supports the argumentation in the main text that the truncation is not overly critical for HIGs.
Revised: This supports the argumentation in the main text that the truncation is not overly critical for HIGs.

Original: As long as numerical noise is suppressed with τ > 10 − 6 , and the actual information about scaling of network parameters and physical variables is not cut off.
Revised: As long as numerical noise is suppressed with τ > 10 − 6 , and the actual information about scaling of network parameters and physical variables is not cut off.

Original: The latter case is visible for an overly large τ = 0 .
Revised: The latter case is visible for an overly large τ = 0 .

Original: 01 in the last graph on the right.
Revised: 01 in the last graph on the right.

Original: Note that many graphs in ﬁgure 10 contain a small plateau at the start of each training run.
Revised: Note that many graphs in ﬁgure 9 contain a small plateau at the start of each training run.

Original: These regions with relatively small progress per wall clock time are caused by the initialization overhead of the underlying deep learning framework (TensorFlow in our case).
Revised: These regions with relatively small progress per wall clock time are caused by the initialization overhead of the underlying deep learning framework (TensorFlow in our case).

Original: As all graphs measure wall clock time, we include the initialization overhead of TensorFlow, which causes a noticeable slow down of the ﬁrst iteration.
Revised: As all graphs measure wall clock time, we include the initialization overhead of TensorFlow, which causes a noticeable slow down of the ﬁrst iteration.

Original: Hence, the relatively slow convergence of the very ﬁrst steps in ﬁgureare not caused by conceptual issues with the HIGs themselves.
Revised: Hence, the relatively slow convergence of the very ﬁrst steps in ﬁgureare not caused by conceptual issues with the HIGs themselves.

Original: Rather, they are a result of the software frameworks and could, e.g., be alleviated with a pre-compilation of the training graphs.
Revised: Rather, they are a result of the software frameworks and could, e.g., be alleviated with a pre-compilation of the training graphs.

Original: In contrast, the initial convergence plateaus of Adam with smaller η in Figure 9 are of a fundamentally  different nature: they are caused by an inherent problem of non-inverting optimizers: their inability to appropriately handle the combination of large and small scale components in the physics of the quantum dipole setup (as outlined in section 3.3).[Figure]
Revised: In contrast, the initial convergence plateaus of Adam with smaller η in Figure 8 are of a fundamentally [Table] different nature: they are caused by an inherent problem of non-inverting optimizers: their inability to appropriately handle the combination of large and small scale components in the physics of the quantum dipole setup (as outlined in section 3.3).

Original: [Figure]

Original: Loss Functions.
Revised: Loss Functions.

Original: While training is evaluated in terms of the regular inner product as loss function:
Revised: While training is evaluated in terms of the regular inner product as loss function:

Original: L (Ψ a , Ψ b ) = 1 − |(cid:104) Ψ a , Ψ b (cid:105)| 2 , we use the following modiﬁed losses to evaluate low- and highenergy states for ﬁgure 4c.
Revised: L (Ψ a , Ψ b ) = 1 − |(cid:104) Ψ a , Ψ b (cid:105)| 2 , we use the following modiﬁed losses to evaluate low- and highenergy states for ﬁgure 4c.

Original: Let Ψ 1 be the ﬁrst excited state, then we deﬁne the low-energy loss as:
Revised: Let Ψ 1 be the ﬁrst excited state, then we deﬁne the low-energy loss as:

Original: [Equation]
Revised: [Equation]

Original: Correspondingly, we deﬁne the high-energy loss with the second excited state Ψ 2 :
Revised: Correspondingly, we deﬁne the high-energy loss with the second excited state Ψ 2 :

Original: [Equation]
Revised: [Equation]

Revised: C A BLATION STUDY

Revised: In this last section, we investigate how the HIG-hyperparameters affect the outcome.

Revised: This includes ablation experiments with respect to κ , β , and τ deﬁned in section 2.2.

Revised: We use the nonlinear oscillator example as the basis for these comparisons.

Revised: Exponent of the diagonal singular value matrix κ :

Revised: We test κ for various values between 1 . 0 , stochastic gradient descent, and − 1 , Gauss-Newton.

Revised: The results are shown in ﬁgure 10a.

Revised: For positive values, curves stagnate early, while for negative κ , the ﬁnal loss values are several orders of magnitude better.

Revised: The HIG curve corresponding to β = − 0 .

Revised: 5 achieves the best result.

Revised: This supports our argumentation that a strong dependence on this parameter exists, and that a choice of κ = − 0 . 5 is indeed a good compromise for scale-correcting updates of reasonable size.

Revised: The strong improvement as soon as κ becomes negative indicates that the collective inversion of the feedback of different data points of the mini-batch is an important ingredient in our method.

Revised: Scalar prefactor term β :

Revised: We test β -values between 0 , no scale correction, and − 0 .

Revised: 5 , which fully normalizes the effect of the largest singular value.

Revised: The results are shown in ﬁgure 10b.

Revised: Compared to the other hyperparameters, we observe that β has only little inﬂuence on the outcome.

Revised: Truncation parameter τ :

Revised: To understand the effect of this parameter, we consider the singular value decomposition (SVD) of the network-solver Jacobian, which is determined by the SVDs of the network Jacobian and the solver Jacobian.

Revised: The singular values of a matrix product AB depend non-trivially on the singular values of the matrices A and B.

Revised: In the simplest case, the singular values of the matrix product are received by multiplication of the individual singular values of both matrix factors.

Revised: In the general case, this depends on how the singular vectors of A and B overlap with each other.

Revised: However, it is likely that singular vectors with a small singular value of A or B overlap signiﬁcantly with singular vectors with a small singular value of AB.

Revised: For this reason, it is important not to truncate too much as this might remove the small-scale physics modes that we are ultimately trying to preserve in order to achieve accurate results.

Revised: On the other hand, less truncation leads to large updates of network weights on a scale beyond the validation of the linear approximation by ﬁrst-order derivatives.

Revised: These uncontrolled network modiﬁcations can lead to over-saturated neurons and prevent further training progress.

Revised: From a practical point of view, we choose τ according to the accuracy of the pure physics optimization problem without a neural network.

Revised: For the quantum dipole training, this value was set to 10 − 5 .

Revised: Trying to solve the pure physics optimization with far smaller values leads to a worse result or no convergence at all.

Revised: The network training behaves in line with this: Figure 9 shows that the network does not learn to control the quantum system with τ -values far smaller than 10 − 5 .

Revised: For the nonlinear oscillator system, the pure physics optimization is stable over a large range of τ -values with similarly good results.

Revised: For the network training, we chose τ to be 10 − 6 .

Revised: We conducted further experiments for the network training with different τ from 10 − 5 to 10 − 10 presented in ﬁgure 11, which show that HIGs have a similar tolerance in τ .

Revised: For a comparison, we also plotted Gauss-Newton curves for different τ .

Revised: We observe that GN curves become more unstable for smaller truncation values τ and diverge in the case 10 − 9 and 10 − 10 while HIG curves achieve overall better loss values and start to converge in this parameter.

Revised: [Figure]

Revised: [Figure]