Original: H ALF -I NVERSE G RADIENTS FOR P HYSICAL D EEP L EARNING
Revised: H ALF -I NVERSE G RADIENTS FOR P HYSICAL D EEP L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent works in deep learning have shown that integrating differentiable physics simulators into the training process can greatly improve the quality of results.
Revised: Recent works in deep learning have shown that integrating differentiable physics simulators into the training process can greatly improve the quality of results.

Original: Although this combination represents a more complex optimization task than supervised neural network training, the same gradient-based optimizers are typically employed to minimize the loss function.
Revised: Although this combination represents a more complex optimization task than supervised neural network training, the same gradient-based optimizers are typically employed to minimize the loss function.

Original: However, the integrated physics solvers have a profound effect on the gradient ﬂow as manipulating scales in magnitude and direction is an inherent property of many physical processes.
Revised: However, the integrated physics solvers have a profound effect on the gradient ﬂow as manipulating scales in magnitude and direction is an inherent property of many physical processes.

Original: Consequently, the gradient ﬂow is often highly unbalanced and creates an environment in which existing gradient-based optimizers perform poorly.
Revised: Consequently, the gradient ﬂow is often highly unbalanced and creates an environment in which existing gradient-based optimizers perform poorly.

Original: In this work, we analyze the characteristics of both physical and neural network optimizations to derive a new method that does not suffer from this phenomenon.
Revised: In this work, we analyze the characteristics of both physical and neural network optimizations to derive a new method that does not suffer from this phenomenon.

Original: Our method is based on a halfinversion of the Jacobian and combines principles of both classical network and physics optimizers to solve the combined optimization task.
Revised: Our method is based on a half-inversion of the Jacobian and combines principles of both classical network and physics optimizers to solve the combined optimization task.

Original: Compared to state-ofthe-art neural network optimizers, our method converges more quickly and yields better solutions, which we demonstrate on three complex learning problems involving nonlinear oscillators, the Schr ¨ odinger equation and the Poisson problem.
Revised: Compared to state-of-the-art neural network optimizers, our method converges more quickly and yields better solutions, which we demonstrate on three complex learning problems involving nonlinear oscillators, the Schr ¨ odinger equation and the Poisson problem.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The groundbreaking successes of deep learning (Krizhevsky et al., 2012; Sutskever et al., 2014;
Revised: The groundbreaking successes of deep learning (Krizhevsky et al., 2012; Sutskever et al., 2014;

Original: Silver et al., 2017) have led to ongoing efforts to study the capabilities of neural networks across all scientiﬁc disciplines.
Revised: Silver et al., 2017) have led to ongoing efforts to study the capabilities of neural networks across all scientiﬁc disciplines.

Original: In the area of physical simulation, neural networks have been used in various ways, such as creating accurate reduced-order models (Morton et al., 2018), inferring improved discretization stencils (Bar-Sinai et al., 2019), or suppressing numerical errors (Um et al., 2020).
Revised: In the area of physical simulation, neural networks have been used in various ways, such as creating accurate reduced-order models (Morton et al., 2018), inferring improved discretization stencils (Bar-Sinai et al., 2019), or suppressing numerical errors (Um et al., 2020).

Original: The long-term goal of these methods is to exceed classical simulations in terms of accuracy and speed, which has been achieved, e.g., for rigid bodies (de Avila Belbute-Peres et al., 2018), physical inverse problems (Holl et al., 2020), and two-dimensional turbulence (Kochkov et al., 2021).
Revised: The long-term goal of these methods is to exceed classical simulations in terms of accuracy and speed, which has been achieved, e.g., for rigid bodies (de Avila Belbute-Peres et al., 2018), physical inverse problems (Holl et al., 2020), and two-dimensional turbulence (Kochkov et al., 2021).

Original: The successful application of deep learning to physical systems naturally hinges on the training setup.
Revised: The successful application of deep learning to physical systems naturally hinges on the training setup.

Original: In recent years, the use of physical loss functions has proven beneﬁcial for the training procedure, yielding substantial improvements over purely supervised training approaches (Tompson et al., 2017; Wu & Tegmark, 2019; Greydanus et al., 2019).
Revised: In recent years, the use of physical loss functions has proven beneﬁcial for the training procedure, yielding substantial improvements over purely supervised training approaches (Tompson et al., 2017; Wu & Tegmark, 2019; Greydanus et al., 2019).

Original: These improvements were shown to stem from three aspects (Battaglia et al., 2016;
Revised: These improvements were shown to stem from three aspects (Battaglia et al., 2016;

Original: Holl et al., 2020): (i)
Revised: Holl et al., 2020): (i)

Original: Incorporating prior knowledge from physical principles facilitates the learning process , (ii) the ambiguities of multimodal cases are resolved naturally, and (iii) simulating the physics at training time can provide more realistic data distributions than pre-computed data sets.
Revised: Incorporating prior knowledge from physical principles facilitates the learning process , (ii) the ambiguities of multimodal cases are resolved naturally, and (iii) simulating the physics at training time can provide more realistic data distributions than pre-computed data sets.

Original: Approaches for training with physical losses can be divided into two categories.
Revised: Approaches for training with physical losses can be divided into two categories.

Original: On the one hand, equation-focused approaches that introduce physical residuals (Tompson et al., 2017; Raissi et al., 2019), and on the other hand, solver-focused approaches that additionally integrate well-established numerical procedures intothe training (Um et al., 2020; Kochkov et al., 2021).
Revised: On the one hand, equation-focused approaches that introduce physical residuals (Tompson et al., 2017; Raissi et al., 2019), and on the other hand, solver-focused approaches that additionally integrate well-established numerical procedures into training (Um et al., 2020; Kochkov et al., 2021).

Original: From a mathematical point of view, training a neural network with a physical loss function bears the difﬁculties of both network training and physics optimization.
Revised: From a mathematical point of view, training a neural network with a physical loss function bears the difﬁculties of both network training and physics optimization.

Original: In order to obtain satisfying results, it is vital to treat ﬂat regions of the optimization landscapes effectively.
Revised: In order to obtain satisfying results, it is vital to treat ﬂat regions of the optimization landscapes effectively.

Original: In learning, the challenging loss landscapes are addressed using gradient-based optimizers with data-based normalizing schemes such as Adam (Kingma & Ba, 2015), whereas in physics, the optimizers of choice are higher-order techniques such as Newton’s method (Gill & Murray, 1978) that inherently make use of inversion processes.
Revised: In learning, the challenging loss landscapes are addressed using gradient-based optimizers with data-based normalizing schemes, such as Adam (Kingma & Ba, 2015), whereas in physics, the optimizers of choice are higher-order techniques, such as Newton’s method (Gill & Murray, 1978), which inherently make use of inversion processes.

Original: However, Holl et al.
Revised: However, Holl et al.

Original: (2021) found that these approaches can not effectively handle the joint optimization of network and physics.
Revised: (2021) found that these approaches can not effectively handle the joint optimization of network and physics.

Original: Gradient-descent-based optimizers suffer from vanishing or exploding gradients, preventing effective convergence, while higher-order methods do not generally scale to the high-dimensional parameter spaces required by deep learning (Goodfellow et al., 2016).
Revised: Gradient-descent-based optimizers suffer from vanishing or exploding gradients, preventing effective convergence, while higher-order methods do not generally scale to the high-dimensional parameter spaces required by deep learning (Goodfellow et al., 2016).

Revised: Inspired by the insight that inversion is crucial for physics problems in learning from Holl et al.

Original: We likewise focus on  inversion-based approach but propose a new method for joint physics and network optimization which we refer to as half-inverse gradients .
Revised: (2021), we focus on an inversion-based approach but propose a new method for joint physics and network optimization which we refer to as half-inverse gradients .

Original: At its core lies a partial matrix inversion, which we derive from the interaction between network and physics both formally and geometrically.
Revised: At its core lies a partial matrix inversion, which we derive from the interaction between network and physics both formally and geometrically.

Original: An important property of our method is that its runtime scales linearly in the number of network parameters.
Revised: An important property of our method is that its runtime scales linearly with the number of network parameters.

Original: To demonstrate the wide-ranging and practical applicability of our method, we show that it yields signiﬁcant improvements in terms of convergence speed and ﬁnal loss values over existing methods.
Revised: To demonstrate the wide-ranging and practical applicability of our method, we show that it yields signiﬁcant improvements in terms of convergence speed and ﬁnal loss values over existing methods.

Original: These improvements are measured both in terms of absolute accuracy as well as wall-clock time.
Revised: These improvements are measured both in terms of absolute accuracy as well as wall-clock time.

Original: We evaluate a diverse set of physical systems, such as the Schr¨odinger equation, a nonlinear chain system and the Poisson problem.
Revised: We evaluate a diverse set of physical systems, such as the Schr¨odinger equation, a nonlinear chain system and the Poisson problem.

Original:  G RADIENTS BASED ON H ALF -I NVERSE J ACOBIANS
Revised:  G RADIENTS BASED ON H ALF -I NVERSE J ACOBIANS

Original: Optimization on continuous spaces can be effectively performed with derivative-based methods, the simplest of which is gradient descent.
Revised: Optimization on continuous spaces can be effectively performed with derivative-based methods, the simplest of which is gradient descent.

Original: For a target function L ( θ ) to be minimized of several variables θ , using bold symbols for vector-valued quantitites in this section, and learning rate η , gradient descent proceeds by repeatedly applying updates
Revised: For a target function L ( θ ) to be minimized of several variables θ , using bold symbols for vector-valued quantities in this section, and learning rate η , gradient descent proceeds by repeatedly applying updates

Original: [Equation]
Revised: [Equation]

Original: For quadratic objectives, this algorithm convergences linearly with the rate of convergence depending on the condition number λ of the Hessian matrix (Lax, 2014).
Revised: For quadratic objectives, this algorithm convergences linearly with the rate of convergence depending on the condition number λ of the Hessian matrix (Lax, 2014).

Original: In the ill-conditioned case λ (cid:29) 1 , ﬂat regions in the optimization landscape can signiﬁcantly slow down the optimization progress.
Revised: In the ill-conditioned case λ (cid:29) 1 , ﬂat regions in the optimization landscape can signiﬁcantly slow down the optimization progress.

Original: This is a ubiquitous problem in non-convex optimization tasks of the generic form:
Revised: This is a ubiquitous problem in non-convex optimization tasks of the generic form:

Original: [Equation]
Revised: [Equation]

Original: Here ( x i , ˆ y i ) denotes the i th data points from a chosen set of measurements, f is a function parametrized by θ to be optimized to model the relationship between the data points y i ( θ ) =
Revised: Here ( x i , ˆ y i ) denotes the i th data points from a chosen set of measurements, f is a function parametrized by θ to be optimized to model the relationship between the data points y i ( θ ) =

Original: f ( x i ; θ ) , and l denotes a loss function measuring the optimization progress.
Revised: f ( x i ; θ ) , and l denotes a loss function measuring the optimization progress.

Original: In the following, we assume the most common case of l ( y i , ˆ y i ) =
Revised: In the following, we assume the most common case of l ( y i , ˆ y i ) =

Original: 12 ||
Revised: 12 ||

Original: y
Revised: y

Original: i − ˆ
Revised: i − ˆ

Original: y
Revised: y

Original: i ||
Revised: i ||

Original: 22 being the squared l2-loss
Revised: 22 being the squared L2-loss.

Original: Physics Optimization.
Revised: Physics Optimization.

Original: Simulating a physical system consists of two steps: (i) mathematically modeling the system by a differential equation, and (ii) discretizing the occuring differential operators to receive a solver usable on a computer.
Revised: Simulating a physical system consists of two steps: (i) mathematically modeling the system by a differential equation, and (ii) discretizing its differential operators to obtain a solver for a computer.

Original: If we then wish to manipulate a physical system by an external force to reach a given conﬁguration,  we have to solve an inverse problem ofthe form 2.
Revised: Optimization tasks occur for instance when manipulating a physical system through an external force to reach a given conﬁguration, for which we have to solve an inverse problem of form 2.

Original: In such a control task, the sum reduces to a single data point ( x , ˆ y ) with x being the initial state, ˆ y the target state and θ the external force we want to ﬁnd.
Revised: In such a control task, the sum reduces to a single data point ( x , ˆ y ) with x being the initial state, ˆ y the target state and θ the external force we want to ﬁnd.

Original: The physical solver corresponds to the function f representing time evolution y ( θ ) =
Revised: The physical solver corresponds to the function f representing time evolution y ( θ ) =

Original: f ( x ; θ ) .
Revised: f ( x ; θ ) .

Original: This single data point sum still includessquared summation over vector components of y − ˆ y in the l2-loss.
Revised: This single data point sum still includes summation over vector components of y − ˆ y in the L2-loss.

Original: Sensitive behavior of the physical system connected to its high-frequency modes is present in the physical solver f , and produces small singular values in its linear approximation.
Revised: Sensitive behavior of the physical system arising from its high-frequency modes is present in the physical solver f , and produces small singular values in its Jacobian.

Original: This leads to an ill-conditioned Jacobian and ﬂat regions in the optimization landscape when minimizing 2.
Revised: This leads to an ill-conditioned Jacobian and ﬂat regions in the optimization landscape when minimizing 2.

Original: This is addressed by using methods that incorporate more information than only the gradient.
Revised: This is addressed by using methods that incorporate more information than only the gradient.

Original: Prominent examples are Newton’s method or the Gauss-Newton’s algorithm
Revised: Prominent examples are Newton’s method or the Gauss-Newton’s algorithm

Original: (Gill & Murray, 1978); the latter one is based on the Jacobian of f and the loss gradient:
Revised: (Gill & Murray, 1978); the latter one is based on the Jacobian of f and the loss gradient:

Original: [Equation]
Revised: [Equation]

Original: Here, the inversion of the Jacobian is done with the pseudoinverse.
Revised: Here the inversion of the Jacobian is calculated with the pseudoinverse.

Original: The Gauss-Newton update maps the steepest descent direction in y space to the parameter space θ .
Revised: The Gauss-Newton update maps the steepest descent direction in y -space to the parameter space θ .

Original: Therefore, to ﬁrst order, the resulting update approximates gradient descent steps in y space, further detailsof which are given in the appendix.
Revised: Therefore, to ﬁrst order, the resulting update approximates gradient descent steps in y -space, further details are given in appendix

Revised: A.2.

Original: An advantage of this formulation is that the update steps in y space are invariant under arbitrary rescaling of the parameters θ , which cancels inherent scales in f and ensures quick progress in the optimization landscape.[Figure] W [Figure]
Revised: An advantage of such higher-order methods is that the update steps in y -space are invariant under arbitrary rescaling of the parameters θ , which cancels inherent scales in f and ensures quick progress in the optimization landscape.

Original: ond iti on e d b)

Original: I ll

Original: [Figure]

Original: c ond iti on e d

Original: [Figure]

Original: Neural Network Training.
Revised: Neural Network Training.

Original: When f represents a neural network in equation 2, the optimization matches the typical supervised learning task.
Revised: For f representing a neural network in equation 2, the optimization matches the typical supervised learning task.

Original: In this context, the problem of ﬂat regions in the optimization landscape is also referred to as pathological curvature (Martens, 2010).
Revised: In this context, the problem of ﬂat regions in the optimization landscape is also referred to as pathological curvature (Martens, 2010).

Original: Solving this problem with higher-order methods is considered to be too expensive given the large number of parameters θ .
Revised: Solving this problem with higher-order methods is considered to be too expensive given the large number of parameters θ .

Original: For learning tasks, widely used optimizers such as Adam instead use gradient information from earlier update steps, for instance in the form of momentum or adaptive learning rate terms, thereby achieving better optimization progress at little additional computational cost.
Revised: For learning tasks, popular optimizers, such as Adam, instead use gradient information from earlier update steps, for instance in the form of momentum or adaptive learning rate terms, thereby improving convergence speed at little additional computational cost.

Original: Additionally, the updates are computed on mini-batches instead of the full data set, which saves computational resources and beneﬁts generalization.
Revised: Furthermore, the updates are computed on mini-batches instead of the full data set, which saves computational resources and beneﬁts generalization (Goodfellow et al., 2016).

Original: Neural Network Training with Physics Objectives.
Revised: Neural Network Training with Physics Objectives.

Original: For the remainder of the paper, we consider joint optimization problems, where f denotes a composition of a neural network parameterized by θ and a physics solver.
Revised: For the remainder of the paper, we consider joint optimization problems, where f denotes a composition of a neural network parameterized by θ and a physics solver.

Original: Using classical network optimizers for minimizing equation 2 is inefﬁcient in this case since data normalization in the network output space is not possible and the classical initialization schemes cannot normalize the effects of the physics solver.
Revised: Using classical network optimizers for minimizing equation 2 is inefﬁcient in this case since data normalization in the network output space is not possible and the classical initialization schemes cannot normalize the effects of the physics solver.

Original: As such, they are unsuited to capture the strong coupling between optimization parameters typically encountered in physics applications.
Revised: As such, they are unsuited to capture the strong coupling between optimization parameters typically encountered in physics applications.

Original: While Gauss-Newton seems promising for these cases, the involved Jacobian inversion tends to result in large overshoots in the updates when the involved physics solver is ill-conditioned.
Revised: While Gauss-Newton seems promising for these cases, the involved Jacobian inversion tends to result in large overshoots in the updates when the involved physics solver is ill-conditioned.

Original: As we will demonstrate, this leads to oversaturation of neurons, hampering the learning capability of the neural network.
Revised: As we will demonstrate, this leads to oversaturation of neurons, hampering the learning capability of the neural network.

Original: 2. A N I LL -C ONDITIONED T OY E XAMPLE
Revised: 2. A N I LL -C ONDITIONED T OY E XAMPLE

Original: To illustrate the argumentation so far, we consider a data set sampled from ˆ y ( x )=(sin(6 x ) , cos(9 x )) for x ∈ [ − 1 , 1] : We train a neural network to describe this data set by using the loss function l ( y , ˆ y ; γ ) =
Revised: To illustrate the argumentation so far, we consider a data set sampled from ˆ y ( x )=(sin(6 x ) , cos(9 x )) for x ∈ [ − 1 , 1] : We train a neural network to describe this data set by using the loss function: [Equation]

Original: 12 (cid:0) y 1 − ˆ y 1 (cid:1) 2 + 12 (cid:0) λ · y 2 − ˆ y 2 (cid:1) 2 , where y 1 and y 2 denote the ﬁrst and second component of the y -vectors.
Revised: Here, we denote vector components by superscripts.

Original: For a scale factor of γ = 1 , we receive the well-conditioned mean squared error loss.
Revised: For a scale factor of γ = 1 , we receive the well-conditioned mean squared error loss.

Original: However, l becomes increasingly ill-conditioned as γ is decreased, which mimics the effects of a physics solver.
Revised: However, l becomes increasingly ill-conditioned as γ is decreased, imitating the effects of a physics solver.

Original: For real-world physics solvers the situation wouldtypically be even more pronounced, as these scales usually vary strongly in direction and magnitude across different data points and optimization steps.
Revised: For real-world physics solvers, the situation would be even more complex since these scales usually vary strongly in direction and magnitude across different data points and optimization steps.

Original: We use a small neural network with a single hidden layer with 7 neurons and a tanh activation.
Revised: We use a small neural network with a single hidden layer with 7 neurons and a tanh activation.

Original: We then compare training with the well-conditioned γ = 1 loss against an ill-conditioned γ = 0 .
Revised: We then compare training with the well-conditioned γ = 1 loss against an ill-conditioned γ = 0 .

Original: 01 loss.
Revised: 01 loss.

Original: In both cases, we train the network using both Adam and Gauss-Newton as representatives of gradient-based and higher-order optimizers, respectively.
Revised: In both cases, we train the network using both Adam and Gauss-Newton as representatives of gradient-based and higher-order optimizers, respectively.

Original: The results are shown in ﬁgure 1.
Revised: The results are shown in ﬁgure 1.

Original: In the well-conditioned case, Adam and Gauss-Newton behave similarly, decreasing the loss by about three orders of magnitude.
Revised: In the well-conditioned case, Adam and Gauss-Newton behave similarly, decreasing the loss by about three orders of magnitude.

Original: However, in the ill-conditioned case, both optimizers fail to minimize the objective beyond a certain point.
Revised: However, in the ill-conditioned case, both optimizers fail to minimize the objective beyond a certain point.

Original: To explain this observation, we ﬁrst illustrate the behavior from the physics viewpoint by considering the trajectory of the network output f ( x ) for a single value x during training (ﬁgure 1, right).
Revised: To explain this observation, we ﬁrst illustrate the behavior from the physics viewpoint by considering the trajectory of the network output f ( x ) for a single value x during training (ﬁgure 1, right).

Original: For γ = 1 , Adam optimizes the network to accurately predict ˆ y ( x ) while for γ = 0 .
Revised: For γ =1 , Adam optimizes the network to accurately predict ˆ y ( x ) while for γ =0 .

Original: 01 , the updates neglect the second component preventing Adam to move efﬁciently along the small-scale coordinate (blue curve in ﬁgure 1b, right).
Revised: 01 , the updates neglect the second component preventing Adam to move efﬁciently along the small-scale coordinate (blue curve in ﬁgure 1b, right).

Original: To illustrate the situation from the viewpoint of the network, we consider the variance in the outputs of speciﬁc neurons over different x (ﬁgure 1, middle).
Revised: To illustrate the situation from the viewpoint of the network, we consider the variance in the outputs of speciﬁc neurons over different x (ﬁgure 1, middle).

Original: When γ = 1 , all neurons process information by producing different outcomes for different x .
Revised: When γ = 1 , all neurons process information by producing different outcomes for different x .

Original: However, for γ = 0 .
Revised: However, for γ = 0 .

Original: 01 , Gauss-Newton’s inversion of the small-scale component y 2 results in large updates, leading to an oversaturation of neurons (red curve in ﬁgure 1b, middle).
Revised: 01 , Gauss-Newton’s inversion of the smallscale component y 2 results in large updates, leading to an oversaturation of neurons (red curve in ﬁgure 1b, middle).

Original: These neurons stop processing information, reducing the effective capacity of the network and preventing the network from accurately ﬁtting ˆ y .
Revised: These neurons stop processing information, reducing the effective capacity of the network and preventing the network from accurately ﬁtting ˆ y .

Original: Facing these problems, a natural questions arises: Is it possible to construct an algorithm that can successfully process the inherently different scales of a physics solver while training a neural network at the same time? 
Revised: Facing these problems, a natural questions arises: Is it possible to construct an algorithm that can successfully process the inherently different scales of a physics solver while training a neural network at the same time? [Figure] W [Figure]

Revised: c ond iti on e d b)

Revised: I ll - c ond iti on e d

Original:  2. U PDATES BASED ON H ALF -I NVERSE J ACOBIANS
Revised: [Figure] 2. U PDATES BASED ON H ALF -I NVERSE J ACOBIANS

Original: We propose a novel method for optimizing neural networks with physics objectives.
Revised: We propose a novel method for optimizing neural networks with physics objectives.

Original: Since pure physics or neural network optimization can be thought of as special cases of the joint optimization, we analogously look for a potential method in the continuum of optimization methods between gradient descent and Gauss-Newton.
Revised: Since pure physics or neural network optimization can be thought of as special cases of the joint optimization, we analogously look for a potential method in the continuum of optimization methods between gradient descent and Gauss-Newton.

Original: We consider both of them to be the most elementary algorithms representing network and physics optimizers, respectively.
Revised: We consider both of them to be the most elementary algorithms representing network and physics optimizers, respectively

Original: The following equation describes an update that lies in between the two.
Revised: The following equation describes updates that lie between the two.

Original: [Equation]
Revised: [Equation]

Original: Here, the exponent< β, κ> of the Jacobian denotes the following procedure deﬁned with the aid of the singular value decomposition J = U Λ V (cid:62) as:
Revised: Here, the exponent κ of the Jacobian denotes the following procedure deﬁned with the aid of the singular value decomposition J = U Λ V (cid:62) :

Original: [Equation]
Revised: [Equation]

Original: Whenβ = 0 and κ = 1 , equation 4 reduces to the well-known form of gradient descent.
Revised: When κ = 1 , equation 5 reduces to the well-known form of gradient descent.

Original: Likewise, the caseβ = 0 and κ = − 1 yields Gauss-Newton since the result of the Jacobian exponentiation then gives the pseudoinverse of the Jacobian.
Revised: Likewise, the case κ = − 1 yields Gauss-Newton since the result of the Jacobian exponentiation then gives the pseudoinverse of the Jacobian.

Original: Unlike other possible interpolations between gradient descent and Gauss-Newton, exponentiation by κ as in equation 4 signiﬁcantly affects the scales inherent in the Jacobian.
Revised: Unlike other possible interpolations between gradient descent and Gauss-Newton, exponentiation by κ as in equation 5 signiﬁcantly affects the scales inherent in the Jacobian.

Original: This is highly important to appropriately inﬂuence physics and neural network scales.
Revised: This is highly important to appropriately inﬂuence physics and neural network scales.

Original: To determine κ , we recall our goal to perform update steps which are normalized in both θ - and y -space.
Revised: To determine κ , we recall our goal to perform update steps which are optimal in both θ - and yspace.

Original: However, since any update ∆ θ and its corresponding effect on the network output ∆ y are connected by the inherent scales encoded in the Jacobian, no single κ exists that normalizes both at the same time.
Revised: However, since any update ∆ θ and its corresponding effect on the solver output ∆ y are connected by the inherent scales encoded in the Jacobian, no single κ exists that normalizes both at the same time.

Original: Instead, we distribute the burden equally between network and physics by choosing κ = − 1 / 2 .
Revised: Instead, we distribute the burden equally between network and physics by choosing κ = − 1 / 2 .

Original: From a geometric viewpoint, the resulting update can be regarded as a steepest descent step measured by the norm given in table 1.
Revised: From a geometric viewpoint, the resulting update can be regarded as a steepest descent step when the norm to measure distance is chosen accordingly.

Original: In addition to the exponentiation, the scalar prefactor inequation 5 enables us to obtain a weaker form of scale invariance: Setting β = − 1 − κ yields algorithms that rescale the largest singular value to 1 , which ensures that the resulting updates cannot produce arbitrarily large updates in y -space.
Revised: This alternative way to approach our method is explained in the appendix (A.2) and summarized in table 1.

Original: Fora batchof size b  , we deﬁne an update with a learning rate η by stacking network-solver Jacobians ∂ y i ∂ θ (cid:12)(cid:12)
Revised: For batch size b and learning rate η , we deﬁne the following update step for our method by stacking network-solver Jacobians ∂ y i ∂ θ (cid:12)(cid:12)

Original: x
Revised: x

Original: i and loss gradients ∂L∂
Revised: i and loss gradients ∂L∂

Original: y i (cid:12)(cid:12) x i , ˆ y i of different data points ( x i , ˆ y i ) to form the following update step for our method:
Revised: y i (cid:12)(cid:12) x i , ˆ y i of different data points ( x i , ˆ y i ) :

Original: [Table]
Revised: [Table]

Original: performs Steepest Descent:
Revised: Steepest Descent:

Original: Norm ( θ -space)
Revised: Norm ( θ -space)

Original: Norm ( y -space)
Revised: Norm ( y -space)

Original: [Table]
Revised: [Table]

Original: in ParameterSpace || · || θ = || · ||
Revised: in Parameter

Original: [Equation]
Revised: [Table]

Original:  − 1 y
Revised: || · || θ = || · || J − 1 y

Original: [Table]
Revised: [Table]

Original: in Physics Space || · || J θ = || · || y
Revised: || · || J θ = || · || y

Original: Ours in Intermediate Space |
Revised: [Table] || ·

Original: [Equation]
Revised: [Equation]

Revised: 3 / 4 θ = || · |

Original: [Equation]
Revised: [Equation]

Original: − 1 /

Original: [Equation]
Revised: [Equation]

Original: 
Revised: y

Original: [Equation]
Revised: [Equation]

Original:  
Revised: 

Original: [Equation]
Revised: [Equation]

Original: ·  
Revised: (cid:12)(cid:12)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:    (6)
Revised: (cid:12)(cid:12) x b   − 1 /·   ∂L∂ y 1 (cid:12)(cid:12) (cid:62) x 1 , ˆ y∂L∂ y 2 (cid:12)(cid:12) (cid:62) x 2 , ˆ y 2 . ∂L∂ y b (cid:12)(cid:12) (cid:62) x b , ˆ y b   (7)

Original: Besides batch size b and learning rate η , we specify a truncation parameter τ as an additional hyperparameter enabling us to suppress numerical noise during the half-inversion process in equation 5.
Revised: Besides batch size b and learning rate η , we specify a truncation parameter τ as an additional hyperparameter enabling us to suppress numerical noise during the half-inversion process in equation 6.

Original: As with the computation of the pseudoinverse via SVD, we set the result of the − 12 -exponentiation of every singular value less than τ to 0 .
Revised: As with the computation of the pseudoinverse via SVD, we set the result of the − 1 2exponentiation of every singular value smaller than τ to 0 .

Original: The use of a half inversion – instead of a full inversion – helps to prevent exploding updates of network parameters while still guaranteeing substantial progress in directions of low curvature.
Revised: The use of a half-inversion – instead of a full inversion – helps to prevent exploding updates of network parameters while still guaranteeing substantial progress in directions of low curvature.

Original: With the procedure outlined above, we arrived at a balanced method that combines the advantages of optimization methods from deep learning and physics.
Revised: With the procedure outlined above, we arrived at a balanced method that combines the advantages of optimization methods from deep learning and physics.

Original: As our method uses  gradientsbased on halfinverse Jacobians, we refer to them in short as half-inverse gradients (HIGs) in the following.
Revised: As our method uses half-inverse Jacobians multiplied with gradients we refer to them in short as half-inverse gradients (HIGs).

Original: Half-inverse Gradients in the Toy Example.
Revised: Half-inverse Gradients in the Toy Example.

Original: With the deﬁnition of HIGs from equation 6, we optimize the toy example introduced in section 2.1.
Revised: With the deﬁnition of HIGs, we optimize the toy example introduced in section 2.1.

Original: The results in ﬁgure 1 show that for γ = 1 , HIGs minimize the objective as well as Adam and Gauss-Newton’s method.
Revised: The results in ﬁgure 1 show that for γ = 1 , HIGs minimize the objective as well as Adam and Gauss-Newton’s method.

Original: More interestingly, HIGs achieve a better result than the other two methods for γ = 0 .
Revised: More interestingly, HIGs achieve a better result than the other two methods for γ = 0 .

Original: On the one hand, the physics trajectory (ﬁgure 1b, right) highlights that HIGs can process information along the smallscale component of y 2 well and successfully progress along this direction.
Revised: On the one hand, the physics trajectory (ﬁgure 1b, right) highlights that HIGs can process information along the small-scale component ywell and successfully progress along this direction.

Original: On the other hand, by checking neuron saturation (ﬁgure 1b, middle), we see that HIGs – in contrast to Gauss Newton – avoid oversaturating neurons.
Revised: On the other hand, by checking neuron saturation (ﬁgure 1b, middle), we see that HIGs – in contrast to Gauss Newton – avoid oversaturating neurons.

Original: 2. P RACTICAL C ONSIDERATIONS
Revised: 2. P RACTICAL C ONSIDERATIONS

Original: Computational Cost.
Revised: Computational Cost.

Original: A HIG update step consists of constructing the stacked Jacobian and computing the half-inversion.
Revised: A HIG update step consists of constructing the stacked Jacobian and computing the half-inversion.

Original: The ﬁrst step is handled via a vectorized Jacobian computation, and therefore induces a runtime cost comparable to regular backpropagation at the expense of higher memory requirements.
Revised: The ﬁrst step can be efﬁciently parallelized on modern GPUs, and therefore induces a runtime cost comparable to regular backpropagation at the expense of higher memory requirements.

Original: The latter is not too problematic since our method achieves its optimal performance at lower batch sizes than the established gradient methods.

Original: In situations where the length of the HIG step is dominated by the half-inversion, memory requirements can be further reduced by parallelizing the Jacobian computation only partially.
Revised: In situations where the computational cost of the HIG step is dominated by the half-inversion, memory requirements can be further reduced by parallelizing the Jacobian computation only partially.

Original: Next, the half-inversion is performed, at the heart of which lies a divide and conquer algorithm for the singular value decomposition (Trefethen & Bau, 1997).
Revised: At the heart of the half-inversion lies a divide and conquer algorithm for the singular value decomposition (Trefethen & Bau, 1997).

Original: Hence, the length of a HIG step scales as O ( | θ | · ( b · | y |) 2 ) , i.e. it linear in the number of network parameters | θ | , and quadratic in the batch size b and the dimension of the physical state | y | .
Revised: Hence, the cost of a HIG step scales as O ( | θ |· b 2 ·| y | 2 ) , i.e. is linear in the number of network parameters | θ | , and quadratic in the batch size b and the dimension of the physical state | y | .

Original: Concrete numbers for memory requirements and duration of a HIG step are listed in the appendix.
Revised: Concrete numbers for memory requirements and duration of a HIG step are listed in the appendix.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: Our method requires choosing several hyperparameters.
Revised: Our method depends on several hyperparameters.

Original: First, we need a suitable choice of the learning rate.
Revised: First, we need a suitable choice of the learning rate.

Original: The normalizing effects of HIGs allow for larger learning rates than commonly used gradient descent variants.
Revised: The normalizing effects of HIGs allow for larger learning rates than commonly used gradient descent variants.

Original: We are able to use η = 1 for many of our experiments.
Revised: We are able to use η = 1 for many of our experiments.

Original: Second, the batch size b affects the number of data points included in the half-inversion process.
Revised: Second, the batch size b affects the number of data points included in the half-inversion process.

Original: It should be noted that the way the feedback of individual data points is processed is fundamentally different from the standard gradient optimizers: Instead of the averaging procedure of individual gradients of a mini batch, our approach constructs an update that is optimized for the complete batch.
Revised: It should be noted that the way the feedback of individual data points is processed is fundamentally different from the standard gradient optimizers: Instead of the averaging procedure of individual gradients of a mini batch, our approach constructs an update that is optimal for the complete batch.

Original: Consequently, the quality of updates increases with higher batch size.
Revised: Consequently, the quality of updates increases with higher batch size.

Original: However, overly large batch sizes can cause the Jacobian to become increasingly ill-conditioned and destabilize the learning progress (Trefethen & Bau, 1997).
Revised: However, overly large batch sizes can cause the Jacobian to become increasingly ill-conditioned and destabilize the learning progress.

Original: Inthe appendix C, we discuss the remaining parameters τ, β , and κ with several ablation experiments to illustrate their effects in detail.
Revised: In appendix C, we discuss the remaining parameters τ and κ with several ablation experiments to illustrate their effects in detail.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method on three physical systems and tasks: controllinga nonlinear oscillator system, the Poisson problem, and a dipole problem from quantum mechanics.
Revised: We evaluate our method on three physical systems: controlling nonlinear oscillators, the Poisson problem, and the quantum dipole problem.

Original: Details of the numerical setups are given in the appendix, together with results for a broad range of hyperparameter choices.
Revised: Details of the numerical setups are given in the appendix along with results for a broad range of hyperparameters.

Original: For a fair comparison, we show results with the best set of hyperparameters for each of the methods below and plot the loss against wall-clock time.
Revised: For a fair comparison, we show results with the best set of hyperparameters for each of the methods below and plot the loss against wall clock time measured in seconds.

Original: All learning curves are recorded on a previously unseen data set.
Revised: All learning curves are recorded on a previously unseen data set.

Original: 3. C ONTROL OF N ONLINEAR O SCILLATORS
Revised: 3. C ONTROL OF N ONLINEAR O SCILLATORS

Original: First, we consider a control task for a system of coupled oscillators with a nonlinear interaction term.
Revised: First, we consider a control task for a system of coupled oscillators with a nonlinear interaction term.

Original: This system is of practical importance in many areas of physics, such as solid state physics (Ibach & L ¨ uth, 2003).
Revised: This system is of practical importance in many areas of physics, such as solid state physics (Ibach & L¨uth, 2003).

Original: Its equations of motions are governed by the Hamiltonian
Revised: Its equations of motions are governed by the Hamiltonian

Original: [Equation]
Revised: [Equation]

Original: where x i and p i denote the Hamiltonian conjugate variables of oscillator i , α the interaction strength. The constant vector c speciﬁes how the control  is applied.
Revised: where x i and p i denote the Hamiltonian conjugate variables of oscillator i , α the interaction strength, and the vector c speciﬁes how to scalar-valued control function u ( t ) is applied.

Original: u ( t ) denotes the scalar-valued control signal to be learned by the neural network. For the integration of the 96 time steps, we use a 4th order Runge-Kutta scheme. The control task is to learn the  signal u ( t ) that transforms a given initial state into a given target state.
Revised: In our setup, we train a neural network to learn the control signal u ( t ) that transforms a given initial state into a given target state with 96 time steps integrated by a 4th order Runge-Kutta scheme.

Original: We use a dense neural network with three hidden layers totalling 2956 trainable parameters and ReLU activations.
Revised: We use a dense neural network with three hidden layers totalling 2956 trainable parameters and ReLU activations.

Original: The Mean-Squared-Error loss is used to quantify differences between predicted and target state.
Revised: The Mean-Squared-Error loss is used to quantify differences between predicted and target state.

Original: A visualization of this control task is shown in ﬁgure 2a.
Revised: A visualization of this control task is shown in ﬁgure 2a.

Original: Optimizer comparison.
Revised: Optimizer comparison.

Original: The goal of our ﬁrst experiments is to give a broad comparison of the proposed HIGs with commonly used optimizers.
Revised: The goal of our ﬁrst experiments is to give a broad comparison of the proposed HIGs with commonly used optimizers.

Original: This includes stochastic gradient descent (SGD), Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), RMSprop (Hinton et al., 2012), Adam (Kingma & Ba, 2015), and Gauss-Newton (GN) applied to mini batches.
Revised: This includes stochastic gradient descent (SGD), Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), RMSprop (Hinton et al., 2012), Adam (Kingma & Ba, 2015), and Gauss-Newton (GN) applied to mini batches.

Original: The results are shown in ﬁgure 2b where all curves show the best runs for each optimizer with suitable hyperparameters independently selected, as explained in the appendix.
Revised: The results are shown in ﬁgure 2b where all curves show the best runs for each optimizer with suitable hyperparameters independently selected, as explained in the appendix.

Original: We ﬁnd that the state-of-the-art optimizers stagnate early, with Adam achieving the best result oscillating around a ﬁnal loss value of 10 − 4 .
Revised: We ﬁnd that the state-of-the-art optimizers stagnate early, with Adam achieving the best result with a ﬁnal loss value of 10 − 4 .

Original: In comparison, our method and GN converge faster, exceeding Adam’s accuracy after about three minutes.
Revised: In comparison, our method and GN converge faster, exceeding Adam’s accuracy after about three minutes.

Original: While
Revised: While

Original: GN exhibits stability problems, the best stable run from our hyperparameter search reaches a loss value of 10 − 6 .
Revised: GN exhibits stability problems, the best stable run from our hyperparameter search reaches a loss value of 10 − 6 .

Original: HIGs, on the other hand, yield the best result with a loss value of 10 − 7 .
Revised: HIGs, on the other hand, yield the best result with a loss value of 10 − 7 .

Original: These results clearly show the potential of our method to process different scales of the physics solver more accurately and robustly.
Revised: These results clearly show the potential of our method to process different scales of the physics solver more accurately and robustly.

Original: They also make clear that the poor result of the widely-used network optimizers cannot be attributed to simple numerical issues as HIG converges to better levels of accuracy with an otherwise identical setup.
Revised: They also make clear that the poor result of the widely-used network optimizers cannot be attributed to simple numerical issues as HIG converges to better levels of accuracy with an otherwise identical setup.

Original: Role of the batch size.
Revised: Role of the batch size.

Original: We conduct multiple experiments using different values for the batch size b as a central parameter of our method.
Revised: We conduct multiple experiments using different values for the batch size b as a central parameter of our method.

Original: The results are shown in ﬁgure 2c.
Revised: The results are shown in ﬁgure 2c.

Original: We observe that for
Revised: We observe that for

Original: [Figure]
Revised: [Figure]

Original: Adam, all runs converge about equally quickly while HIGs, and GN, shownotable improvements from larger batch sizes.
Revised: Adam, all runs converge about equally quickly while HIGs and GN show improvements from larger batch sizes.

Original: This illustrates an important difference between Adam and HIG: Adam uses an average ofpoint-wise gradients  in the mini batch. As soon as the batch size is large enough, this average becomes representative of the expectation.
Revised: This illustrates an important difference between Adam and HIG: Adam uses an average of gradients of data points in the mini batch, which approaches its expectation for large b .

Original: Further increasing the batch size has little inﬂuence on the updates.
Revised: Further increasing the batch size has little inﬂuence on the updates.

Original: Our method, on the other hand, includes the individual data point gradients without averaging.
Revised: In contrast, our method includes the individual data point gradients without averaging.

Original: As shown in equation 6, we construct updates that are optimized for the whole batch by solving a linear system.
Revised: As shown in equation 7, we construct updates that are optimized for the whole batch by solving a linear system.

Original: This gives our method the ability to hit target states very accurately with increasing batch size.
Revised: This gives our method the ability to hit target states very accurately with increasing batch size.

Original: To provide further insights into the workings of HIGs, we focus on detailed comparisons with Adam as the most widely used gradient descent variant.
Revised: To provide further insights into the workings of HIGs, we focus on detailed comparisons with Adam as the most popular gradient descent variant.

Original: 3. P OISSON P ROBLEM
Revised: 3. P OISSON P ROBLEM

Original: Next we consider Poisson’s equation to illustrate advantages and current limitations of HIGs.
Revised: Next we consider Poisson’s equation to illustrate advantages and current limitations of HIGs.

Original: Poisson problems play an important role in electrostatics, Newtonian gravity, and ﬂuid dynamics (Ames, 2014).
Revised: Poisson problems play an important role in electrostatics, Newtonian gravity, and ﬂuid dynamics (Ames, 2014).

Original: For a source distribution ρ ( x ) , the goal is to ﬁnd the corresponding potential ﬁeld φ ( x ) fulﬁlling the following differential equation:
Revised: For a source distribution ρ ( x ) , the goal is to ﬁnd the corresponding potential ﬁeld φ ( x ) fulﬁlling the following differential equation:

Original: [Equation]
Revised: [Equation]

Original: Classically, Poisson problems are solved by solving the corresponding system of linear equations on the chosen grid resolution.
Revised: Classically, Poisson problems are solved by solving the corresponding system of linear equations on the chosen grid resolution.

Original: Instead, we train a dense neural network with three hidden layers and 41408 trainable parameters to solve the Poisson problem for a given right hand side ρ .
Revised: Instead, we train a dense neural network with three hidden layers and 41408 trainable parameters to solve the Poisson problem for a given right hand side ρ .

Original: We consider a two-dimensional system with a spatial discretization of 8 × 8 degrees of freedom.
Revised: We consider a two-dimensional system with a spatial discretization of 8 × 8 degrees of freedom.

Original: An example distribution and solution for the potential ﬁeld are shown in ﬁgure 3a.
Revised: An example distribution and solution for the potential ﬁeld are shown in ﬁgure 3a.

Original: Convergence and Runtime.
Revised: Convergence and Runtime.

Original: Figure 3b showsthe learning curves  when training the network with Adam and HIGs for different learning rates.
Revised: Figure 3b shows learning curves for different learning rates when training the network with Adam and HIGs.

Original: As we consider a two-dimensional system, this optimization task is challenging for both methods and requires longer training runs.
Revised: As we consider a two-dimensional system, this optimization task is challenging for both methods and requires longer training runs.

Original: We ﬁnd that both
Revised: We ﬁnd that both

Original: Adam and HIGs are able to minimize the loss by up to three orders of magnitude.
Revised: Adam and HIGs are able to minimize the loss by up to three orders of magnitude.

Original: The performance of Adam varies, and its two runs with larger η quickly slow down.
Revised: The performance of Adam varies, and its two runs with larger η quickly slow down.

Original: In terms of absolute convergence per time, the Adam curve with the smallest η(in light blue) shows advantages in this scenario.
Revised: In terms of absolute convergence per time, the Adam curve with the smallest η shows advantages in this scenario.

Original: However, choosing a log-scale for the time axis reveals that both methods have not fully converged.
Revised: However, choosing a log-scale for the time axis reveals that both methods have not fully converged.

Original: In particular, while the Adam curve begins to ﬂatten at the end, the slope of the HIG curve remains constant and decreases with a steeper slope than Adam.
Revised: In particular, while the Adam curve begins to ﬂatten at the end, the slope of the HIG curve remains constant and decreases with a steeper slope than Adam.

Original: The performance of Adamin this example can be explained by two reasons.
Revised: The performance of Adam can be explained by two reasons.

Original: First, the time to compute a single Adam update is much smaller than for HIGs, which requires the SVD solve from equation 5.
Revised: First, the time to compute a single Adam update is much smaller than for HIGs, which requires the SVD solve from equation 6.

Original: While these could potentially be sped up with appropriate methods (Foster et al., 2011; Allen-Zhu & Li, 2016), the absolute convergence per iteration, shown in the appendix in ﬁgure 7, shows how much each HIG update improves over Adam.
Revised: While these could potentially be sped up with appropriate methods (Foster et al., 2011; Allen-Zhu & Li, 2016), the absolute convergence per iteration, shown in the appendix in ﬁgure 7, shows how much each HIG update improves over Adam.

Original: Second, compared to the other examples, the Poisson problem is relatively simple, requiring only a single matrix inversion.
Revised: Second, compared to the other examples, the Poisson problem is relatively simple, requiring only a single matrix inversion.

Original: This represents a level of difﬁculty which Adam is still able to handle relatively well.
Revised: This represents a level of difﬁculty which Adam is still able to handle relatively well.

Original: [Figure]

Original: HIGs with Adam Pretraining.
Revised: HIGs with Adam Pretraining.

Original: To further investigate the potential of HIGs, we repeat the training, this time usinga state of the  Adam model from ﬁgure 3b forthe network initialization.
Revised: To further investigate the potential of HIGs, we repeat the training, this time using the best Adam model from ﬁgure 3b for network initialization.

Original: While Adam progresses slowly, the HIG training is able to quickly improve the state of the neural network, resulting in  a signiﬁcant drop of the loss values, followed by a faster descent than Adam.
Revised: While Adam progresses slowly, HIGs are able to quickly improve the state of the neural network, resulting in [Figure] a signiﬁcant drop of the loss values, followed by a faster descent than Adam.

Original: Interestingly, this experiment indicates that the HIG updates are able to improve aspects of the solution which Adam is agnostic to.
Revised: Interestingly, this experiment indicates that the HIG updates are able to improve aspects of the solution which Adam is agnostic to.

Original: Despite outlining the potential gains from faster SVD calculations, this example also highlights the quality of the HIG updates for simpler, ﬁeld-based PDEs.
Revised: Despite outlining the potential gains from faster SVD calculations, this example also highlights the quality of the HIG updates for simpler PDEs.

Original: 3. Q UANTUM D IPOLE
Revised: 3. Q UANTUM D IPOLE

Original: As a ﬁnal example, we target the quantum dipole problem,which is a standard control task formulated on the Schr ¨ odinger equation andis highly relevant in quantum physics (Von Neumann, 2018).
Revised: As a ﬁnal example, we target the quantum dipole problem, a standard control task formulated on the Schr ¨ odinger equation and highly relevant in quantum physics (Von Neumann, 2018).

Original: Given an initial and a target state, we train a neural network to compute the temporal transition function u ( t ) in an inﬁnite-well potential V according the evolution equation of the physical state Ψ :
Revised: Given an initial and a target state, we train a neural network to compute the temporal transition function u ( t ) in an inﬁnite-well potential V according the evolution equation of the physical state Ψ :

Original: [Equation]
Revised: [Equation]

Original: We employ a modiﬁed Crank-Nicolson scheme (Winckel et al., 2009) for the discretization of spatial and temporal derivatives.
Revised: We employ a modiﬁed Crank-Nicolson scheme (Winckel et al., 2009) for the discretization of spatial and temporal derivatives.

Original: Thus, each training iteration consists of multiple implicit time integration steps – 384 in our setup – for the forward as well as the backward pass of each mini-batch.
Revised: Thus, each training iteration consists of multiple implicit time integration steps – 384 in our setup – for the forward as well as the backward pass of each mini-batch.

Original: The control task consists of inferring a signal that converts the ground state to a given randomized linear combination of the ﬁrst and the second excited state.
Revised: The control task consists of inferring a signal that converts the ground state to a given randomized linear combination of the ﬁrst and the second excited state.

Original: We use a dense neural network with three hidden layers, 9484 trainable parameters and tanh activations.
Revised: We use a dense neural network with three hidden layers, 9484 trainable parameters and tanh activations.

Original: As loss function, we use the inner product of quantum mechanics L (Ψ a , Ψ b ) = 1 − |(cid:104) Ψ a , Ψ b (cid:105)| 2 .
Revised: Similarity in quantum theories is quantiﬁed with inner products; therefore, our loss function is given by L (Ψ a , Ψ b ) = 1 −|(cid:104) Ψ a , Ψ b (cid:105)| 2 .

Original: A visualization of this control task is shown in ﬁgure 4a.
Revised: A visualization of this control task is shown in ﬁgure 4a.

Original: Speed and Accuracy.
Revised: Speed and Accuracy.

Original: We observe that HIGs minimize the loss faster and reach a better ﬁnal level of accuracy than Adam (ﬁgure 4b).
Revised: We observe that HIGs minimize the loss faster and reach a better ﬁnal level of accuracy than Adam (ﬁgure 4b).

Original: While the Adam run with the largest learning rate drops faster initially, its ﬁnal performance is worse than all other runs.
Revised: While the Adam run with the largest learning rate drops faster initially, its ﬁnal performance is worse than all other runs.

Original: In this example, the difference between the ﬁnal loss values is not as large as for the previous experiments.
Revised: In this example, the difference between the ﬁnal loss values is not as large as for the previous experiments.

Original: This is due to the numerical accuracy achievable by a pure physics optimization, which for our choice of parameters is around 10 − 6 .
Revised: This is due to the numerical accuracy achievable by a pure physics optimization, which for our choice of parameters is around 10 − 6 .

Original: Hence, we can not expect to improve upon this lower bound for derived learning problems.
Revised: Hence, we can not expect to improve beyond this lower bound for derived learning problems.

Original: Our results indicate that the partial inversion of the Jacobian successfully leads to the observed improvements in convergence speed and accuracy.
Revised: Our results indicate that the partial inversion of the Jacobian successfully leads to the observed improvements in convergence speed and accuracy.

Original: Low and High Energy Components.
Revised: Low and High Energy Components.

Original: The quantum control problem also serves to highlight the weakness of gradient-based optimizers in appropriately processing different scales of the solutions.
Revised: The quantum control problem also serves to highlight the weakness of gradient-based optimizers in appropriately processing different scales of the solutions.

Original: In the initial training stage, the Adam curves stagnate at a loss value of 0 . 5 .
Revised: In the initial training stage, the Adam curves stagnate at a loss value of 0 . 5 .

Original: This is most pronounced for η = 10 − 4 in dark blue.
Revised: This is most pronounced for η = 10 − 4 in dark blue.

Original: To explain this effect, we recall that our learning objective targets transitions to combinations of the 1st and 2nd excited quantum states, and both states appear on average with equal weight in the training data.
Revised: To explain this effect, we recall that our learning objective targets transitions to combinations of the 1st and 2nd excited quantum states, and both states appear on average with equal weight in the training data.

Original: The transition to the energetically higher state is more difﬁcult and connected to smaller scales in the physics solver causing Adam to ﬁt the lower-energetic component ﬁrst.
Revised: Transitions to the energetically higher states are more difﬁcult and connected to smaller scales in the physics solver, causing Adam to ﬁt the lowerenergetic component ﬁrst.

Original: In contrast, our method is constructed to consider all scales via the partial inversion of the Jacobian and can therefore process the small scales more efﬁciently.
Revised: In contrast, our method is constructed to process small scales in the Jacobian via the half-inversion more efﬁciently.

Original: As a consequence, it does not stagnate at 0 .
Revised: As a consequence, the loss curves decrease faster below 0 .

Original: We support this explanation by explicitly plotting separate loss curves in ﬁgure 4c quantifying how well the low and high energy component of the target state was learned.
Revised: We support this explanation by explicitly plotting separate loss curves in ﬁgure 4c quantifying how well the low and high energy component of the target state was learned.

Original: Not only does Adam prefer to minimize the low-energy loss, it also increases the same loss again before it is able to minimize the high-energy loss.
Revised: Not only does Adam prefer to minimize the low-energy loss, it also increases the same loss again before it is able to minimize the high-energy loss.

Original: In contrast, we observe that the HIG minimizes both losses uniformly.
Revised: In contrast, we observe that HIGs minimize both losses uniformly.

Original: This is another indication that the theory outlined above carries over to an  even processing of different gradient magnitudes in joint physics and neural network objectives.
Revised: This is another indication for the correctness of the theory outlined above of an more even processing of different scales in joint physics and neural network objectives through our method.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Optimization algorithms.
Revised: Optimization algorithms.

Original: Optimization on continuous spaces is a huge ﬁeld that offers a vast range of techniques (Ye et al., 2019).
Revised: Optimization on continuous spaces is a huge ﬁeld that offers a vast range of techniques (Ye et al., 2019).

Original: Famous examples are gradient descent (Curry, 1944), GaussNewton’s method (Gill & Murray, 1978), Conjugate Gradient (Hestenes et al., 1952), or the limitedmemory BFGS algorithm (Liu & Nocedal, 1989).
Revised: Famous examples are gradient descent (Curry, 1944), Gauss-Newton’s method (Gill & Murray, 1978), Conjugate Gradient (Hestenes et al., 1952), or the limited-memory BFGS algorithm (Liu & Nocedal, 1989).

Original: In deep learning, the preferred methods instead rely on ﬁrst order information in the form of the gradient, such as SGD (Bottou, 2010) and RMSProp (Hinton et al., 2012).
Revised: In deep learning, the preferred methods instead rely on ﬁrst order information in the form of the gradient, such as SGD (Bottou, 2010) and RMSProp (Hinton et al., 2012).

Original: Several methods approximate the diagonal of the Hessian to improve scaling behavior, such as Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), and most prominently, Adam (Kingma & Ba, 2015).
Revised: Several methods approximate the diagonal of the Hessian to improve scaling behavior, such as Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), and most prominently, Adam (Kingma & Ba, 2015).

Original: However, due to neglecting all inter-dependencies ofthe parameters, these methods are limited in their capabilities to handle uneven learning objectives.
Revised: However, due to neglecting interdependencies of parameters, these methods are limited in their capabilities to handle physical learning objectives.

Original: Despite the computational cost, higher-order methods have also been studied in deep learning (Pascanu & Bengio, 2013) .
Revised: Despite the computational cost, higher-order methods have also been studied in deep learning (Pascanu & Bengio, 2013) .

Original: Practical methods have been suggested by using aKroenecker-factorization of the
Revised: Practical methods have been suggested by using a

Original:  Fisher matrix (Martens & Grosse, 2015) via iterative linear solversin (Martens, 2010), or by recursive approximations of the Hessian (Botev et al., 2017).
Revised: Kroenecker-factorization of the Fisher matrix (Martens & Grosse, 2015), iterative linear solvers (Martens, 2010), or by recursive approximations of the Hessian (Botev et al., 2017).

Original: To the best of our knowledge, the only other technique speciﬁcally targeting optimization of neural networks with physics objectives are physical gradients (PGs) (Holl et al., 2021).
Revised: To the best of our knowledge, the only other technique speciﬁcally targeting optimization of neural networks with physics objectives is the inversion approach from Holl et al.

Original: Despite targeting inversions, they fundamentally differ from our approach as PGs are based on inverse physics solvers, while we address the problem by treating network and solver as an entity and half-inverting its Jacobian.
Revised: However, their updates are based on inverse physics solvers, while we address the problem by treating network and solver as an entity and half-inverting its Jacobian.

Original: Thus, we work on the level of linear approximations while PGs are able to use higher-order information provided that an higher-order inverse solver exists.
Revised: Thus, we work on the level of linear approximations while updates based on physics inversion are able to harness higher-order information provided that an higher-order inverse solver exists.

Original: Additionally, PGs compute their update by averaging gradients over different data points, in line with typical gradient-based neural network optimizers.
Revised: Additionally, they compute their update by averaging gradients over different data points, in line with typical gradient-based neural network optimizers.

Original: HIGs instead process the feedback of different data points via collective inversion.
Revised: HIGs instead process the feedback of different data points via collective inversion.

Original: Incorporating physics.
Revised: Incorporating physics.

Original: Many works involve differentiable formulations of physical models, e.g., for robotics (Toussaint et al., 2018), to enable deep architectures (Chen et al., 2018), as a means for scene understanding (Battaglia et al., 2013; Santoro et al., 2017), or the control of rigid body environments de Avila Belbute-Peres et al.
Revised: Many works involve differentiable formulations of physical models, e.g., for robotics (Toussaint et al., 2018), to enable deep architectures (Chen et al., 2018), as a means for scene understanding (Battaglia et al., 2013; Santoro et al., 2017), or the control of rigid body environments de Avila Belbute-Peres et al.

Original: Additional works have shown the advantages of physical loss formulations Greydanus et al. ; Cranmer et al.
Revised: Additional works have shown the advantages of physical loss formulations (Greydanus et al., 2019; Cranmer et al., 2020).

Original: Differentiable simulation methods were proposed for a variety of phenomena, e.g. for ﬂuids (Schenck & Fox, 2018), PDE discretizations (Bar-Sinai et al., 2019), molecular dynamics (Wang et al., 2020), reducing numerical errors (Um et al., 2020), and cloth (Liang et al., 2019; Rasheed et al., 2020).
Revised: Differentiable simulation methods were proposed for a variety of phenomena, e.g. for ﬂuids (Schenck & Fox, 2018), PDE discretizations (Bar-Sinai et al., 2019), molecular dynamics (Wang et al., 2020), reducing numerical errors (Um et al., 2020), and cloth (Liang et al., 2019; Rasheed et al., 2020).

Original: It is worth noting that none of these works question the use  standard deep learning optimizers, such as Adam.
Revised: It is worth noting that none of these works question the use of standard deep learning optimizers, such as Adam.

Original: In addition, by now a variety of specialized software frameworks are available to realize efﬁcient implementations (Hu et al., 2020; Schoenholz & Cubuk, 2019; Holl et al., 2020).
Revised: In addition, by now a variety of specialized software frameworks are available to realize efﬁcient implementations (Hu et al., 2020; Schoenholz & Cubuk, 2019; Holl et al., 2020).

Original:  D ISCUSSION AND O UTLOOK
Revised:  D ISCUSSION AND O UTLOOK

Original: We investigated the training of neural networks for physical simulations. As including a physics solver greatly affects the gradient ﬂow, we questioned the current practice of using the standard gradient-based network optimizers for training.
Revised: We have considered optimization problems of neural networks in combination with physical solvers and questioned the current practice of using the standard gradient-based network optimizers for training.

Original: By considering ways to smoothly transition from  gradient descent to the Gauss-Newton algorithm, we identiﬁed a procedure to better address the joint optimization problem. Compared to existing optimizers, our  method can achieve a faster and more accurate minimization of the learning objectives.
Revised: Derived from an analysis of smooth transitions between gradient descent and GaussNewton’s method, our novel method learns physics modes more efﬁciently without overly straining the network through large weight updates, leading to a faster and more accurate minimization of the learning objective. This was demonstrated with a range of experiments.

Original: We believe that our work can provide a starting point for further research into improved learning methods for physical problems.
Revised: We believe that our work provides a starting point for further research into improved learning methods for physical problems.

Original: In the context of HIGs, highly interesting avenues for future work are efﬁcient methods for the inversion of the Jacobian matrix, or including unstructured solvers.
Revised: Highly interesting avenues for future work are efﬁcient methods for the half-inversion of the Jacobian matrix, or applying HIGs to physical systems exhibiting chaotic behavior or to more sophisticated training setups (Battaglia et al., 2013; Ummenhofer et al., 2020; Pfaff et al., 2020). A CKNOWLEDGEMENTS

Original: The combination of ﬁnite elements (Zienkiewicz et al., 1977) with graph-neural networks (Battaglia et al., 2013; Ummenhofer et al., 2020; Pfaff et al., 2020) provides a sound foundation that is especially interesting for future applications of HIGs.
Revised: This work was supported by the ERC Consolidator Grant CoG-2019-863850 SpaTe and by the DFG SFB-Transregio 109 DGD . We would also like to express our gratitude to the reviewers and the area chair for their helpful feedback.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To ensure reproducibility of our work we will publish all source code that is necessary to reproduce the results of our paper upon acceptance.
Revised: Our code for the experiments presented in this paper is publicly available at https://github.

Original: This includes data generation, neural network training and evaluation. Hyperparameters are also provided in the appendix for completeness
Revised: com/tum-pbs/half-inverse-gradients . Additionally, the chosen hyperparameters are listed in the appendix along with the hardware used to run our simulations.

Original: A PPENDIX A G AUSS -N EWTON ’ S M ETHOD IN THE C ONTEXT OF P HYSICAL D EEP L EARNING
Revised: A PPENDIX A F URTHER DETAILS ON OPTIMIZATION ALGORITHMS

Original: Our work considers optimization algorithms for functions of the form f ( x ; θ ) =
Revised: Our work considers optimization algorithms for functions of the form f ( x ; θ ) =

Original: y with θ , ∆ θ ∈ R t , denoting weight vector and weight update vector, respectively, while x ∈ R n and y ∈ R m denote input and output.
Revised: y with θ , ∆ θ ∈ R t , denoting weight vector and weight update vector, respectively, while x ∈ R n and y ∈ R m denote input and output.

Original: The learning process solves the minimization problem argmin θ L ( f ( x ; θ ) , ˆ y ) via a sequence θ k +

Revised: The learning process solves the minimization problem argmin θ L ( f ( x ; θ ) , ˆ y ) via a sequence θ k +

Original: =
Revised: =

Original: θ k + η ∆ θ .
Revised: θ k + η ∆ θ .

Original: Here, ˆ y are the reference solutions, and we target losses of the form L ( x , ˆ y ; θ ) = (cid:80)
Revised: Here, ˆ y are the reference solutions, and we target losses of the form L ( x , ˆ y ; θ ) = (cid:80)

Original: i l (cid:0) f
Revised: i l (cid:0) f

Original: ( x i ; θ ) , ˆ y i (cid:1) with i being an index for multiple data points (i.e., observations).
Revised: ( x i ; θ ) , ˆ y i (cid:1) with i being an index for multiple data points (i.e., observations).

Original: l denotes the l2-loss (cid:80)
Revised: l denotes the L2-loss (cid:80)

Original: j || x
Revised: j || x

Original: j − ˆ y j || 22 with j referencing the entries of a mini batch of size b .
Revised: j − ˆ y j || 2 with j referencing the entries of a mini batch of size b .

Original: Update Step. With this notation, the update step of the Gauss-Newton algorithm (Adby, 2013) for η = 1 is given by:
Revised: Using this notation, the update step of the Gauss-Newton algorithm (Adby, 2013) for η = 1 is given by:

Original: [Equation]
Revised: [Equation]

Original: The size of the Jacobian matrix is given by  y -space and parameter dimensions.
Revised: The size of the Jacobian matrix is given by the dimensions of y - and θ -space.

Original: For a full-rank
Revised: For a full-rank

Original: Jacobian corresponding to non-constrained optimization, the Gauss-Newton update is equivalent to:
Revised: Jacobian corresponding to non-constrained optimization, the Gauss-Newton update is equivalent to:

Original: [Equation]
Revised: [Equation]

Revised: Even in a constrained setting, we can reparametrize the coordinates to obtain an unconstrained optimization problem on the accessible manifold and rewrite ∆ θ GN similarly.

Original: This shortened form of the update step is given in equation 3, and is the basis for our discussion in the main text.
Revised: This shortened form of the update step is given in equation 3, and is the basis for our discussion in the main text.

Original: Geometric Interpretation.
Revised: It is well-known that the negative gradient of a function L ( θ ) points in the direction of steepest descent leading to the interpretation of gradient descent as a steepest descent algorithm. However, the notion of steepest descent requires deﬁning a measure of distance, which is in this case the usual L2-norm in θ .

Revised: By using different metrics, we can regard Gauss-Newton and HIG steps as steepest descent algorithms as well.

Revised: Gauss-Newton updates.

Original: These updates  can be regarded asa gradient descent inthe y-variables up to ﬁrst order in the update step as can be seen with the following equation:
Revised: The updates ∆ θ GN can be regarded as gradient descent in y up to ﬁrst order in the update step. This can be seen with a simple equation by considering how these updates change y .

Original: [Equation]
Revised: [Equation]

Original: In a constrained setting, we can reparametrize the coordinates to obtain an unconstrained optimization problem on the accessible manifold.

Original: We can then repeat the argument above. Hence, even in the constrained setting the Gauss-Newton algorithm can be seen as gradient descent on the y -manifold. In ﬁgure 1 of the main paper this behavior is visible inboth the well- and ill-conditioned case in terms of the straight line that the GN curve takes towards the target even in  thepresence of a neural network.
Revised: In ﬁgure 1 of the main paper, this property is visible in the physics trajectories for the wellconditioned case, where L ( y ) is a uniform L2-loss and hence, gradient descent in y produces a straight line to the target point. The Gauss-Newton curve ﬁrst shows several steps in varying directions as the higher-order terms from the neural network cannot be neglected yet.

Original: The ﬁrst few, seemingly erratic, movements in y -space which are observable for both GN and HIG stem from the inversion operations.
Revised: However, after this initial phase the curve exhibits the expected linear motion.

Original: The behavior of GN to perform steepest descent in y -space stands in contrast to gradient descent methods, which instead perform steepest descent on the θ -manifold.
Revised: The behavior of GN to perform steepest descent on the y -manifold stands in contrast to gradient descent methods, which instead perform steepest descent on the θ -manifold.

Original: In combination, this geometric viewpoint, as summarized in table 1 above motivates our construction of  the HIGs
Revised: This geometric view is the basis for an alternative way to derive our method that is presented below. HIG updates. HIG updates can be regarded as a steepest descent algorithm, again up to ﬁrst order in the update step, when measuring distances of θ -vectors with the following semi-norm: [Equation]

Revised: Here || · || denotes the usual L2-norm and J = ∂ y ∂ θ the Jacobian of network and solver.

Revised: The exponentiation is performed as explained in the main text, with J

Original: Stability of Inversions in the Context of Physical Deep Learning.
Revised: = U Λ V (cid:62) being the SVD, and J 3 / 4 given by V Λ 3 / 4 U (cid:62) . Additionally, we will use the natural map between dual vector and vector (cid:104)· , ·(cid:105) and the loss gradient g = ∂L∂ y . To prove the claim above, we expand the loss around an arbitrary starting point θ 0 :

Revised: [Equation]

Revised: The ﬁrst term on the right-hand side is constant and the third term is neglected according to the assumptions of the claim.

Revised: Hence, we investigate for which ﬁxed-length ∆ θ the second term decreases the most:

Revised: [Equation]

Revised: In the ﬁrst step above, we split the Jacobian J (cid:62) =

Revised: V Λ U (cid:62) =

Revised: ( V Λ 1 / 4 V (cid:62) )

Revised: ( V Λ 3 / 4 U (cid:62) )

Revised: =

Revised: J 1 / 4 J 3 / 4 .

Revised: γ denotes the angle between J 1 / 4 g (cid:62) and J 3 / 4 ∆ θ .

Revised: This expression is minimized for γ = − π , meaning the two vectors have to be antiparallel:

Revised: [Equation]

Revised: This requirement is fulﬁlled by the HIG update ∆ θ HIG = − J 1 / 2 g (cid:62) , and is therefore a steepest descent method, which concludes our proof.

Revised: This presents another approach to view HIGs as an interpolation between gradient descent and

Revised: Gauss-Newton’s method.

Revised: More precisely, gradient descent performs steepest descent in the usual

Revised: L2-norm in θ -space ( || θ || ).

Revised: Considering only terms up to linear order, Gauss-Newton performs steepest descent in the L2-norm in y -space ( || J θ || ).

Revised: The HIG update ( || J 3 / 4 θ || ) lies between these two methods.

Revised: The quarter factors in the exponents result from the additional factor of 2 that has to be compensated for when considering L2-norms.

Revised: A. S TABILITY OF I NVERSIONS IN THE C ONTEXT OF P HYSICAL D EEP L EARNING .

Original: In the following, we illustrate how the full inversion of GN can lead to instabilities at training time.
Revised: In the following, we illustrate how the full inversion of GN can lead to instabilities at training time.

Original: Interestingly, physical solvers are not the only cause of small singular values in the Jacobian.
Revised: Interestingly, physical solvers are not the only cause of small singular values in the Jacobian.

Original: They can also occur when applying equation 11 to a mini batch to train a neural network and are not caused by numerical issues.
Revised: They can also occur when applying equation 12 to a mini batch to train a neural network and are not caused by numerical issues.

Original: Consider the simple case of two data points ( x 1 , ˆ y 1 ) and ( x 2 , ˆ y 2 ) and a one-dimensional output.
Revised: Consider the simple case of two data points ( x 1 , ˆ y 1 ) and ( x 2 , ˆ y 2 ) and a one-dimensional output.

Original: Let f be the neural network and J the Jacobian, which is in this case the gradient of the network output.
Revised: Let f be the neural network and J the Jacobian, which is in this case the gradient of the network output.

Original: Then equation 11 yields:
Revised: Then equation 12 yields:

Original: [Equation]
Revised: [Equation]

Original: Next, we linearly approximate the second row by using the Hessian H by assuming the function to be learned is ˆ f , i.e. ˆ f ( x 1 ) =
Revised: Next, we linearly approximate the second row by using the Hessian H by assuming the function to be learned is ˆ f , i.e. ˆ f ( x 1 ) =

Original: y 1 and ˆ f ( x 2 )
Revised: y 1 and ˆ f ( x 2 )

Original: =
Revised: =

Original: y 2 .
Revised: y 2 .

Original: Neglecting terms beyond the linear approximation, we receive:
Revised: Neglecting terms beyond the linear approximation, we receive:

Original: [Equation]
Revised: [Equation]

Original: Considering the case of two nearby data points, i.e. x 2 − x 1 being small, the two row vectors in the stacked Jacobian on the left-hand side are similar, i.e. the angle between them is small.
Revised: Considering the case of two nearby data points, i.e. x 2 − x 1 being small, the two row vectors in the stacked Jacobian on the left-hand side are similar, i.e. the angle between them is small.

Original: This leads to a small singular value of the stacked Jacobian.
Revised: This leads to a small singular value of the stacked Jacobian.

Original: In the limit of x 2 = x 1 both row vectors are linearly dependant and hence, one singular value becomes zero.
Revised: In the limit of x 2 = x 1 both row vectors are linearly dependant and hence, one singular value becomes zero.

Original: Moreover, even if x 2 is not close to x 1 , small singular values can occur if the batch size increases: for a growing number of row vectors it becomes more and more likely that the Jacobian contains similar or linearly dependent vectors.
Revised: Moreover, even if x 2 is not close to x 1 , small singular values can occur if the batch size increases: for a growing number of row vectors it becomes more and more likely that the Jacobian contains similar or linearly dependent vectors.

Original: After inversion, a small singular value becomes large.
Revised: After inversion, a small singular value becomes large.

Original: This leads to a large update ∆ θ GN when the right-hand side of equation 14 overlaps with the corresponding singular vector.
Revised: This leads to a large update ∆ θ GN when the right-hand side of equation 19 overlaps with the corresponding singular vector.

Original: This can easily happen if the linear approximation of the right-hand side is poor, for instance when ˆ f is a solution to an inverse physics problem.
Revised: This can easily happen if the linear approximation of the right-hand side is poor, for instance when ˆ f is a solution to an inverse physics problem.

Original: Then ˆ f can have multiple modes and can, even within a mode, exhibit highly sensitive or even singular behavior.
Revised: Then ˆ f can have multiple modes and can, even within a mode, exhibit highly sensitive or even singular behavior.

Original: In turn, applying large updates to the network weights naturally can lead to the oversaturation of neurons, as illustrated above, and diverging training runs in general.
Revised: In turn, applying large updates to the network weights naturally can lead to the oversaturation of neurons, as illustrated above, and diverging training runs in general.

Original: As illustrated in the main paper, these inherent problems of GN are alleviated by the partial inversion of the HIG.
Revised: As illustrated in the main paper, these inherent problems of GN are alleviated by the partial inversion of the HIG.

Original: It yields a fundamentally different order of scaling via its square-root inversion, which likewise does not guarantee that small singular values lead to overshoots (hence the truncation), but in general strongly stabilizes the training process.
Revised: It yields a fundamentally different order of scaling via its square-root inversion, which likewise does not guarantee that small singular values lead to overshoots (hence the truncation), but in general strongly stabilizes the training process.

Original: B E XPERIMENTAL D ETAILS
Revised: B E XPERIMENTAL D ETAILS

Original: In the following, we provide details of the physical simulations used for our experiments in  the main paper.
Revised: In the following, we provide details of the physical simulations used for our experiments in section 3 of the main paper.

Original: For the different methods, we use the following abbreviations: half-inverse gradients (HIG), Gauss-Newton’s method (GN), and stochastic gradient descent (GD).
Revised: For the different methods, we use the following abbreviations: half-inverse gradients (HIG), Gauss-Newton’s method (GN), and stochastic gradient descent (GD).

Original: Learning rates are denoted by η , batch sizes by b , and truncation parameters for HIG and GN by τ .
Revised: Learning rates are denoted by η , batch sizes by b , and truncation parameters for HIG and GN by τ .

Original: All loss results are given for the average loss over a test set with samples distinct from the training data set.
Revised: All loss results are given for the average loss over a test set with samples distinct from the training data set.

Original: For each method, we run a hyperparameter search for every experiment, varying the learning rate by several orders of magnitude, and the batch size in factors of two.
Revised: For each method, we run a hyperparameter search for every experiment, varying the learning rate by several orders of magnitude, and the batch size in factors of two.

Original: Unless noted otherwise, the best runs in terms of ﬁnal test loss were selected and shown in the main text.
Revised: Unless noted otherwise, the best runs in terms of ﬁnal test loss were selected and shown in the main text.

Original: The following sections contain several examples from the hyperparameter search to illustrate how the different methods react to the changed settings.
Revised: The following sections contain several examples from the hyperparameter search to illustrate how the different methods react to the changed settings.

Original: Runtime Measurements Runtimes for the non-linear chain and quantum dipole were measured on a machine with Intel Xeon 6240 CPUs and NVIDIA GeForce RTX 2080 Ti GPUs.
Revised: Runtime Measurements Runtimes for the non-linear chain and quantum dipole were measured on a machine with Intel Xeon 6240 CPUs and NVIDIA GeForce RTX 2080 Ti GPUs.

Revised: The

Original: The Poisson experiments used an Intel Xeon W-2235 CPU with NVIDIA Quadro RTX 8000 GPU.We experimentally veriﬁed that these platforms yield an on-par performance for our implementation.
Revised: Poisson experiments used an Intel Xeon W-2235 CPU with NVIDIA Quadro RTX 8000 GPU. We experimentally veriﬁed that these platforms yield an on-par performance for our implementation.

Original: As deep learning API we used TensorFlow version 2.5.
Revised: As deep learning API we used TensorFlow version 2.5.

Original: If not stated otherwise, each experiment retained the default settings.
Revised: If not stated otherwise, each experiment retained the default settings.

Original: All runtime graphs in the main paper and appendix contain wall-clock measurements that include all steps of a learning run, such as initialization, in addition to the evaluation time of each epoch.
Revised: All runtime graphs in the main paper and appendix contain wall-clock measurements that include all steps of a learning run, such as initialization, in addition to the evaluation time of each epoch.

Original: However, the evaluations of the test sets to determine the performance in terms of loss are not included.
Revised: However, the evaluations of the test sets to determine the performance in terms of loss are not included.

Original: As optimizers such as Adam typically performs a larger number of update steps including these evaluations would have put these optimizers at an unnecessary disadvantage.[Table] [Table]
Revised: As optimizers such as Adam typically performs a larger number of update steps including these evaluations would have put these optimizers at an unnecessary disadvantage.

Original:  T OY E XAMPLE ( SECTION 2.1)
Revised: B. T OY E XAMPLE ( SECTION 2.1)

Original: For the toy example, the target function is given by ˆ f ( x ) =
Revised: For the toy example, the target function is given by ˆ f ( x ) =

Original: (sin(6 x ) , cos(9 x )) .
Revised: (sin(6 x ) , cos(9 x )) .

Original: We used a dense neural network consisting of one hidden layer with 7 neurons and tanh activation, and an output layer with 2 neurons and linear activation.
Revised: We used a dense neural network consisting of one hidden layer with 7 neurons and tanh activation, and an output layer with 2 neurons and linear activation.

Original: For training, we use 1024 data points uniformly sampled  from the [ − 1 , 1] interval, and a batch size of 256.
Revised: For training, we use 1024 data points uniformly sampled [Table] [Table] from the [ − 1 , 1] interval, and a batch size of 256.

Original: The test set consists of 1024 new samples.

Original: For the optimizers, the following hyperparameters were used for both the well-conditioned loss and the ill-conditioned loss: Adam η = 0 .
Revised: For the optimizers, the following hyperparameters were used for both the well-conditioned loss and the ill-conditioned loss: Adam η = 0 .

Original: 3 ;  has no learning rate (equivalent to η = 1 ), τ = 10 − 4 ; HIG η = 1 . 0 , τ = 10 − 6 .
Revised: 3 ; GN has no learning rate (equivalent to η = 1 ), τ = 10 − 4 ; HIG η = 1 . 0 , τ = 10 − 6 .

Original: B. C ONTROL OF N ONLINEAR O SCILLATORS ( SECTION 3.1)
Revised: B. C ONTROL OF N ONLINEAR O SCILLATORS ( SECTION 3.1)

Original: The Hamiltonian function given in equation 7 leads to the following equations of motions:¨
Revised: The Hamiltonian function given in equation 8 leads to the following equations of motions:

Original: [Equation]
Revised: [Equation]

Original: The simulations of the nonlinear oscillators were performed for two mass points and a time interval of 12 units with a time step ∆ t = 0 .
Revised: The simulations of the nonlinear oscillators were performed for two mass points and a time interval of 12 units with a time step ∆ t = 0 .

Original: This results in 96 time steps via 4th order Runge-Kutta per learning iteration.
Revised: This results in 96 time steps via 4th order Runge-Kutta per learning iteration.

Original: We generated 4096 data points for a control vector c = (0 . 0 , 3 . 0) , and an interaction strength α = 1 . 0 with randomized conjugate variables x and p .
Revised: We generated 4096 data points for a control vector c = (0 . 0 , 3 . 0) , and an interaction strength α = 1 . 0 with randomized conjugate variables x and p .

Original: The test set consists of 4096 new data points.
Revised: The test set consists of 4096 new data points.

Original: For the neural network, we set up a fully-connected network with ReLU activations passing inputs through three hidden layers with 20 neurons in each layer before being mapped to a 96 output layer with linear activation.
Revised: For the neural network, we set up a fully-connected network with ReLU activations passing inputs through three hidden layers with 20 neurons in each layer before being mapped to a 96 output layer with linear activation.

Original: For the comparison with other optimizers (ﬁgure 2b) we performed a broad hyperparameter search for each method, as outlined above, to determine suitable settings.
Revised: For the comparison with other optimizers (ﬁgure 2b) we performed a broad hyperparameter search for each method, as outlined above, to determine suitable settings.

Original: The parameters for Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), RMSprop (Hinton et al., 2012), Gauss-Newton (Gill & Murray, 1978), HIGs, and stochastic gradient descent (Curry, 1944) are summarized in table 2.
Revised: The parameters for Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), RMSprop (Hinton et al., 2012), Gauss-Newton (Gill & Murray, 1978), HIGs, and stochastic gradient descent (Curry, 1944) are summarized in table 2.

Original: For ﬁgure 2c the following hyperparameters were used: η = 3 · 10 − 4 for
Revised: For ﬁgure 2c the following hyperparameters were used: η = 3 · 10 − 4 for

Original: Adam, and η = 1 . 0 , τ = 10 − 6 for HIG.
Revised: Adam, and η = 1 . 0 , τ = 10 − 6 for HIG.

Original: Further Experiments.
Revised: Further Experiments.

Original: Figure 5 and ﬁgure 6 contain additional runs with different hyperparameters for the method comparison of ﬁgure 2b in the main paper.
Revised: Figure 5 and ﬁgure 6 contain additional runs with different hyperparameters for the method comparison of ﬁgure 2b in the main paper.

Original: The graphs illustrate that all ﬁve method do not change their behavior signiﬁcantly for the different batch sizes in each plot, but become noticeably unstable for larger learning rates η (plots on the right sides of each section).
Revised: The graphs illustrate that all ﬁve method do not change their behavior signiﬁcantly for the different batch sizes in each plot, but become noticeably unstable for larger learning rates η (plots on the right sides of each section).

Original: Details on the memory footprint and update durations can be found in table 3.
Revised: Details on the memory footprint and update durations can be found in table 3.

Original: Since our simulations were not limited by memory, we used an implementation for the Jacobian computation of HIGs, which scales quadratically in the batch size.
Revised: Since our simulations were not limited by memory, we used an implementation for the Jacobian computation of HIGs, which scales quadratically in the batch size.

Original: Should this become a bottleneck, this scaling could potentially be made linear by exploiting that the Jacobian of the physical solver for multiple data points is blockdiagonal.[Figure] [Figure]
Revised: Should this become a bottleneck, this scaling could potentially be made linear by exploiting that the Jacobian of the physical solver for multiple data points is blockdiagonal.

Original: B. P OISSON P ROBLEM ( SECTION 3.2)
Revised: B. P OISSON P ROBLEM ( SECTION 3.2)

Original: We discretize Poisson’s equation on a regular grid for a two-dimensional domain Ω = [0 , 8] × [0 , 8] with a grid spacing of ∆ x = 1 .
Revised: We discretize Poisson’s equation on a regular grid for a two-dimensional domain Ω = [0 , 8] × [0 , 8] with a grid spacing of ∆ x = 1 .

Original: Dirichlet boundary conditions of φ = 0 are imposed on all foursides of Ω .
Revised: Dirichlet boundary conditions of φ = 0 are imposed on all four

Original: The Laplace operator is discretized with a ﬁnite difference stencil (Ames, 2014).
Revised: [Figure] [Figure]

Original: For the neural network, we set up a fully-connected network with tanh activation functions.
Revised: For the neural network, we set up a fully-connected network with tanh activation functions.

Original: The 8x8 inputs pass through three hidden layers with 64, 256 and 64 neurons, respectively, before being mapped to 8x8 in the output layer.
Revised: The 8x8 inputs pass through three hidden layers with 64, 256 and 64 neurons, respectively, before being mapped to 8x8 in the output layer.

Original: For training, source distributions ρ are sampled from random frequencies in Fourier space, and transformed to real space via the inverse Fourier transform.
Revised: For training, source distributions ρ are sampled from random frequencies in Fourier space, and transformed to real space via the inverse Fourier transform.

Original: The mean value is normalized to zero.
Revised: The mean value is normalized to zero.

Original: We sample data on-the-ﬂy, resulting in an effectively inﬁnite data set.
Revised: We sample data on-the-ﬂy, resulting in an effectively inﬁnite data set.

Original: This makes a separate test set redundant as all training data is previously unseen.[Table] [Figure] [Table]
Revised: This makes a separate test set redundant as all training data is previously unseen.

Original: [Figure]

Original: [Table]

Original: Further Experiments.
Revised: Further Experiments.

Original: Figure 7a shows Adam and HIG runs from ﬁgure 3b over epochs.
Revised: Figure 7a shows Adam and HIG runs from ﬁgure 3b over epochs.

Original: The HIG runs converge faster per iteration, which indicates that HIGs perform qualitatively better updates.
Revised: The HIG runs converge faster per iteration, which indicates that HIGs perform qualitatively better updates.

Original: Additionally, we use the pretrained HIG run from ﬁgure 3c as a starting point for further Adam training.
Revised: Additionally, we use the pretrained HIG run from ﬁgure 3c as a starting point for further Adam training.

Original: The results are shown in 7b.
Revised: The results are shown in 7b.

Original: We observe that the network quickly looses the progress the HIGs have made and continues with a loss value similar to the orginal Adam run. 
Revised: We observe that the network quickly looses the progress the HIGs have made, and continues with a loss value similar to the orginal Adam run. This again [Table] [Figure]

Revised: [Table]

Original: This again supports our intuition that Adam, in contrast to HIGs, cannot harness the full potential of the physics solver.
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: supports our intuition that Adam, in contrast to HIGs, cannot harness the full potential of the physics solver.

Original: Details on the memory footprint and update durations can be found in table
Revised: Details on the memory footprint and update durations can be found in table

Original: B. Q UANTUM D IPOLE ( SECTION3.3)
Revised: B. Q UANTUM D IPOLE ( SECTION

Original: For the quantum dipole problem, we discretize the Schr¨odinger equation on a spatial domain Ω = [0 , 2] with a spacing of ∆ x = 0 .
Revised: For the quantum dipole problem we discretize the Schr ¨ odinger equation on a spatial domain Ω = [0 , 2] with a spacing of ∆ x = 0 .

Original: 133 resulting in 16 discretization points.
Revised: 133 resulting in 16 discretization points.

Original: In time, we simulate up to a time of 19 . 2 with a time step of ∆ t = 0 .
Revised: We simulate up to a time of 19 . 2 with a time step of ∆ t = 0 .

Original: 05 , which yields 384 time steps.
Revised: 05 , which yields 384 time steps.

Original: Spatial and temporal discretization use a modiﬁed Crank-Nicolson scheme (Winckel et al., 2009) which is tailored to quantum simulations.
Revised: Spatial and temporal discretization use a modiﬁed Crank-Nicolson scheme (Winckel et al., 2009) which is tailored to quantum simulations.

Original: The training data set consists of 1024 randomized superpositions of the ﬁrst and second excited state, while the test set contains a new set of 1024 randomized superpositions.
Revised: The training data set consists of 1024 randomized superpositions of the ﬁrst and second excited state, while the test set contains a new set of 1024 randomized superpositions.

Original: For the neural network, we set up a fully-connected network with tanh activations passing the inputs through three hidden layers with 20 neurons in each layer before being mapped to a 384  output layer with linear activation.
Revised: For the neural network, we set up a fully-connected network with tanh activations passing the inputs through three hidden layers with 20 neurons in each layer before being mapped to a 384 neuron output layer with linear activation.

Original: Overall, the network contains 9484 trainable parameters.
Revised: Overall, the network contains 9484 trainable parameters.

Original: Experimental details.
Revised: Experimental details.

Original: For the training runs in ﬁgure 4b, Adam used b = 16 , while for HIG b = 16 , and τ = 10 − 5 were used.
Revised: For the training runs in ﬁgure 4b, Adam used b = 16 , while for HIG b = 16 , and τ = 10 − 5 were used.

Original: For the training runs in ﬁgure 4c, Adam used b = 16 , η = 0 .
Revised: For the training runs in ﬁgure 4c, Adam used b = 16 , η = 0 .

Original: 0001 , while HIGs used b = 16 , τ = 10 − 5 , and η = 0 .
Revised: 0001 , while HIGs used b = 16 , τ = 10 − 5 , and η = 0 .

Original: Details on the memory footprint and update durations can be found in table
Revised: Details on the memory footprint and update durations can be found in table

Original: [Figure] [Figure] Figure 8 and ﬁgure 9 show the performance of both methods for a broader range of τ settings for
Revised: Figure 8 and ﬁgure 9 show the performance of both methods for a broader range of τ settings for

Original: HIGs, and η for Adam.
Revised: HIGs, and η for Adam.

Original: For Adam, a trade-off between slow convergence and oscillating updates exists.
Revised: For Adam, a trade-off between slow convergence and oscillating updates exists.

Original: The HIGs yield high accuracy in training across a wide range of values for τ , ranging from 10 − 5 to 10 − 3 .
Revised: The HIGs yield high accuracy in training across a wide range of values for τ , ranging from 10 − 5 to 10 − 3 .

Original: This supports the argumentation in the main text that the truncation is not  overly critical for HIGs.
Revised: This supports the argumentation in the main text that the truncation is not [Figure] [Figure] overly critical for HIGs.

Original: As long as numerical noise is suppressed with τ > 10 − 6 , and the actual information about scaling of network parameters and physical variables is not cut off.
Revised: As long as numerical noise is suppressed with τ > 10 − 6 , and the actual information about scaling of network parameters and physical variables is not cut off.

Original: The latter case is visible for an overly large τ = 0 .
Revised: The latter case is visible for an overly large τ = 0 .

Original: 01 in the last graph on the right.
Revised: 01 in the last graph on the right

Original:  Note that many graphs in ﬁgure 9 contain a small plateau at the start of each training run.
Revised: [Figure] Note that many graphs in ﬁgure 9 contain a small plateau at the start of each training run.

Original: These regions with relatively small progress per wall clock time are caused by the initialization overhead of the underlying deep learning framework (TensorFlow in our case).
Revised: These regions with relatively small progress per wall clock time are caused by the initialization overhead of the underlying deep learning framework (TensorFlow in our case).

Original: As all graphs measure wall clock time, we include the initialization overhead of TensorFlow, which causes a noticeable slow down of the ﬁrst iteration.
Revised: As all graphs measure wall clock time, we include the initialization overhead of TensorFlow, which causes a noticeable slow down of the ﬁrst iteration.

Original: Hence, the relatively slow convergence of the very ﬁrst steps in ﬁgureare not caused by conceptual issues with the HIGs themselves.
Revised: Hence, the relatively slow convergence of the very ﬁrst steps in ﬁgureare not caused by conceptual issues with the HIGs themselves.

Original: Rather, they are a result of the software frameworks and could, e.g., be alleviated with a pre-compilation of the training graphs.
Revised: Rather, they are a result of the software frameworks and could, e.g., be alleviated with a pre-compilation of the training graphs.

Original: In contrast, the initial convergence plateaus of Adam with smaller η in Figure 8 are of a fundamentally 
Revised: In contrast, the initial convergence plateaus of Adam with smaller η in Figure 8 are of a fundamentally different nature: they are caused by an inherent problem of non-inverting optimizers: their inability to appropriately handle the combination of large and small scale components in the physics of the quantum dipole setup (as outlined in section 3.3).

Original: [Table]
Revised: [Table]

Original: different nature: they are caused by an inherent problem of non-inverting optimizers: their inability to appropriately handle the combination of large and small scale components in the physics of the quantum dipole setup (as outlined in section 3.3).
Revised: [Figure] [Figure]

Original: Loss Functions.
Revised: Loss Functions.

Original: While training is evaluated in terms of the regular inner product as loss function:
Revised: While training is evaluated in terms of the regular inner product as loss function:

Original: L (Ψ a , Ψ b ) = 1 − |(cid:104) Ψ a , Ψ b (cid:105)| 2 , we use the following modiﬁed losses to evaluate low- and highenergy states for ﬁgure 4c.
Revised: L (Ψ a , Ψ b ) = 1 − |(cid:104) Ψ a , Ψ b (cid:105)| 2 , we use the following modiﬁed losses to evaluate low- and highenergy states for ﬁgure 4c.

Original: Let Ψ 1 be the ﬁrst excited state, then we deﬁne the low-energy loss as:
Revised: Let Ψ 1 be the ﬁrst excited state, then we deﬁne the low-energy loss as:

Original: [Equation]
Revised: [Equation]

Original: Correspondingly, we deﬁne the high-energy loss with the second excited state Ψ 2 :
Revised: Correspondingly, we deﬁne the high-energy loss with the second excited state Ψ 2 :

Original: [Equation]
Revised: [Equation]

Revised: Additional Experiments with a Convolutional Neural Network.

Original: C A BLATION STUDY
Revised: Our method is agnostic to speciﬁc network architectures. To illustrate this, we conduct additional experiments with a convolutional neural network. The setup is the same as before, only the fully-connected neural network is replaced by a network with 6 hidden convolutional layers each with kernel size 3,features and tanh activation, followed by an 384 neuron dense output layer with linear activation giving the network a total of 21984 trainable parameters. The results of these experiments are plotted in ﬁgure 10 and 11.

Revised: We ﬁnd that HIGs behave in line with the fully-connected network case (ﬁgure 9).

Revised: There exists a range τ -values from around 10 − 5 to 10 − 3 for which stable training is possible.

Revised: Regarding optimization with Adam, we likewise observe a faster and more accurate minimization of the loss function for the best HIG run ( η = 0 .

Revised: 7 , b = 16 , τ = 10 − 4 )

Revised: compared to the best Adam run ( η = 0 . 0002 , b = 256 ).

Revised: C A BLATION STUDY

Original: In this last section, we investigate how the HIG-hyperparameters affect the outcome.
Revised: In this last section, we investigate how the HIG-hyperparameters affect the outcome.

Original: This includes ablation experiments with respect to κ, β , and τ deﬁned in section 2.2.
Revised: This includes ablation experiments with respect to κ and τ deﬁned in section 2.2.

Original: We use the nonlinear oscillator example as the basis for these comparisons.
Revised: We use the nonlinear oscillator example as the basis for these comparisons and consider the following HIG update step: [Equation]

Revised: Here, the exponent < β, κ > of the Jacobian denotes the following procedure deﬁned with the aid of the singular value decomposition J = U Λ V (cid:62) as:

Revised: [Equation]

Revised: Compared to the HIG update 5 in the main text, update 21 has an additional scalar prefactor with an parameter β resulting from earlier experiments with our method.

Original: Exponent of the diagonal singular value matrix κ  :
Revised: Setting β = − 1 − κ yields algorithms that rescale the largest singular value to 1 , which ensures that the resulting updates cannot produce arbitrarily large updates in y -space. This can be thought of as a weaker form of scale invariance. Just as 5, equation 21 deﬁnes an interpolation between gradient descent ( β = 0 , κ = 1 ) and the Gauss-Newton method ( β = 0 , κ = − 1 ) as well. Scalar prefactor term β :

Revised: We test β -values between 0 , no scale correction, and − 0 .

Revised: 5 , which fully normalizes the effect of the largest singular value for κ = − 0 .

Revised: The results are shown in ﬁgure 12a.

Revised: Compared to the other hyperparameters, we observe that β has only little inﬂuence on the outcome, which is why we decided to present the method without this parameter in the main text.

Revised: Exponent of the diagonal singular value matrix κ :

Original: We test κ for various values between 1 . 0 , stochastic gradient descent, and − 1 , Gauss-Newton.
Revised: We test κ for various values between 1 . 0 , stochastic gradient descent, and − 1 , Gauss-Newton.

Original: The results are shown in ﬁgure 10a.
Revised: The results are shown in ﬁgure 12b.

Original: For positive values, curves stagnate early, while for negative κ , the ﬁnal loss values are several orders of magnitude better.
Revised: For positive values, curves stagnate early, while for negative κ , the ﬁnal loss values are several orders of magnitude better.

Original: The HIG curve corresponding to β = − 0 .
Revised: The HIG curve corresponding to β = − 0 .

Original: 5 achieves the best result.
Revised: 5 achieves the best result.

Original: This supports our argumentation that a strong dependence on this parameter exists, and that a choice of κ = − 0 . 5 is indeed a good compromise for scale-correcting updates of reasonable size.
Revised: This supports our argumentation that a strong dependence on this parameter exists, and that a choice of κ = − 0 . 5 is indeed a good compromise for scale-correcting updates of reasonable size.

Original: The strong improvement as soon as κ becomes negative indicates that the collective inversion of the feedback of different data points of the mini-batch is an important ingredient in our method.
Revised: The strong improvement as soon as κ becomes negative indicates that the collective inversion of the feedback of different data points of the mini-batch is an important ingredient in our method.

Original: Scalar prefactor term β :

Original: We test β -values between 0 , no scale correction, and − 0 .

Original: 5 , which fully normalizes the effect of the largest singular value.

Original: The results are shown in ﬁgure 10b.

Original: Compared to the other hyperparameters, we observe that β has only little inﬂuence on the outcome.

Original: Truncation parameter τ :
Revised: Truncation parameter τ :

Original: To understand the effect of this parameter, we consider the singular value decomposition (SVD) of the network-solver Jacobian, which is determined by the SVDs of the network Jacobian and the solver Jacobian.
Revised: To understand the effect of this parameter, we consider the singular value decomposition (SVD) of the network-solver Jacobian, which is determined by the SVDs of the network Jacobian and the solver Jacobian.

Original: The singular values of a matrix product AB depend non-trivially on the singular values of the matrices A and B.
Revised: The singular values of a matrix product AB depend non-trivially on the singular values of the matrices A and B.

Original: In the simplest case, the singular values of the matrix product are received by multiplication of the individual singular values of both matrix factors.
Revised: In the simplest case, the singular values of the matrix product are received by multiplication of the individual singular values of both matrix factors.

Original: In the general case, this depends on how the singular vectors of A and B overlap with each other.
Revised: In the general case, this depends on how the singular vectors of A and B overlap with each other.

Original: However, it is likely that singular vectors with a small singular value of A or B overlap signiﬁcantly with singular vectors with a small singular value of AB.
Revised: However, it is likely that singular vectors with a small singular value of A or B overlap signiﬁcantly with singular vectors with a small singular value of AB.

Original: For this reason, it is important not to truncate too much as this might remove the small-scale physics modes that we are ultimately trying to preserve in order to achieve accurate results.
Revised: For this reason, it is important not to truncate too much as this might remove the small-scale physics modes that we are ultimately trying to preserve in order to achieve accurate results.

Original: On the other hand, less truncation leads to large updates of network weights on a scale beyond the validation of the linear approximation by ﬁrst-order derivatives.
Revised: On the other hand, less truncation leads to large updates of network weights on a scale beyond the validation of the linear approximation by ﬁrst-order derivatives.

Original: These uncontrolled network modiﬁcations can lead to over-saturated neurons and prevent further training progress.
Revised: These uncontrolled network modiﬁcations can lead to over-saturated neurons and prevent further training progress.

Original: From a practical point of view, we choose τ according to the accuracy of the pure physics optimization problem without a neural network.
Revised: From a practical point of view, we choose τ according to the accuracy of the pure physics optimization problem without a neural network.

Original: For the quantum dipole training, this value was set to 10 − 5 .
Revised: For the quantum dipole training, this value was set to 10 − 5 .

Original: Trying to solve the pure physics optimization with far smaller values leads to a worse result or no convergence at all.
Revised: Trying to solve the pure physics optimization with far smaller values leads to a worse result or no convergence at all.

Original: The network training behaves in line with this: Figure 9 shows that the network does not learn to control the quantum system with τ -values far smaller than 10 − 5 .
Revised: The network training behaves in line with this: Figure 9 shows that the network does not learn to control the quantum system with τ -values far smaller than 10 − 5 .

Original: For the nonlinear oscillator system, the pure physics optimization is stable over a large range of τ -values with similarly good results.
Revised: For the nonlinear oscillator system, the pure physics optimization is stable over a large range of τ -values with similarly good results.

Original: For the network training, we chose τ to be 10 − 6 .
Revised: For the network training, we chose τ to be 10 − 6 .

Original: We conducted further experiments for the network training with different τ from 10 − 5 to 10 − 10 presented in ﬁgure 11, which show that HIGs have a similar tolerance in τ .
Revised: We conducted further experiments for the network training with different τ from 10 − 5 to 10 − 10 presented in ﬁgure 13, [Figure] [Figure] which show that HIGs have a similar tolerance in τ .

Original: For a comparison, we also plotted Gauss-Newton curves for different τ .
Revised: For a comparison [Figure] we also plotted GaussNewton curves for different τ .

Original: We observe that GN curves become more unstable for smaller truncation values τ and diverge in the case 10 − 9 and 10 − 10 while HIG curves achieve overall better loss values and start to converge in this parameter.[Figure] [Figure]
Revised: We observe that GN curves become more unstable for smaller truncation values τ and diverge in the case 10 − 9 and 10 − 10 while HIG curves achieve overall better loss values and start to converge in this parameter.