Original: P HYSICAL G RADIENTS FOR D EEP L EARNING
Revised: P HYSICAL G RADIENTS FOR D EEP L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Solving inverse problems, such as parameter estimation and optimal control, is a vital part of science.
Revised: Solving inverse problems, such as parameter estimation and optimal control, is a vital part of science.

Original: Many experiments repeatedly collect data and rely on machine learning algorithms to quickly infer solutions to the associated inverse problems.
Revised: Many experiments repeatedly collect data and rely on machine learning algorithms to quickly infer solutions to the associated inverse problems.

Original: We ﬁnd that state-of-the-art training techniques are not well-suited to many problems that involve physical processes since the magnitude and direction of the gradients can vary strongly.
Revised: We ﬁnd that state-of-the-art training techniques are not well-suited to many problems that involve physical processes since the magnitude and direction of the gradients can vary strongly.

Original: We propose a novel hybrid training approach that combines higher-order optimization methods with machine learning techniques.
Revised: We propose a novel hybrid training approach that combines higher-order optimization methods with machine learning techniques.

Original: We take updates of a higher-order or domain-speciﬁc optimizer and embed them into the gradient-descent-based learning pipeline as a physical gradient , replacing the regular gradient of the physical process.
Revised: We take updates of a higher-order or domain-speciﬁc optimizer and embed them into the gradient-descent-based learning pipeline as a physical gradient , replacing the regular gradient of the physical process.

Original: This also allows us to introduce domain knowledge into training by incorporating priors about the solution space into the gradients.
Revised: This also allows us to introduce domain knowledge into training by incorporating priors about the solution space into the gradients.

Original: We demonstrate the capabilities of our method on a variety of canonical physical systems, showing that physical gradients yield signiﬁcant improvements on a wide range of optimization and learning problems.
Revised: We demonstrate the capabilities of our method on a variety of canonical physical systems, showing that physical gradients yield signiﬁcant improvements on a wide range of optimization and learning problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Inverse problems that involve physical systems play a central role in computational science.
Revised: Inverse problems that involve physical systems play a central role in computational science.

Original: This class of problems includes parameter estimation (Tarantola, 2005) and optimal control (Zhou et al., 1996).
Revised: This class of problems includes parameter estimation (Tarantola, 2005) and optimal control (Zhou et al., 1996).

Original: Solving inverse problems is integral in determining the age of the universe (Freedman et al., 2001), measuring the amount of dark matter (Padmanabhan & White, 2009), detecting gravitational waves (George & Huerta, 2018b), controlling plasma ﬂows (Maingi et al., 2019), searching for neutrinoless double-beta decay (Agostini et al., 2013; Aalseth et al., 2018), reconstructing particle trajectories and energies (Belayneh et al., 2020), and testing general relativity (Dyson et al., 1920;
Revised: Solving inverse problems is integral in determining the age of the universe (Freedman et al., 2001), measuring the amount of dark matter (Padmanabhan & White, 2009), detecting gravitational waves (George & Huerta, 2018b), controlling plasma ﬂows (Maingi et al., 2019), searching for neutrinoless double-beta decay (Agostini et al., 2013; Aalseth et al., 2018), reconstructing particle trajectories and energies (Belayneh et al., 2020), and testing general relativity (Dyson et al., 1920;

Original: Kraniotis & Whitehouse, 2003).
Revised: Kraniotis & Whitehouse, 2003).

Original: Decades of research in optimization have produced a wide range of iterative methods and algorithms for solving inverse problems (Press et al., 2007).
Revised: Decades of research in optimization have produced a wide range of iterative methods and algorithms for solving inverse problems (Press et al., 2007).

Original: Higher-order methods such as limited-memory
Revised: Higher-order methods such as limited-memory

Original: BFGS (Liu & Nocedal, 1989) have been especially successful.
Revised: BFGS (Liu & Nocedal, 1989) have been especially successful.

Original: These methods have two fundamental drawbacks.
Revised: These methods have two fundamental drawbacks.

Original: First, they require a good initial guess of the solution.
Revised: First, they require a good initial guess of the solution.

Original: When the initial guess is far away from the true solution in parameter space, iterative methods may diverge or converge to a local optimum instead.
Revised: When the initial guess is far away from the true solution in parameter space, iterative methods may diverge or converge to a local optimum instead.

Original: Second, their computation cost depends on the problem they are solving as the gradient and possibly even Hessian matrix of the problem needs to be evaluated for each iteration.
Revised: Second, their computational cost is strongly problem-dependent, as the gradient and possibly even Hessian matrix need to be evaluated for each iteration of the optimization.

Original: This may be negligible for simple tasks but when the optimization involves a physical simulation, each iteration can take seconds.
Revised: This may be negligible for simple tasks but when the optimization involves a physical simulation, each iteration can take seconds or longer.

Original: MHowever, many experiments continuously or repeatedly collect data, yielding data sets with millions or billions of individual events, each posing an inverse problem.
Revised: However, many experiments continuously or repeatedly collect data, yielding data sets with millions or billions of individual events, each posing an inverse problem.

Original: Solving all of these with iterative solvers poses a major challenge which is why, i In such settings, data analysis has often turned to machine learning algorithms to quickly infer approximate solutions given the observations (Carleo et al., 2019; Delaquis et al., 2018; George & Huerta, 2018b; Agostini et al., 2013).
Revised: Solving all of these with iterative solvers poses a major challenge which is why, in such settings, data analysis has often turned to machine learning algorithms to quickly infer approximate solutions given the observations (Carleo et al., 2019; Delaquis et al., 2018; George & Huerta, 2018b; Agostini et al., 2013).

Original: The inferred solutions can  be used to ﬁlter data sets for interesting events and can serve as the basis for  more accurate analysis (CMS Collaboration, 2020; George & Huerta, 2018a;b).
Revised: The inferred solutions can then, for example, be used to ﬁlter data sets for interesting events or can serve as the starting point for a more accurate analysis (CMS Collaboration, 2020; George & Huerta, 2018a;b).

Original: Interestingly, using machine learning models to solve the inverse problems also reduces the chances of divergence or ﬁnding local optima.
Revised: Interestingly, using machine learning models to solve the inverse problems also reduces the chances of divergence or convergence to undesired local optima.

Original: Unlike iterative solvers, learning models are jointly trained on all data which allows updates from one example to avoid local minima in a different example.
Revised: Unlike iterative solvers, learning models are jointly trained on large collections of data. The shared parameters allow updates from one example to avoid local minima from a different example.

Original: Fig.shows a generic case of this.
Revised: Fig. 1 shows a generic case illustrating this effect:

Original: The task is to localize a wave packet A · sin( f · x ) · exp ( − 12 ( x − x 0 ) 2 /σ 2 ) and determine its amplitude from a noisy recorded time series (grey curve).
Revised: The task is to localize a wave packet A · sin( f · x ) · exp ( − 12 ( x − x 0 ) 2 /σ 2 ) and determine its amplitude from a noisy recorded time series (grey curve).

Original: Iterative solvers fail to reconstruct the wave packet when the initial guess is not already close to the true parameters, even diverging in 0.12% of our cases.
Revised: Iterative solvers fail to reconstruct the wave packet when the initial guess is not already close to the true parameters, even diverging in 0.12% of our cases.

Original: In contrast, a neural network trained on the same data using the same objective quickly learns to ﬁt all wave packets.
Revised: In contrast, a neural network trained on the same data using the same objective quickly learns to ﬁt all wave packets.

Original: Details are given in appendix B.1.
Revised: Details are given in appendix B.1.

Original: In this paper, we derive a hybrid optimization algorithm that integrates higher-order inverse solvers into the traditional deep learning pipeline without imposing these limitations and without discarding the progress made in ﬁrst-order optimization.
Revised: In this paper, we derive a hybrid optimization algorithm that integrates higher-order inverse solvers into the traditional deep learning pipeline without imposing these limitations and without discarding the progress made in ﬁrst-order optimization.

Original: Our method can work with arbitrary second-order solvers when the solution space of the inverse problem allows for computation of the Hessian.
Revised: Our method can work with arbitrary second-order solvers when the solution space of the inverse problem allows for computation of the Hessian.

Original: Otherwise, it requires additional domain knowledge which is used to formulate the inverse solver in a more efﬁcient manner.
Revised: Otherwise, it requires additional domain knowledge which is used to formulate the inverse solver in a more efﬁcient manner.

Original: The result of the embedded inverse-problem solver takes the place of the ﬁrstorder gradient of the physical process, and we refer to it as the physical gradient (PG).
Revised: The result of the embedded inverse-problem solver takes the place of the ﬁrst-order gradient of the physical process, and we refer to it as the physical gradient (PG).

Original: When used in conjunction with training deep learning models, physical gradients allow each update step to encode nonlinear physics, which yields substantially more accurate optimization directions.
Revised: When used in conjunction with training deep learning models, physical gradients allow each update step to encode nonlinear physics, which yields substantially more accurate optimization directions.

Original: A key advantage of our approach is its compatibility with acceleration schemes (Duchi et al., 2011; Kingma & Ba, 2015) and stabilization techniques (Ioffe & Szegedy, 2015; Huang et al., 2016; Ba et al., 2016) developed for training deep learning models.
Revised: A key advantage of our approach is its compatibility with acceleration schemes (Duchi et al., 2011; Kingma & Ba, 2015) and stabilization techniques (Ioffe & Szegedy, 2015; Huang et al., 2016; Ba et al., 2016) developed for training deep learning models.

Original: [Figure]
Revised: [Figure]

Original: The presented hybrid training approach differs fundamentally from pure ﬁrst-order training.
Revised: The presented hybrid training approach differs fundamentally from pure ﬁrst-order training.

Original: While PGs take the place of regular gradients in the optimization algorithm, they are based on real physical states instead of the adjoint ones calculated in backpropagation.
Revised: While PGs take the place of regular gradients in the optimization algorithm, they are based on real physical states instead of the adjoint ones calculated in backpropagation.

Original: This feature makes it possible to integrate solution priors into the training without restricting the model architecture (Kim et al., 2019), adding soft constraints to the objective (Raissi et al., 2018), or relying exclusively on ﬁrstorder updates (Rackauckas et al., 2020).
Revised: This feature makes it possible to integrate solution priors into the training without restricting the model architecture (Kim et al., 2019), adding soft constraints to the objective (Raissi et al., 2018), or relying exclusively on ﬁrstorder updates (Rackauckas et al., 2020).

Original: We test our method on a wide variety of inverse problems including the highly challenging Navier-Stokes equations.
Revised: We test our method on a wide variety of inverse problems including the highly challenging Navier-Stokes equations.

Original: In all of our experiments, neural networks trained with PGs significantly outperform state-of-the-art training schemes, yielding accuracy gains of up to three orders of magnitude.
Revised: In all of our experiments, neural networks trained with PGs significantly outperform state-of-the-art training schemes, yielding accuracy gains of up to three orders of magnitude.

Original:  M ETHOD
Revised:  M ETHOD

Original: We consider unconstrained inverse problems that involve a differentiable physical process P : X ⊂ R d x → Y ⊂ R d
Revised: We consider unconstrained inverse problems that involve a differentiable physical process P : X ⊂ R d x → Y ⊂ R d

Original: y which can be simulated.
Revised: y which can be simulated.

Original: Here X denotes the physical parameter space and Y the space of possible observed outcomes.
Revised: Here X denotes the physical parameter space and Y the space of possible observed outcomes.

Original: Given an observed or desired output y ∗ ∈ Y , the inverse problem consists of ﬁnding optimal parameters
Revised: Given an observed or desired output y ∗ ∈ Y , the inverse problem consists of ﬁnding optimal parameters

Original: [Equation]
Revised: [Equation]

Original: We are interested in approximating this problem using a neural network, x ∗ =
Revised: We are interested in approximating this problem using a neural network, x ∗ =

Original: NN( y ∗ | θ ) , parameterized by θ .
Revised: NN( y ∗ | θ ) , parameterized by θ .

Original: Let Y ∗ =
Revised: Let Y ∗ =

Original: { y ∗ i | i = 1 , ..., N } denote a set of N inverse problems, all sharing the same physical process P .
Revised: { y ∗ i | i = 1 , ..., N } denote a set of N inverse problems, all sharing the same physical process P .

Original: Then training the network is equivalent to ﬁnding
Revised: Then training the network is equivalent to ﬁnding

Original: [Equation]
Revised: [Equation]

Original: This training scheme is unsupervised, i.e. no labels are required over the training data Y .
Revised: This training scheme is unsupervised, i.e. no labels are required over the training data Y .

Original: 2. I TERATIVE OPTIMIZATION
Revised: 2. I TERATIVE OPTIMIZATION

Original: Inverse problems as in Eq.
Revised: Inverse problems as in Eq.

Original: 1 are classically solved by starting with an initial guess x 0 and iteratively applying updates ∆ x k .
Revised: 1 are classically solved by starting with an initial guess x 0 and iteratively applying updates ∆ x k .

Original: This yields an estimate of the inverse of the physical model P , i.e., x ∗ ≈
Revised: This yields an estimate of the inverse of the physical model P , i.e., x ∗ ≈

Original: P − 1 n ∗ ( y ∗ | x 0 ) where n ∗ denotes the number of iterations required to reach a chosen convergence threshold.
Revised: P − 1 n ∗ ( y ∗ | x 0 ) where n ∗ denotes the number of iterations required to reach a chosen convergence threshold.

Original: A key decision in such an iterative optimization scheme is how to derive the update ∆ x k .
Revised: A key decision in such an iterative optimization scheme is how to derive the update ∆ x k .

Original: Figure 2 visually compares gradient descent, Newton’s method and PGs which we will discuss and compare in the following paragraphs.
Revised: Figure 2a visually compares gradient descent, Newton’s method and PGs which we will introduce in section 2.2. [Figure]

Original: First-order methods The scale of typical machine learning datasets necessitates the use of minibatches, which has led popular optimization methods in machine learning to rely only on ﬁrstorder derivatives of L .
Revised: First-order methods The scale of typical machine learning datasets necessitates the use of minibatches, which has led popular optimization methods in machine learning to rely only on ﬁrstorder derivatives of L .

Original: These Hessian-free methods are popular in deep learning due to their low[Figure] computational cost and stable convergence (Goodfellow et al., 2016).
Revised: These Hessian-free methods are popular in deep learning due to their low computational cost and stable convergence (Goodfellow et al., 2016).

Original: The simplest such method is gradient descent (GD) (Curry, 1944) and its stochastic version (SGD) where ∆ x = − η · (cid:0) ∂L∂x (cid:1) T .
Revised: The simplest such method is gradient descent (GD) (Curry, 1944) and its stochastic version (SGD) where ∆ x = − η · (cid:0) ∂L∂x (cid:1) T .

Original: Note that ∂L∂x , and therefore of ∆ x , scales inversely with x .
Revised: Note that ∂L∂x , and therefore of ∆ x , scales inversely with x .

Original: When x carries physical units, neither ∆ x nor ∆ x T lie in the same space as x .
Revised: When x carries physical units, neither ∆ x nor ∆ x T lie in the same space as x .

Original: This scaling behavior is a problem when dealing with sensitive or insensitive functions, i.e. functions whose gradient magnitudes (cid:12)(cid:12) ∂L∂x (cid:12)(cid:12) are far from 1.
Revised: This scaling behavior is a problem when dealing with sensitive or insensitive functions, i.e. functions whose gradient magnitudes (cid:12)(cid:12) ∂L∂x (cid:12)(cid:12) are far from 1.

Original: Large gradient magnitudes – when small changes in x cause large changes in L – counter-intuitively lead to large updates ∆ x when using GD, resulting in even larger changes to L .
Revised: When small changes in x cause large changes in L , this counter-intuitively leads to large updates ∆ x when using GD, resulting in even larger changes to L .

Original: This is often referred to as exploding gradients, and the opposite, vanishing gradients, occurs with insensitive functions.
Revised: This is often referred to as exploding gradients, and the opposite effect, vanishing gradients, occurs with insensitive functions.

Original: Machine learning models can be tuned to behave well given these updates (Ioffe & Szegedy, 2015; Loshchilov & Hutter, 2019) but in physics-based optimization, where functions may be extremely sensitive in certain parts of the parameter space or on subsets of training examples, GD often prescribes suboptimal optimization directions.
Revised: Machine learning models can be tuned to behave well given these updates (Ioffe & Szegedy, 2015; Loshchilov & Hutter, 2019) but in physics-based optimization, where functions may be extremely sensitive in certain parts of the parameter space or on subsets of training examples, GD often prescribes suboptimal optimization directions.

Original: While there are a number of ﬁrst-order optimizers that approximate higher-order information to improve convergence (Kingma & Ba, 2015; Hestenes et al., 1952; Duchi et al., 2011; Martens &
Revised: While there are a number of ﬁrst-order optimizers that approximate higher-order information to improve convergence (Kingma & Ba, 2015; Hestenes et al., 1952; Duchi et al., 2011; Martens &

Original: Grosse, 2015), they cannot take nonlinearities into account, which results in suboptimal optimization directions when optimizing nonlinear functions.
Revised: Grosse, 2015), they cannot take nonlinearities into account, which results in suboptimal optimization directions when optimizing nonlinear functions.

Original: Second-order methods Newton’s method (Atkinson, 2008) uses the inverse Hessian to determine the optimization update ∆ x = − η · (cid:16) ∂ 2 L ∂x 2 (cid:17) − 1 (cid:0) ∂L∂x (cid:1) T .
Revised: Second-order methods Newton’s method (Atkinson, 2008) uses the inverse Hessian to determine the optimization update ∆ x = − η · (cid:16) ∂ 2 L ∂x 2 (cid:17) − 1 (cid:0) ∂L∂x (cid:1) T .

Original: Direct computation of the Hessian adds a signiﬁcant computational cost to each iteration.
Revised: Direct computation of the Hessian adds a signiﬁcant computational cost to each iteration.

Original: Quasi-Newton methods (Broyden, 1970; Liu &
Revised: Quasi-Newton methods (Broyden, 1970; Liu &

Original: Nocedal, 1989; Gill & Murray, 1978; Mor´e, 1978; Powell, 1970; Berndt et al., 1974; Conn et al., 1991; Avriel, 2003) alleviate this issue by approximating the inverse Hessian instead.
Revised: Nocedal, 1989; Gill & Murray, 1978; Mor´e, 1978; Powell, 1970; Berndt et al., 1974; Conn et al., 1991; Avriel, 2003) alleviate this issue by approximating the inverse Hessian instead.

Original: Unlike GD,
Revised: Unlike GD,

Original: Newton-type methods prescribe small updates to sensitive functions and large updates to insensitive ones.
Revised: Newton-type methods prescribe small updates to sensitive functions and large updates to insensitive ones.

Original: The resulting update steps typically progress towards optima much faster than ﬁrst-order methods (Ye et al., 2019).
Revised: The resulting update steps typically progress towards optima much faster than ﬁrst-order methods (Ye et al., 2019).

Original: However, the difﬁculty of approximating the Hessian for batched data sets (Schraudolph et al., 2007) and the high cost of evaluating the Hessian directly have largely prevented second-order methods from being utilized in machine learning research (Goodfellow et al., 2016).
Revised: However, the difﬁculty of approximating the Hessian for batched data sets (Schraudolph et al., 2007) and the high cost of evaluating the Hessian directly have largely prevented second-order methods from being utilized in machine learning research (Goodfellow et al., 2016).

Original: Domain-speciﬁc inverse solvers Domain knowledge can often be used to formulate more efﬁcient optimization updates or to directly approximate a solution.
Revised: Domain-speciﬁc inverse solvers Domain knowledge can often be used to formulate more efﬁcient optimization updates or to directly approximate a solution.

Original: A classical example would be preconditioners that, when chosen correctly, allow linear solvers to converge much more quickly and reliably.
Revised: A classical example would be preconditioners that, when chosen correctly, allow linear solvers to converge much more quickly and reliably.

Original: In nonlinear settings, part of the governing equations may be inverted analytically or priors about the solution space may be incorporated into the solver, for example.
Revised: In nonlinear settings, part of the governing equations may be inverted analytically or priors about the solution space may be incorporated into the solver, for example.

Original: A time-reversed simulation may be able to more accurately recover prior states than a generic inverse solver.
Revised: A time-reversed simulation may be able to more accurately recover prior states than a generic inverse solver.

Original: These methodsand shortcuts are commonly used in optimization but their use has eluded machine learning applications so far due to the ﬁxed gradient-descent pipeline.
Revised: These methods are commonly used in optimization but their use has eluded machine learning applications so far due to the ﬁxed gradient-descent pipeline.

Original: When learning with PGs, domain-speciﬁc optimizers can be integrated into training like any other optimizers, allowing for greatly-increased training convergence.
Revised: When learning with PGs, domain-speciﬁc optimizers can be integrated into training like any other optimizers, allowing for greatly-increased training convergence.

Original: Appendix appendix A.2 lists necessary conditions that need to be fulﬁlled.
Revised: Appendix appendix A.2 lists necessary conditions that need to be fulﬁlled.

Original: 2. P HYSICAL GRADIENTS FOR DEEP LEARNING
Revised: 2. P HYSICAL GRADIENTS FOR DEEP LEARNING

Original: We now consider the problem of learning solutions to the inverse problems as in Eq.
Revised: We now consider the problem of learning solutions to the inverse problems as in Eq.

Original: As discussed above, GD-based optimizers fail to take the nonlinearity and potentially strongly varying sensitivity of the joint problem into account while higher-order optimizers are hard to apply directly.
Revised: As discussed above, GD-based optimizers fail to take the nonlinearity and potentially strongly varying sensitivity of the joint problem into account while higher-order optimizers are hard to apply directly.

Original: Instead, we aim to leverage the advantages of higher-order optimizers in a deep learning setting by embedding arbitrary inverse-problem solvers as physical gradients (PGs) into the ﬁrst-order optimization pipeline.
Revised: Instead, we aim to leverage the advantages of higher-order optimizers in a deep learning setting by embedding arbitrary inverse-problem solvers as physical gradients (PGs) into the ﬁrst-order optimization pipeline.

Original: The network updates ∆ θ can then be computed using any ﬁrst-order optimizer such as SGD or Adam.
Revised: The network updates ∆ θ can then be computed using any ﬁrst-order optimizer such as SGD or Adam.

Original: Additionally, all state-of-the-art deep learning techniques, such as normalization Ioffe & Szegedy (2015); Ba et al.
Revised: Additionally, all state-of-the-art deep learning techniques, such as normalization Ioffe & Szegedy (2015); Ba et al.

Original: (2016) or dropout Srivastava et al.
Revised: (2016) or dropout Srivastava et al.

Original: (2014) can be employed, which often improves training speed and generalization performance Keskar et al.
Revised: (2014) can be employed, which often improves training speed and generalization performance Keskar et al.

Original: Our method can be applied whenever one of the following is available: (i) ∂ 2 P ∂x 2 can be computed numerically, (ii) a higher-order update such as (cid:16) ∂ 2 P ∂x 2 (cid:17) − 1 ∂ P ∂x can be derived analytically, or (iii) an inverse-physics solver can be derived using the available domain knowledge.
Revised: Our method can be applied whenever one of the following is available: (i) ∂ 2 P ∂x 2 can be computed numerically, (ii) a higher-order update such as (cid:16) ∂ 2 P ∂x 2 (cid:17) − 1 ∂ P ∂x can be derived analytically, or (iii) an inverse-physics solver can be derived using the available domain knowledge.

Original: If (i) is fulﬁlled, we can explicitly compute the Newton direction in in x space.
Revised: If (i) is fulﬁlled, we can explicitly compute the Newton direction in in x space.

Original: Thus, with any of (i), (ii) or (iii) we have access to better updates ∆ x than ∆ x GD ∝ ∂ P ∂x .
Revised: Thus, with any of (i), (ii) or (iii) we have access to better updates ∆ x than ∆ x GD ∝ ∂ P ∂x .

Original: These ∆ x are the basis for our method which we describe in the following.
Revised: These ∆ x are the basis for our method which we describe in the following.

Original: We derive our method starting from Eq. 2 which describes the combined problem of ﬁnding both solutions x ∗ to the individual inverse problems
Revised: We derive our method starting from Eq. 2 which describes the combined problem of ﬁnding both solutions x ∗ to the individual inverse problems

Original: y ∗ ∈
Revised: y ∗ ∈

Original: Y ∗ and network weights θ that approximate them.
Revised: Y ∗ and network weights θ that approximate them.

Original: One way to solve this problem is to ﬁrst consider all inverse problems separately and precompute corresponding solutions X sv
Revised: One way to solve this problem is to ﬁrst consider all inverse problems separately and precompute corresponding solutions X sv

Original: = { x sv i : P ( x sv i ) =
Revised: = { x sv i : P ( x sv i ) =

Original: y ∗ i } using an iterative optimizer.
Revised: y ∗ i } using an iterative optimizer.

Original: Then, X sv could be used as labels for supervised training of a neural network:
Revised: Then, X sv could be used as labels for supervised training of a neural network:

Original: [Equation]
Revised: [Equation]

Original: This precomputation makes it possible to use an efﬁcient optimization method for X sv while training the neural network with a ﬁrst-order optimizer, retaining the ability to employ state-of-the-art deep learning techniques, such as normalization (Ioffe & Szegedy, 2015; Ba et al., 2016) or dropout (Srivastava et al., 2014).
Revised: This precomputation makes it possible to use an efﬁcient optimization method for X sv while training the neural network with a ﬁrst-order optimizer, retaining the ability to employ state-of-the-art deep learning techniques, such as normalization (Ioffe & Szegedy, 2015; Ba et al., 2016) or dropout (Srivastava et al., 2014).

Original: However, this approach has severe drawbacks.
Revised: However, this approach has severe drawbacks.

Original: For one, the individual optimizations can more easily get stuck in local optima than the combined training (Holl et al., 2020).
Revised: For one, the individual optimizations can more easily get stuck in local optima than the combined training (Holl et al., 2020).

Original: Also, many inverse problems are inherently multi-modal, i.e. multiple solutions x ∗ exist for one y ∗ .
Revised: Also, many inverse problems are inherently multi-modal, i.e. multiple solutions x ∗ exist for one y ∗ .

Original: In such settings x sv heavily depends on the initial guess x 0 used for precomputation, which may cause the network to interpolate between possible solutions, leading to subpar convergence and generalization performance.
Revised: In such settings x sv heavily depends on the initial guess x 0 used for precomputation, which may cause the network to interpolate between possible solutions, leading to subpar convergence and generalization performance.

Original: To avoid these problems, we alter the training procedure from Eq.
Revised: To avoid these problems, we alter the training procedure from Eq.

Original: 3 in two ways.
Revised: 3 in two ways.

Original: First, we treat the inverse problem solver P − 1 as part of a coupled optimization and consider it as part of the training loop, yielding
Revised: First, we treat the inverse problem solver P − 1 as part of a coupled optimization and consider it as part of the training loop, yielding

Original: [Equation]
Revised: [Equation]

Original: Next we can condition the embedded inverse problem solver on the neural network prediction,
Revised: Next we can condition the embedded inverse problem solver on the neural network prediction,

Original: P − 1 n ∗ i ( y ∗ i )
Revised: P − 1 n ∗ i ( y ∗ i )

Original: → P − 1 n ∗ i ( y ∗ i | NN( y ∗ i | θ )) , by using it as an initial guess.
Revised: → P − 1 n ∗ i ( y ∗ i | NN( y ∗ i | θ )) , by using it as an initial guess.

Original: The embedded optimizer can now ﬁnd a minimum close to the prediction, which allows the neural network to choose any solution of a multi-modal problem.
Revised: The embedded optimizer can now ﬁnd a minimum close to the prediction, which allows the neural network to choose any solution of a multi-modal problem.

Original: Also, since all inverse problems from Y ∗ are optimized jointly, this reduces the likelihood that an individual solution gets stuck in a local minimum.
Revised: Also, since all inverse problems from Y ∗ are optimized jointly, this reduces the likelihood that an individual solution gets stuck in a local minimum.

Original: It is furthermore not necessary to run the iterative inverse solver to convergence.
Revised: It is furthermore not necessary to run the iterative inverse solver to convergence.

Original: Any choice of n ∈ N steps likely leads to convergence (proof given in appendix A.2), and choosing small n can increase the training speed.
Revised: Any choice of n ∈ N steps likely leads to convergence (proof given in appendix A.2), and choosing small n can increase the training speed.

Original: We also observe this in practice. In all our experiments, the case n =converges reliably with any physics optimizer we tried.
Revised: We also observe this in practice: Optimizations with n = 1 converged reliably in all our experiments.

Original: With these modiﬁcations, we arrive at
Revised: With these modiﬁcations, we arrive at

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The central quantity here is the difference between prediction and correction obtained from P − 1 n , which we refer to as the physical gradient .
Revised: The central quantity here is the difference between prediction and correction obtained from P − 1 n , which we refer to as the physical gradient .

Original: It encodes a valid physical state and takes the place of (cid:0) ∂ P ∂x (cid:1) T , which would otherwise be computed by backpropagation.
Revised: It encodes a valid physical state and takes the place of (cid:0) ∂ P ∂x (cid:1) T , which would otherwise be computed by backpropagation.

Original: Since the PG stems from a  higher-order or  domain-speciﬁc solver, the resulting updates ∆ θ will also be non-linear without computing the Hessian w.r.t.
Revised: Since the PG stems from a generic higher-order or a domain-speciﬁc solver, the resulting updates ∆ θ will also be non-linear without computing the Hessian w.r.t.

Original: θ (see Fig. 2b).
Revised: θ .

Original: Assuming the physics optimizer can account for varying function sensitivity in its updates ∆ x , the resulting PG updates ∆ θ will inherit that property since the network is generally tuned to retain gradient magnitude. 
Revised: Assuming the physics optimizer can account for varying function sensitivity in its updates ∆ x , the resulting PG updates ∆ θ will inherit that property since the network is generally tuned to retain gradient magnitude. The properties of

Revised: PG as well as GD and Newton’s method are summarized in Fig. 2b.

Original: While the L 2 measure in Eq.
Revised: While the L 2 measure in Eq.

Original: 5 resembles a supervised objective in x space, the actual network optimization does not behave as such.
Revised: 5 looks like a supervised objective in x space, the actual network optimization does not behave as such.

Original: To see this more clearly, consider P − 1 performing a single
Revised: To see this more clearly, consider P − 1 performing a single

Original: GD step with unit learning rate, P − 1 ( y ∗ | x ) = x − (cid:0)
Revised: GD step with unit learning rate, P − 1 ( y ∗ | x ) = x − (cid:0)

Original: ∂ P ∂x (cid:1) T ( P ( x ) − y ∗ ) .
Revised: ∂ P ∂x (cid:1) T ( P ( x ) − y ∗ ) .

Original: Then the gradient for Eq. 5 is (cid:0) ∂ P ∂x ∂x∂θ (cid:1) T ( P ( x ) − y ∗ ) , which is identical to the unsupervised training in Eq.
Revised: Then the gradient for Eq.is (cid:0) ∂ P ∂x ∂x∂θ (cid:1) T ( P ( x ) − y ∗ ) , which is identical to the unsupervised training in Eq.

Original: 2 (proof in appendix A.1).
Revised: 2 (the proof is given in appendix A.1).

Original: do x 0 ← NN( y ∗ | θ )
Revised: do x 0 ← NN( y ∗ | θ )

Original: [Table] for k = 1 , ..., n do x k ← P − 1 ( y ∗ | x k − 1 ) Accumulate iterative [Table]
Revised: Solution inference with NN for k = 1 , ..., n do updates

Original: for [Table] end ∆ θ ← (cid:0) ∂x∂θ (cid:1) T · ( x 0 − x n ) T [Table]
Revised: Backpropagation through NN

Original: θ ← Step( θ, ∆ θ, η ) [Table] end
Revised: Parameter update end

Original: Algorithm 1 details the corresponding neural network training procedure.
Revised: Algorithm 1 details the corresponding neural network training procedure.

Original: There, the effective gradient ∆ θ can also be written as ∂∂θ 12 (cid:107) x 0 − x n (cid:107) 22 with x n taken as constant as in Eq. 5, which lends itself to straightforward integration into machine learning frameworks where a simple L 2 loss can be used to optimize the network.
Revised: There, the effective gradient ∆ θ can also be written as ∂∂θ 12 (cid:107) x 0 − x n (cid:107) 22 with x n taken as constant as in Eq. 5, which lends itself to straightforward integration into machine learning frameworks where a simple L 2 loss can be used to optimize the network.

Original: This L 2 loss acts as a proxy to embed the physical gradient into the network training pipeline and is not to be confused with a traditional supervised loss in x space (see appendix A.1 for a more detailed discussion).
Revised: This L 2 loss acts as a proxy to embed the physical gradient into the network training pipeline and is not to be confused with a traditional supervised loss in x space (see appendix A.1 for a more detailed discussion).

Original:  R ESULTS
Revised:  R ESULTS

Original: We ﬁrst demonstrate the differences between various neural network training schemes on a simple one-parameter example before moving on to physical systems described by partial differential equations.
Revised: We ﬁrst demonstrate the differences between various neural network training schemes on a simple one-parameter example before moving on to physical systems described by partial differential equations.

Original: We speciﬁcally consider Poisson’s equation, the heat equation, and the NavierStokes equations.
Revised: We speciﬁcally consider Poisson’s equation, the heat equation, and the NavierStokes equations.

Original: This selection covers ubiquitous physical processes with diffusion, transport, and strongly non-local effects, featuring both explicit and implicit solvers.
Revised: This selection covers ubiquitous physical processes with diffusion, transport, and strongly non-local effects, featuring both explicit and implicit solvers.

Original: In each experiment, we train identically-initialized networks using different gradient schemes for P to examine the relative differences in convergence behavior.
Revised: In each experiment, we train identically-initialized networks using different gradient schemes for P to examine the relative differences in convergence behavior.

Original: For the weight update, we primarily employ the Adam optimizer (Kingma & Ba, 2015), a standard method in deep learning that outperforms SGD in our experiments.
Revised: For the weight update, we primarily employ the Adam optimizer (Kingma & Ba, 2015), a standard method in deep learning that outperforms SGD in our experiments.

Original: All training data are randomly generated on the ﬂy, resulting in effectively inﬁnite data sets Y .
Revised: All training data are randomly generated on the ﬂy, resulting in effectively inﬁnite data sets Y .

Original: All shown learning curves are therefore representative of the performance on unseen data.
Revised: All shown learning curves are therefore representative of the performance on unseen data.

Original: A detailed description of our experiments along with additional visualizations and performance measurements can be found in appendix B. 
Revised: A detailed description of our experiments along with additional visualizations and performance measurements can be found in appendix B. [Figure]

Original: 3. S INGLE - PARAMETER OPTIMIZATION
Revised: 3. S INGLE - PARAMETER OPTIMIZATION

Original: First, we consider the task of ﬁnding solutions x ∗ ∈ R to the inverse problem (Eq. 1)
Revised: First, we consider the task of ﬁnding solutions x ∗ ∈ R to the inverse problem (Eq. 1)

Original: P a ( x ) = sin( ax ) , y ∗ = − 1 , given a ∈ [0 . 1 , 10 . 1) .
Revised: P a ( x ) = sin( ax ) , y ∗ = − 1 , given a ∈ [0 . 1 , 10 . 1) .

Original: This problem may seem simple at ﬁrst glance but
Revised: This problem may seem simple at ﬁrst glance but

Original: P shares some properties with chaotic physical systems, which makes this problem hard to optimize.
Revised: P shares some properties with chaotic physical systems, which makes this problem hard to optimize.

Original: We train a neural network with three fully-connected hidden layers to solve this task; the learning curves are shown in Fig.
Revised: We train a neural network with three fully-connected hidden layers to solve this task; the learning curves are shown in Fig.

Original: Supervised training schemes (Eq. 3) perform poorly on this task due to multi-modality.[Figure] [Figure]
Revised: Supervised training schemes (Eq. 3) perform poorly on this task due to multi-modality.

Original: p t i m . M AE [Figure] Using Adam for the joint optimization – with backpropagation through the network and physics (Eq. 2) – avoids the aforementioned problem as the gradients dynamically guide the optimization towards a proximal minimum.
Revised: Using Adam for the joint optimization – with backpropagation through the network and physics (Eq. 2) – avoids the aforementioned problem as the gradients dynamically guide the optimization towards a proximal minimum.

Original: However, ∂ P ∂x oscillates without bound when x → 0 , which causes overﬂow errors in the optimization.
Revised: However, ∂ P ∂x oscillates without bound when x → 0 , which causes overﬂow errors in the optimization.

Original: To avoid this, we clip the gradients ∂ P ∂x to [ − 1 , 1] , which keeps the optimization relatively stable, although in most cases it never fully converges.
Revised: To avoid this, we clip the gradients ∂ P ∂x to [ − 1 , 1] , which keeps the optimization relatively stable, although in most cases it never fully converges.

Original: This demonstrates one of the key weaknesses of ﬁrst-order learning: when the magnitude of the gradients varies strongly between examples, the optimization is either slow or unstable, depending on the learning rate.
Revised: This demonstrates one of the key weaknesses of ﬁrst-order learning: when the magnitude of the gradients varies strongly between examples, the optimization is either slow or unstable, depending on the learning rate.

Original: Finally, we test two variants of physical gradients (Eq. 5) on this problem: Newton’s method and an analytic solver.
Revised: Finally, we test two variants of physical gradients (Eq. 5) on this problem: Newton’s method and an analytic solver.

Original: As Fig.
Revised: As Fig.

Original: 3 shows, the PG methods vastly outperform both supervised and unsupervised ﬁrst-order training, converging to an accuracy of around 10 − 5 .
Revised: 3 shows, the PG methods vastly outperform both supervised and unsupervised ﬁrst-order training, converging to an accuracy of around 10 − 5 .

Original: For this example, using
Revised: For this example, using

Original: PG with the inverted physics results in a 1000x improvement in accuracy over conventional network training, while being more stable.
Revised: PG with the inverted physics results in a 1000x improvement in accuracy over conventional network training, while being more stable.

Original: The gradients of all variants are visualized in appendix B.2.
Revised: The gradients of all variants are visualized in appendix B.2.

Original: 3. P OISSON ’ S EQUATION
Revised: P OISSON ’ S EQUATION

Original: Poisson’s equation, P ( x ) = ∇ − 2 x , plays an important role in electrostatics, Newtonian gravity, and ﬂuid dynamics (Ames, 2014).
Revised: Poisson’s equation, P ( x ) = ∇ − 2 x , plays an important role in electrostatics, Newtonian gravity, and ﬂuid dynamics (Ames, 2014).

Original: It has the property that local changes in x can affect P ( x ) globally.
Revised: It has the property that local changes in x can affect P ( x ) globally.

Original: [Figure] Here we consider a two-dimensional system and train a U-net (Ronneberger et al., 2015) to solve inverse problems (Eq. 1) on pseudo-randomly generated y ∗ .
Revised: Here we consider a two-dimensional system and train a U-net (Ronneberger et al., 2015) to solve inverse problems (Eq. 1) on pseudo-randomly generated y ∗ .

Original: Fig.
Revised: Fig.

Original: 4c shows the learning curves.
Revised: 4c shows the learning curves.

Original: We construct PGs based on the analytic inverse and couple them with Adam for training the neural network.
Revised: We construct PGs based on the analytic inverse and couple them with Adam for training the neural network.

Original: When learning with Adam or SGD with momentum, learning drastically slows after around 200 and 300 iterations, respectively.
Revised: When learning with Adam or SGD with momentum, learning drastically slows after around 200 and 300 iterations, respectively.

Original: Meanwhile, the learning curve involving PGs closely resembles an exponential curve, which indicates linear convergence, the ideal case for ﬁrst-order methods optimizing an L 2 objective.
Revised: Meanwhile, the learning curve involving PGs closely resembles an exponential curve, which indicates linear convergence, the ideal case for ﬁrst-order methods optimizing an L 2 objective.

Original: During all of training, the PG variant converges exponentially faster than the ﬁrst-order alternatives, its relative performance difference compared to Adam continually increasing from a factor of 3 at iteration 60 to a full order of magnitude after 5k iterations.
Revised: During all of training, the PG variant converges exponentially faster than the ﬁrst-order alternatives, its relative performance difference compared to Adam continually increasing from a factor of 3 at iteration 60 to a full order of magnitude after 5k iterations.

Original: 3. H EAT EQUATION
Revised: 3. H EAT EQUATION

Original: Next, we consider a system with fundamentally non-invertible dynamics.
Revised: Next, we consider a system with fundamentally non-invertible dynamics.

Original: The heat equation, ∂u∂t = ν · ∇ 2 u , models heat ﬂow in solids but also plays a part in many diffusive systems (Droniou, 2014).
Revised: The heat equation, ∂u∂t = ν · ∇ 2 u , models heat ﬂow in solids but also plays a part in many diffusive systems (Droniou, 2014).

Original: It gradually destroys information as the temperature equilibrium is approached (Grayson, 1987), causing ∇P to become near-singular.
Revised: It gradually destroys information as the temperature equilibrium is approached (Grayson, 1987), causing ∇P to become near-singular.

Original: Inspired by heat conduction in microprocessors, we generate examples x
Revised: Inspired by heat conduction in microprocessors, we generate examples x

Original: GT by randomly scattering four to ten heat generating rectangular regions on a plane and  simulating the heat proﬁle y ∗
Revised: GT by randomly scattering four to ten heat generating rectangular regions on a plane and [Figure] [Figure] simulating the heat proﬁle y ∗

Original: =
Revised: =

Original: P ( x GT ) as observed from outside a heat-conducting casing.
Revised: P ( x GT ) as observed from outside a heat-conducting casing.

Original: We train a U-net (Ronneberger et al., 2015) to solve the corresponding inverse problem (Eq. 1).
Revised: We train a U-net (Ronneberger et al., 2015) to solve the corresponding inverse problem (Eq. 1).

Original: The learning curves are shown in Fig.
Revised: The learning curves are shown in Fig.

Original: 5c.
Revised: 5c.

Original: Unlike with the previous example, here ∂ P ∂x is stable, and single inverse problems can be optimized with η = 1 using GD (see Fig.
Revised: Unlike with the previous example, here ∂ P ∂x is stable, and single inverse problems can be optimized with η = 1 using GD (see Fig.

Original: 5b).
Revised: 5b).

Original: For the unsupervised network training (Eq. 2), we use Adam with η = 10 − 3 and observe that the distance to the solution starts rapidly decreasing before decelerating between iterations 30 and 40 to a slow but mostly stable convergence.
Revised: For the unsupervised network training (Eq. 2), we use Adam with η = 10 − 3 and observe that the distance to the solution starts rapidly decreasing before decelerating between iterations 30 and 40 to a slow but mostly stable convergence.

Original: The sudden deceleration is rooted in the adjoint problem, which is also a diffusion problem.
Revised: The sudden deceleration is rooted in the adjoint problem, which is also a diffusion problem.

Original: Backpropagation through P removes detail from the gradients, which makes it hard for ﬁrst-order methods to recover the solution.
Revised: Backpropagation through P removes detail from the gradients, which makes it hard for ﬁrst-order methods to recover the solution.

Original: For training with PGs (Eq. 5), we consider L-BFGS-B and the analytically derived inverse-physics solver P − 1 .
Revised: For training with PGs (Eq. 5), we consider L-BFGS-B and the analytically derived inverse-physics solver P − 1 .

Original: As with the previous example, L-BFGS-B outperforms GD when optimizing single inverse problems but reaches a limit as to the accuracy it can reach due to ill-conditioning (see
Revised: As with the previous example, L-BFGS-B outperforms GD when optimizing single inverse problems but reaches a limit as to the accuracy it can reach due to ill-conditioning (see

Original: Fig. 5b).
Revised: Fig. 5b).

Original: While P can be inverted analytically, the resulting operator produces numerically highly unstable results.
Revised: While P can be inverted analytically, the resulting operator produces numerically highly unstable results.

Original: Instead, we use the available domain knowledge to formulate an inverse problem solver P − 1 that ﬁnds approximate x ∗ in a stable manner by introducing probabilistic reasoning into[Figure] with η = 1 , averaged the algorithm (see appendix B.4).
Revised: Instead, we use the available domain knowledge to formulate an inverse problem solver P − 1 that ﬁnds approximate x ∗ in a stable manner by introducing probabilistic reasoning into the algorithm (see appendix B.4).

Original: This increases the computation time per training iteration by about 7% compared to evaluating the regular gradient.
Revised: This increases the computation time per training iteration by about 7% compared to evaluating the regular gradient.

Original: Using L-BFGS-B with n = 32 as a PG to train the network leads to solutions that are about 20% more accurate and noticeably sharper after the same number of training iterations.
Revised: Using L-BFGS-B with n = 32 as a PG to train the network leads to solutions that are about 20% more accurate and noticeably sharper after the same number of training iterations.

Original: This advantage is achieved early during training and stays relatively constant throughout.
Revised: This advantage is achieved early during training and stays relatively constant throughout.

Original: From iteration 40 on,
Revised: From iteration 40 on,

Original: L-BFGS-B training converges about as fast as pure ﬁrst-order training.
Revised: L-BFGS-B training converges about as fast as pure ﬁrst-order training.

Original: However, the 17x longer computation time per iteration largely negates these advantages.
Revised: However, the 17x longer computation time per iteration largely negates these advantages.

Original: The PGs based on the inverse-physics solver, on the other hand, manage to improve the convergence speed substantially.
Revised: The PGs based on the inverse-physics solver, on the other hand, manage to improve the convergence speed substantially.

Original: They perform on par with L-BFGS-B training for the ﬁrst ∼ 35 iterations.
Revised: They perform on par with L-BFGS-B training for the ﬁrst ∼ 35 iterations.

Original: However, afterwards the increase in accuracy does not slow down as much as with the other methods, resulting in an exponentially faster convergence.
Revised: However, afterwards the increase in accuracy does not slow down as much as with the other methods, resulting in an exponentially faster convergence.

Original: At iteration 100, the predictions are around 34% more accurate compared to Adam, and the difference increases to 130% after 10k iterations.
Revised: At iteration 100, the predictions are around 34% more accurate compared to Adam, and the difference increases to 130% after 10k iterations.

Original: To reach the accuracy of regular Adam training with 10k iterations, L-BFGS-B training requiresiterations and training with the inverse-physics solver requires only 138 iterations.
Revised: To reach the accuracy of regular Adam training with 10k iterations, L-BFGS-B training requiresiterations and training with the inverse-physics solver requires only 138 iterations.

Original: Additional examples and learning curves are shown in section appendix B.4.
Revised: Additional examples and learning curves are shown in section appendix B.4.

Original: 3. N AVIER -S TOKES EQUATIONS
Revised: 3. N AVIER -S TOKES EQUATIONS

Original: Fluids and turbulence are among the most challenging and least understood areas of physics due to their highly nonlinear behavior and chaotic nature (Galdi, 2011).
Revised: Fluids and turbulence are among the most challenging and least understood areas of physics due to their highly nonlinear behavior and chaotic nature (Galdi, 2011).

Original: We consider a two-dimensional system governed by the incompressible Navier-Stokes equations: [Equation] − ∇ · ∇ v − ∇ p , ∇· v = 0 , ∇× p = 0 , where p denotes pressure and ν the viscosity.
Revised: We consider a two-dimensional system governed by the incompressible Navier-Stokes equations: ∂v∂t = ν ∇ 2 v − ∇ · ∇ v − ∇ p , ∇· v = 0 , ∇× p = 0 , where p denotes pressure and ν the viscosity.

Original: At t = 0 , a region of the ﬂuid is randomly marked with a massless colorant m 0 that passively moves with the ﬂuid, ∂m∂t = − v · ∇ m , and after a time t , the marker is observed again, m t .
Revised: At t = 0 , a region of the ﬂuid is randomly marked with a massless colorant m 0 that passively moves with the ﬂuid, ∂m∂t = − v · ∇ m , and after a time t , the marker is observed again, m t .

Original: An example observation pair y ∗ = { m 0 , m t } is shown in Fig.
Revised: An example observation pair y ∗ = { m 0 , m t } is shown in Fig.

Original: 6a.
Revised: 6a.

Original: We target the inverse problem (Eq. 1) of ﬁnding an initial ﬂuid velocity x ≡ vsuch that the ﬂuid simulation P matches m t at time t .
Revised: We target the inverse problem (Eq. 1) of ﬁnding an initial ﬂuid velocity x ≡ vsuch that the ﬂuid simulation P matches m t at time t .

Original: Since P is deterministic, x encodes the complete ﬂuid ﬂow from 0 to t .
Revised: Since P is deterministic, x encodes the complete ﬂuid ﬂow from 0 to t .

Original: We deﬁne the objective in frequency space with lower frequencies being weighted more strongly.
Revised: We deﬁne the objective in frequency space with lower frequencies being weighted more strongly.

Original: This deﬁnition considers the marker distribution match on all scales, from the coarse global match to ﬁne details, and is compatible with the deﬁnition in Eq.
Revised: This deﬁnition considers the marker distribution match on all scales, from the coarse global match to ﬁne details, and is compatible with the deﬁnition in Eq.

Original: We train a U-net (Ronneberger et al., 2015) to solve these inverse problems; the learning curves are shown in
Revised: We train a U-net (Ronneberger et al., 2015) to solve these inverse problems; the learning curves are shown in

Original: Fig.
Revised: Fig.

Original: 6c.
Revised: 6c.

Original: For unsupervised training (Eq. 2) with Adam, the error decreases for the ﬁrst 100 iterations while the network learns to infer velocities that lead to an approximate match.
Revised: For unsupervised training (Eq. 2) with Adam, the error decreases for the ﬁrst 100 iterations while the network learns to infer velocities that lead to an approximate match.

Original: The error then proceeds to[Table] decline at a much lower rate, nearly coming to a standstill.
Revised: The error then proceeds to decline at a much lower rate, nearly coming to a standstill.

Original: This is caused by an overshoot in terms of vorticity, as visible in Fig.
Revised: This is caused by an overshoot in terms of vorticity, as visible in Fig.

Original: 6a right.
Revised: 6a right.

Original: While the resulting dynamics can roughly approximate the shape of the observed m t , they fail to match its detailed structure.
Revised: While the resulting dynamics can roughly approximate the shape of the observed m t , they fail to match its detailed structure.

Original: Moving from this local optimum to the global optimum is very hard for the network as the distance in x space is large and the gradients become very noisy due to the highly non-linear physics.
Revised: Moving from this local optimum to the global optimum is very hard for the network as the distance in x space is large and the gradients become very noisy due to the highly non-linear physics.

Original: A similar behavior can also be seen when optimizing single inverse problems with GD (Fig. 6b).
Revised: A similar behavior can also be seen when optimizing single inverse problems with GD (Fig. 6b).

Original: Here the strongly varying magnitude of the gradients poses an additional challenge for the ﬁrst-order optimization, causing its progress to stall.
Revised: Here the strongly varying magnitude of the gradients poses an additional challenge for the ﬁrst-order optimization, causing its progress to stall.

Original: With η = 1 , it takes more than 20K iterations for GD to converge for single problems.
Revised: With η = 1 , it takes more than 20K iterations for GD to converge for single problems.

Original: For training with PGs (Eq. 5), we exploit our knowledge about the ﬂow – e.g. smoothness – to formulate an inverse simulator P − 1 that incorporates corresponding priors on the solution space.
Revised: For training with PGs (Eq. 5), we exploit our knowledge about the ﬂow – e.g. smoothness – to formulate an inverse simulator P − 1 that incorporates corresponding priors on the solution space.

Original: It runs the simulation in reverse, starting with m t , then estimates x by comparing the marker densities from the forward and reverse pass (see appendix B.5 for details).
Revised: It runs the simulation in reverse, starting with m t , then estimates x by comparing the marker densities from the forward and reverse pass (see appendix B.5 for details).

Original: This method converges for single inverse problems within around 100 iterations and its computational costs are on the same level as evaluating the gradient.
Revised: This method converges for single inverse problems within around 100 iterations and its computational costs are on the same level as evaluating the gradient.

Original: When used for network training, we observe vastly improved convergence behavior compared to conventional ﬁrst-order training.
Revised: When used for network training, we observe vastly improved convergence behavior compared to conventional ﬁrst-order training.

Original: The error rapidly decreases during the ﬁrst 200 iterations, at which point the inferred solutions are more accurate than pure Adam training by a factor of 2.3.
Revised: The error rapidly decreases during the ﬁrst 200 iterations, at which point the inferred solutions are more accurate than pure Adam training by a factor of 2.3.

Original: The error then continues to improve at a slower but still exponentially faster rate,  reaching a relative accuracy advantage of 5x after 20K iterations.
Revised: The error then continues to improve at a slower but still exponentially faster rate, [Figure] [Table] reaching a relative accuracy advantage of 5x after 20K iterations.

Original: To match the network trained with pure Adam for 20K iterations, the PG variant only requires 55 iterations.
Revised: To match the network trained with pure Adam for 20K iterations [Table] the PG variant only requires 55 iterations.

Original: This improvement is possible because the inverse-physics solver and associated PG does not suffer from the strongly-varying gradient strengths and directions, which drown the ﬁrst-order signal in noise.
Revised: This improvement is possible because the inverse-physics solver and associated PG does not suffer from the strongly-varying gradient strengths and directions, which drown the ﬁrst-order signal in noise.

Original: Instead, physical gradients behave much more smoothly, both in magnitude and direction.
Revised: Instead, physical gradients behave much more smoothly, both in magnitude and direction.

Original: To reach the same solution quality as the  network prediction, our domain knowledge solver needs more than 10,000 times as long because it needs to run the  forward and backward simulation a couple of times.
Revised: To reach the same solution quality as the neural network prediction [Table] the domain knowledge solver needs more than 10,000 times as long. This difference is caused by solver having to run the full forward and backward simulation several times.

Original: Table 1 lists measured training and inference times.
Revised: Table 1 lists measured training and inference times

Original:  With both iterative solver and network, we used a batch size of 64 and divided the total time by the batch size.
Revised: [Table] With both iterative solver and network [Table] we used a batch size of 64 and divided the total time by the batch size.

Original: More examples from the ﬂuid data set including time evolution sequences are shown in appendix B.5.
Revised: More examples from the ﬂuid data set including time evolution sequences are shown in appendix B.5.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have shown how to integrate arbitrary optimization schemes for inverse problems into machine learning settings.
Revised: We have shown how to integrate arbitrary optimization schemes for inverse problems into machine learning settings.

Original: Our results show that ﬁrst-order optimization is suboptimal in many situations, especially when the magnitude or direction of gradients can vary during the optimization or from example to example.
Revised: Our results show that ﬁrst-order optimization is suboptimal in many situations, especially when the magnitude or direction of gradients can vary during the optimization or from example to example.

Original: While more advanced optimizers such as Adam alleviate this behavior, they fail to determine good optimization directions for many problems.
Revised: While more advanced optimizers such as Adam alleviate this behavior, they fail to determine good optimization directions for many problems.

Original: By embedding higher-order optimizers – and especially inverse-physics solvers – into the ﬁrst-order training pipeline, we were able to mitigate these shortcomings without discarding the progress made in the ﬁeld of machine learning.
Revised: By embedding higher-order optimizers – and especially inverse-physics solvers – into the ﬁrst-order training pipeline, we were able to mitigate these shortcomings without discarding the progress made in the ﬁeld of machine learning.

Original: The resulting training scheme converges exponentially faster than traditional ﬁrst-order training on a wide variety of inverse problems involving partial differential equations while being on par with regular gradient evaluation in terms of computational cost.
Revised: The resulting training scheme converges exponentially faster than traditional ﬁrst-order training on a wide variety of inverse problems involving partial differential equations while being on par with regular gradient evaluation in terms of computational cost.

Original: We anticipate our method to be applicable to a wide range of differentiable processes.
Revised: We anticipate our method to be applicable to a wide range of differentiable processes.

Original: Beyond physics, interesting avenues lie in applying PGs to differentiable rendering or training invertible neural networks.
Revised: Beyond physics, interesting avenues lie in applying PGs to differentiable rendering or training invertible neural networks.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We will publish the full source code and trained network models upon acceptance.
Revised: We will publish the full source code and trained network models upon acceptance.

Original: They will be sufﬁcient to retrain the networks and plot the ﬁgures shown in this work.
Revised: They will be sufﬁcient to retrain the networks and plot the ﬁgures shown in this work.

Original: All data sets were generated randomly, so slight variations are expected.
Revised: All data sets were generated randomly, so slight variations are expected.

Original: Repeated experiments are shown in the appendix.
Revised: Repeated experiments are shown in the appendix.

Original: E MBEDDING THE PHYSICAL GRADIENT INTO THE GRADIENT DESCENT PIPELINE
Revised: E MBEDDING THE PHYSICAL GRADIENT INTO THE GRADIENT DESCENT PIPELINE

Original: The key for embedding inverse-physics optimizers in the machine learning pipeline is the proxy Lloss in Eq.
Revised: The key for embedding inverse-physics optimizers in the machine learning pipeline is the proxy Lloss in Eq.

Original: 5 which passes the gradients from the physics optimizer to the GD-based neural network optimizer.
Revised: 5 which passes the gradients from the physics optimizer to the GD-based neural network optimizer.

Original: This l 2 loss may look like a supervised loss in x space but due to the dependency of the labels P −on the prediction, it behaves very differently.
Revised: This l 2 loss may look like a supervised loss in x space but due to the dependency of the labels P −on the prediction, it behaves very differently.

Original: To demonstrate this, we consider the case that GD is being used as the physics optimizer.
Revised: To demonstrate this, we consider the case that GD is being used as the physics optimizer.

Original: Then the total loss is purely deﬁned in y space, reducing to a regular ﬁrst-order optimization.
Revised: Then the total loss is purely deﬁned in y space, reducing to a regular ﬁrst-order optimization.

Original: The proxy L 2 loss simply connects the computational graphs used in backpropagation.
Revised: The proxy L 2 loss simply connects the computational graphs used in backpropagation.

Original: Physical gradients based on gradient descent are equal to unsupervised training With the shorthands x i =
Revised: Physical gradients based on gradient descent are equal to unsupervised training With the shorthands x i =

Original: NN( y ∗ i | θ ) , y i
Revised: NN( y ∗ i | θ ) , y i

Original: = P ( x i ) and ∆ y i
Revised: = P ( x i ) and ∆ y i

Original: =
Revised: =

Original: y i − y ∗
Revised: y i − y ∗

Original: i , we can write the objective functions of unsupervised training and physical gradient training as
Revised: i , we can write the objective functions of unsupervised training and physical gradient training as

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: Minimizing U ( θ ) is identical to minimizing M ( θ ) using any gradient-descent-based optimizer if P − 1 is implemented as a single gradient descent step with η = 1 .
Revised: Minimizing U ( θ ) is identical to minimizing M ( θ ) using any gradient-descent-based optimizer if P − 1 is implemented as a single gradient descent step with η = 1 .

Original: Proof.
Revised: Proof.

Original: The gradient of the unsupervised objective function is
Revised: The gradient of the unsupervised objective function is

Original: [Equation]
Revised: [Equation]

Original: For the mixed training, we recall that the objective function of a single inverse problem is L ( x )
Revised: For the mixed training, we recall that the objective function of a single inverse problem is L ( x )

Original: =
Revised: =

Original: 12 ||P ( x ) − y ∗
Revised: 12 ||P ( x ) − y ∗

Original: || 22 .
Revised: || 22 .

Original: The corresponding gradient is ∂L∂x = ∆ y · ∂y∂x .
Revised: The corresponding gradient is ∂L∂x = ∆ y · ∂y∂x .

Original: Inserting a gradient descent step for
Revised: Inserting a gradient descent step for

Original: P − 1 , the gradient of the mixed objective function becomes
Revised: P − 1 , the gradient of the mixed objective function becomes

Original: [Equation]
Revised: [Equation]

Original: which is equal to ∂U∂θ for η = 1 .
Revised: which is equal to ∂U∂θ for η = 1 .

Original: This generic L 2 formulation is even applicable in settings where different parts of an end-to-end pipeline are computed by different software frameworks or hardware accelerators.
Revised: This generic L 2 formulation is even applicable in settings where different parts of an end-to-end pipeline are computed by different software frameworks or hardware accelerators.

Original: A. C ONVERGENCE WITH PHYSICAL GRADIENTS
Revised: A. C ONVERGENCE WITH PHYSICAL GRADIENTS

Original: We consider the convergence of a coupled optimization of L i ( x i ) with x i =
Revised: We consider the convergence of a coupled optimization of L i ( x i ) with x i =

Original: f θ ( y i ) involving gradient descent optimization of a general function approximator f θ and some other optimization scheme
Revised: f θ ( y i ) involving gradient descent optimization of a general function approximator f θ and some other optimization scheme

Original: P for minimizing objective functions L i ( x i ) .
Revised: P for minimizing objective functions L i ( x i ) .

Original: Here, i denotes the index of a speciﬁc example y from a data set of size N .
Revised: Here, i denotes the index of a speciﬁc example y from a data set of size N .

Original: In the context of learning to solve inverse problems, P is the inverse-problem solver on which physical gradient P ( x ) − x is based, and f θ represents the neural network.
Revised: In the context of learning to solve inverse problems, P is the inverse-problem solver on which physical gradient P ( x ) − x is based, and f θ represents the neural network.

Original: In the main text, we considered objectives of the form L i =
Revised: In the main text, we considered objectives of the form L i =

Original: 12 ||P ( x i ) − y ∗
Revised: 12 ||P ( x i ) − y ∗

Original: i || 22 , but now we consider a more general form.
Revised: i || 22 , but now we consider a more general form.

Original: [Figure]
Revised: [Figure]

Original: Assumptions
Revised: Assumptions

Original: Let L =
Revised: Let L =

Original: { L i : R d → R | i = 1 , ..., N } be a ﬁnite set of functions and let x ∗
Revised: { L i : R d → R | i = 1 , ..., N } be a ﬁnite set of functions and let x ∗

Original: i ∈ R
Revised: i ∈ R

Original: d be any global minima of L i . 2.
Revised: d be any global minima of L i . 2.

Original: Let P
Revised: Let P

Original: =
Revised: =

Original: { P i : R d → R d | i = 1 , ..., N } be a set of update functions for L i such that ∃ τ > 0 : ∀ x ∈ R
Revised: { P i : R d → R d | i = 1 , ..., N } be a set of update functions for L i such that ∃ τ > 0 : ∀ x ∈ R

Original: d
Revised: d

Original: : L i ( x ) − L i ( P i ( x )) ≥ τ ( L ( x ) − L ( x ∗ )) and ∃ K > 0 : ∀ x ∈ R d : || P i ( x ) − x || ≤ K ( L ( x ) − L ( x ∗ )) .
Revised: : L i ( x ) − L i ( P i ( x )) ≥ τ ( L ( x ) − L ( x ∗ )) and ∃ K > 0 : ∀ x ∈ R d : || P i ( x ) − x || ≤ K ( L ( x ) − L ( x ∗ )) .

Original: Let f θ : R m → R d be differentiable w.r.t.
Revised: Let f θ : R m → R d be differentiable w.r.t.

Original: θ with the property that ∃ η > 0 : ∀ i ∈ 1 , ..., N ∀ x i ∈ R d ∀ (cid:15) > 0 ∃ n ∈ N : || f
Revised: θ with the property that ∃ η > 0 : ∀ i ∈ 1 , ..., N ∀ x i ∈ R d ∀ (cid:15) > 0 ∃ n ∈ N : || f

Original: θ xin ( y i ) − x i || 2 ≤ (cid:15) where θ xn is the sequence of gradient descent steps with θ xn
Revised: θ xin ( y i ) − x i || 2 ≤ (cid:15) where θ xn is the sequence of gradient descent steps with θ xn

Original: +

Revised: +

Original: = θ xn − η (cid:16)
Revised: = θ xn − η (cid:16)

Original: ∂f θ ∂θ (cid:17) T ( f θ n − x ) , η > 0 .
Revised: ∂f θ ∂θ (cid:17) T ( f θ n − x ) , η > 0 .

Original: Here we make minimal assumptions about L ; not even continuity is required as long as we have access to some form of a physical gradient that can optimize it in x .
Revised: Here we make minimal assumptions about L ; not even continuity is required as long as we have access to some form of a physical gradient that can optimize it in x .

Original: We denote k repeated applications of P by P k ( x ) .
Revised: We denote k repeated applications of P by P k ( x ) .

Original: The assumption in (3) states that the function f θ is ﬂexible enough to ﬁt our problem.
Revised: The assumption in (3) states that the function f θ is ﬂexible enough to ﬁt our problem.

Original: It must be able to converge to every point x for all examples.
Revised: It must be able to converge to every point x for all examples.

Original: This property is fulﬁlled even with shallow neural networks, given that sufﬁciently many parameters are present Cybenko (1989).
Revised: This property is fulﬁlled even with shallow neural networks, given that sufﬁciently many parameters are present Cybenko (1989).

Original: We now show that P can be used in combination with gradient descent to optimize the coupled problem.
Revised: We now show that P can be used in combination with gradient descent to optimize the coupled problem.

Original: Theorem 2.
Revised: Theorem 2.

Original: For all k ∈ N , there exists an update strategy θ n +

Revised: For all k ∈ N , there exists an update strategy θ n +

Original: =
Revised: =

Original: U k
Revised: U k

Original: ( θ n ) based on a single evaluation of P k for which L ( f θ n ( y i )) converges to a minimum x ∗ i or minimum region of L i ∀ i
Revised: ( θ n ) based on a single evaluation of P k for which L ( f θ n ( y i )) converges to a minimum x ∗ i or minimum region of L i ∀ i

Original: = 1 , ..., N .
Revised: = 1 , ..., N .

Original: Proof.
Revised: Proof.

Original: Dropping the example index i , we denote ˜ x n ≡ P k ( x n ) and ∆ L ∗ =
Revised: Dropping the example index i , we denote ˜ x n ≡ P k ( x n ) and ∆ L ∗ =

Original: L ( x n ) − L ( x ∗ ) .
Revised: L ( x n ) − L ( x ∗ ) .

Original: Let Idenote the open set of all x for which L ( x ) − L ( x n ) > τ 2 ∆ L ∗ Deﬁnition (2) provides that ˜ x n ∈ I 2 .
Revised: Let Idenote the open set of all x for which L ( x ) − L ( x n ) > τ 2 ∆ L ∗ Deﬁnition (2) provides that ˜ x n ∈ I 2 .

Original: Since I 2 is open, ∃ (cid:15) > 0 : ∀ x ∈ B (cid:15) (˜ x n ) :
Revised: Since I 2 is open, ∃ (cid:15) > 0 : ∀ x ∈ B (cid:15) (˜ x n ) :

Original: L ( x n ) − L ( x ) > τ 2 ∆ L ∗ where B (cid:15) ( x ) denotes the open set containing all x (cid:48) ∈ R d for which || x (cid:48) − x
Revised: L ( x n ) − L ( x ) > τ 2 ∆ L ∗ where B (cid:15) ( x ) denotes the open set containing all x (cid:48) ∈ R d for which || x (cid:48) − x

Original: || 2 < (cid:15) , i.e. there exists a small ball around ˜ x n which is fully contained in I 2 (see sketch in Fig. 7).
Revised: || 2 < (cid:15) , i.e. there exists a small ball around ˜ x n which is fully contained in I 2 (see sketch in Fig. 7).

Original: Using the convergence assumption for f θ , deﬁnition (3), we can ﬁnd a ﬁnite n ∈ N for which f θ n ∈ B (cid:15) (˜ x n ) and therefore L ( x n ) − L ( f θ n ) > τ 2 ∆ L ∗ .
Revised: Using the convergence assumption for f θ , deﬁnition (3), we can ﬁnd a ﬁnite n ∈ N for which f θ n ∈ B (cid:15) (˜ x n ) and therefore L ( x n ) − L ( f θ n ) > τ 2 ∆ L ∗ .

Original: We can thus use the following strategy
Revised: We can thus use the following strategy

Original: U k for minimizing L ( f θ ) : First, compute ˜ x n =
Revised: U k for minimizing L ( f θ ) : First, compute ˜ x n =

Original: P k ( x n ) .
Revised: P k ( x n ) .

Original: Then perform gradient descent steps in θ with the effective objective function 12 || f θ − ˜ x n || 22 until L ( x n ) − L ( f θ ) ≥ τ 2 ∆ L ∗ .
Revised: Then perform gradient descent steps in θ with the effective objective function 12 || f θ − ˜ x n || 22 until L ( x n ) − L ( f θ ) ≥ τ 2 ∆ L ∗ .

Original: With this strategy, each application of U k reduces the loss to ∆ L ∗ n +1 ≤ (1 − τ 2 )∆ L ∗ n so any arbitrary value of L > L ( x ∗ ) can be reached within a ﬁnite number of steps.
Revised: With this strategy, each application of U k reduces the loss to ∆ L ∗ n +1 ≤ (1 − τ 2 )∆ L ∗ n so any arbitrary value of L > L ( x ∗ ) can be reached within a ﬁnite number of steps.

Original: Deﬁnition (2) also ensures that || P ( x ) − x || → 0 as the optimization progresses which guarantees that the optimization converges to a minimum region.
Revised: Deﬁnition (2) also ensures that || P ( x ) − x || → 0 as the optimization progresses which guarantees that the optimization converges to a minimum region.

Original: While this guarantees convergence, it requires potentially many gradient descent steps in θ for each physical gradient evaluation.
Revised: While this guarantees convergence, it requires potentially many gradient descent steps in θ for each physical gradient evaluation.

Original: This can be advantageous in special circumstances, e.g. when the physical gradient is more expensive to compute than an update in θ , and θ is far away from a solution.
Revised: This can be advantageous in special circumstances, e.g. when the physical gradient is more expensive to compute than an update in θ , and θ is far away from a solution.

Original: However, in many cases, we want to update the physical gradient after each update to θ .
Revised: However, in many cases, we want to update the physical gradient after each update to θ .

Original: Without additional assumptions about P and f θ , there is no guarantee in this case that L will decrease every iteration, even for inﬁnitesimally small step sizes.
Revised: Without additional assumptions about P and f θ , there is no guarantee in this case that L will decrease every iteration, even for inﬁnitesimally small step sizes.

Original: Despite this, there is good reason to assume that the optimization decreases L over time.
Revised: Despite this, there is good reason to assume that the optimization decreases L over time.

Original: When performing a gradient descent step in θ for the objective 12 || f θ − ˜ x n || 22 , the next value of f θ must lie closer to ˜ x n than x n .
Revised: When performing a gradient descent step in θ for the objective 12 || f θ − ˜ x n || 22 , the next value of f θ must lie closer to ˜ x n than x n .

Original: Assuming that, on average, the gradient descent steps do not prefer a certain direction, x n +1 is more likely to lie in I = { x ∈ R d : L ( x ) <
Revised: Assuming that, on average, the gradient descent steps do not prefer a certain direction, x n +1 is more likely to lie in I = { x ∈ R d : L ( x ) <

Original: L ( x n ) } than outside it.
Revised: L ( x n ) } than outside it.

Original: The region of increasing loss is shaded orange in Fig.
Revised: The region of increasing loss is shaded orange in Fig.

Original: Since this region always ﬁlls less than half of the sphere around ˜ x n , L is more likely to decrease than increase.
Revised: Since this region always ﬁlls less than half of the sphere around ˜ x n , L is more likely to decrease than increase.

Original: While this shows that the loss should decrease on average, it does not guarantee convergence.
Revised: While this shows that the loss should decrease on average, it does not guarantee convergence.

Original: We now look at two speciﬁc formulations of P for which convergence to the correct solution is guaranteed.
Revised: We now look at two speciﬁc formulations of P for which convergence to the correct solution is guaranteed.

Original: The ﬁrst involves the physical gradient pointing directly towards a solution x ∗ .
Revised: The ﬁrst involves the physical gradient pointing directly towards a solution x ∗ .

Original: This is, for example, the case for unimodal problems when we choose k large enough.
Revised: This is, for example, the case for unimodal problems when we choose k large enough.

Original: In the second case, we consider the physical gradient being aligned with the gradient descent vector in x space.
Revised: In the second case, we consider the physical gradient being aligned with the gradient descent vector in x space.

Original: Theorem 3.
Revised: Theorem 3.

Original: If ∀ x ∈ R d ∃ λ ∈ (0 , 1] : P ( x ) = x + λ ( x ∗ − x ) , then the sequence f θ n with θ n +1 = θ n − η (cid:16) ∂f∂θ (cid:17) T ( x − P ( x )) converges to x ∗ .
Revised: If ∀ x ∈ R d ∃ λ ∈ (0 , 1] : P ( x ) = x + λ ( x ∗ − x ) , then the sequence f θ n with θ n +1 = θ n − η (cid:16) ∂f∂θ (cid:17) T ( x − P ( x )) converges to x ∗ .

Original: Proof.
Revised: Proof.

Original: Rewriting x − P ( x ) =
Revised: Rewriting x − P ( x ) =

Original: ∂∂x (cid:0) λ
Revised: ∂∂x (cid:0) λ

Original: 2 || x − x ∗
Revised: 2 || x − x ∗

Original: || 22 (cid:1) yields the update θ n +1 − θ n = − η (cid:16) ∂
Revised: || 22 (cid:1) yields the update θ n +1 − θ n = − η (cid:16) ∂

Original: ˜
Revised: ˜

Original: L∂x ∂f∂θ (cid:17) T where ˜ L =
Revised: L∂x ∂f∂θ (cid:17) T where ˜ L =

Original: λ 2 || x − x ∗
Revised: λ 2 || x − x ∗

Original: || 22 .
Revised: || 22 .

Original: This describes pure gradient descent towards x ∗ with the gradients scaled by λ .
Revised: This describes pure gradient descent towards x ∗ with the gradients scaled by λ .

Original: Since λ ∈ (0 , 1] , the convergence proof of gradient descent applies.
Revised: Since λ ∈ (0 , 1] , the convergence proof of gradient descent applies.

Original: Theorem 4.
Revised: Theorem 4.

Original: If ∀ x ∈ R d ∃ λ ∈ (0 , 1] : P ( x ) = x − λ (cid:0) ∂L∂x (cid:1) T , then the sequence f θ n with θ n +1 = θ n − η (cid:16) ∂f∂θ (cid:17) T ( x − P ( x )) converges to minima of L .
Revised: If ∀ x ∈ R d ∃ λ ∈ (0 , 1] : P ( x ) = x − λ (cid:0) ∂L∂x (cid:1) T , then the sequence f θ n with θ n +1 = θ n − η (cid:16) ∂f ∂θ (cid:17) T ( x − P ( x )) converges to minima of L .

Original: Proof.
Revised: Proof.

Original: This is equivalent to gradient descent in L ( θ ) .
Revised: This is equivalent to gradient descent in L ( θ ) .

Original: Rewriting the update yields θ n +1 − θ n = − η (cid:16) ∂f∂θ (cid:17) T (cid:16) x − (cid:16) x − (cid:0) λ ∂L∂x (cid:1) T (cid:17)(cid:17)
Revised: Rewriting the update yields θ n +1 − θ n = − η (cid:16) ∂f∂θ (cid:17) T (cid:16) x − (cid:16) x − (cid:0) λ ∂L∂x (cid:1) T (cid:17)(cid:17)

Original: = − ηλ (cid:16) ∂L∂x ∂f∂θ (cid:17) T which is the gradient descent update scaled by λ .
Revised: = − ηλ (cid:16) ∂L∂x ∂f∂θ (cid:17) T which is the gradient descent update scaled by λ .

Original: B E XPERIMENTS
Revised: B E XPERIMENTS

Original: Here, we give a more detailed description of our experiments including setup and analysis.
Revised: Here, we give a more detailed description of our experiments including setup and analysis.

Original: The implementation of our experiments is based on the Φ Flow (PhiFlow) framework Holl et al.
Revised: The implementation of our experiments is based on the Φ Flow (PhiFlow) framework Holl et al.

Original: and uses TensorFlow Abadi et al.
Revised: and uses TensorFlow Abadi et al.

Original: and PyTorch Paszke et al.
Revised: and PyTorch Paszke et al.

Original: (2017) for automatic differentiation.
Revised: (2017) for automatic differentiation.

Original: Our code is open source and will be made available upon acceptance.
Revised: Our code is open source and will be made available upon acceptance.

Original: {+   ,WHUDWLRQV   /RVV *XHVVLQJ /%)*6% 1HWZRUN    ,WHUDWLRQV     
+}
Revised: {+   ,WHUDWLRQV   /RVV *XHVVLQJ /%)*6% 1HWZRUN    ,WHUDWLRQV     
+}

Revised: 0$( *XHVVLQJ /%)*6% 1HWZRUN    ,WHUDWLRQV    

Revised: A 0$( *XHVVLQJ /%)*6% 1HWZRUN

Original: B. W AVE PACKET FIT
Revised: B. W AVE PACKET FIT

Original: This experiment is  a generic curve ﬁt example.
Revised: This experiment is an instance of a generic curve ﬁtting problem.

Original: The task is to ﬁnd the parameters that result in least mean squared error between two curves.
Revised: The task is to ﬁnd the parameters that result in least mean squared error between two curves.

Original: [-   ,WHUDWLRQV   /RVV *XHVVLQJ /%)*6% 1HWZRUN    ,WHUDWLRQV     
 0$( *XHVVLQJ /%)*6% 1HWZRUN    ,WHUDWLRQV     A 0$( *XHVVLQJ /%)*6% 1HWZRUN-]
Revised: [-   ,WHUDWLRQV   /RVV *XHVVLQJ /%)*6% 1HWZRUN    ,WHUDWLRQV     
 0$( *XHVVLQJ /%)*6% 1HWZRUN    ,WHUDWLRQV     A 0$( *XHVVLQJ /%)*6% 1HWZRUN-]We use this problem to illustrate the advantages of using neural networks as inverse solvers over classical optimization algorithms.

Original: Data generation.
Revised: Data generation.

Original: We simulate an observed time series y ∗ from a random ground truth position ξand amplitude  A  .
Revised: We simulate an observed time series y ∗ from a random ground truth position and amplitude x ∗ = { ξ 0 , A } .

Original: Each time series contains 256 entries and is made up of the wave packet and noise.
Revised: Each time series contains 256 entries and is made up of the wave packet and noise.

Original: For the wave packet, we sample ξ 0 ∈ [25 . 6 , 230 . 4) and A ∈ [1 , 4) from uniform distributions.
Revised: For the wave packet, we sample ξ 0 ∈ [25 . 6 , 230 . 4) and A ∈ [1 , 4) from uniform distributions.

Original: The wave packet has the functional form
Revised: The wave packet has the functional form

Original: [Equation]
Revised: [Equation]

Original: where we set f = 45 and σ = 20 constant for all data.
Revised: where we set f = 45 and σ = 20 constant for all data.

Original: For the noise, we superimpose random values sampled from the normal distribution N (0 , 25 ) at each sample.
Revised: For the noise, we superimpose random values sampled from the normal distribution N (0 , 25 ) at each sample.

Original: Network architecture.
Revised: Network architecture.

Original: We construct the neural network from convolutional blocks, followed by fully-connected layers, all using the ReLU activation function.
Revised: We construct the neural network from convolutional blocks, followed by fully-connected layers, all using the ReLU activation function.

Original: The input is ﬁrst processed by ﬁve blocks, each containing a max pooling operation and two 1D convolutions with kernel size 3.
Revised: The input is ﬁrst processed by ﬁve blocks, each containing a max pooling operation and two 1D convolutions with kernel size 3.

Original: Each convolution outputs 16 feature maps.
Revised: Each convolution outputs 16 feature maps.

Original: The downsampled result is then passed to two fully connected layers with 64 and 32 and 2 neurons, respectively, before a third fully-connected layer produces the predicted ξ 0 and A .
Revised: The downsampled result is then passed to two fully connected layers with 64 and 32 and 2 neurons, respectively, before a third fully-connected layer produces the predicted ξ 0 and A .

Original: ξ 0 is passed through a Sigmoid activation function and normalized to values between 25.6 and 230.4.
Revised: ξ 0 is passed through a Sigmoid activation function and normalized to values between 25.6 and 230.4.

Original: Training and ﬁtting.
Revised: Training and ﬁtting.

Original: We ﬁt the data using L-BFGS-B with an initial guess of ξ 0 = 128 and A =and the network output is offset by the same amount.
Revised: We ﬁt the data using L-BFGS-B with an initial guess of ξ 0 = 128 and A =and the network output is offset by the same amount.

Original: Both network and L-BFGS-B minimize the squared loss ||
Revised: Both network and L-BFGS-B minimize the squared loss ||

Original: y ( ξ 0 , A ) − y ∗ || 22 and the resulting performance curves are shown in Fig.
Revised: y ( ξ 0 , A ) − y ∗ || 2 2 and the resulting performance curves are shown in Fig.

Original: We observe that L-BFGS-B manages to ﬁt the wave packet when it is reasonably close to the center where the initial guess predicts it.
Revised: We observe that L-BFGS-B manages to ﬁt the wave packet when it is reasonably close to the center where the initial guess predicts it.

Original: When the wave packet is located to either side, L-BFGS-B does not ﬁnd it and instead ﬁts the noise near the center.
Revised: When the wave packet is located to either side, L-BFGS-B does not ﬁnd it and instead ﬁts the noise near the center.

Original: The neural network is trained using Adam with learning rate 0.001 and batch size of 100.
Revised: The neural network is trained using Adam with learning rate 0.001 and batch size of 100.

Original: Despite the simpler, Hessian-free, optimization updates, the network quickly learns to localize all wave packets, outperforming L-BFGS-B after ... training iterations.
Revised: Despite the simpler, Hessian-free, optimization updates, the network quickly learns to localize all wave packets, outperforming L-BFGS-B after 30 to 40 training iterations.

Original: This improvement stems from the network’s reparameterization of the problem, allowing for joint training using all data.
Revised: This improvement is possible because of the network’s reparameterization of the problem, allowing for joint parameter optimization using all data.

Revised: When the prediction for one example is close to a local optimum, updates from different examples can prevent it from converging to that sub-optimal solution.

Original: B. S INGLE - PARAMETER OPTIMIZATION
Revised: B. S INGLE - PARAMETER OPTIMIZATION

Original: The optimization consists of ﬁnding x ∗ = argmin x sin( ax ) .
Revised: The optimization consists of ﬁnding x ∗ = argmin x sin( ax ) .

Original: Fig.
Revised: Fig.

Original: 9 shows the gradients for the case a = 1 .
Revised: 9 shows the gradients for the case a = 1 .

Original: The PG as well as the supervised methods require ﬁnding a solution x ∗ closest to an initial guess x 0 .
Revised: The PG as well as the supervised methods require ﬁnding a solution x ∗ closest to an initial guess x 0 .

Original: We solve this analytically by ﬁrst determining the index of the closest minimum, n = a 2 π · x 0 − 34 which we round to the closest integer.
Revised: We solve this analytically by ﬁrst determining the index of the closest minimum, n = [Equation] − 34 which we round to the closest integer.

Original: The furthest-out minima ( n = 0 and n = − 1 ) require special handling.
Revised: The furthest-out minima ( n = 0 and n = − 1 ) require special handling.

Original: Then the position of the minimum corresponding to n is x∗ =
Revised: Then the position of the minimum corresponding to n is x

Original: [Equation]
Revised: [Equation]

Original: n . Newton’s method The analytic form of a Newton update is
Revised: Newton’s method The analytic form of a Newton update is

Original: [Equation]
Revised: [Equation]

Original: which approaches both minima and maxima.
Revised: which approaches both minima and maxima.

Original: We alter this update to walk only towards minimum points by ﬂipping the optimization direction so that the update points in the same direction as ∂ P ∂x .
Revised: We alter this update to walk only towards minimum points by ﬂipping the optimization direction so that the update points in the same direction as ∂ P ∂x .

Original: Since Newton’s method relies on an inversion of the Hessian, the produced physical gradients diverge where ∂ 2 P ∂x 2 = 0 .
Revised: Since Newton’s method relies on an inversion of the Hessian, the produced physical gradients diverge where ∂ 2 P ∂x 2 = 0 .

Original: To avoid overﬂow, we clip the gradients ∂ P ∂x to [ − 1 , 1] before they are backpropagated through the neural network.
Revised: To avoid overﬂow, we clip the gradients ∂ P ∂x to [ − 1 , 1] before they are backpropagated through the neural network.

Original: Inverse gradients
Revised: Inverse gradients

Original: In addition to gradient inversion via the Hessian, we can invert the gradients directly.
Revised: In addition to gradient inversion via the Hessian, we can invert the gradients directly.

Original: The principle of inverting the Jacobian is not new.
Revised: The principle of inverting the Jacobian is not new.

Original: Inverse kinematics applications such as controlling robots also employ ﬁrst-order inversion Craig (2005).
Revised: Inverse kinematics applications such as controlling robots also employ ﬁrst-order inversion Craig (2005).

Original: This results in a similar behavior concerning dimensions and function sensitivity as Newton-type methods.
Revised: This results in a similar behavior concerning dimensions and function sensitivity as Newton-type methods.

Original: However, this ﬁrst-order inversion cannot directly be applied to general optimization tasks because the Jacobian of a scalar objective function is a row vector.
Revised: However, this ﬁrst-order inversion cannot directly be applied to general optimization tasks because the Jacobian of a scalar objective function is a row vector.

Original: For this speciﬁc inverse problem, however, the gradient can be inverted because only a single parameter is optimized.
Revised: For this speciﬁc inverse problem, however, the gradient can be inverted because only a single parameter is optimized.

Original: The physical gradient is computed as
Revised: The physical gradient is computed as

Original: [Equation]
Revised: [Equation]

Original: where h = sin (cid:0) ax (cid:1) + 1 measures the height above y ∗ .
Revised: where h = sin (cid:0) ax (cid:1) + 1 measures the height above y ∗ .

Original: It approaches ±∞ when sin( ax ) is maximal.
Revised: It approaches ±∞ when sin( ax ) is maximal.

Original: This inversion requires domain knowledge since h is based on the fact that min P = − 1 .
Revised: This inversion requires domain knowledge since h is based on the fact that min P = − 1 .

Original: Like with
Revised: Like with

Original: Newton’s method, we apply gradient clipping to bound diverging inverse gradients.
Revised: Newton’s method, we apply gradient clipping to bound diverging inverse gradients.

Original: Neural network training We set up a multilayer perceptron with a single input and output as well as three hidden layers containing 10, 20 and 10 neurons, respectively.
Revised: Neural network training We set up a multilayer perceptron with a single input and output as well as three hidden layers containing 10, 20 and 10 neurons, respectively.

Original: This adds up to 461 total trainable parameters.
Revised: This adds up to 461 total trainable parameters.

Original: A bias is applied at each layer and the ReLU activation function is applied after each hidden layer.
Revised: A bias is applied at each layer and the ReLU activation function is applied after each hidden layer.

Original: The network is trained using Adam with a learning rate of 0.002 and minibatches containing 1000 uniformly sampled a ∈ [0 . 1 , 10 . 1] .
Revised: The network is trained using Adam with a learning rate of 0.002 and minibatches containing 1000 uniformly sampled a ∈ [0 . 1 , 10 . 1] .

Original: We use Φ Flow together with PyTorch to train the neural network in this experiment.
Revised: We use Φ Flow together with PyTorch to train the neural network in this experiment.

Original: Learning curves for different network initializations are shown in Fig.
Revised: Learning curves for different network initializations are shown in Fig.

Original: 10 and the recorded computation times are shown in Fig.
Revised: 10 and the recorded computation times are shown in Fig.

Original: While supervised training shows very consistent behavior across training runs, the convergence curves of Adam show more erratic behavior.
Revised: While supervised training shows very consistent behavior across training runs, the convergence curves of Adam show more erratic behavior.

Original: Training with PGs based on the analytic solver converges in all cases, but does not always reach the same level of accuracy.
Revised: Training with PGs based on the analytic solver converges in all cases, but does not always reach the same level of accuracy.

Original: While yielding superior accuracy than pure Adam for the large majority of tested initializations, it sometimes stops converging to an accuracy larger than 10 − 4 .
Revised: While yielding superior accuracy than pure Adam for the large majority of tested initializations, it sometimes stops converging to an accuracy larger than 10 − 4 .

Original: This is most likely caused by numerical effects of the inverse solver.
Revised: This is most likely caused by numerical effects of the inverse solver.

Original: Training with physical gradients based on Newton’s method or inverse gradients yields a convergence that strongly depends on the network initialization.
Revised: Training with physical gradients based on Newton’s method or inverse gradients yields a convergence that strongly depends on the network initialization.

Original: This variant converges only for certain initializations.
Revised: This variant converges only for certain initializations.

Original: However, upon convergence, it typically reaches an accuracy similar to the inverse physics variant.
Revised: However, upon convergence, it typically reaches an accuracy similar to the inverse physics variant.

Original: B. P OISSON ’ S EQUATION
Revised: B. P OISSON ’ S EQUATION

Original: We consider Poisson’s equation, ∇ 2 y = x where x is the initial state and y is the output of the simulator.
Revised: We consider Poisson’s equation, ∇ 2 y = x where x is the initial state and y is the output of the simulator.

Original: We set up a two-dimensional simulation with 80 by 60 cubic cells.
Revised: We set up a two-dimensional simulation with 80 by 60 cubic cells.

Original: Our simulator computes y = P ( x ) = ∇ − 2 x implicitly via the conjugate gradient method.

Original: The inverse problem consists of ﬁnding an initial value x ∗ for a given target y ∗ such that ∇ 2 y ∗ = x ∗ .

Original: We formulate this problem as minimizing L ( x ) = [Equation] | [Equation]
Revised: Our simulator computes

Original: P

Original: [Equation]

Original: x )

Original: [Equation]

Original: ||

Original: [Equation]

Original: ||∇ − 2 ( x − x ∗ )

Original: || 22 .

Original: We now investigate the computed updates ∆ x of various optimization methods for this problem.

Original:      6XSHUYLVHG    
Revised:      6XSHUYLVHG    

Original:  *'      ,*    
Revised:  *'      ,*    

Original:  1HZWRQ    
Revised:  1HZWRQ    

Original:  3*  
Revised:  3*  

Original: [Figure]
Revised: [Figure]

Original:    
Revised:    

Original: VLQ
Revised: VLQ

Original: [
Revised: [

Original:                  6HHG 3* *' 1HZWRQ ,* 6XSHUYLVHG                  6HHG 3* *' 1HZWRQ ,* 6XSHUYLVHG                  6HHG 3* *' 1HZWRQ ,* 6XSHUYLVHG                  6HHG 3* *' 1HZWRQ ,* 6XSHUYLVHG 
Revised:                  6HHG 3* *' 1HZWRQ ,* 6XSHUYLVHG                  6HHG 3* *' 1HZWRQ ,* 6XSHUYLVHG                  6HHG 3* *' 1HZWRQ ,* 6XSHUYLVHG                  6HHG 3* *' 1HZWRQ ,* 6XSHUYLVHG y

Revised: = P ( x ) = ∇ − 2 x implicitly via the conjugate gradient method.

Revised: The inverse problem consists of ﬁnding an initial value x ∗ for a given target y ∗ such that ∇ 2 y ∗ = x ∗ .

Revised: We formulate this problem as minimizing L ( x ) = 12 ||P ( x ) − y ∗ || 22 = 12 ||∇ − 2 ( x − x ∗ )

Revised: || 22 .

Revised: We now investigate the computed updates ∆ x of various optimization methods for this problem.

Original: Gradient descent Gradient descent prescribes the update ∆ x 
Revised: Gradient descent Gradient descent prescribes the update ∆ x = − η · (cid:0) ∂L∂x (cid:1) T =

Original: [Equation]
Revised: [Equation]

Original:  which requires an additional implicit solve for each optimization step.
Revised: 2 ( y − y ∗ ) which requires an additional implicit solve for each optimization step.

Original: This backward solve produces much larger values than the forward solve, causing GD-based methods to diverge from oscillations unless η is very small.
Revised: This backward solve produces much larger values than the forward solve, causing GD-based methods to diverge from oscillations unless η is very small.

Original: We found that GD requires η ≤ 2 · 10 − 5 , while the momentum in Adam allows for larger η .
Revised: We found that GD requires η ≤ 2 · 10 − 5 , while the momentum in Adam allows for larger η .

Original: For both GD and Adam, the optimization converges extremely slowly, making
Revised: For both GD and Adam, the optimization converges extremely slowly, making

Original: GD-based methods unfeasible for this problem.
Revised: GD-based methods unfeasible for this problem.

Original: Physical gradients via analytic inversion Poisson’s equation can easily be inverted analytically, yielding x = ∇ 2 y .
Revised: Physical gradients via analytic inversion Poisson’s equation can easily be inverted analytically, yielding x = ∇ 2 y .

Original: Correspondingly, we formulate the update step as ∆ x = − η · ∂x∂y · ( y − y ∗ )
Revised: Correspondingly, we formulate the update step as ∆ x = − η · ∂x∂y · ( y − y ∗ )

Original: = − η · ∇ 2 ( y − y ∗ ) which directly points to x ∗ for η = 1 .
Revised: = − η · ∇ 2 ( y − y ∗ ) which directly points to x ∗ for η = 1 .

Original: Here the Laplace operator appears in the computation of the optimization direction.
Revised: Here the Laplace operator appears in the computation of the optimization direction.

Original: This is much easier to compute numerically than the
Revised: This is much easier to compute numerically than the

Original: Poisson operator used by gradient descent.
Revised: Poisson operator used by gradient descent.

Original: Consequently, no additional implicit solve is required for the optimization and the cost per iteration is less than with gradient descent.
Revised: Consequently, no additional implicit solve is required for the optimization and the cost per iteration is less than with gradient descent.

Original: This computational advantages also carries over to neural network training where this method can be integrated into the backpropagation pipeline as a physical gradient.
Revised: This computational advantages also carries over to neural network training where this method can be integrated into the backpropagation pipeline as a physical gradient.

Original: Neural network training We ﬁrst generate ground truth solutions x ∗ by adding ﬂuctuations of varying frequencies with random amplitudes.
Revised: Neural network training We ﬁrst generate ground truth solutions x ∗ by adding ﬂuctuations of varying frequencies with random amplitudes.

Original: From these x ∗ , we compute y ∗ =
Revised: From these x ∗ , we compute y ∗ =

Original: P ( x ∗ ) to form the set of target states Y .
Revised: P ( x ∗ ) to form the set of target states Y .

Original: Both generation of x ∗ and y ∗ is performed on the ﬂy, resulting in a data set of effectively inﬁnite size, |Y| = ∞ .
Revised: Both generation of x ∗ and y ∗ is performed on the ﬂy, resulting in a data set of effectively inﬁnite size, |Y| = ∞ .

Original: This has the advantage that learning curves are representative of both test performance as well as training performance.
Revised: This has the advantage that learning curves are representative of both test performance as well as training performance.

Original: The top of Fig.
Revised: The top of Fig.

Original: 11 shows some examples generated this way.
Revised: 11 shows some examples generated this way.

Original: We train a U-net Ronneberger et al.
Revised: We train a U-net Ronneberger et al.

Original: with a total of 4 resolution levels and skip connections.
Revised: with a total of 4 resolution levels and skip connections.

Original: The network receives the feature map y ∗ as input.
Revised: The network receives the feature map y ∗ as input.

Original: Max pooling is used for downsampling and bilinear interpolation for upsampling.
Revised: Max pooling is used for downsampling and bilinear interpolation for upsampling.

Original: After each downsampling or upsampling operation, two blocks consisting of 2D convolution with kernel size of 3x3, batch normalization and ReLU activation are performed.
Revised: After each downsampling or upsampling operation, two blocks consisting of 2D convolution with kernel size of 3x3, batch normalization and ReLU activation are performed.

Original: All of these convolutions output 16 feature maps and a ﬁnal 1x1 convolution brings the output down to one feature map.
Revised: All of these convolutions output 16 feature maps and a ﬁnal 1x1 convolution brings the output down to one feature map.

Original: The network contains a total of 37, trainable parameters.
Revised: The network contains a total of 37, trainable parameters.

Original: For SGD and Adam training, the composite gradient of NN ◦ P is computed with TensorFlow or
Revised: For SGD and Adam training, the composite gradient of NN ◦ P is computed with TensorFlow or

Original: PyTorch, enabling an end-to-end optimization.
Revised: PyTorch, enabling an end-to-end optimization.

Original: The learning rate is set to η = 10 − 3 with Adam and η = 10 − 9 for SGD.
Revised: The learning rate is set to η = 10 − 3 with Adam and η = 10 − 9 for SGD.

Original: The extremely small learning rate for SGD is required to balance out the large gradients and is consistent with the behavior of gradient-descent optimization on single examples where an η = 2 · 10 − 5 was required.
Revised: The extremely small learning rate for SGD is required to balance out the large gradients and is consistent with the behavior of gradient-descent optimization on single examples where an η = 2 · 10 − 5 was required.

Original: We use a typical value of 0.9 for the momentum of SGD and
Revised: We use a typical value of 0.9 for the momentum of SGD and

Original: Adam.
Revised: Adam.

Original: For the training using Adam with physical gradients, we compute ∆ x as described above and keep η = 10 − 3 .
Revised: For the training using Adam with physical gradients, we compute ∆ x as described above and keep η = 10 − 3 .

Original: For each case, we set the learning rate to the maximum value that consistently converges.
Revised: For each case, we set the learning rate to the maximum value that consistently converges.

Original: The learning curves for three additional random network initializations are shown at the bottom of Fig.
Revised: The learning curves for three additional random network initializations are shown at the bottom of Fig.

Original: 11, while Fig.
Revised: 11, while Fig.

Original: 14 shows the computation time per iteration.
Revised: 14 shows the computation time per iteration.

Original: B. H EAT EQUATION
Revised: B. H EAT EQUATION

Original: We consider a two-dimensional system governed by the heat equation ∂u∂t = ν · ∇ 2 u .
Revised: We consider a two-dimensional system governed by the heat equation ∂u∂t = ν · ∇ 2 u .

Original: Given an initial state x = u 0 at t 0 , the simulator computes the state at a later time t ∗ via y = u ∗ =
Revised: Given an initial state x = u 0 at t 0 , the simulator computes the state at a later time t ∗ via y = u ∗ =

Original: P ( x ) .
Revised: P ( x ) .

Original: Exactly inverting this system is only possible for t · ν = 0 and becomes increasingly unstable for larger t · ν because initially distinct heat levels even out over time, drowning the original information in noise.
Revised: Exactly inverting this system is only possible for t · ν = 0 and becomes increasingly unstable for larger t · ν because initially distinct heat levels even out over time, drowning the original information in noise.

Original: Hence the Jacobian of the physics ∂y∂x is near-singular.
Revised: Hence the Jacobian of the physics ∂y ∂x is near-singular.

Original: In our experiment we set t · ν = 8 on a domain consisting of 64x64 cells of unit length.
Revised: In our experiment we set t · ν = 8 on a domain consisting of 64x64 cells of unit length.

Original: This level of diffusion is challenging, and diffuses most details while leaving the large-scale structure intact.
Revised: This level of diffusion is challenging, and diffuses most details while leaving the large-scale structure intact.

Original: We apply periodic boundary conditions and compute the result in frequency space where the physics can be computed analytically as ˆ y
Revised: We apply periodic boundary conditions and compute the result in frequency space where the physics can be computed analytically as ˆ y

Original: = ˆ x · e − k 2 ( t ∗ − t 0 ) where ˆ y k ≡ F ( y ) k denotes the k -th element of the Fourier-transformed vector y .
Revised: = ˆ x · e − k 2 ( t ∗ − t 0 ) where ˆ y k ≡ F ( y ) k denotes the k -th element of the Fourier-transformed vector y .

Original: Here, high frequencies are dampened exponentially.
Revised: Here, high frequencies are dampened exponentially.

Revised: The inverse

Revised: [Figure]

Original: The inverse problem can thus be written as minimizing L 
Revised: problem can thus be written as minimizing L ( [Equation] =

Revised: [Equation]

Revised: P

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: Gradient descent Using the analytic formulation, we can compute the gradient descent update as
Revised: Gradient descent Using the analytic formulation, we can compute the gradient descent update as

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: GD applies the forward physics to the gradient vector itself, which results in updates that are stable but lack high frequency spatial information.
Revised: GD applies the forward physics to the gradient vector itself, which results in updates that are stable but lack high frequency spatial information.

Original: Consequently, GD-based optimization methods converge slowly on this task after ﬁtting the coarse structure and have severe problems in recovering high-frequency details.
Revised: Consequently, GD-based optimization methods converge slowly on this task after ﬁtting the coarse structure and have severe problems in recovering high-frequency details.

Original: This is not because the information is fundamentally missing but because
Revised: This is not because the information is fundamentally missing but because

Original: GD cannot adequately process high-frequency details.
Revised: GD cannot adequately process high-frequency details.

Original: Stable physical gradients The frequency formulation of the heat equation can be inverted analytically, yielding ˆ x k
Revised: Stable physical gradients The frequency formulation of the heat equation can be inverted analytically, yielding ˆ x k

Original: = ˆ y k · e k 2 ( t ∗ − t 0 ) .
Revised: = ˆ y k · e k 2 ( t ∗ − t 0 ) .

Original: This allows us to deﬁne the update
Revised: This allows us to deﬁne the update

Original: [Equation]
Revised: [Equation]

Original: Here, high frequencies are multiplied by exponentially large factors, resulting in numerical instabilities.
Revised: Here, high frequencies are multiplied by exponentially large factors, resulting in numerical instabilities.

Original: When applying this formula directly to the gradients, it can lead to large oscillations in ∆ x .
Revised: When applying this formula directly to the gradients, it can lead to large oscillations in ∆ x .

Original: This is the opposite behavior compared to Poisson’s equation where the GD updates were unstable and the PG stable.
Revised: This is the opposite behavior compared to Poisson’s equation where the GD updates were unstable and the PG stable.

Original: The numerical instabilities here can, however, be avoided by taking a probabilistic viewpoint.
Revised: The numerical instabilities here can, however, be avoided by taking a probabilistic viewpoint.

Original: The observed values y contain a certain amount of noise n , with the remainder constituting the signal s = y − n .
Revised: The observed values y contain a certain amount of noise n , with the remainder constituting the signal s = y − n .

Original: For the noise, we assume a normal distribution n ∼ N (0 , (cid:15) · y ) with (cid:15) > 0 and for the signal, we assume that it arises from reasonable values of x so that y ∼ N (0 , δ · e − k 2 ) with δ > 0 .
Revised: For the noise, we assume a normal distribution n ∼ N (0 , (cid:15) · y ) with (cid:15) > 0 and for the signal, we assume that it arises from reasonable values of x so that y ∼ N (0 , δ · e − k 2 ) with δ > 0 .

Original: With this, we can estimate the probability of an observed value arising from the signal using Bayes’ theorem p ( s | v ) =
Revised: With this, we can estimate the probability of an observed value arising from the signal using Bayes’ theorem p ( s | v ) =

Original: [Equation] | n ) · p ( n ) where we assume the priors p ( s ) = p ( n ) = 12 .
Revised: p ( v | s ) · p ( s ) p ( v | s ) · p ( s )+ p ( v | n ) · p ( n ) where we assume the priors p ( s ) = p ( n ) = 12 .

Original: Based on this probability, we dampen the ampliﬁcation of the inverse physics which yields a stable inverse.
Revised: Based on this probability, we dampen the ampliﬁcation of the inverse physics which yields a stable inverse.

Original: Gradients computed in this way hold as much high-frequency information as can be extracted given the noise that is present.
Revised: Gradients computed in this way hold as much high-frequency information as can be extracted given the noise that is present.

Original: This leads to a much faster convergence and more precise solution than any generic optimization method.
Revised: This leads to a much faster convergence and more precise solution than any generic optimization method.

Original: Neural network training For training, we generate x ∗ by randomly placing between 4 andhot rectangles of random size and shape in the domain and computing y
Revised: Neural network training For training, we generate x ∗ by randomly placing between 4 andhot rectangles of random size and shape in the domain and computing y

Original: = P ( x ∗ ) .
Revised: = P ( x ∗ ) .

Original: Both operations are executed on the ﬂy so that |Y| = ∞ and the learning curves are representative of both training and test performance.
Revised: Both operations are executed on the ﬂy so that |Y| = ∞ and the learning curves are representative of both training and test performance.

Original: For the neural network, we use the same U-net architecture as in the previous experiment.
Revised: For the neural network, we use the same U-net architecture as in the previous experiment.

Original: We train with a batch size of 128 and a constant learning rate of η = 10 − 3 .
Revised: We train with a batch size of 128 and a constant learning rate of η = 10 − 3 .

Original: The network updates are computed with TensorFlow’s or PyTorch’s automatic differentiation.
Revised: The network updates are computed with TensorFlow’s or PyTorch’s automatic differentiation.

Original: Fig.shows two examples from the data set, along with the corresponding inferred solutions, as well as the network learning curves for two network initializations.
Revised: Fig.shows two examples from the data set, along with the corresponding inferred solutions, as well as the network learning curves for two network initializations.

Original: The measured computation time per iteration is shown in Fig.
Revised: The measured computation time per iteration is shown in Fig.

Original: B. N AVIER -S TOKES EQUATIONS
Revised: B. N AVIER -S TOKES EQUATIONS

Original: Here, we give additional details on the simulation, data generation, physical gradients and network training procedure for the ﬂuid experiment.
Revised: Here, we give additional details on the simulation, data generation, physical gradients and network training procedure for the ﬂuid experiment.

Original: Simulation details We simulate the ﬂuid dynamics using a direct numerical solver.
Revised: Simulation details We simulate the ﬂuid dynamics using a direct numerical solver.

Original: We adopt the marker-in-cell (MAC) method Harlow & Welch (1965); Harlow (1972) which guarantees stable simulations even for large velocities or time increments.
Revised: We adopt the marker-in-cell (MAC) method Harlow & Welch (1965); Harlow (1972) which guarantees stable simulations even for large velocities or time increments.

Original: The velocity vectors are sampled in staggered form at the face centers of grid cells while the marker density is sampled at the cell centers.
Revised: The velocity vectors are sampled in staggered form at the face centers of grid cells while the marker density is sampled at the cell centers.

Original: The initial velocity v 0 is speciﬁed at cell centers and resampled to a staggered grid for the simulation.
Revised: The initial velocity v 0 is speciﬁed at cell centers and resampled to a staggered grid for the simulation.

Original: Our simulation employs a second-order advection scheme Selle et al.
Revised: Our simulation employs a second-order advection scheme Selle et al.

Original: (2008) to transport both the marker and the velocity vectors.
Revised: (2008) to transport both the marker and the velocity vectors.

Original: This step introduces signiﬁcant amount of numerical diffusion which can clearly be seen in the ﬁnal marker distributions.
Revised: This step introduces signiﬁcant amount of numerical diffusion which can clearly be seen in the ﬁnal marker distributions.

Original: Hence, we do not numerically solve for adding  additional viscosity.
Revised: Hence, we do not numerically solve for adding [Figure] additional viscosity.

Original: Incompressibility is achieved via Helmholz decomposition of the velocity ﬁeld using a conjugate gradient solve.
Revised: Incompressibility is achieved via Helmholz decomposition of the velocity ﬁeld using a conjugate gradient solve

Original: Neither pressure projection nor advection are energy-conserving operations.
Revised: Neither pressure projection nor advection are energy-conserving operations.

Original: While specialized energy-conserving simulation schemes for ﬂuids exist Gluhovsky & Tong (1999); Mullen et al.
Revised: While specialized energy-conserving simulation schemes for ﬂuids exist Gluhovsky & Tong (1999); Mullen et al.

Original: (2009), we instead enforce energy conservation by normalizing the velocity ﬁeld at each time step to the total energy of the previous time step.
Revised: (2009), we instead enforce energy conservation by normalizing the velocity ﬁeld at each time step to the total energy of the previous time step.

Original: Here, the energy is computed as E = (cid:82) R 2 dx v ( x )since we assume constant ﬂuid density.[Figure]
Revised: Here, the energy is computed as E = (cid:82) R 2 dx v ( x )since we assume constant ﬂuid density.

Original: Data generation The data set consists of marker pairs { m 0 , m t } which are randomly generated on-the-ﬂy.
Revised: Data generation The data set consists of marker pairs { m 0 , m t } which are randomly generated on-the-ﬂy.

Original: For each example, a center position for m 0 is chosen on a grid of 64x64 cells.
Revised: For each example, a center position for m 0 is chosen on a grid of 64x64 cells.

Original: m 0 is then generated from discretized noise ﬂuctuations to ﬁll half the domain size in each dimension.
Revised: m 0 is then generated from discretized noise ﬂuctuations to ﬁll half the domain size in each dimension.

Original: The number of marked cells is random.
Revised: The number of marked cells is random.

Original: Next, a ground truth initial velocity v 0 is generated from three components.
Revised: Next, a ground truth initial velocity v 0 is generated from three components.

Original: First, a uniform velocity ﬁeld moves the marker towards the center of the domain to avoid boundary collisions.
Revised: First, a uniform velocity ﬁeld moves the marker towards the center of the domain to avoid boundary collisions.

Original: Second, a large vortex with random strength and direction is added.
Revised: Second, a large vortex with random strength and direction is added.

Original: The velocity magnitude of the vortex falls off with a Gaussian function depending on the distance from the vortex center.
Revised: The velocity magnitude of the vortex falls off with a Gaussian function depending on the distance from the vortex center.

Original: Third, smaller-scale vortices of random strengths and sizes are added additionally perturb the ﬂow ﬁelds.
Revised: Third, smaller-scale vortices of random strengths and sizes are added additionally perturb the ﬂow ﬁelds.

Original: These are generated by assigning a random amplitude and phase to each frequency making up the velocity ﬁeld.
Revised: These are generated by assigning a random amplitude and phase to each frequency making up the velocity ﬁeld.

Original: The range from which the amplitudes are sampled depends on the magnitude frequency.
Revised: The range from which the amplitudes are sampled depends on the magnitude frequency.

Original: Given m 0 and v 0 , a ground truth simulation is run for t = 2 with ∆ t = 0 .
Revised: Given m 0 and v 0 , a ground truth simulation is run for t = 2 with ∆ t = 0 .

Original: The resulting marker density is then used as the target for the optimization.
Revised: The resulting marker density is then used as the target for the optimization.

Original: This ensures that there exists a solution for each example.
Revised: This ensures that there exists a solution for each example.

Original: Computation of physical gradients To compute the physical gradients for this example, we construct an explicit formulation ˆ v 0 =
Revised: Computation of physical gradients To compute the physical gradients for this example, we construct an explicit formulation ˆ v 0 =

Original: P − 1 ( m 0 , m t | x 0 ) that produces an estimate for v 0 given an initial guess x 0 by locally inverting the physics.
Revised: P − 1 ( m 0 , m t | x 0 ) that produces an estimate for v 0 given an initial guess x 0 by locally inverting the physics.

Original: From this information, it ﬁts the coarse velocity, i.e. the uniform velocity and the vortex present in the data.
Revised: From this information, it ﬁts the coarse velocity, i.e. the uniform velocity and the vortex present in the data.

Original: This use of domain knowledge, i.e., enforcing the translation and rotation components of the velocity ﬁeld as a prior, is what allows it to produce a much better estimate of v 0 than the regular gradient.
Revised: This use of domain knowledge, i.e., enforcing the translation and rotation components of the velocity ﬁeld as a prior, is what allows it to produce a much better estimate of v 0 than the regular gradient.

Original: More formally, it assumes that the solution lies on a manifold that is much more low-dimensional than v 0 .
Revised: More formally, it assumes that the solution lies on a manifold that is much more low-dimensional than v 0 .

Original: On the other hand, this estimator ignores the small-scale velocity ﬂuctuations which limits the accuracy it can achieve.
Revised: On the other hand, this estimator ignores the small-scale velocity ﬂuctuations which limits the accuracy it can achieve.

Original: However, the difﬁculty of ﬁtting the full velocity ﬁeld without any assumptions outweighs this limitation.
Revised: However, the difﬁculty of ﬁtting the full velocity ﬁeld without any assumptions outweighs this limitation.

Original: Nevertheless, GD could eventually lead to better results if trained for an extremely long time.
Revised: Nevertheless, GD could eventually lead to better results if trained for an extremely long time.

Original: To estimate the vortex strength, the estimator runs a reverse Navier-Stokes simulation.
Revised: To estimate the vortex strength, the estimator runs a reverse Navier-Stokes simulation.

Original: The reverse simulation is initialized with the marker m rev t = m t and velocity v rev t = v t from the forward simulation.
Revised: The reverse simulation is initialized with the marker m rev t = m t and velocity v rev t = v t from the forward simulation.

Original: The reverse simulation then computes m rev and v rev for all time steps by performing simulation steps with ∆ t = − 0 .
Revised: The reverse simulation then computes m rev and v rev for all time steps by performing simulation steps with ∆ t = − 0 .

Original: Then, the update to the vortex strength is computed from the differences m rev − m at each time step and an estimate of the vortex location at these time steps.
Revised: Then, the update to the vortex strength is computed from the differences m rev − m at each time step and an estimate of the vortex location at these time steps.

Original: Neural network training We train a U-net Ronneberger et al.
Revised: Neural network training We train a U-net Ronneberger et al.

Original: (2015) similar to the previous experiments but with 5 resolution levels.
Revised: (2015) similar to the previous experiments but with 5 resolution levels.

Original: The network contains a total of 49,570 trainable parameters.
Revised: The network contains a total of 49,570 trainable parameters.

Original: The network is given the observed markers m 0 and m t , resulting in an input consisting of two feature maps.
Revised: The network is given the observed markers m 0 and m t , resulting in an input consisting of two feature maps.

Original: It outputs two feature maps which are interpreted as a velocity ﬁeld sampled at cell centers.
Revised: It outputs two feature maps which are interpreted as a velocity ﬁeld sampled at cell centers.

Original: The objective function is deﬁned as |F ( P ( x ) − y ∗ ) | · w where F denotes the two-dimensional
Revised: The objective function is deﬁned as |F ( P ( x ) − y ∗ ) | · w where F denotes the two-dimensional

Original: Fourier transform and w is a weighting vector that factors high frequencies exponentially less than low frequencies.
Revised: Fourier transform and w is a weighting vector that factors high frequencies exponentially less than low frequencies.

Original: We train the network using Adam with a learning rate of 0.005 and mini-batches containing 64 examples each, using PyTorch’s automatic differentiation to compute the weight updates.
Revised: We train the network using Adam with a learning rate of 0.005 and mini-batches containing 64 examples each, using PyTorch’s automatic differentiation to compute the weight updates.

Original: We found that second-order optimizers like L-BFGS-B yield no signiﬁcant advantage over gradient descent, and typically overshoot in terms of high-frequency motions.
Revised: We found that second-order optimizers like L-BFGS-B yield no signiﬁcant advantage over gradient descent, and typically overshoot in terms of high-frequency motions.

Original: Example trajectories and reconstructions are shown in Fig.
Revised: Example trajectories and reconstructions are shown in Fig.

Original: 13 and performance measurements are shown in Fig.
Revised: 13 and performance measurements are shown in Fig.

Original: G T
Revised: G T

Original: A +
Revised: A +

Original: P G
Revised: P G

Original: A d a m
Revised: A d a m

Original: G T
Revised: G T

Original: A +
Revised: A +

Original: P G
Revised: P G

Original: A d a m
Revised: A d a m

Original: G T
Revised: G T

Original: A +
Revised: A +

Original: P G
Revised: P G

Original: A d a m
Revised: A d a m

Original: [Figure]
Revised: [Figure]

Original: e c .
Revised: e c .

Original: / i t e r a t i o n
Revised: / i t e r a t i o n

Original: [Figure]
Revised: [Figure]

Original: (Newton
Revised: (Newton

Original: [Figure]
Revised: [Figure]

Original: S e c .
Revised: S e c .

Original: / i t e r a t i o n
Revised: / i t e r a t i o n

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: [Figure]
Revised: [Figure]

Original: / i t e r a t i o n
Revised: / i t e r a t i o n

Original: [Figure]
Revised: [Figure]