Original: B I - LEVEL P HYSICS -I NFORMED N EURAL N ETWORKS FOR PDE C ONSTRAINED O PTIMIZATION USING B ROYDEN ’ S H YPERGRADIENTS
Revised: B I - LEVEL P HYSICS -I NFORMED N EURAL N ETWORKS FOR PDE C ONSTRAINED O PTIMIZATION USING B ROYDEN ’ S H YPERGRADIENTS

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep learning based approaches like Physics-informed neural networks (PINNs) and DeepONets have shown promise on solving PDE constrained optimization (PDECO) problems.
Revised: Deep learning based approaches like Physics-informed neural networks (PINNs) and DeepONets have shown promise on solving PDE constrained optimization (PDECO) problems.

Original: However, existing methods are insufficient to handle those
Revised: However, existing methods are insufficient to handle those

Original: PDE constraints that have a complicated or nonlinear dependency on optimization targets.
Revised: PDE constraints that have a complicated or nonlinear dependency on optimization targets.

Original: In this paper, we present a novel bi-level optimization framework to resolve the challenge by decoupling the optimization of the targets and constraints.
Revised: In this paper, we present a novel bi level optimization framework to resolve the challenge by decoupling the optimization of the targets and constraints

Original: For the inner loop optimization, we adopt PINNs to solve the PDE constraints only.
Revised: For the inner loop optimization we adopt PINNs to solve the PDE constraints only

Original: For the outer loop, we design a novel method by using Broyden’s method based on the
Revised: For the outer loop we design a novel method by using Broyden’s method based on the

Original: Implicit Function Theorem (IFT), which is efficient and accurate for approximating hypergradients.
Revised: Implicit Function Theorem (IFT) which is efficient and accurate for approximating hypergradients.

Original: We further present theoretical explanations and error analysis of the hypergradients computation.
Revised: We further present theoretical explanations and error analysis of the hypergradients computation.

Original: Extensive experiments on multiple large-scale and nonlinear PDE constrained optimization problems demonstrate that our method achieves state-of-the-art results compared with strong baselines.
Revised: Extensive experiments on multiple large-scale and nonlinear PDE constrained optimization problems demonstrate that our method achieves state-of-the-art results compared with strong baselines

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: PDE constrained optimization (PDECO) aims at optimizing the performance of a physical system constrained by partial differential equations (PDEs) with desired properties.
Revised: PDE constrained optimization (PDECO) aims at optimizing the performance of a physical system constrained by partial differential equations (PDEs) with desired properties.

Original: It is a fundamental task in numerous areas of science (Chakrabarty & Hanson, 2005; Ng & Dubljevic, 2012) and engineering (Hicks & Henne, 1978;
Revised: It is a fundamental task in numerous areas of science (Chakrabarty & Hanson, 2005; Ng & Dubljevic, 2012) and engineering (Hicks & Henne, 1978;

Original: Chen et al., 2009), with a wide range of important applications including image denoising in computer vision (De los Reyes & Schönlieb, 2013), design of aircraft wings in aerodynamics (Hicks & Henne, 1978), and drug delivery (Chakrabarty & Hanson, 2005) in biology etc.
Revised: Chen et al., 2009), with a wide range of important applications including image denoising in computer vision (De los Reyes & Schönlieb, 2013), design of aircraft wings in aerodynamics (Hicks & Henne, 1978), and drug delivery (Chakrabarty & Hanson, 2005) in biology etc.

Original: These problem have numerous inherent challenges due to the diversity and complexity of physical constraints and practical problems.
Revised: These problem have numerous inherent challenges due to the diversity and complexity of physical constraints and practical problems.

Original: Traditional numerical methods like adjoint methods (Herzog & Kunisch, 2010) based on finite element methods (FEMs) (Zienkiewicz et al., 2005) have been studied for decades.
Revised: Traditional numerical methods like adjoint methods (Herzog & Kunisch, 2010) based on finite element methods (FEMs) (Zienkiewicz et al., 2005) have been studied for decades.

Original: They could be divided into continuous and discretized adjoint methods (Mitusch et al., 2019).
Revised: They could be divided into continuous and discretized adjoint methods (Mitusch et al., 2019).

Original: The former one requires complex handcraft derivation of adjoint PDEs and the latter one is more flexible and more frequently used.
Revised: The former one requires complex handcraft derivation of adjoint PDEs and the latter one is more flexible and more frequently used.

Original: However, the computational cost of FEMs grows quadratically to cubically (Xue et al., 2020) w.r.t mesh sizes.
Revised: However, the computational cost of FEMs grows quadratically to cubically (Xue et al., 2020) w.r.t mesh sizes.

Original: Thus compared with other constrained optimization problems, it is much more expensive or even intractable to solve high dimensional PDECO problems with a large search space or mesh size.
Revised: Thus compared with other constrained optimization problems, it is much more expensive or even intractable to solve high dimensional PDECO problems with a large search space or mesh size.

Original: To mitigate this problem, neural network methods like DeepONet (Lu et al., 2019) have been proposed as surrogate models of FEMs recently.
Revised: To mitigate this problem, neural network methods like DeepONet (Lu et al., 2019) have been proposed as surrogate models of FEMs recently.

Original: DeepONet learns a mapping from control (decision) variables to solutions of PDEs and further replaces PDE constraints with the operator network.
Revised: DeepONet learns a mapping from control (decision) variables to solutions of PDEs and further replaces PDE constraints with the operator network.

Original: But these methods require pretraining a large operator network which is non-trivial and inefficient.
Revised: But these methods require pretraining a large operator network which is non-trivial and inefficient.

Original: Moreover, its performance may deteriorate if the optimal solution is out of the training distribution (Lanthaler et al., 2022).
Revised: Moreover, its performance may deteriorate if the optimal solution is out of the training distribution (Lanthaler et al., 2022).

Original: Another approach of neural methods (Lu et al., 2021; Mowlavi & Nabi, 2021) proposes to use a single PINN (Raissi et al., 2019) to solve the PDECO problem instead of pretraining an operator network.
Revised: Another approach of neural methods (Lu et al., 2021; Mowlavi & Nabi, 2021) proposes to use a single PINN (Raissi et al., 2019) to solve the PDECO problem instead of pretraining an operator network.

Original: It uses the method of Lagrangian multipliers to treat the PDE constraints as regularization terms, and thus optimize the objective and PDE loss simultaneously.
Revised: It uses the method of Lagrangian multipliers to treat the PDE constraints as regularization terms, and thus optimize the objective and PDE loss simultaneously.

Original: However, such methods introduce a trade-off between optimization targets and regularization terms (i.e., PDE losses) which is crucial for the performance (Nandwani et al., 2019).
Revised: However, such methods introduce a trade-off between optimization targets and regularization terms (i.e., PDE losses) which is crucial for the performance (Nandwani et al., 2019).

Original: It is generally non-trivial to set proper weights for balancing these terms due to the lack of theoretical guidance.
Revised: It is generally non-trivial to set proper weights for balancing these terms due to the lack of theoretical guidance.

Original: Existing heuristic approaches for selecting the weights may usually yield an unstable training process.
Revised: Existing heuristic approaches for selecting the weights may usually yield an unstable training process.

Original: Therefore, it is imperative to develop an effective strategy to handle PDE constraints for solving PDECO problems.
Revised: Therefore, it is imperative to develop an effective strategy to handle PDE constraints for solving PDECO problems.

Original: To address the aforementioned challenges, we propose a novel bi-level optimization framework named
Revised: To address the aforementioned challenges, we propose a novel bi-level optimization framework named

Original: Bi-level Physics-informed Neural networks with Broyden’s hypergradients (BPN) for solving PDE constrained optimization problems.
Revised: Bi-level Physics-informed Neural networks with Broyden’s hypergradients (BPN) for solving PDE constrained optimization problems.

Original: Specifically, we first present a bi-level formulation of the PDECO problems, which decouples the optimization of the targets and PDE constraints, thereby naturally addressing the challenge of loss balancing in regularization based methods.
Revised: Specifically, we first present a bi-level formulation of the PDECO problems, which decouples the optimization of the targets and PDE constraints, thereby naturally addressing the challenge of loss balancing in regularization based methods.

Original: To solve the bi-level optimization problem, we develop an iterative method that optimizes PINNs with PDE constraints in the inner loop while optimizes the control variables for objective functions in the outer loop using hypergradients.
Revised: To solve the bi-level optimization problem, we develop an iterative method that optimizes PINNs with PDE constraints in the inner loop while optimizes the control variables for objective functions in the outer loop using hypergradients.

Original: In general, it is nontrivial to compute hypergradients in bi-level optimization for control variables especially if the inner loop optimization is complicated (Lorraine et al., 2020).
Revised: In general, it is nontrivial to compute hypergradients in bi-level optimization for control variables especially if the inner loop optimization is complicated (Lorraine et al., 2020).

Original: To address this issue, we further propose a novel strategy based on implicit differentiation using
Revised: To address this issue, we further propose a novel strategy based on implicit differentiation using

Original: Broyden’s method which is a scalable and efficient quasi-Newton method in practice (Kelley, 1995;
Revised: Broyden’s method which is a scalable and efficient quasi-Newton method in practice (Kelley, 1995;

Original: Bai et al., 2020).
Revised: Bai et al., 2020).

Original: We then theoretically prove an upper bound for the approximation of hypergradients under mild assumptions.
Revised: We then theoretically prove an upper bound for the approximation of hypergradients under mild assumptions.

Original: Extensive experiments on several benchmark PDE constrained optimization problems show that our method is more effective and efficient compared with the alternative methods.
Revised: Extensive experiments on several benchmark PDE constrained optimization problems show that our method is more effective and efficient compared with the alternative methods.

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: • To the best of our knowledge, it is the first attempt that solves general PDECO problems based on deep learning using a bi-level optimization framework that enjoys scalability and theoretical guarantee.
Revised: • To the best of our knowledge, it is the first attempt that solves general PDECO problems based on deep learning using a bi-level optimization framework that enjoys scalability and theoretical guarantee.

Original: • We propose a novel and efficient method for hypergradients computation using Broyden’s method to solve the bi-level optimization.
Revised: • We propose a novel and efficient method for hypergradients computation using Broyden’s method to solve the bi-level optimization.

Original: • We conduct extensive experiments and achieve state-of-the-art results among deep learning methods on several challenging PDECO problems with complex geometry or non-linear Naiver-Stokes equations.
Revised: • We conduct extensive experiments and achieve state-of-the-art results among deep learning methods on several challenging PDECO problems with complex geometry or non-linear Naiver-Stokes equations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Networks Approaches for PDE Constrained Optimization.
Revised: Neural Networks Approaches for PDE Constrained Optimization.

Original: Surrogate modeling is an important class of methods for PDE constrained optimization (Queipo et al., 2005).
Revised: Surrogate modeling is an important class of methods for PDE constrained optimization (Queipo et al., 2005).

Original: Physics-informed neural networks (PINNs) are powerful and flexible surrogates to represent the solutions of PDEs (Raissi et al., 2019).
Revised: Physics-informed neural networks (PINNs) are powerful and flexible surrogates to represent the solutions of PDEs (Raissi et al., 2019).

Original: hPINN (Lu et al., 2021) treats PDE constraints as regularization terms and optimizes the control variables and states simultaneously.
Revised: hPINN (Lu et al., 2021) treats PDE constraints as regularization terms and optimizes the control variables and states simultaneously.

Original: It uses the penalty method and the
Revised: It uses the penalty method and the

Original: Lagrangian method to adjust the weights of multipliers.
Revised: Lagrangian method to adjust the weights of multipliers.

Original: (Mowlavi & Nabi, 2021) also adopts the same formulation but uses a line search to find the largest weight when the PDE error is within a certain range.
Revised: (Mowlavi & Nabi, 2021) also adopts the same formulation but uses a line search to find the largest weight when the PDE error is within a certain range.

Original: The key limitation of these approaches is that heuristically choosing methods for tuning weights of multipliers might be sub-optimal and unstable.
Revised: The key limitation of these approaches is that heuristically choosing methods for tuning weights of multipliers might be sub-optimal and unstable.

Original: Another class of methods train an operator network from control variables to solutions of PDEs or objective functions.
Revised: Another class of methods train an operator network from control variables to solutions of PDEs or objective functions.

Original: Several works (Xue et al., 2020; Sun et al., 2021; Beatson et al., 2020) use mesh-based methods and predict states on all mesh points from control variables at the same time.
Revised: Several works (Xue et al., 2020; Sun et al., 2021; Beatson et al., 2020) use mesh-based methods and predict states on all mesh points from control variables at the same time.

Original: PI-DeepONet (Wang et al., 2021a;c) adopts the architecture of DeepONet (Lu et al., 2019) and trains the network using physics-informed losses (PDE losses).
Revised: PI-DeepONet (Wang et al., 2021a;c) adopts the architecture of DeepONet (Lu et al., 2019) and trains the network using physics-informed losses (PDE losses).

Original: However, they produce unsatisfactory results if the optimal solution is out of the distribution (Lanthaler et al., 2022).
Revised: However, they produce unsatisfactory results if the optimal solution is out of the distribution (Lanthaler et al., 2022).

Original: Bi-level Optimization in Machine Learning.
Revised: Bi-level Optimization in Machine Learning.

Original: Bi-level optimization is widely used in various machine learning tasks, e.g., neural architecture search (Liu et al., 2018), meta learning (Rajeswaran et al., 2019) and hyperparameters optimization (Lorraine et al., 2020; Bao et al., 2021).
Revised: Bi-level optimization is widely used in various machine learning tasks, e.g., neural architecture search (Liu et al., 2018), meta learning (Rajeswaran et al., 2019) and hyperparameters optimization (Lorraine et al., 2020; Bao et al., 2021).

Original: One of the key challenges is to compute hypergradients with respect to the inner loop optimization (Liu et al., 2021).
Revised: One of the key challenges is to compute hypergradients with respect to the inner loop optimization (Liu et al., 2021).

Original: Some previous works (Maclaurin et al., 2015; Liu et al., 2018) use unrolled optimization or truncated unrolled optimization which is to differentiate the optimization process.
Revised: Some previous works (Maclaurin et al., 2015; Liu et al., 2018) use unrolled optimization or truncated unrolled optimization which is to differentiate the optimization process.

Original: However, this is not scalable if the inner loop optimization is a complicated process.
Revised: However, this is not scalable if the inner loop optimization is a complicated process.

Original: Some other works (Lorraine et al., 2020; Clarke et al., 2021; Rajeswaran et al., 2019) compute the hypergradient based on implicit function theorem.
Revised: Some other works (Lorraine et al., 2020; Clarke et al., 2021; Rajeswaran et al., 2019) compute the hypergradient based on implicit function theorem.

Original: This requires computing of the inverse hessian-vector product (inverse-HVP).
Revised: This requires computing of the inverse hessian-vector product (inverse-HVP).

Original: In (Lorraine et al., 2020) it proposes to use neumann series to approximate hypergradients.
Revised: In (Lorraine et al., 2020) it proposes to use neumann series to approximate hypergradients.

Original: Some works also use the conjugated gradient method (Pedregosa, 2016).
Revised: Some works also use the conjugated gradient method (Pedregosa, 2016).

Original: The approximation for implicit differentiation is crucial for the accuracy of hypergradients computation (Grazzi et al., 2020).
Revised: The approximation for implicit differentiation is crucial for the accuracy of hypergradients computation (Grazzi et al., 2020).

Original:  M ETHODOLOGY 3. P RELIMINARIES
Revised:  M ETHODOLOGY 3. P RELIMINARIES

Original: Let Y, U, V be three Banach spaces.
Revised: Let Y, U, V be three Banach spaces.

Original: The solution fields of PDEs are called state variables, i.e., y ∈ Y ad ⊂ Y , and functions or variables we can control are control variables, i.e., u ∈ U ad ⊂ U where Y ad , U ad are called admissible spaces, e.g. a subspace parameterized by neural networks or finite element basis.
Revised: The solution fields of PDEs are called state variables, i.e., y ∈ Y ad ⊂ Y , and functions or variables we can control are control variables, i.e., u ∈ U ad ⊂ U where Y ad , U ad are called admissible spaces, e.g. a subspace parameterized by neural networks or finite element basis.

Original: The PDE constrained optimization can be formulated as:
Revised: The PDE constrained optimization can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where J : Y × U → R is the objective function and e : Y × U
Revised: where J : Y × U → R is the objective function and e : Y × U

Original: → V are PDE constraints.
Revised: → V are PDE constraints.

Original: Usually, the PDE system of e ( y, u ) = 0 contains multiple equations and boundary/initial conditions as
Revised: Usually, the PDE system of e ( y, u ) = 0 contains multiple equations and boundary/initial conditions as

Original: [Equation]
Revised: [Equation]

Original: where F : Y × U → (Ω → R d 1 ) is the differential operator representing PDEs and B : Y × U →
Revised: where F : Y × U → (Ω → R d 1 ) is the differential operator representing PDEs and B : Y × U →

Original: (Ω → R d 2 ) represents boundary/initial conditions.
Revised: (Ω → R d 2 ) represents boundary/initial conditions.

Original: Existing methods based on regularization (e.g. the penalty method) solve the PDECO problem by minimizing the following objective (Lu et al., 2021):
Revised: Existing methods based on regularization (e.g. the penalty method) solve the PDECO problem by minimizing the following objective (Lu et al., 2021):

Original: [Equation]
Revised: [Equation]

Original: where the solutions y and control variables u are respectively parameterized by w ∈ R m and θ ∈ R n with w being the weights of PINNs.
Revised: where the solutions y and control variables u are respectively parameterized by w ∈ R m and θ ∈ R n with w being the weights of PINNs.

Original: λ i ∈
Revised: λ i ∈

Original: R
Revised: R

Original: d i are hyper-parameters balancing these terms of the optimization targets.
Revised: d i are hyper-parameters balancing these terms of the optimization targets.

Original: One main difficulty is that λ i are hard to set and the results are sensitive to them due to the complex nature of regularization terms (PDE constraints).
Revised: One main difficulty is that λ i are hard to set and the results are sensitive to them due to the complex nature of regularization terms (PDE constraints).

Original: In general, large λ i makes it difficult to optimize the objective J , while small λ i can result in a nonphysical solution of y w .
Revised: In general, large λ i makes it difficult to optimize the objective J , while small λ i can result in a nonphysical solution of y w .

Original: Besides, the optimal λ i may also vary with the different phases of training.
Revised: Besides, the optimal λ i may also vary with the different phases of training.

Original: 3. R EFORMULATING PDECO AS B I -L EVEL O PTIMIZATION
Revised: 3. R EFORMULATING PDECO AS B I -L EVEL O PTIMIZATION

Original: To resolve the above challenges of regularization based methods, we first present a new perspective that interprets PDECO as a bi-level optimization problem (Liu et al., 2021), which can facilitate a new solver consequentially.
Revised: To resolve the above challenges of regularization based methods, we first present a new perspective that interprets PDECO as a bi-level optimization problem (Liu et al., 2021), which can facilitate a new solver consequentially.

Original: Specifically, we solve the following bi-level optimization problem:
Revised: Specifically, we solve the following bi-level optimization problem:

Original: [Equation]
Revised: [Equation]

Original: In the outer loop, we only minimize J with respect to θ given the optimal value of w ∗ , and we optimize PDE losses using PINNs in the inner loop with the fixed θ .
Revised: In the outer loop, we only minimize J with respect to θ given the optimal value of w ∗ , and we optimize PDE losses using PINNs in the inner loop with the fixed θ .

Original: The objective E of the inner loop sub-problem is:
Revised: The objective E of the inner loop sub-problem is:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: By transforming the problem in Eq.
Revised: By transforming the problem in Eq.

Original: equation 1 into a bi-level optimization problem, the optimization of PDEs’ state variables and control variables can be decoupled, which relieves the headache of setting a proper hyper-parameter of λ i in Eq.
Revised: equation 1 into a bi-level optimization problem, the optimization of PDEs’ state variables and control variables can be decoupled, which relieves the headache of setting a proper hyper-parameter of λ i in Eq.

Original: equation 3.
Revised: equation 3.

Original: To solve this bi-level optimization problem, we design inner loops and outer loops that are executed iteratively.
Revised: To solve this bi-level optimization problem, we design inner loops and outer loops that are executed iteratively.

Original: As shown in Figure 1, we train PINNs in the inner loop with PDE losses in Eq.
Revised: As shown in Figure 1, we train PINNs in the inner loop with PDE losses in Eq.

Original: equation 5.
Revised: equation 5.

Original: In the outer loop, we compute the hypergradients based on Implicit Function Differentiation inspired  by Lorraine et al.
Revised: In the outer loop, we compute the hypergradients based on Implicit Function Differentiation inspired [Figure] by Lorraine et al.

Original: Along this line, we need to calculate a highly complex inverse HessianJacobian product.
Revised: Along this line, we need to calculate a highly complex inverse HessianJacobian product.

Original: To address this issue, we propose to use Broyden’s method which provides an[Figure] efficient approximation at a superlinear convergence speed (Rodomanov & Nesterov, 2021).
Revised: To address this issue, we propose to use Broyden’s method which provides an efficient approximation at a superlinear convergence speed (Rodomanov & Nesterov, 2021).

Original: In particular, we fine-tune the PINNs using the PDE losses in each iteration of the inner loop.
Revised: In particular, we fine-tune the PINNs using the PDE losses in each iteration of the inner loop.

Original: We further compute the gradients of J with respect to parameters of control variables θ in the outer loop, which is also recognized as hypergradients in bi-level optimization and detailed in the following section.
Revised: We further compute the gradients of J with respect to parameters of control variables θ in the outer loop, which is also recognized as hypergradients in bi-level optimization and detailed in the following section.

Original: 3. H YPERGRADIENTS C OMPUTATION USING B ROYDEN I TERATIONS
Revised: 3. H YPERGRADIENTS C OMPUTATION USING B ROYDEN I TERATIONS

Original: The upper-level objective J depends on the optimal w ∗ of the lower level optimization, i.e.,
Revised: The upper-level objective J depends on the optimal w ∗ of the lower level optimization, i.e.,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Thus we need to consider the Jacobian of w ∗ with respect to θ when calculating the hypergradients.
Revised: Thus we need to consider the Jacobian of w ∗ with respect to θ when calculating the hypergradients.

Original: Since w ∗ minimizes the lower level problem, we can derive ∂w ∗ ∂θ by applying Cauchy Implicit
Revised: Since w ∗ minimizes the lower level problem, we can derive ∂w ∗ ∂θ by applying Cauchy Implicit

Original: Function Theorem (Lorraine et al., 2020) as,
Revised: Function Theorem (Lorraine et al., 2020) as,

Original: Proposition 1 (Proof in Appendix B.1) .
Revised: Proposition 1 (Proof in Appendix B.1) .

Original: If for some ( w ′ , θ ′ ) , the lower level optimization is solved, i.e.
Revised: If for some ( w ′ , θ ′ ) , the lower level optimization is solved, i.e.

Original: ∂ E ∂w | ( w ′ ,θ ′ )
Revised: ∂ E ∂w | ( w ′ ,θ ′ )

Original: = 

Revised: =

Original: and ( ∂ 2 E ∂w∂w ⊤ ) − 1 is invertible, then there exists a function w ∗ =
Revised: and ( ∂ 2 E ∂w∂w ⊤ ) − 1 is invertible, then there exists a function w ∗ =

Original: w ∗ ( θ ) surrounding ( w ′ , θ ′ )
Revised: w ∗ ( θ ) surrounding ( w ′ , θ ′ )

Original: s.t.
Revised: s.t.

Original: ∂ E ∂w | ( w ∗ ( θ ′ ) ,θ ′ )
Revised: ∂ E ∂w | ( w ∗ ( θ ′ ) ,θ ′ )

Original: = 

Revised: =

Original: and we have:
Revised: and we have:

Original: [Equation]
Revised: [Equation]

Original: By Proposition 1, we could compute the hypergradients analytically as
Revised: By Proposition 1, we could compute the hypergradients analytically as

Original: [Equation]
Revised: [Equation]

Original: [Equation] However, computing the inverse of Hessian matrix, i.e., (cid:16) ∂ 2 E ∂w∂w ⊤ (cid:17) − 1 , is intractable for parameters of neural networks.
Revised: However, computing the inverse of Hessian matrix, i.e., (cid:16) ∂ 2 E ∂w∂w ⊤ (cid:17) − 1 , is intractable for parameters of neural networks.

Original: To handle this challenge, we can first compute z ∗ ≜
Revised: To handle this challenge, we can first compute z ∗ ≜

Original: ∂ J ∂w ∗ · (cid:16) ∂ 2 E ∂w∂w ⊤ (cid:17) − 1 which is also called the inverse vector-Hessian product.
Revised: ∂ J ∂w ∗ · (cid:16) ∂ 2 E ∂w∂w ⊤ (cid:17) − 1 which is also called the inverse vector-Hessian product.

Original: Previous works (Lorraine et al., 2020) use the
Revised: Previous works (Lorraine et al., 2020) use the

Original: Neumann series to approximate z ∗ with linear convergence speed.
Revised: Neumann series to approximate z ∗ with linear convergence speed.

Original: However, in practice this approach is usually a coarse and imprecise estimation of the hypergradients (Grazzi et al., 2020).
Revised: However, in practice this approach is usually a coarse and imprecise estimation of the hypergradients (Grazzi et al., 2020).

Original: Here we employ a more efficient and effective approach to compute the inverse vector-Hessian product which enjoys superlinear convergence speed (Rodomanov & Nesterov, 2021).
Revised: Here we employ a more efficient and effective approach to compute the inverse vector-Hessian product which enjoys superlinear convergence speed (Rodomanov & Nesterov, 2021).

Original: It equals to finding the root z ∗ for the following linear equation as
Revised: It equals to finding the root z ∗ for the following linear equation as

Original: [Equation]
Revised: [Equation]

Original: Note that for each evaluation of g w ( z ) , we only need to compute two Jacobian-vector products with a low computational cost, which does not need to create a giant instance of Hessian matrix.
Revised: Note that for each evaluation of g w ( z ) , we only need to compute two Jacobian-vector products with a low computational cost, which does not need to create a giant instance of Hessian matrix.

Original: Specifically, we use a low rank Broyden’s method (Broyden, 1965;
Revised: Specifically, we use a low rank Broyden’s method (Broyden, 1965;

Original: Rodomanov & Nesterov, 2021) to iteratively approximate the solution z ∗ .
Revised: Rodomanov & Nesterov, 2021) to iteratively approximate the solution z ∗ .

Original: In each iteration, we first approximate the inverse of
Revised: In each iteration, we first approximate the inverse of

Revised: [Equation]

Revised: as

Original: [Equation]
Revised: [Equation]

Original: We update u, v and z according to the following rules,
Revised: We update u, v and z according to the following rules,

Original: [Equation]
Revised: [Equation]

Original: where ∆ z i +

Revised: where ∆ z i +

Original: =
Revised: =

Original: z i +1 − z i , ∆ g
Revised: z i +1 − z i , ∆ g

Original: i +1 =
Revised: i +1 =

Original: g i +1 − g i , and α is the step size (usually set to 1 or using line search).
Revised: g i +1 − g i , and α is the step size (usually set to 1 or using line search).

Original: In summary, the inversion of the Hessian could be updated by
Revised: In summary, the inversion of the Hessian could be updated by

Original: [Equation]
Revised: [Equation]

Original: After m iterations, we use z m as the approximation of z ∗ .
Revised: After m iterations, we use z m as the approximation of z ∗ .

Original: We store u
Revised: We store u

Original: i , v i in low rank matrices which is displayed as two red thin matrices in Figure 1.
Revised: i , v i in low rank matrices which is displayed as two red thin matrices in Figure 1.

Original: Since we use a low rank approximation of the inverse of Hessian matrix, we no longer need to store the whole matrix and only need to record u k and v k ,
Revised: Since we use a low rank approximation of the inverse of Hessian matrix, we no longer need to store the whole matrix and only need to record u k and v k ,

Original: where k
Revised: where k

Original: = 1 . . .
Revised: = 1 . . .

Original: K and K is a tunable parameter depending on the memory limit.
Revised: K and K is a tunable parameter depending on the memory limit.

Original: We run
Revised: We run

Original: Broyden iterations until the maximum iteration is reached or the error is within a threshold.
Revised: Broyden iterations until the maximum iteration is reached or the error is within a threshold.

Original: Our method is named Bi-level Physics-informed Neural networks with Broyden’s hypergradients (BPN), of which the pseudo code is outlined in Algorithm 1.
Revised: Our method is named Bi-level Physics-informed Neural networks with Broyden’s hypergradients (BPN), of which the pseudo code is outlined in Algorithm 1.

Original: Given a PDECO problem, we initialize
Revised: Given a PDECO problem, we initialize

Original: PINNs with random parameters w 0 and a guess on control parameters θ 0 .
Revised: PINNs with random parameters w 0 and a guess on control parameters θ 0 .

Original: First, we train PINNs under initial control θ 0 for N w epochs as warm up (influence of hyperparameters will be discussed in Appendix C).
Revised: First, we train PINNs under initial control θ 0 for N w epochs as warm up (influence of hyperparameters will be discussed in Appendix C).

Original: Then, we compute the hypergradients for θ using Broyden’s method and update it with gradient descent.
Revised: Then, we compute the hypergradients for θ using Broyden’s method and update it with gradient descent.

Original: After that, we fine-tune PINNs under θ for N f epochs.
Revised: After that, we fine-tune PINNs under θ for N f epochs.

Original: These two steps are iteratively executed until convergence.
Revised: These two steps are iteratively executed until convergence.

Original: finetune
Revised: 8: end end

Original:  T HEORETICAL A NALYSIS AND D ISCUSSION
Revised:  T HEORETICAL A NALYSIS AND D ISCUSSION

Original: Error analysis of hypergradients approximation.
Revised: Error analysis of hypergradients approximation.

Original: We now present analyses of the hypergradients approximation using BPN.
Revised: We now present analyses of the hypergradients approximation using BPN.

Original: Due to the complexity of the problem and computational resources, it is extremely difficult to calculate the hypergradients exactly (Grazzi et al., 2020).
Revised: Due to the complexity of the problem and computational resources, it is extremely difficult to calculate the hypergradients exactly (Grazzi et al., 2020).

Original: Our BPN using Broyden’s method based on Implicit Function Differentiation is also an approximation for hypergradients.
Revised: Our BPN using Broyden’s method based on Implicit Function Differentiation is also an approximation for hypergradients.

Original: Then the approximation error or the convergence rate for hypergradients is one of the key factors that determine the performance.
Revised: Then the approximation error or the convergence rate for hypergradients is one of the key factors that determine the performance.

Original: Here we show that the approximation error could be bounded under mild assumptions using Broyden’s method.
Revised: Here we show that the approximation error could be bounded under mild assumptions using Broyden’s method.

Original: Inspired by the idea of (Grazzi et al., 2020; Ji et al., 2021), we prove that the approximation error could be decomposed into two parts, the former error term is induced by the inner loop optimization and the latter term is caused by the linear solver.
Revised: Inspired by the idea of (Grazzi et al., 2020; Ji et al., 2021), we prove that the approximation error could be decomposed into two parts, the former error term is induced by the inner loop optimization and the latter term is caused by the linear solver.

Original: Moreover, since our hypergradients are approximated by solving the linear equations using
Revised: Moreover, since our hypergradients are approximated by solving the linear equations using

Original: Broyden’s method, it enjoys a superlinear convergence rate (Rodomanov & Nesterov, 2021), which is superior compared with other methods such as Neumann Series (Lorraine et al., 2020).
Revised: Broyden’s method, it enjoys a superlinear convergence rate (Rodomanov & Nesterov, 2021), which is superior compared with other methods such as Neumann Series (Lorraine et al., 2020).

Original: Specifically, we state the assumptions to be satisfied as below.
Revised: Specifically, we state the assumptions to be satisfied as below.

Original: Assumption 
Revised: Assumption 1.

Original: [Equation]
Revised: We denote ∇ 1 = ∂∂w , ∇ 2 = ∂∂θ , d 1 = dd w , d 2 = dd θ for all ( w, θ ) ,

Original: • ∇ 21 E is invertible and ∥∇ [Equation] E∥ 
Revised: • ∇ 21 E is invertible and ∥∇ 21 E∥ 2 ≥ µ .

Original: [Equation] • ∇ 21 E and ∇ 1 ∇ 2 E are Lipschitz continuous with constants ρ and τ respectively.
Revised: • ∇ 21 E and ∇ 1 ∇ 2 E are Lipschitz continuous with constants ρ and τ respectively.

Original: • ∇ 1 J , ∇ 2 J , ∇ 1 E and ∇ 2 E are Lipschitz continuous with constant L .
Revised: • ∇ 1 J , ∇ 2 J , ∇ 1 E and ∇ 2 E are Lipschitz continuous with constant L .

Original: • The inner loop optimization is solved by an iterative algorithm that satisfies ∥ w t − w ∥ 2 ≤ p t
Revised: • The inner loop optimization is solved by an iterative algorithm that satisfies ∥ w t − w ∥ 2 ≤ p t

Original: ∥ w ∥ 2 and p t < 1 , p t → 0 as t → ∞ where t is the number of iterations.
Revised: ∥ w ∥ 2 and p t < 1 , p t → 0 as t → ∞ where t is the number of iterations.

Original: The above assumptions hold for most practical PDECO problems.
Revised: The above assumptions hold for most practical PDECO problems.

Original: For the linear equation in
Revised: For the linear equation in

Original: Eq.
Revised: Eq.

Original: equation 9, we use a m step Broyden’s method to solve it.
Revised: equation 9, we use a m step Broyden’s method to solve it.

Original: Denote κ = Lµ and d 2 J t , d 2 J are the computed and real hypergradients for θ respectively, we have the following Theorem.
Revised: Denote κ = Lµ and d 2 J t , d 2 J are the computed and real hypergradients for θ respectively, we have the following Theorem.

Original: Theorem 1 (Proof in Appendix B.2) .
Revised: Theorem 1 (Proof in Appendix B.2) .

Original: If Assumption 1 holds and the linear equation equation 9 is solved with Broyden’s method, there exists a constant M > 0 , such that the following inequality holds at iteration t ,[Equation]
Revised: If Assumption 1 holds and the linear equation equation 9 is solved with Broyden’s method, there exists a constant M > 0 , such that the following inequality holds at iteration t ,

Original: [Equation]
Revised: [Equation]

Original: The theorem above provides a rigorous theoretical guarantee that the hypergradients are close to real hypergradients when the inner loop optimization, i.e., PINNs’ training, could be solved.
Revised: The theorem above provides a rigorous theoretical guarantee that the hypergradients are close to real hypergradients when the inner loop optimization, i.e., PINNs’ training, could be solved.

Original: We observe that the convergence rate relies on the inner loop optimization p t which is usually locally linear (Jiao et al., 2022) for gradient descent and a superlinear term introduced by Broyden’s method (Rodomanov & Nesterov, 2021).
Revised: We observe that the convergence rate relies on the inner loop optimization p t which is usually locally linear (Jiao et al., 2022) for gradient descent and a superlinear term introduced by Broyden’s method (Rodomanov & Nesterov, 2021).

Original: The high convergence speed could nearly eliminate this term in a few iterations, so the main error is usually dominated by the first term which is also verified in our experiments.
Revised: The high convergence speed could nearly eliminate this term in a few iterations, so the main error is usually dominated by the first term which is also verified in our experiments.

Original: Connections to traditional adjoint methods.
Revised: Connections to traditional adjoint methods.

Original: Our BPN and adjoint methods (Herzog & Kunisch, 2010) share a similar idea of solving a transposed linear system in Eq.
Revised: Our BPN and adjoint methods (Herzog & Kunisch, 2010) share a similar idea of solving a transposed linear system in Eq.

Original: equation 9 to reduce the computational cost.
Revised: equation 9 to reduce the computational cost.

Original: But our bi-level optimization is a more general framework compared with constrained optimization in adjoint methods.
Revised: But our bi-level optimization is a more general framework compared with constrained optimization in adjoint methods.

Original: It allows more flexible design choices like replacing
Revised: It allows more flexible design choices like replacing

Original: FEM solvers with PINNs.
Revised: FEM solvers with PINNs.

Original: Additionally, Eq.
Revised: Additionally, Eq.

Original: (9) is different from the adjoint equation (Herzog & Kunisch, 2010).
Revised: (9) is different from the adjoint equation (Herzog & Kunisch, 2010).

Original: Our Eq (9) solves a system in the parameter space of neural networks but the traditional adjoint equation corresponds to a real field defined on admissible space U or meshes.
Revised: Our Eq (9) solves a system in the parameter space of neural networks but the traditional adjoint equation corresponds to a real field defined on admissible space U or meshes.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct extensive experiments on practical PDE constrained optimization problems to show the effectiveness of our method.
Revised: In this section, we conduct extensive experiments on practical PDE constrained optimization problems to show the effectiveness of our method.

Original: 5. E XPERIMENTAL S ETUP AND E VALUATION P ROTOCOL
Revised: 5. E XPERIMENTAL S ETUP AND E VALUATION P ROTOCOL

Original: Benchmark problems.
Revised: Benchmark problems.

Original: We choose several classic and challenging PDE constrained optimization problems containing both boundary, domain and time distributed optimization for both linear and non-linear equations.
Revised: We choose several classic and challenging PDE constrained optimization problems containing both boundary, domain and time distributed optimization for both linear and non-linear equations.

Original: More details of problems are listed in Appendix A.
Revised: More details of problems are listed in Appendix A.

Original: (1) Poisson’s Equations.
Revised: (1) Poisson’s Equations.

Original: We solve this classic problem on an irregular domain which could be viewed as a prototype of the heat exchanger (Diersch et al., 2011) that is widely used in multiple domains.
Revised: We solve this classic problem on an irregular domain which could be viewed as a prototype of the heat exchanger (Diersch et al., 2011) that is widely used in multiple domains.

Original: We denote this problem as Poisson’s 2d CG in Table 1.
Revised: We denote this problem as Poisson’s 2d CG in Table 1.

Original: (2) Heat Equations.
Revised: (2) Heat Equations.

Original: A time-distributed control problem for two dimensional heat equation similar to (Wang et al., 2021a) (denoted by Heat 2d).
Revised: A time-distributed control problem for two dimensional heat equation similar to (Wang et al., 2021a) (denoted by Heat 2d).

Original: (3) Burgers Equations.
Revised: (3) Burgers Equations.

Original: A time-distributed control problem for burgers equations (Burgers 1d) which are used in nonlinear acoustics and gas dynamics.
Revised: A time-distributed control problem for burgers equations (Burgers 1d) which are used in nonlinear acoustics and gas dynamics.

Original: (4) ∼ (6) Navier-Stokes Equations.
Revised: (4) ∼ (6) Navier-Stokes Equations.

Original: They are highly nonlinear equations characterizing flow of fluids.
Revised: They are highly nonlinear equations characterizing flow of fluids.

Original: We solve one shape optimization problem similar to (Wang et al., 2021a
Revised: We solve one shape optimization problem similar to (Wang et al., 2021a)

Original: ) (denoted by NS Shape in
Revised: (denoted by NS Shape in

Original: Table 1)

Original: and

Original: two boundary

Original:  control problems denoted by NS 2inlets and NS backstep in Table 1) ( Mowlavi & Nabi
Revised: Table 1) and two boundary control problems (denoted by NS 2inlets and NS backstep in Table 1) (Mowlavi & Nabi, 2021).

Original: Baselines.
Revised: Baselines.

Original: To demonstrate the effectiveness and superiority of our method, we compare it with several neural methods for PDECO recently proposed.
Revised: To demonstrate the effectiveness and superiority of our method, we compare it with several neural methods for PDECO recently proposed.

Original: (1) ∼ (2) hPINN (Lu et al., 2021):
Revised: (1) ∼ (2) hPINN (Lu et al., 2021):

Original: It trains PINNs with PDE losses and objective functions jointly with adaptive weights as a regularization term.
Revised: It trains PINNs with PDE losses and objective functions jointly with adaptive weights as a regularization term.

Original: There are two strategies, i.e. the penalty method and the augmented Lagragian method to update weights and we use hPINN-P and hPINN-A to denote them respectively.
Revised: There are two strategies, i e. the penalty method and the augmented Lagragian method to update weights and we use hPINN-P and hPINN-A to denote them respectively.

Original: (3) PINN-LS (Mowlavi & Nabi, 2021):
Revised: (3) PINN-LS (Mowlavi & Nabi, 2021):

Original: It also treats the objective function as regularization term with adaptive weights.
Revised: It also treats the objective function as regularization term with adaptive weights.

Original: And it uses line-search rules to find the maximum tolerated weights for the objective function.
Revised: And it uses line-search rules to find the maximum tolerated weights for the objective function.

Original: (4) PI-DeepONet (Wang et al., 2021a):
Revised: (4) PI-DeepONet (Wang et al., 2021a):

Original: As a two-stage method, it first trains a DeepONet using physics-informed losses and then optimizes the objective using the operator network.
Revised: As a two-stage method, it first trains a DeepONet using physics-informed losses and then optimizes the objective using the operator network.

Original: Apart from the methods mentioned above we also implement several other bi-level optimization algorithms and measure their performance on these PDECO tasks.
Revised: Apart from the methods mentioned above, we also implement several other bi-level optimization algorithms and measure their performance on these PDECO tasks.

Original: (5) Truncated Unrolled Differentiation(TRMD)
Revised: (5) Truncated Unrolled Differentiation(TRMD)

Original: (Shaban et al., 2019):
Revised: (Shaban et al., 2019):

Original: It assumes that the inner loop optimization is solved by several gradient updates and uses reverse mode backpropagation to calculate hypergradients.
Revised: It assumes that the inner loop optimization is solved by several gradient updates and uses reverse mode backpropagation to calculate hypergradients.

Original: (6) T 1 − T 2 (Luketina et al., 2016):
Revised: (6) T 1 − T 2 (Luketina et al., 2016):

Original: It computes the hypergradients using Implicit Differentiation Theorem and uses an identity matrix instead of calculating the exact inversion of Hessian matrix.
Revised: It computes the hypergradients using Implicit Differentiation Theorem and uses an identity matrix instead of calculating the exact inversion of Hessian matrix.

Original: (7) Neumann Series (Neumann) (Lorraine et al., 2020):
Revised: (7) Neumann Series (Neumann) (Lorraine et al., 2020):

Original: It proposed to use the Neumann series to iteratively approximates the inverse of Hessian and uses the vector-Jacobian product and the vector-Hessian product to avoid instantiation of Hessian matrices.
Revised: It proposed to use the Neumann series to iteratively approximates the inverse of Hessian and uses the vector-Jacobian product and the vector-Hessian product to avoid instantiation of Hessian matrices.

Original: Beyond these baselines, we also choose adjoint method (Herzog & Kunisch, 2010) which is a traditional PDECO method as a reference.
Revised: Beyond these baselines, we also choose adjoint method (Herzog & Kunisch, 2010) which is a traditional PDECO method as a reference.

Original: We run adjoint methods based on high fidelity finite element solvers (Mitusch et al., 2019) to calculate the reference solutions of these tasks.
Revised: We run adjoint methods based on high fidelity finite element solvers (Mitusch et al., 2019) to calculate the reference solutions of these tasks.

Original: For linear problems, it could be viewed as the ground truth solution.
Revised: For linear problems, it could be viewed as the ground truth solution.

Original: However, it is computationally expensive and sensitive to initial guesses since it solves the whole system in each iteration.
Revised: However, it is computationally expensive and sensitive to initial guesses since it solves the whole system in each iteration.

Original: Hyperparameters and Evaluation Protocol.
Revised: Hyperparameters and Evaluation Protocol.

Original: We use multi-layer perceptrons (MLP) with width from 64 ∼ 128 and depth from 3 ∼ 5 for different problems, and train them using Adam optimizer (Kingma & Ba, 2014) with a learning rate 10 − 3 .
Revised: We use multi-layer perceptrons (MLP) with width from 64 ∼ 128 and depth from 3 ∼ 5 for different problems, and train them using Adam optimizer (Kingma & Ba, 2014) with a learning rate 10 − 3 .

Original: Since the accuracy of PINNs of regularization based methods could not be guaranteed, we resort to finite element methods (FEM) to evaluate the performance.
Revised: Since the accuracy of PINNs of regularization based methods could not be guaranteed, we resort to finite element methods (FEM) to evaluate the performance.

Original: The evaluation metric is the objective function for each problem which is specified in Appendix A.
Revised: The evaluation metric is the objective function for each problem which is specified in Appendix A.

Original: We save and interpolate the control variables and solve the system using FEM and calculate the objective function numerically in every validation epoch.
Revised: We save and interpolate the control variables and solve the system using FEM and calculate the objective function numerically in every validation epoch.

Original: Other details and method-specific hyperparameters are reported in Appendix F.
Revised: Other details and method-specific hyperparameters are reported in Appendix F.

Original: We run experiments on a single 2080 Ti GPU.
Revised: We run experiments on a single 2080 Ti GPU.

Original: 5. M AIN R ESULTS
Revised: 5. M AIN R ESULTS

Original: Based on the experimental results for all PDECO tasks in Table 1, we have the following observations.
Revised: Based on the experimental results for all PDECO tasks in Table 1, we have the following observations.

Original: First, our BPN achieves state-of-the-art performance compared with all baselines.
Revised: First, our BPN achieves state-of-the-art performance compared with all baselines.

Original: It shows that bi-level optimization is an effective and scalable way for large-scale PDECO problems.
Revised: It shows that bi-level optimization is an effective and scalable way for large-scale PDECO problems.

Original: Second, we observe that our BPN reaches nearly the global optimal result for linear equations (Poisson’s and
Revised: Second, we observe that our BPN reaches nearly the global optimal result for linear equations (Poisson’s and

Original: Heat equation).
Revised: Heat equation).

Original: For these equations, the problem is convex and reference values provided by adjoint methods could be viewed as the ground truth.
Revised: For these equations, the problem is convex and reference values provided by adjoint methods could be viewed as the ground truth.

Original: For non-linear problems, the reference values provide locally optimal solutions and our BPN sometimes outperforms adjoint methods.
Revised: For non-linear problems, the reference values provide locally optimal solutions and our BPN sometimes outperforms adjoint methods.

Original: Third, the update strategy of loss weights is critical for regularization based methods like hPINN-P, hPINN-A and 
Revised: Third, the update strategy of loss weights is critical for regularization based methods like hPINN-P, hPINN-A and [Table]

Revised: [Table]

Original: PINN-LS which limits their performance without theoretical guidelines.
Revised: PINN-LS which limits their performance without theoretical guidelines.

Original: A possible reason is that the balance between the objective function and PDE losses is sensitive and unstable for complex PDEs.[Table] [Table]
Revised: A possible reason is that the balance between the objective function and PDE losses is sensitive and unstable for complex PDEs.

Original: 5. C OMPARISON WITH O THER B I -L EVEL O PTIMIZATION S TRATEGIES
Revised: 5. C OMPARISON WITH O THER B I -L EVEL O PTIMIZATION S TRATEGIES

Original: We list the performance of all bi-level optimization methods in Table 2.
Revised: We list the performance of all bi-level optimization methods in Table 2.

Original: First, we observe that our method using Broyden’s method for hypergradients computation achieves the best results compared with other methods.
Revised: First, we observe that our method using Broyden’s method for hypergradients computation achieves the best results compared with other methods.

Original: This is a natural consequence since our method gives the most accurate approximation for the response gradients.
Revised: This is a natural consequence since our method gives the most accurate approximation for the response gradients.

Original: Second, we could see that all bi-level optimization methods are effective on solving PDECO problems.
Revised: Second, we could see that all bi-level optimization methods are effective on solving PDECO problems.

Original: Third, the results are better if the hypergradients are more accurate in most cases.
Revised: Third, the results are better if the hypergradients are more accurate in most cases.

Original: For example, Neumann series and TRMD use better approximation for inverse Hessian and they perform better compared with T 1 − T 2 .
Revised: For example, Neumann series and TRMD use better approximation for inverse Hessian and they perform better compared with T 1 − T 2 .

Original: 5. E XPERIMENTS ON I TERATION E FFICIENCY
Revised: 5. E XPERIMENTS ON I TERATION E FFICIENCY

Original: To show that our method is computationally efficient, we conduct efficiency experiments by comparing the iterations required.
Revised: To show that our method is computationally efficient, we conduct efficiency experiments by comparing the iterations required.

Original: Since PI-DeepONet is a two-stage method and FEMs are not based on gradient descent, we only choose three regularization based methods as baselines.
Revised: Since PI-DeepONet is a two-stage method and FEMs are not based on gradient descent, we only choose three regularization based methods as baselines.

Original: Note that for our BPN we count total inner loop iterations for fairness.
Revised: Note that for our BPN we count total inner loop iterations for fairness.

Original: We plot values of objective functions J in Figure 22.
Revised: We plot values of objective functions J in Figure 22.

Original: We found that our BPN is much more efficient with a nearly linear convergence speed for linear problems.
Revised: We found that our BPN is much more efficient with a nearly linear convergence speed for linear problems.

Original: This shows that the hypergradients provided by bi-level optimization are more stable and effective compared with regularization based methods.
Revised: This shows that the hypergradients provided by bi-level optimization are more stable and effective compared with regularization based methods.

Original: Besides, we found that PINN-LS is more efficient compared with hPINNs.
Revised: Besides, we found that PINN-LS is more efficient compared with hPINNs.

Original: However, it is a common drawback that regularization based methods are not stable which might take a lot of effort to find suitable loss weights.
Revised: However, it is a common drawback that regularization based methods are not stable which might take a lot of effort to find suitable loss weights.

Original: 5. F IDELITY OF H YPERGRADIENTS AND A BLATION S TUDIES
Revised: 5. F IDELITY OF H YPERGRADIENTS AND A BLATION S TUDIES

Original: Fidelity of hypergradients compared with other methods.
Revised: Fidelity of hypergradients compared with other methods.

Original: In this experiment, we aim to show how accurate the hypergradients are for these bi-level optimization methods.
Revised: In this experiment, we aim to show how accurate the hypergradients are for these bi-level optimization methods.

Original: However, all tasks in our main experiments do not have a closed form solution for response gradients.
Revised: However, all tasks in our main experiments do not have a closed form solution for response gradients.

Original: Here we conduct this experiment on a toy task of one dimensional Poisson’s equation with an analytical solution in
Revised: Here we conduct this experiment on a toy task of one dimensional Poisson’s equation with an analytical solution in

Original: Appendix A. We first compare our method with several other bi-level optimization methods.
Revised: Appendix A. We first compare our method with several other bi-level optimization methods.

Original: We measure the cosine similarity, which is defined as x · y ∥ x ∥∥ y ∥ for any two vectors x, y , between computed hypergradients and analytical hypergradients and the results are shown in Figure 3.
Revised: We measure the cosine similarity, which is defined as x · y ∥ x ∥∥ y ∥ for any two vectors x, y , between computed hypergradients and analytical hypergradients and the results are shown in Figure 3.

Original: Note that the data are collected in the first 75 outer iterations since all methods converge fast on this problem.
Revised: Note that the data are collected in the first 75 outer iterations since all methods converge fast on this problem.

Original: The left part of the figure shows that Broyden’s method gives the most accurate hypergradients with similarity  close to 1.
Revised: The left part of the figure shows that Broyden’s method gives the most accurate hypergradients with similarity [Figure] close to 1.

Original: The other methods also provide a positive approximation that helps to optimize the control variable.
Revised: The other methods also provide a positive approximation that helps to optimize the control variable.

Original: However, the stability of Neumann series is not good and in some iterations it provides a negative estimation of hypergradients.
Revised: However, the stability of Neumann series is not good and in some iterations it provides a negative estimation of hypergradients.

Original: [Figure]

Original: Fidelity of hypergradients using different number of Broyden iterations.
Revised: Fidelity of hypergradients using different number of Broyden iterations.

Original: Since Broyden’s method is an iterative algorithm for solving the optimization problem, there is a trade-off between efficiency and performance.
Revised: Since Broyden’s method is an iterative algorithm for solving the optimization problem, there is a trade-off between efficiency and performance.

Original: Here we compare the fidelity of hypergradients using different numbers of iterations for Broyden’s method.
Revised: Here we compare the fidelity of hypergradients using different numbers of iterations for Broyden’s method.

Original: We also use box plots for this experiment and the results are in the right part of Figure 3.
Revised: We also use box plots for this experiment and the results are in the right part of Figure 3.

Original: We observe that as the number of Broyden iterations increases, the hypergradients become more accurate.
Revised: We observe that as the number of Broyden iterations increases, the hypergradients become more accurate.

Original: And the median cosine similarity of hypergradients is more than 0.9 even if we only use 8 iterations which shows the efficiency of Broyden’s method.
Revised: And the median cosine similarity of hypergradients is more than 0.9 even if we only use 8 iterations which shows the efficiency of Broyden’s method.

Original: We also find that the increment is minor after 16 iterations for this simple problem.
Revised: We also find that the increment is minor after 16 iterations for this simple problem.

Original: This shows that with the number of iterations increases, Broyden’s method converges fast and the rest of the error is dominated by the term caused by inexactness of inner loop optimization in Theorem 1 and Eq.
Revised: This shows that with the number of iterations increases, Broyden’s method converges fast and the rest of the error is dominated by the term caused by inexactness of inner loop optimization in Theorem 1 and Eq.

Original: equation 35.
Revised: equation 35.

Original: We conduct more ablation studies including the impact of hyperparameters in Broyden’s method and PINNs’ optimization in Appendix C, the comparison between our BPN and continuous adjoint method with adjoint PDE solved with PINNs in Appendix G, comparison of running time in Appendix
Revised: We conduct more ablation studies including the impact of hyperparameters in Broyden’s method and PINNs’ optimization in Appendix C, the comparison between our BPN and continuous adjoint method with adjoint PDE solved with PINNs in Appendix G, comparison of running time in Appendix

Original: G, and provide more visualization results for these PDECO problems in Appendix E.
Revised: G, and provide more visualization results for these PDECO problems in Appendix E.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we proposed a novel bi-level optimization framework named Bi-level Physics-informed
Revised: In this paper, we proposed a novel bi-level optimization framework named Bi-level Physics-informed

Original: Neural networks (BPN) for solving PDE constrained optimization problems.
Revised: Neural networks (BPN) for solving PDE constrained optimization problems.

Original: We used PINNs for solving inner loop optimization and Broyden’s method for computing hypergradients.
Revised: We used PINNs for solving inner loop optimization and Broyden’s method for computing hypergradients.

Original: Experiments on multiplereal scales and nonlinear PDECO tasks, including complex geometry or non-linear Naiver-Stokes equations, verified the effectiveness of our method.
Revised: Experiments on multiple PDECO tasks, including complex geometry or non-linear Naiver-Stokes equations, verified the effectiveness of our method.

Original: As for potential negative social impact, the interpretability of PINNs is not comparable with traditional numerical solvers and there is still room for improvement on more complex problems, which is left for future work.
Revised: As for potential negative social impact, the interpretability of PINNs is not comparable with traditional numerical solvers, which is left for future work.

Original:  R EPRODUCIBILITY STATEMENT
Revised:  R EPRODUCIBILITY STATEMENT

Original: We ensure the reproducibility of our paper from three aspects.
Revised: We ensure the reproducibility of our paper from three aspects.

Original: (1) Experiment: The implementation of our experiment is described in Sec. 5.1.
Revised: (1) Experiment: The implementation of our experiment is described in Sec. 5.1.

Original: Ablation study for our experiments is in Sec. 5.5.
Revised: Ablation study for our experiments is in Sec. 5.5.

Original: Further details are in Appendix A and Appendix C. (2) Code: Our code is included in supplementary materials.
Revised: Further details are in Appendix A and Appendix C. (2) Code: Our code is included in supplementary materials.

Original: (3) Theory and Method: A complete proof of the theoretical results described is provided in Appendix B.
Revised: (3) Theory and Method: A complete proof of the theoretical results described is provided in Appendix B.

Original:  E THICS STATEMENT
Revised:  E THICS STATEMENT

Original: PDE constrained optimization has a wide range of real-world applications in science and engineering including physics, fluids dynamics, heat engineering and aerospace industry, etc.
Revised: PDE constrained optimization has a wide range of real-world applications in science and engineering including physics, fluids dynamics, heat engineering and aerospace industry, etc.

Original: Our BPN is a general framework for PDECO and thus might accelerate the development of these fields.
Revised: Our BPN is a general framework for PDECO and thus might accelerate the development of these fields.

Original: The potential negative impact is that methods based on neural networks like PINNs lack theoretical guarantee and interpretability.
Revised: The potential negative impact is that methods based on neural networks like PINNs lack theoretical guarantee and interpretability.

Original: Accident investigation becomes more difficult if these unexplainable are deployed in risk-sensitive areas.
Revised: Accident investigation becomes more difficult if these unexplainable are deployed in risk-sensitive areas.

Original: A possible solution to mitigate this impact is to develop more explainable and robust methods with better theoretical guidance or corner case protection when they are applied to risk-sensitive areas.
Revised: A possible solution to mitigate this impact is to develop more explainable and robust methods with better theoretical guidance or corner case protection when they are applied to risk-sensitive areas.

Original: A CKNOWLEDGEMENT
Revised: A CKNOWLEDGEMENT

Original: This work was supported by the National Key Research and Development Program of China (2020AAA0106000, 2020AAA0106302, 2021YFB2701000), NSFC Projects (Nos. 62061136001, 62076147, U19B2034, U1811461, U19A2081, 61972224), BNRist (BNR2022RC01006), Tsinghua
Revised: This work was supported by the National Key Research and Development Program of China (2020AAA0106000, 2020AAA0106302, 2021YFB2701000), NSFC Projects (Nos. 62061136001, 62076147, U19B2034, U1811461, U19A2081, 61972224), BNRist (BNR2022RC01006), Tsinghua

Original: Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University.
Revised: Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University.

Revised: J.Z was also supported by the XPlorer Prize.

Original: A D ETAILS OF PDECO TASKS
Revised: A D ETAILS OF PDECO TASKS

Original: In this section, we give a mathematical description of these PDECO tasks in detail.
Revised: In this section, we give a mathematical description of these PDECO tasks in detail.

Original: A. 2 D P OISSON ’ S EQUATION WITH COMPLEX GEOMETRY (P OISSON ’ S 2 D CG)
Revised: A. 2 D P OISSON ’ S EQUATION WITH COMPLEX GEOMETRY (P OISSON ’ S 2 D CG)

Original: In this problem, we optimize a two-dimensional Poisson’s equation defined on a complex geometry which is a prototype of heat exchanger (Diersch et al., 2011).
Revised: In this problem, we optimize a two-dimensional Poisson’s equation defined on a complex geometry which is a prototype of heat exchanger (Diersch et al., 2011).

Original: We solve the equation on a rectangle area Ω r =
Revised: We solve the equation on a rectangle area Ω r =

Original: [ − 4 , 4] 2 minus four circles Ω ci = { ( x, y ) : ( x − x i ) 2 + ( y − y i )
Revised: [ − 4 , 4] 2 minus four circles Ω ci = { ( x, y ) : ( x − x i ) 2 + ( y − y i )

Original: 2 ⩽ r 2 i } where x i , y i , r
Revised: 2 ⩽ r 2 i } where x i , y i , r

Original: i are ± 2 . 4 , ± 2 . 4 , 0 . 8 .
Revised: i are ± 2 . 4 , ± 2 . 4 , 0 . 8 .

Original: We define a field variable u ( x
Revised: We define a field variable u ( x

Original: , y ) ∈ R on domain Ω = Ω r \ (cid:83) 4 i =1 Ω ci .
Revised: , y ) ∈ R on domain Ω = Ω r \ (cid:83) 4 i =1 Ω ci .

Original: The control source f ( x, y ) ∈ R is distributed on a circle χ
Revised: The control source f ( x, y ) ∈ R is distributed on a circle χ

Original: = { ( x, y ) : x 2 + y 2 ⩽ 1 .
Revised: = { ( x, y ) : x 2 + y 2 ⩽ 1 .

Original: Our goal is to solve the following optimization problem,
Revised: Our goal is to solve the following optimization problem,

Original: [Equation]
Revised: [Equation]

Original: We could visualize the geometry of this problem in Figure 4.
Revised: We could visualize the geometry of this problem in Figure 4.

Original: The source function is distributed in a circle at the origin with radius of 1.6 (not displayed here).
Revised: The source function is distributed in a circle at the origin with radius of 1.6 (not displayed here).

Original: A. T IME DISTRIBUTED CONTROL OF 2 D HEAT EQUATION (H EAT 2 D )
Revised: A. T IME DISTRIBUTED CONTROL OF 2 D HEAT EQUATION (H EAT 2 D )

Original: In this problem, our goal is to solve an optimal control task of a system governed by the heat equation.
Revised: In this problem, our goal is to solve an optimal control task of a system governed by the heat equation.

Original: The temperature field u ( x, y, t ) ∈ R is defined on a rectangle domain Ω = [0 , 1] 2 with time t ∈ [0 , 2] and the control signal is a function f
Revised: The temperature field u ( x, y, t ) ∈ R is defined on a rectangle domain Ω = [0 , 1] 2 with time t ∈ [0 , 2] and the control signal is a function f

Original: ( t ) ∈ R depends on time but does not depend on spatial coordinates x and y .
Revised: ( t ) ∈ R depends on time but does not depend on spatial coordinates x and y .

Original: Our goal is to make u close to a target function ˆ u ( x, y, t )
Revised: Our goal is to make u close to a target function ˆ u ( x, y, t )

Original: [Equation]
Revised: [Equation]

Original: The coefficient is ν = 0 .
Revised: The coefficient is ν = 0 .

Original: 001 and the target function is chosen as ˆ u = 32 x (1 − x ) y (1 − y ) sin πt .
Revised: 001 and the target function is chosen as ˆ u = 32 x (1 − x ) y (1 − y ) sin πt .

Original: We choose f ( t ) = 0 .
Revised: We choose f ( t ) = 0 .

Original: 1 as the initial guess for the problem.
Revised: 1 as the initial guess for the problem.

Original: The solution at timestep t = 2 . 0 is shown in
Revised: The solution at timestep t = 2 . 0 is shown in

Original: Figure 6.
Revised: Figure 6.

Original: A. T IME DISTRIBUTED CONTROL OF 1 D BURGERS EQUATION (B URGERS 1 D )
Revised: A. T IME DISTRIBUTED CONTROL OF 1 D BURGERS EQUATION (B URGERS 1 D )

Original: Burgers equation is a nonlinear PDE widely used in various areas like applied mathematics, fluid mechanics, gas dynamics, traffic flow and nonlinear acoustics.
Revised: Burgers equation is a nonlinear PDE widely used in various areas like applied mathematics, fluid mechanics, gas dynamics, traffic flow and nonlinear acoustics.

Original: Here we use viscous Burgers equations as an example which is a dissipative system.
Revised: Here we use viscous Burgers equations as an example which is a dissipative system.

Original: The field variable u ( x, t ) is defined on Ω =
Revised: The field variable u ( x, t ) is defined on Ω =

Original: [ − 1 , 1] and t ∈ [0 , 1] .
Revised: [ − 1 , 1] and t ∈ [0 , 1] .

Original: The control variable f ( t ) depends only on time.
Revised: The control variable f ( t ) depends only on time.

Original: We aim to solve the following optimization problem,
Revised: We aim to solve the following optimization problem,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The diffusion coefficient ν = 0 .
Revised: The diffusion coefficient ν = 0 .

Original: 01 and the target function is chosen as
Revised: 01 and the target function is chosen as

Original: [Equation]
Revised: [Equation]

Original: which is a symmetric wave.
Revised: which is a symmetric wave.

Original: We use f ( t ) = 

Revised: We use f ( t ) =

Original: as the initial guess for this problem.
Revised: as the initial guess for this problem.

Original: The visualization of the initial solution is shown in the Figure
Revised: The visualization of the initial solution is shown in the Figure

Original: A. I NLET FLOW CONTROL FOR 2 D STEADY N AIVER -S TOKES E QUATIONS (NS 2 INLETS )
Revised: A. I NLET FLOW CONTROL FOR 2 D STEADY N AIVER -S TOKES E QUATIONS (NS 2 INLETS )

Original: Naiver-Stokes equations are one of the most important equations in fluid mechanics, aerodynamics and applied mathematics which are notoriously difficult to solve due to the high non-linearity.
Revised: Naiver-Stokes equations are one of the most important equations in fluid mechanics, aerodynamics and applied mathematics which are notoriously difficult to solve due to the high non-linearity.

Original: In this problem, we solve NS equations in a pipeline and aim to find the best inlet flow distribution f ( y ) to make the outlet flow as uniform as possible.
Revised: In this problem, we solve NS equations in a pipeline and aim to find the best inlet flow distribution f ( y ) to make the outlet flow as uniform as possible.

Original: The flow velocity field is u = ( u, v ) and the pressure field is p and they are defined on a rectangle domain Ω
Revised: The flow velocity field is u = ( u, v ) and the pressure field is p and they are defined on a rectangle domain Ω

Original: = [0 , 1 . 5] × [0 , 1 . 0] .
Revised: = [0 , 1 . 5] × [0 , 1 . 0] .

Original: We have two inlets and two outlets and several walls for this domain,
Revised: We have two inlets and two outlets and several walls for this domain,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The whole problem is as follows,
Revised: The whole problem is as follows,

Original: [Equation]
Revised: [Equation]

Original: The target function is a parabolic function ˆ u ( x, y )
Revised: The target function is a parabolic function ˆ u ( x, y )

Original: = 4 y (1 − y ) and the velocity field on the second inlet and outlet is v 2 ( x ) = 

Revised: = 4 y (1 − y ) and the velocity field on the second inlet and outlet is v 2 ( x ) =

Original: ( x − 0 .
Revised: ( x − 0 .

Original: 5)(1 − x ) .
Revised: 5)(1 − x ) .

Original: The Reynold number is set to 100 in this problem.
Revised: The Reynold number is set to 100 in this problem.

Original: We initialize the solution f ( y ) the same with the target function, i.e. f ( y )
Revised: We initialize the solution f ( y ) the same with the target function, i.e. f ( y )

Original: = 4 y (1 − y ) .
Revised: = 4 y (1 − y ) .

Original: In this case, the outlet velocity and the whole velocity field is shown in the following pictures 8.
Revised: In this case, the outlet velocity and the whole velocity field is shown in the following pictures 8.

Original: A. D RAG MINIMIZATION OVER AN OBSTACLE OF NS EQUATIONS (NS SHAPE )
Revised: A. D RAG MINIMIZATION OVER AN OBSTACLE OF NS EQUATIONS (NS SHAPE )

Original: This problem is a shape optimization task that is to find the best shape of the obstacle that minimizes the drag forces from the flow.
Revised: This problem is a shape optimization task that is to find the best shape of the obstacle that minimizes the drag forces from the flow.

Original: The inlet is the left side of the area and the outlet is the right side of the area,
Revised: The inlet is the left side of the area and the outlet is the right side of the area,

Original: [Equation]
Revised: [Equation]

Original: The flow field is defined on a domain Ω = [0 , 8] 2 \ Ω o and Ω o is the obstacle.
Revised: The flow field is defined on a domain Ω = [0 , 8] 2 \ Ω o and Ω o is the obstacle.

Original: The shape of the obstacle is a ellipse parameterized by a parameter a ∈ [0 . 5 , 2 . 5] .
Revised: The shape of the obstacle is a ellipse parameterized by a parameter a ∈ [0 . 5 , 2 . 5] .

Original: The goal is to minimize the following objective.
Revised: The goal is to minimize the following objective.

Original: [Equation]
Revised: [Equation]

Original: A.
Revised: A.

Original: I NLET FLOW CONTROL FOR BACKSTEP FLOW (NS BACKSTEP )
Revised: I NLET FLOW CONTROL FOR BACKSTEP FLOW (NS BACKSTEP )

Original: The backstep flow is a classic example that might exhibits turbulent flow.
Revised: The backstep flow is a classic example that might exhibits turbulent flow.

Original: In this problem, we also aim to find the best inlet flow distribution f ( y ) to make the outlet flow symmetric and uniform.
Revised: In this problem, we also aim to find the best inlet flow distribution f ( y ) to make the outlet flow symmetric and uniform.

Original: The geometry of backstep could be viewed as a union of two rectangles Ω
Revised: The geometry of backstep could be viewed as a union of two rectangles Ω

Original: =
Revised: =

Original: [0 , 1] × [0 , 0 . 5] ∪ [1 , 2] × [0 , 1] .
Revised: [0 , 1] × [0 , 0 . 5] ∪ [1 , 2] × [0 , 1] .

Original: The inlet is the left side of the area and the outlet is the right side of the area,
Revised: The inlet is the left side of the area and the outlet is the right side of the area,

Original: [Equation]
Revised: [Equation]

Original: We are going to optimize the velocity fields of the outlet,
Revised: We are going to optimize the velocity fields of the outlet,

Original: [Equation]
Revised: [Equation]

Original: The target velocity field is ˆ u ( x, y )
Revised: The target velocity field is ˆ u ( x, y )

Original: = 3 y (1 − y )
Revised: = 3 y (1 − y )

Original: and we initialize the inlet velocity f ( y ) to be 8 y (0 .
Revised: and we initialize the inlet velocity f ( y ) to be 8 y (0 .

Original: 5 − y ) .
Revised: 5 − y ) .

Original: The Reynold number is also set to 100 .
Revised: The Reynold number is also set to 100 .

Original: The visualization of the geometry of this problem is shown in Figure 9.
Revised: The visualization of the geometry of this problem is shown in Figure 9.

Original: A solution of the initial guess is shown in Figure 10.
Revised: A solution of the initial guess is shown in Figure 10.

Original: A. D ISTRIBUTED FORCE CONTROL OF 2 D N AVIER -S TOKES EQUATIONS
Revised: A. D ISTRIBUTED FORCE CONTROL OF 2 D N AVIER -S TOKES EQUATIONS

Original: In previous examples of Naiver-Stokes equations, the control variable is either 1d function (inlet flow) or vector (shape parameters).
Revised: In previous examples of Naiver-Stokes equations, the control variable is either 1d function (inlet flow) or vector (shape parameters).

Original: In this problem, we consider a more challenging setting to control a system governed by NS equation by optimizing distributed force (or source) terms over a domain.
Revised: In this problem, we consider a more challenging setting to control a system governed by NS equation by optimizing distributed force (or source) terms over a domain.

Original: The dimension of the control variable (function) in this problem is 2.
Revised: The dimension of the control variable (function) in this problem is 2.

Original: The geometry and boundary condition is the same with NS2inlet problem.
Revised: The geometry and boundary condition is the same with NS2inlet problem.

Original: The flow velocity field is u = ( u, v ) and the pressure field is p and they are defined on a rectangle domain Ω
Revised: The flow velocity field is u = ( u, v ) and the pressure field is p and they are defined on a rectangle domain Ω

Original: = [0 , 1 . 5] × [0 , 1 . 0] .
Revised: = [0 , 1 . 5] × [0 , 1 . 0] .

Original: We have two inlets and two outlets and several walls for this domain,
Revised: We have two inlets and two outlets and several walls for this domain,

Original: [Equation]
Revised: [Equation]

Original: The whole problem is as follows,
Revised: The whole problem is as follows,

Original: [Equation]
Revised: [Equation]

Original: A. P OISSION ’ S 1 D T OY PROBLEM FOR EVALUATING HYPERGRADIENTS SIMILARITY .
Revised: A. P OISSION ’ S 1 D T OY PROBLEM FOR EVALUATING HYPERGRADIENTS SIMILARITY .

Original: This is a simple problem with an analytic solution which is a boundary control for one-dimensional
Revised: This is a simple problem with an analytic solution which is a boundary control for one-dimensional

Original: Poission’s equation.
Revised: Poission’s equation.

Original: We use this problem for evaluating the fidelity of hypergradients,
Revised: We use this problem for evaluating the fidelity of hypergradients,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The exact solution of this Poission’s equation is,
Revised: The exact solution of this Poission’s equation is,

Original: [Equation]
Revised: [Equation]

Original: And the objective function J is a quadratic form about θ 

Revised: And the objective function J is a quadratic form about θ

Original: and θ 1 ,
Revised: and θ 1 ,

Original: [Equation]
Revised: [Equation]

Original: The gradient for this function is
Revised: The gradient for this function is

Original: [Equation]
Revised: [Equation]

Original: Thus we could compare the approximated hypergradients with the real gradients to study the effectiveness of different bi-level optimization strategies.
Revised: Thus we could compare the approximated hypergradients with the real gradients to study the effectiveness of different bi-level optimization strategies.

Original: The optimal solution of this problem is
Revised: The optimal solution of this problem is

Original: [Equation]
Revised: [Equation]

Original: and the corresponding objective function J ∗ = 0 .
Revised: and the corresponding objective function J ∗ = 0 .

Original: B P ROOF OF T HEOREMS
Revised: B P ROOF OF T HEOREMS

Original: In this section, we prove theorems in the main text.
Revised: In this section, we prove theorems in the main text.

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: Theorem 1. 

Revised: Theorem 1.

Original: If for some ( w ′ , θ ′ ) , the lower level optimization is solved, i.e. ∂ E ∂w | ( w ′ ,θ ′ )
Revised: If for some ( w ′ , θ ′ ) , the lower level optimization is solved, i.e. ∂ E ∂w | ( w ′ ,θ ′ )

Original: = 

Revised: =

Original: and regularity conditions are satisfied, then there exists a function w ∗ =
Revised: and regularity conditions are satisfied, then there exists a function w ∗ =

Original: w ∗ ( θ ) surrounding ( w ′ , θ ′ )
Revised: w ∗ ( θ ) surrounding ( w ′ , θ ′ )

Original: s.t.
Revised: s.t.

Original: ∂ E ∂w | ( w ∗ ( θ ′ ) ,θ ′ )
Revised: ∂ E ∂w | ( w ∗ ( θ ′ ) ,θ ′ )

Original: = 

Revised: =

Original: and we have:
Revised: and we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Cauchy’s Implicit Function Theorem, if f ( w ′ , θ ′ )
Revised: By Cauchy’s Implicit Function Theorem, if f ( w ′ , θ ′ )

Original: = 0 is satisfied and its Jacobian ∂f∂w | ( w ′ ,θ ′ ) is invertible , then there exists an open set U including θ ′ and a continuous differentiable function w ∗ satisfying that w ∗ ( θ ′ )
Revised: = 0 is satisfied and its Jacobian ∂f∂w | ( w ′ ,θ ′ ) is invertible , then there exists an open set U including θ ′ and a continuous differentiable function w ∗ satisfying that w ∗ ( θ ′ )

Original: = w ′ and f ( w ∗ ( θ ) , θ ) = 0 , ∀ θ ∈ U .
Revised: = w ′ and f ( w ∗ ( θ ) , θ ) = 0 , ∀ θ ∈ U .

Original: Moreover, the partial derivatives could be computed by
Revised: Moreover, the partial derivatives could be computed by

Original: [Equation]
Revised: [Equation]

Original: Here we take f ( w ′ , θ ′ ) =
Revised: Here we take f ( w ′ , θ ′ ) =

Original: ∂ E ∂w ⊤ ( w ′ , θ ′ ) and we get the following result
Revised: ∂ E ∂w ⊤ ( w ′ , θ ′ ) and we get the following result

Original: [Equation]
Revised: [Equation]

Original: Thus we have proven the theorem.
Revised: Thus we have proven the theorem.

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: If assumptions 1 hold and the linear equation 9 is solved with Broyden method, there exists a constant M > 0 , such that the following inequality holds,
Revised: If assumptions 1 hold and the linear equation 9 is solved with Broyden method, there exists a constant M > 0 , such that the following inequality holds,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, since we assume the inner loop optimization is a contraction map,
Revised: First, since we assume the inner loop optimization is a contraction map,

Original: [Equation]
Revised: [Equation]

Original: Then for all t , the weights w t are in a compact set W = { w ′ :
Revised: Then for all t , the weights w t are in a compact set W = { w ′ :

Original: ∥ w ′ ∥
Revised: ∥ w ′ ∥

Original: ⩽ 2 ∥ w ∥} .
Revised: ⩽ 2 ∥ w ∥} .

Original: Note that J is continous and differentiable, then there exists a constant M and J is Lipschitz continous with M , i.e. for all w t ∈ W we have
Revised: Note that J is continous and differentiable, then there exists a constant M and J is Lipschitz continous with M , i.e. for all w t ∈ W we have

Original: [Equation]
Revised: [Equation]

Original: We define the following notations to simplify the proof,
Revised: We define the following notations to simplify the proof,

Original: [Equation]
Revised: [Equation]

Original: and z t,m is the approximation after m iterations of Broyden method.
Revised: and z t,m is the approximation after m iterations of Broyden method.

Original: The error of hypergradients are as follows,
Revised: The error of hypergradients are as follows,

Original: [Equation]
Revised: [Equation]

Original: Then if we could bound these terms respectively, the total error could be controlled.
Revised: Then if we could bound these terms respectively, the total error could be controlled.

Original: For the first and second term, it is caused by the inexactness of the inner loop optimization, since ∇ 2 J is Lipschitz with constant L
Revised: For the first and second term, it is caused by the inexactness of the inner loop optimization, since ∇ 2 J is Lipschitz with constant L

Original: [Equation]
Revised: [Equation]

Original: And ∇ 1 E , ∇ 2 E is Lipschitz continuous with constant L ,
Revised: And ∇ 1 E , ∇ 2 E is Lipschitz continuous with constant L ,

Original: [Equation]
Revised: [Equation]

Original: And these two terms could be bounded by,
Revised: And these two terms could be bounded by,

Original: [Equation]
Revised: [Equation]

Original: Thus we have,
Revised: Thus we have,

Original: [Equation]
Revised: [Equation]

Original: The last term is mixed term caused by both the approximation error when solving the linear system and the inexactness of the inner loop optimization,
Revised: The last term is mixed term caused by both the approximation error when solving the linear system and the inexactness of the inner loop optimization,

Original: [Equation]
Revised: [Equation]

Original: since Broyden method enjoys a superlinear convergence rate (Rodomanov & Nesterov, 2021)
Revised: since Broyden method enjoys a superlinear convergence rate (Rodomanov & Nesterov, 2021)

Original: [Equation]
Revised: [Equation]

Original: we have the following convergence bound,
Revised: we have the following convergence bound,

Original: [Equation]
Revised: [Equation]

Original: Rank
Revised: Rank

Original: Iters     0.
Revised: Iters     0.

Original: Thus we sum these error terms together and get,
Revised: Thus we sum these error terms together and get,

Original: [Equation]
Revised: [Equation]

Original: Thus we have proven the main theorem for error analysis of the hypergradients computation using
Revised: Thus we have proven the main theorem for error analysis of the hypergradients computation using

Original: Broyden method.
Revised: Broyden method.

Original: C S UPPLEMENTARY ABLATION EXPERIMENTS AND DETAILS .
Revised: C S UPPLEMENTARY ABLATION EXPERIMENTS AND DETAILS .

Original: In this section, we provide the results of some ablation studies about our BPN and baselines.
Revised: In this section, we provide the results of some ablation studies about our BPN and baselines.

Original: We choose Heat 2d and NS backstep as examples for these experiments.
Revised: We choose Heat 2d and NS backstep as examples for these experiments.

Original: C. P ERFORMANCE WITH THE DIFFERENT NUMBER OF B ROYDEN ITERATIONS AND MAXIMUM MEMORY STEPS .
Revised: C. P ERFORMANCE WITH THE DIFFERENT NUMBER OF B ROYDEN ITERATIONS AND MAXIMUM MEMORY STEPS .

Original: In this subsection, we investigate the influence of different iterations and memory limit of Broyden method (Broyden, 1965).
Revised: In this subsection, we investigate the influence of different iterations and memory limit of Broyden method (Broyden, 1965).

Original: We conduct experiments on Heat 2d problem and NS backstep problem.
Revised: We conduct experiments on Heat 2d problem and NS backstep problem.

Original: Note that the maximum memory limit should be less than the maximum number of iterations.
Revised: Note that the maximum memory limit should be less than the maximum number of iterations.

Original: The results of Heat 2d equation are shown in Table 3.
Revised: The results of Heat 2d equation are shown in Table 3.

Original: We see that roughly the performance improves (the objective function decreases) with the increase of maximum memory limit and maximum iterations.
Revised: We see that roughly the performance improves (the objective function decreases) with the increase of maximum memory limit and maximum iterations.

Original: In this problem, the rank for approximating the inverse Hessian more than 16 is able to handle this problem.
Revised: In this problem, the rank for approximating the inverse Hessian more than 16 is able to handle this problem.

Original: There is no significant gain if we use more than 32 iterations.
Revised: There is no significant gain if we use more than 32 iterations.

Original: The results of NS backstep problem are shown in Table 4.
Revised: The results of NS backstep problem are shown in Table 4.

Original: The trend is similar to Heat 2d problem and roughly the performance improves with the increase of Broyden iterations.
Revised: The trend is similar to Heat 2d problem and roughly the performance improves with the increase of Broyden iterations.

Original: However, we see that this problem is much more difficult and the performance deteriorates if we only use 8 memory steps even though we use many iterations.
Revised: However, we see that this problem is much more difficult and the performance deteriorates if we only use 8 memory steps even though we use many iterations.

Original: C. P ERFORMANCE WITH DIFFERENT NUMBERS OF FINETUNING EPOCHS .
Revised: C. P ERFORMANCE WITH DIFFERENT NUMBERS OF FINETUNING EPOCHS .

Original: In this subsection, we investigate the influence of different finetune epochs for the inner loop of
Revised: In this subsection, we investigate the influence of different finetune epochs for the inner loop of

Original: PINNs, i.e. N f in Algorithm 1.
Revised: PINNs, i.e. N f in Algorithm 1.

Original: We choose Heat 2d problem for this experiment and the results are listed in Table 5.
Revised: We choose Heat 2d problem for this experiment and the results are listed in Table 5.

Original: We also plot the results in Figure 11.
Revised: We also plot the results in Figure 11.

Original: We see that this hyperparameter has a considerable influence on the convergence speed from about 0.2k ∼ 0.6k.
Revised: We see that this hyperparameter has a considerable influence on the convergence speed from about 0.2k ∼ 0.6k.

Original: However, its impact on the final performance is minor within a broad range.
Revised: However, its impact on the final performance is minor within a broad range.

Original: We also observe that a small N f might deteriorate final performance.
Revised: We also observe that a small N f might deteriorate final performance.

Original: Thus choosing a moderate N f is efficient and effective.
Revised: Thus choosing a moderate N f is efficient and effective.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C. P ERFORMANCE WITH DIFFERENT NUMBERS OF WARMUP EPOCHS .
Revised: C. P ERFORMANCE WITH DIFFERENT NUMBERS OF WARMUP EPOCHS .

Original: In this subsection, we investigate the influence of different finetune epochs for the inner loop of
Revised: In this subsection, we investigate the influence of different finetune epochs for the inner loop of

Original: PINNs, i.e. N w in Algorithm 1.
Revised: PINNs, i.e. N w in Algorithm 1.

Original: We also choose Heat 2d problem for this experiment.
Revised: We also choose Heat 2d problem for this experiment.

Original: The results with different N w are shown in Table 6.
Revised: The results with different N w are shown in Table 6.

Original: The observations for this experiment are similar to the ablation study on N f .
Revised: The observations for this experiment are similar to the ablation study on N f .

Original: We see that convergence speed might be different for different optimization schedules of PINNs in the inner loop, but the final performance does not change a lot.
Revised: We see that convergence speed might be different for different optimization schedules of PINNs in the inner loop, but the final performance does not change a lot.

Original: In summary,
Revised: In summary,

Original: BPN is robust to the choice of these two parameters on Heat 2d problem.
Revised: BPN is robust to the choice of these two parameters on Heat 2d problem.

Original: D BPN FOR S HAPE O PTIMIZATION .
Revised: D BPN FOR S HAPE O PTIMIZATION .

Original: In this subsection, we show that BPN could also be used for a special class of PDECO tasks which are called shape optimization.
Revised: In this subsection, we show that BPN could also be used for a special class of PDECO tasks which are called shape optimization.

Original: The main challenge for this is the PDE losses E and optimization target J does not directly depend on the shape parameters, but they depend on the collocations points when we sample from the region in practice.
Revised: The main challenge for this is the PDE losses E and optimization target J does not directly depend on the shape parameters, but they depend on the collocations points when we sample from the region in practice.

Original: Thus calculating gradients for shape parameters is a challenge for shape optimization tasks.
Revised: Thus calculating gradients for shape parameters is a challenge for shape optimization tasks.

Original: Another viewpoint is that the shape parameters contribute to the integral domain but not the objective function.
Revised: Another viewpoint is that the shape parameters contribute to the integral domain but not the objective function.

Original: This means that the gradients with respect to θ will not appear in the computational graph.
Revised: This means that the gradients with respect to θ will not appear in the computational graph.

Original: Specifically, if J is also an integral over the domain, we have
Revised: Specifically, if J is also an integral over the domain, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: And the PDE losses could be written as (here we assume the control parameters only influence the geometric shape Ω( θ ) ),
Revised: And the PDE losses could be written as (here we assume the control parameters only influence the geometric shape Ω( θ ) ),

Original: [Equation]
Revised: [Equation]

Original: We need to compute the gradients of ∂ J ∂θ , ∂
Revised: We need to compute the gradients of ∂ J ∂θ , ∂

Original: [Equation]
Revised: [Equation]

Original: ∂w∂θ T which rely on the gradients with respect to θ that is not contained in the computational graph.
Revised: ∂w∂θ T which rely on the gradients with respect to θ that is not contained in the computational graph.

Original: Inspired by the idea from traditional shape optimization, we have the following rules,
Revised: Inspired by the idea from traditional shape optimization, we have the following rules,

Original: [Equation]
Revised: [Equation]

Original: where n is normal direction for boundary ∂ Ω( θ ) .
Revised: where n is normal direction for boundary ∂ Ω( θ ) .

Original: For the PDE loss, we have
Revised: For the PDE loss, we have

Original: [Equation]
Revised: [Equation]

Original: where κ is the culvature of the boundary shape for two dimensional problems.
Revised: where κ is the culvature of the boundary shape for two dimensional problems.

Original: However, in order to compute the hypergradients, we need to compute the following vector-Hessian product,
Revised: However, in order to compute the hypergradients, we need to compute the following vector-Hessian product,

Original: [Equation]
Revised: [Equation]

Original: We need to compute the directional derivatives for a vector function ∂ E ∂θ T .
Revised: We need to compute the directional derivatives for a vector function ∂ E ∂θ T .

Original: Direct computation of this term requires n times of backpropagation which is extremely inefficient.
Revised: Direct computation of this term requires n times of backpropagation which is extremely inefficient.

Original: Here we use a small trick that only requires a constant computational cost by transforming this into a vector-Jacobian product,
Revised: Here we use a small trick that only requires a constant computational cost by transforming this into a vector-Jacobian product,

Original: [Equation]
Revised: [Equation]

Original: · ∂w ∗ ∂θ = ∂ ∂v (cid:18) z · (cid:18) ∂ ∂w (cid:18) v · ∂ E ∂θ (cid:19)(cid:19)(cid:19) .
Revised: · ∂w ∗ ∂θ = ∂ ∂v (cid:18) z · (cid:18) ∂ ∂w (cid:18) v · ∂ E ∂θ (cid:19)(cid:19)(cid:19) .

Original: We could see that ∂∂w (cid:0) v · ∂ E ∂θ (cid:1) could be computed using a single vector-Jacobian product and ∂∂v (cid:0) z · (cid:0) ∂∂w (cid:0) v · ∂ E ∂θ (cid:1)(cid:1)(cid:1) could then be computed by two vector-Jacobian products.
Revised: We could see that ∂∂w (cid:0) v · ∂ E ∂θ (cid:1) could be computed using a single vector-Jacobian product and ∂∂v (cid:0) z · (cid:0) ∂∂w (cid:0) v · ∂ E ∂θ (cid:1)(cid:1)(cid:1) could then be computed by two vector-Jacobian products.

Original: This trick is easily implemented and efficient in practice.
Revised: This trick is easily implemented and efficient in practice.

Original: E V ISUALIZATION OF O PTIMIZATION R ESULTS
Revised: E V ISUALIZATION OF O PTIMIZATION R ESULTS

Original: In this subsection, we show the results of the PDECO problems.
Revised: In this subsection, we show the results of the PDECO problems.

Original: We also compare our optimized result with the reference results solved by high fidelity adjoint solvers based on FEMs.
Revised: We also compare our optimized result with the reference results solved by high fidelity adjoint solvers based on FEMs.

Original: To make sure the evaluation is fair and accurate, we discretize the control variables into meshes.
Revised: To make sure the evaluation is fair and accurate, we discretize the control variables into meshes.

Original: Then we load them into the FEM solver and interpolation the control variable using an interpolation curve or surface.
Revised: Then we load them into the FEM solver and interpolation the control variable using an interpolation curve or surface.

Original: We solve these systems using high-fidelity FEM solvers.
Revised: We solve these systems using high-fidelity FEM solvers.

Original: E. P OISSION ’ S 2 D E QUATION WITH COMPLEX GEOMETRY .
Revised: E. P OISSION ’ S 2 D E QUATION WITH COMPLEX GEOMETRY .

Original: We show the results of the control variable of Poisson’s 2d CG problem solved using our BPN and adjoint method (reference) in Figure 12.
Revised: We show the results of the control variable of Poisson’s 2d CG problem solved using our BPN and adjoint method (reference) in Figure 12.

Original: We plot the distance between final solution y and the target function in the Figure 14.
Revised: We plot the distance between final solution y and the target function in the Figure 14.

Original: We observe that the optimization result using BPN looks different with using adjoint solvers.
Revised: We observe that the optimization result using BPN looks different with using adjoint solvers.

Original: Although the objective function is nearly the same in this problem, the control variables are different.
Revised: Although the objective function is nearly the same in this problem, the control variables are different.

Original: The results of BPN are not symmetric and is smoother compared with adjoint solvers.
Revised: The results of BPN are not symmetric and is smoother compared with adjoint solvers.

Original: A possible reason is that randomness in training PINNs leads to a breaking of symmetry.
Revised: A possible reason is that randomness in training PINNs leads to a breaking of symmetry.

Original: And the inductive biases of representing control variables using neural networks tend to find smoother solutions.
Revised: And the inductive biases of representing control variables using neural networks tend to find smoother solutions.

Original: E. T EMPORAL CONTROL FOR HEAT EQUATIONS .
Revised: E. T EMPORAL CONTROL FOR HEAT EQUATIONS .

Original: We show the results of the control variable of Heat 2d problem solved using our BPN and adjoint method (reference) in Figure 15.
Revised: We show the results of the control variable of Heat 2d problem solved using our BPN and adjoint method (reference) in Figure 15.

Original: The solution field of u at time t = 1 . 0 is shown in Figure 16.
Revised: The solution field of u at time t = 1 . 0 is shown in Figure 16.

Original: And the distance to the target function is shown in Figure 17.
Revised: And the distance to the target function is shown in Figure 17.

Original: E. N ON - LINEAR N AVIER -S TOKES EQUATIONS .
Revised: E. N ON - LINEAR N AVIER -S TOKES EQUATIONS .

Original: Here we only choose the NS backstep problem as an example for visualization of PDECO on NavierStokes equations.
Revised: Here we only choose the NS backstep problem as an example for visualization of PDECO on NavierStokes equations.

Original: This problem is more complex due to the non-linearity of Navier-Stokes equations.
Revised: This problem is more complex due to the non-linearity of Navier-Stokes equations.

Original: Both the results of our BPN and adjoint solvers might not be the ground truth solution.
Revised: Both the results of our BPN and adjoint solvers might not be the ground truth solution.

Original: We visualize the velocity field u, v of the solution in Figure 18 and Figure 19.
Revised: We visualize the velocity field u, v of the solution in Figure 18 and Figure 19.

Original: The pressure field is shown in 20.
Revised: The pressure field is shown in 20.

Original: We see that both methods allocate a higher velocity to the lower part of the inlet.
Revised: We see that both methods allocate a higher velocity to the lower part of the inlet.

Original: But the details are different since there are many degrees of freedom for this problem.
Revised: But the details are different since there are many degrees of freedom for this problem.

Original: It is also possible there are many local minima for this problem.
Revised: It is also possible there are many local minima for this problem.

Original: F O THER DETAILS AND HYPERPARAMETERS FOR BASELINES .
Revised: F O THER DETAILS AND HYPERPARAMETERS FOR BASELINES .

Original: In this section, we specify some details and hyperparameters for baselines
Revised: In this section, we specify some details and hyperparameters for baselines

Original: . F. H YPERPARAMETERS FOR REGULARIZATION BASED METHODS .
Revised: . F. H YPERPARAMETERS FOR REGULARIZATION BASED METHODS .

Original: For these methods, we need to adjust the weights of the Lagrangian multipliers after several finetune epochs of PINNs.
Revised: For these methods, we need to adjust the weights of the Lagrangian multipliers after several finetune epochs of PINNs.

Original: We choose the finetune steps from { 1000 , 2000 , 4000 , 6000 } .
Revised: We choose the finetune steps from { 1000 , 2000 , 4000 , 6000 } .

Original: For hPINN, the weights are set to 0.001 for initialization and they are amplified by the ratio of 2 after each finetuning stage.
Revised: For hPINN, the weights are set to 0.001 for initialization and they are amplified by the ratio of 2 after each finetuning stage.

Original: The maximum weights are 1e3.
Revised: The maximum weights are 1e3.

Original: For the line search strategy of PINN-LS, the maximum loss threshold is selected from { 10 − 4 , 10 − 3 , 10 − 2 } for different problems.
Revised: For the line search strategy of PINN-LS, the maximum loss threshold is selected from { 10 − 4 , 10 − 3 , 10 − 2 } for different problems.

Original: F. H YPERPARAMETERS FOR PI-D EEP ON ET .
Revised: F. H YPERPARAMETERS FOR PI-D EEP ON ET .

Original: We use modified MLPs as network architectures for PI-DeepONets proposed by (Wang et al., 2021b;a).
Revised: We use modified MLPs as network architectures for PI-DeepONets proposed by (Wang et al., 2021b;a).

Original: The number of layers is selected from 3 ∼ 8 and the layer width is selected from { 32 , 64 , 128 , 256 } .
Revised: The number of layers is selected from 3 ∼ 8 and the layer width is selected from { 32 , 64 , 128 , 256 } .

Original: We use a Gaussian random field for generating the training functions (Wang et al., 2021a).
Revised: We use a Gaussian random field for generating the training functions (Wang et al., 2021a).

Original: We generate another 1000 input functions also from the GRF as the test functions.
Revised: We generate another 1000 input functions also from the GRF as the test functions.

Original: We pretrain the
Revised: We pretrain the

Original: PI-DeepONet until convergence.
Revised: PI-DeepONet until convergence.

Original: In the second stage, we replace the PDE constraints with the
Revised: In the second stage, we replace the PDE constraints with the

Original: PI-DeepONet and optimize the control variables using the Adam optimizer with a learning rate of 10 − 3 .
Revised: PI-DeepONet and optimize the control variables using the Adam optimizer with a learning rate of 10 − 3 .

Original: F. D ETAILS OF O THER BI - LEVEL OPTIMIZATION METHODS .
Revised: F. D ETAILS OF O THER BI - LEVEL OPTIMIZATION METHODS .

Original: TRMD .
Revised: TRMD .

Original: It uses a truncated reverse-mode automatic differentiation for computing the hypergradients.
Revised: It uses a truncated reverse-mode automatic differentiation for computing the hypergradients.

Original: If we use the SGD optimizer with L steps as the inner loop update algorithm,
Revised: If we use the SGD optimizer with L steps as the inner loop update algorithm,

Original: [Equation]
Revised: [Equation]

Original: We need to compute the following Jacobian matrix,
Revised: We need to compute the following Jacobian matrix,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Instead of create an instance of Jacobian matrix, we calculate the following Jacobian-vector product,
Revised: Instead of create an instance of Jacobian matrix, we calculate the following Jacobian-vector product,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We could iteratively calculate it with a truncated history of m steps.
Revised: We could iteratively calculate it with a truncated history of m steps.

Original: The steps is chosen from { 1 , 5 , 10 , 20 } in experiments.
Revised: The steps is chosen from { 1 , 5 , 10 , 20 } in experiments.

Original: Neumann Series.
Revised: Neumann Series.

Original: It also approximates the hypergradients based on Implicit Function Differentiation.
Revised: It also approximates the hypergradients based on Implicit Function Differentiation.

Original: But the difference is that it uses the following rule to approximate the inverse Hessian matrix,
Revised: But the difference is that it uses the following rule to approximate the inverse Hessian matrix,

Original: [Equation]
Revised: [Equation]

Original: And the inverse Hessian-Jacobian product is approximated by,
Revised: And the inverse Hessian-Jacobian product is approximated by,

Original: [Equation]
Revised: [Equation]

Original: The convergence speed is decided by two hyperparameters α and K .
Revised: The convergence speed is decided by two hyperparameters α and K .

Original: We choose α from { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 } and K from { 1 , 2 , 4 , 8 , 16 , 32 } .
Revised: We choose α from { 10 − 6 , 10 − 5 , 10 − 4 , 10 − 3 , 10 − 2 } and K from { 1 , 2 , 4 , 8 , 16 , 32 } .

Original: In theory, α should satisfy α < || (cid:16) ∂ 2 E ∂w∂w ⊤ (cid:17) ||
Revised: In theory, α should satisfy α < || (cid:16) ∂ 2 E ∂w∂w ⊤ (cid:17) ||

Original: We also experimentally find that large α and large K might lead to divergence depending on the problems.
Revised: We also experimentally find that large α and large K might lead to divergence depending on the problems.

Original: For too small α it might degenerate to the method of T 1 − T 2 .
Revised: For too small α it might degenerate to the method of T 1 − T 2 .

Original: G S UPPLEMENTARY EXPERIMENTS
Revised: G S UPPLEMENTARY EXPERIMENTS

Original: In this section, we provide supplementary experimental results to show the effectiveness of our method.
Revised: In this section, we provide supplementary experimental results to show the effectiveness of our method.

Original: G. E FFICIENCY E XPERIMENTS
Revised: G. E FFICIENCY E XPERIMENTS

Original: In this subsection, we conduct efficiency experiments to show that our BPN is more efficient compared with baselines.
Revised: In this subsection, we conduct efficiency experiments to show that our BPN is more efficient compared with baselines.

Original: We measure objective functions varying with both time and number of iterations for baselines on Poisson’s 2d CG problem and Heat 2d problem.
Revised: We measure objective functions varying with both time and number of iterations for baselines on Poisson’s 2d CG problem and Heat 2d problem.

Original: The results are shown in the following two figures.
Revised: The results are shown in the following two figures.

Original: G. R UN T IME A NALYSIS AND E RROR B ARS
Revised: G. R UN T IME A NALYSIS AND E RROR B ARS

Original: Here we report experimental results measuring error bars and wall clock runtime for these methods on POisson’s 2d CG problem and Heat 2d problem.
Revised: Here we report experimental results measuring error bars and wall clock runtime for these methods on POisson’s 2d CG problem and Heat 2d problem.

Original: • We run all experiments with 5 random seeds and compute the mean and std of these results.
Revised: • We run all experiments with 5 random seeds and compute the mean and std of these results.

Original: • We measure all wall clock time for comparison.
Revised: • We measure all wall clock time for comparison.

Original: For neural networks based methods, we use a single 2080 Ti GPU.
Revised: For neural networks based methods, we use a single 2080 Ti GPU.

Original: The reference FEM are runned on a laptop with 8 threads of CPU (11-th Gen Intel Core i5-11300 H @3.10 GHz ).
Revised: The reference FEM are runned on a laptop with 8 threads of CPU (11-th Gen Intel Core i5-11300 H @3.10 GHz ).

Original: The mesh size of reference FEM is roughly 500x500 for Poisson’s 2d CG and 400x400x20 for Heat 2d.
Revised: The mesh size of reference FEM is roughly 500x500 for Poisson’s 2d CG and 400x400x20 for Heat 2d.

Original: We omit PI-DeepONet for time comparison because it uses a pre-trained operator network.
Revised: We omit PI-DeepONet for time comparison because it uses a pre-trained operator network.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G. C OMPARISON WITH CONJUGATE GRADIENT METHOD .
Revised: G. C OMPARISON WITH CONJUGATE GRADIENT METHOD .

Original: We compare the time cost and the performance on Poisson’s 2d CG and Heat 2d problems using the conjugate gradient for computing hypergradients.
Revised: We compare the time cost and the performance on Poisson’s 2d CG and Heat 2d problems using the conjugate gradient for computing hypergradients.

Original: The results are shown in the following table.
Revised: The results are shown in the following table.

Original: We use 50 iterations for Broyden’s method.
Revised: We use 50 iterations for Broyden’s method.

Original: We have two observations from these results.
Revised: We have two observations from these results.

Original: First, the performance using Broyden’s method is better than using conjugated gradients.
Revised: First, the performance using Broyden’s method is better than using conjugated gradients.

Original: We found the main reason is that the hypergradients are more accurately approximated than using the conjugate gradient.
Revised: We found the main reason is that the hypergradients are more accurately approximated than using the conjugate gradient.

Original: Second, the performance of conjugated gradients improves with the increase in the number of iterations of CG.
Revised: Second, the performance of conjugated gradients improves with the increase in the number of iterations of CG.

Original: However, it is not efficient enough to use conjugated descent without preconditioning.
Revised: However, it is not efficient enough to use conjugated descent without preconditioning.

Original: But since we cannot store the whole hessian matrix, we are not able to use effective preconditioning techniques like incomplete Cholesky decomposition or Gauss-Seidel preconditioning.
Revised: But since we cannot store the whole hessian matrix, we are not able to use effective preconditioning techniques like incomplete Cholesky decomposition or Gauss-Seidel preconditioning.

Original: In summary, using a quasi-Netwon method like
Revised: In summary, using a quasi-Netwon method like

Original: Broyden’s method is more efficient for solving such a high dimensional and dense linear equation.
Revised: Broyden’s method is more efficient for solving such a high dimensional and dense linear equation.

Original: G. C OMPARISON WITH ADJOINT SIMULATION WITH PINN S .
Revised: G. C OMPARISON WITH ADJOINT SIMULATION WITH PINN S .

Original: Adjoint PDEs for all cases in our main experiments, exist and they can be analytically derived.
Revised: Adjoint PDEs for all cases in our main experiments, exist and they can be analytically derived.

Original: We also provide supplementary experiments comparing our algorithm with adjoint method simulated by using PINNs.
Revised: We also provide supplementary experiments comparing our algorithm with adjoint method simulated by using PINNs.

Original: We denote the method computes gradients from adjoint simulation of PDEs using
Revised: We denote the method computes gradients from adjoint simulation of PDEs using

Original: PINNs as PINN-adjoint.
Revised: PINNs as PINN-adjoint.

Original: We found that PINN-adjoint is also effective for linear problems.
Revised: We found that PINN-adjoint is also effective for linear problems.

Original: However, by checking residual loss of PDEs and adjoint PDEs, we found that adjoint PDEs are more difficult to solve with PINNs, especially for complex cases like nonlinear Naiver-Stokes equations.
Revised: However, by checking residual loss of PDEs and adjoint PDEs, we found that adjoint PDEs are more difficult to solve with PINNs, especially for complex cases like nonlinear Naiver-Stokes equations.

Original: In summary, we see that
Revised: In summary, we see that

Original: our
Revised: our

Original: BPN is still competitive and outperforms PINN-adjoint in average.
Revised: BPN is still competitive and outperforms PINN-adjoint in average.

Original: Poisson’s 2d CG
Revised: Poisson’s 2d CG

Original: Heat 2d
Revised: Heat 2d

Original: NS backstep
Revised: NS backstep

Original: Objective
Revised: Objective

Original: [Table]
Revised: [Table]

Original: PINN-adjoint 0.
Revised: PINN-adjoint 0.

Original: 24min 0.
Revised: 24min 0.

Original: [Table]
Revised: [Table]

Original: 16min 0.
Revised: 16min 0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: G. S UPPLEMENTARY EXPERIMENTS ON NS2 INLET FORCE PROBLEM .
Revised: G. S UPPLEMENTARY EXPERIMENTS ON NS2 INLET FORCE PROBLEM .

Original: We have conducted a supplementary experiment on controlling a system governed by NS equation by optimizing a force (source) distributed over a 2d domain.
Revised: We have conducted a supplementary experiment on controlling a system governed by NS equation by optimizing a force (source) distributed over a 2d domain.

Original: The results are shown in the following
Revised: The results are shown in the following

Original: Table 10.
Revised: Table 10.

Original: G. D ETAILS OF EVALUATION PROTOCOL .
Revised: G. D ETAILS OF EVALUATION PROTOCOL .

Original: Here we specify more details of the evaluation protocol.
Revised: Here we specify more details of the evaluation protocol.

Original: For the main experiments, we measure the performance using the final prediction when the algorithm stops rather than the best solution it founds.
Revised: For the main experiments, we measure the performance using the final prediction when the algorithm stops rather than the best solution it founds.

Original: The motivation for this metric is that for practical problems, estimating the ground truth objective is expensive.
Revised: The motivation for this metric is that for practical problems, estimating the ground truth objective is expensive.

Original: For efficiency experiments, we measure their performance in each validation epoch to compare the efficiency for these methods.
Revised: For efficiency experiments, we measure their performance in each validation epoch to compare the efficiency for these methods.