Original: C ONSTRAINED P HYSICAL -S TATISTICS M ODELS FOR D YNAMICAL S YSTEM I DENTIFICATION AND P REDICTION
Revised: C ONSTRAINED P HYSICAL -S TATISTICS M ODELS FOR D YNAMICAL S YSTEM I DENTIFICATION AND P REDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modeling dynamical systems combining prior physical knowledge and machine learning (ML) is promising in scientiﬁc problems when the underlying processes are not fully understood, e.g. when the dynamics is partially known.
Revised: Modeling dynamical systems combining prior physical knowledge and machine learning (ML) is promising in scientiﬁc problems when the underlying processes are not fully understood, e.g. when the dynamics is partially known.

Original: A common practice to identify the respective parameters of the physical and ML components is to formulate the problem as supervised learning on observed trajectories.
Revised: A common practice to identify the respective parameters of the physical and ML components is to formulate the problem as supervised learning on observed trajectories.

Original: However, this formulation leads to an inﬁnite number of possible decompositions.
Revised: However, this formulation leads to an inﬁnite number of possible decompositions.

Original: To solve this ill-posedness, we reformulate the learning problem by introducing an upper bound on the prediction error of a physical-statistical model.
Revised: To solve this ill-posedness, we reformulate the learning problem by introducing an upper bound on the prediction error of a physical-statistical model.

Original: This allows us to control the contribution of both the physical and statistical components to the overall prediction.
Revised: This allows us to control the contribution of both the physical and statistical components to the overall prediction.

Original: This framework generalizes several existing hybrid schemes proposed in the literature.
Revised: This framework generalizes several existing hybrid schemes proposed in the literature.

Original: We provide theoretical guarantees on the wellposedness of our formulation along with a proof of convergence in a simple afﬁne setting.
Revised: We provide theoretical guarantees on the wellposedness of our formulation along with a proof of convergence in a simple afﬁne setting.

Original: For more complex dynamics, we validate our framework experimentally.
Revised: For more complex dynamics, we validate our framework experimentally.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Dynamical systems prediction and identiﬁcation ﬁnd crucial applications ranging from medicine and the study of tumors (Hanahan & Weinberg, 2011; Lu & Fei, 2014) to oceanic and climate forecasting (Oreskes et al., 1994;
Revised: Dynamical systems prediction and identiﬁcation ﬁnd crucial applications ranging from medicine and the study of tumors (Hanahan & Weinberg, 2011; Lu & Fei, 2014) to oceanic and climate forecasting (Oreskes et al., 1994;

Original: Caers, 2011).
Revised: Caers, 2011).

Original: The modeling of such systems traditionally rely on ordinary or partial differential equations (ODE/PDE) (Madec, 2008; Marti et al., 2010), and their resolution via numerical solvers and data assimilation (Ghil & Malanotte-Rizzoli, 1991).
Revised: The modeling of such systems traditionally rely on ordinary or partial differential equations (ODE/PDE) (Madec, 2008; Marti et al., 2010), and their resolution via numerical solvers and data assimilation (Ghil & Malanotte-Rizzoli, 1991).

Original: In real world applications, two main pitfalls occur: ﬁrst the dynamics may only be partially known and thus do not fully represent the studied phenomena (Rio & Santoleri, 2018); second, the system state may only be partially observed as in ocean models (Gaultier et al., 2013).
Revised: In real world applications, two main pitfalls occur: ﬁrst the dynamics may only be partially known and thus do not fully represent the studied phenomena (Rio & Santoleri, 2018); second, the system state may only be partially observed as in ocean models (Gaultier et al., 2013).

Original: Machine learning (ML) has become a complementary approach to traditional physics based models (denoted MB for model based)
Revised: Machine learning (ML) has become a complementary approach to traditional physics based models (denoted MB for model based)

Original: (Reichstein et al., 2019; Dueben & Bauer, 2018).
Revised: (Reichstein et al., 2019; Dueben & Bauer, 2018).

Original: Both offer advantages: whereas
Revised: Both offer advantages: whereas

Original: MB approaches generalize and extrapolate better, ML high expressivity approaches beneﬁt from the ongoing explosive growth of available data such as satellite observations, with reduced costs compared to classical data assimilation.
Revised: MB approaches generalize and extrapolate better, ML high expressivity approaches beneﬁt from the ongoing explosive growth of available data such as satellite observations, with reduced costs compared to classical data assimilation.

Original: In that perspective, recent lines of work tackle the learning of hybrid models relying on prior physical knowledge and machine learning (Yin et al., 2021; Mehta et al., 2020).
Revised: In that perspective, recent lines of work tackle the learning of hybrid models relying on prior physical knowledge and machine learning (Yin et al., 2021; Mehta et al., 2020).

Original: Efﬁciently learning such decompositions actually means solving two different tasks: system identiﬁcation, i.e. estimating the parameters of the physical model, and prediction, i.e. recovering the trajectories associated to the dynamics.
Revised: Efﬁciently learning such decompositions actually means solving two different tasks: system identiﬁcation, i.e. estimating the parameters of the physical model, and prediction, i.e. recovering the trajectories associated to the dynamics.

Original: Both are essential for hybrid MB/ML models of dynamical systems.
Revised: Both are essential for hybrid MB/ML models of dynamical systems.

Original: Whereas prediction aims at robust extrapolation, identiﬁcation accounts for physical interpretability of the MB/ML model.
Revised: Whereas prediction aims at robust extrapolation, identiﬁcation accounts for physical interpretability of the MB/ML model.

Original: While solving both problems using model-based formulation admits well-known numerical solutions, for example using the adjoint method (Le Dimet & Talagrand, 1986; Courtier et al., 1994), the combination of physical models and deep learning is still an open area of research.
Revised: While solving both problems using model-based formulation admits well-known numerical solutions, for example using the adjoint method (Le Dimet & Talagrand, 1986; Courtier et al., 1994), the combination of physical models and deep learning is still an open area of research.

Original: In this context, ML applications mainly focus on the prediction task, at the expense of the system identiﬁcation: Willard et al.
Revised: In this context, ML applications mainly focus on the prediction task, at the expense of the system identiﬁcation: Willard et al.

Original: (2020) underlines the lack of generalizability of black-box ML models and their inability to produce physically sound results.
Revised: (2020) underlines the lack of generalizability of black-box ML models and their inability to produce physically sound results.

Original: Indeed, Ayed et al.
Revised: Indeed, Ayed et al.

Original: (2020) show that without any prior knowledge, the recovered estimates of a dynamical system states are not physically plausible despite accurate predictions.
Revised: (2020) show that without any prior knowledge, the recovered estimates of a dynamical system states are not physically plausible despite accurate predictions.

Original: Moreover, as noted by Yin et al.
Revised: Moreover, as noted by Yin et al.

Original: (2021), learning a linear MB/ML decompo- sition with the sole supervision on the system trajectories is ill-posed and admits an inﬁnite number of decompositions.
Revised: (2021), learning a linear MB/ML decompo- sition with the sole supervision on the system trajectories is ill-posed and admits an inﬁnite number of decompositions.

Original: Such observations highlight the need to incorporate physically motivated constraints in the learning of hybrid models, e.g. through regularization penalties, and several works propose additional constraints to guide the model towards physical solutions (Jia et al., 2019;
Revised: Such observations highlight the need to incorporate physically motivated constraints in the learning of hybrid models, e.g. through regularization penalties, and several works propose additional constraints to guide the model towards physical solutions (Jia et al., 2019;

Original: Yin et al., 2021; Linial et al., 2021).
Revised: Yin et al., 2021; Linial et al., 2021).

Original: Finally, to complete prior dynamical knowledge with a data-driven component and ensure interpretability of the decomposition, we work out a principled framework that generalizes previous attempts in the regularization of hybrid models.
Revised: Finally, to complete prior dynamical knowledge with a data-driven component and ensure interpretability of the decomposition, we work out a principled framework that generalizes previous attempts in the regularization of hybrid models.

Original: Our contributions are :
Revised: Our contributions are :

Original: • In section 3.1, we introduce a novel way to recover well-posedness and interpretability in the learning of hybrid MB/ML models through the control of an upper bound.
Revised: • In section 3.1, we introduce a novel way to recover well-posedness and interpretability in the learning of hybrid MB/ML models through the control of an upper bound.

Original: We further extend our framework to incorporate auxiliary data when available to handle complex realworld problems.
Revised: We further extend our framework to incorporate auxiliary data when available to handle complex realworld problems.

Original: • In section 3.2, we propose a novel alternate-optimization algorithm to learn hybrid models.
Revised: • In section 3.2, we propose a novel alternate-optimization algorithm to learn hybrid models.

Original: • In section 3.3, we provide an analysis of the convergence of the proposed algorithm on a simpliﬁed case and experimentally evidence the soundness of our approach on more complex settings of increasing difﬁculty including challenging real world problems (section 4).
Revised: • In section 3.3, we provide an analysis of the convergence of the proposed algorithm on a simpliﬁed case and experimentally evidence the soundness of our approach on more complex settings of increasing difﬁculty including challenging real world problems (section 4).

Original:  B ACKGROUND AND P ROBLEM S ETUP
Revised:  B ACKGROUND AND P ROBLEM S ETUP

Original: We consider a dynamical system with state at time t denoted Z t =
Revised: We consider a dynamical system with state at time t denoted Z t =

Original: Z ( t ) .
Revised: Z ( t ) .

Original: Z t might be fully or only partially observed: we write Z t =
Revised: Z t might be fully or only partially observed: we write Z t =

Original: ( X t , Y t ) , where X t is the observed component and Y t the unobserved one.
Revised: ( X t , Y t ) , where X t is the observed component and Y t the unobserved one.

Original: The evolution of Z is governed by a differential equation with dynamics :
Revised: The evolution of Z is governed by a differential equation with dynamics :

Original: [Equation]
Revised: [Equation]

Original: The objective is to predict trajectories of X , i.e. to model the evolution of the observable part following dX t dt = f
Revised: The objective is to predict trajectories of X , i.e. to model the evolution of the observable part following dX t dt = f

Original: X ( Z t ) .
Revised: X ( Z t ) .

Original: For simplicity, we omit the index X in f X and write f ( . )
Revised: For simplicity, we omit the index X in f X and write f ( . )

Original: ∆ = f X ( . ) .
Revised: ∆ = f X ( . ) .

Original: Dynamical Hypothesis We assume partial knowledge of the dynamics of the observed X t :
Revised: Dynamical Hypothesis We assume partial knowledge of the dynamics of the observed X t :

Original: [Equation]
Revised: [Equation]

Original: where f k ∈ H k is a known operator with unknown parameters θ ∗ , and f u ∈ H u is the unknown residual dynamics.
Revised: where f k ∈ H k is a known operator with unknown parameters θ ∗ , and f u ∈ H u is the unknown residual dynamics.

Original: H k and H u denote function spaces.
Revised: H k and H u denote function spaces.

Original: Note that the additive hypothesis in eq.
Revised: Note that the additive hypothesis in eq.

Original: (2) is not restrictive (see discussion in appendix B).
Revised: (2) is not restrictive (see discussion in appendix B).

Original: Learning Problem
Revised: Learning Problem

Original: Our objective is to approximate f with a function h learned from the observed data.
Revised: Our objective is to approximate f with a function h learned from the observed data.

Original: According to eq.
Revised: According to eq.

Original: (2), we assume h = h k
Revised: (2), we assume h = h k

Original: + h u .
Revised: + h u .

Original: h k ∈ H k belongs to the same hypothesis space as f k , i.e. it has the same parametric form.
Revised: h k ∈ H k belongs to the same hypothesis space as f k , i.e. it has the same parametric form.

Original: Its parameters are denoted θ k .
Revised: Its parameters are denoted θ k .

Original: Note that h k ( ., θ ∗ ) =
Revised: Note that h k ( ., θ ∗ ) =

Original: f k .
Revised: f k .

Original: h u ∈ H u is represented by a free form functional with parameters θ u , e.g. a neural network.
Revised: h u ∈ H u is represented by a free form functional with parameters θ u , e.g. a neural network.

Original: The learning problem is then to estimate from data the parameters of h k so that they match the true physical ones and those of h u so as to approximate at best the unknown dynamics f .
Revised: The learning problem is then to estimate from data the parameters of h k so that they match the true physical ones and those of h u so as to approximate at best the unknown dynamics f .

Original: In this regard, an intuitive training objective is to minimize a distance d between h = h k + h u and f :
Revised: In this regard, an intuitive training objective is to minimize a distance d between h = h k + h u and f :

Original: [Equation]
Revised: [Equation]

Original: where p Z is the distribution of the state values Z that accounts for varying initial states.
Revised: where p Z is the distribution of the state values Z that accounts for varying initial states.

Original: Each Z then deﬁnes a training sample.
Revised: Each Z then deﬁnes a training sample.

Original: Minimizing eq.
Revised: Minimizing eq.

Original: (3) with h = h k
Revised: (3) with h = h k

Original: + h u enables to predict accurate trajectories but may have an inﬁnite number of solutions.
Revised: + h u enables to predict accurate trajectories but may have an inﬁnite number of solutions.

Original: For instance, h u may bypass the physical hypothesis h k .
Revised: For instance, h u may bypass the physical hypothesis h k .

Original: Thus, interpretability is not guaranteed.
Revised: Thus, interpretability is not guaranteed.

Original: We develop in the following section a method to overcome this ill-posedness.
Revised: We develop in the following section a method to overcome this ill-posedness.

Original:  M ETHOD
Revised:  M ETHOD

Original: In hybrid modeling, two criteria are essentials: 1.
Revised: In hybrid modeling, two criteria are essentials: 1.

Original: identiﬁability, i.e. the estimated parameters of h k should correspond to the true physical ones; 2.
Revised: identiﬁability, i.e. the estimated parameters of h k should correspond to the true physical ones; 2.

Original: prediction power, i.e. the statistical component h u should complete h k
Revised: prediction power, i.e. the statistical component h u should complete h k

Original: so that h = h k
Revised: so that h = h k

Original: + h u performs accurate prediction over the system states.
Revised: + h u performs accurate prediction over the system states.

Original: To control the contribution of each term h k and h u , we work upper bounds out of eq.
Revised: To control the contribution of each term h k and h u , we work upper bounds out of eq.

Original: (3) (section 3.1).
Revised: (3) (section 3.1).

Original: We then propose to minimize d( h, f ) while constraining the upper bounds, which provide us with a well-posed learning framework (section 3.2).
Revised: We then propose to minimize d( h, f ) while constraining the upper bounds, which provide us with a well-posed learning framework (section 3.2).

Original: Besides, we show that several previous works that introduced constrained optimization to solve related problems are speciﬁc cases of our formulation (Yin et al., 2021;
Revised: Besides, we show that several previous works that introduced constrained optimization to solve related problems are speciﬁc cases of our formulation (Yin et al., 2021;

Original: Jia et al., 2019; Linial et al., 2021).
Revised: Jia et al., 2019; Linial et al., 2021).

Original: Finally, we introduce an alternate optimization algorithm which convergence is shown in section 3.3 for a linear approximation of f .
Revised: Finally, we introduce an alternate optimization algorithm which convergence is shown in section 3.3 for a linear approximation of f .

Original: 3. S TRUCTURAL C ONSTRAINTS FOR D YNAMICAL S YSTEMS
Revised: 3. S TRUCTURAL C ONSTRAINTS FOR D YNAMICAL S YSTEMS

Original: To ensure identiﬁability, we derive regularizations on h k and h u ﬂowing from the control of an upper bound of d( h, f ) .
Revised: To ensure identiﬁability, we derive regularizations on h k and h u ﬂowing from the control of an upper bound of d( h, f ) .

Original: In particular, to minimize d( h k , f k ) would enable us to accurately interpret h k as the true f k , and thus h u as the residual dynamics f u .
Revised: In particular, to minimize d( h k , f k ) would enable us to accurately interpret h k as the true f k , and thus h u as the residual dynamics f u .

Original: However, since we do not access the parameters of f k , computing d( h k , f k ) is not tractable.
Revised: However, since we do not access the parameters of f k , computing d( h k , f k ) is not tractable.

Original: We then consider two possible situations.
Revised: We then consider two possible situations.

Original: In the ﬁrst one, the only available information on the physical system is the parametric form of f k (or equivalently of h k ), training thus only relies on observed trajectories (section 3.1.1).
Revised: In the ﬁrst one, the only available information on the physical system is the parametric form of f k (or equivalently of h k ), training thus only relies on observed trajectories (section 3.1.1).

Original: In the second one, we consider that auxiliary information about f k is available and will be used to minimize the distance between h k and f k (section 3.1.2).
Revised: In the second one, we consider that auxiliary information about f k is available and will be used to minimize the distance between h k and f k (section 3.1.2).

Original: While the ﬁrst setting is the more general, the physical prior it relies on is often insufﬁcient to effectively handle real world situations.
Revised: While the ﬁrst setting is the more general, the physical prior it relies on is often insufﬁcient to effectively handle real world situations.

Original: The second setting makes use of more informative priors and better corresponds to real cases as will be shown in the experimental section (see section 4.2).
Revised: The second setting makes use of more informative priors and better corresponds to real cases as will be shown in the experimental section (see section 4.2).

Original: 3.1. C ONTROLLING THE ML C OMPONENT AND THE MB H YPOTHESIS
Revised: 3.1. C ONTROLLING THE ML C OMPONENT AND THE MB H YPOTHESIS

Original: We propose a general approach to constrain the learning of hybrid models when one solely access the functional form of h k .
Revised: We propose a general approach to constrain the learning of hybrid models when one solely access the functional form of h k .

Original: In this case, to make h k accountable in our observed phenomena, a solution is to minimize d( h k , f ) .
Revised: In this case, to make h k accountable in our observed phenomena, a solution is to minimize d( h k , f ) .

Original: Following the triangle inequality we link up both errors d( h, f ) and d( h k , f ) (computations available in appendix C.1):
Revised: Following the triangle inequality we link up both errors d( h, f ) and d( h k , f ) (computations available in appendix C.1):

Original: [Equation]
Revised: [Equation]

Original: We want the physical-statistical model h
Revised: We want the physical-statistical model h

Original: = h k
Revised: = h k

Original: + h u to provide high quality forecasts.
Revised: + h u to provide high quality forecasts.

Original: Minimizing the sole upper bound does not ensure such aim, as h u is only penalized through d( h u , 0) and is not optimized to contribute to predictions.
Revised: Minimizing the sole upper bound does not ensure such aim, as h u is only penalized through d( h u , 0) and is not optimized to contribute to predictions.

Original: We thus propose to minimize d( h, f ) while controlling both d( h u , 0) and d( h k , f ) .
Revised: We thus propose to minimize d( h, f ) while controlling both d( h u , 0) and d( h k , f ) .

Original: Such a control of the upper bound of eq.
Revised: Such a control of the upper bound of eq.

Original: (4) amounts to balancing the contribution of the ML and the MB components.
Revised: (4) amounts to balancing the contribution of the ML and the MB components.

Original: This will be formally introduced in section 3.2.
Revised: This will be formally introduced in section 3.2.

Revised: Link to the Literature

Revised: The least action principle on the ML component i.e. constraining d( h u , 0) is invoked for a geometric argument in (Yin et al., 2021), and appears as a co-product of the introduction of d( h k , f ) in eq.

Revised: Optimizing d( h k , f ) to match the physical model with observations is investigated in (Forssell & Lindskog, 1997).

Original: Such a general approach  allows us to perform prediction (via h ) and system identiﬁcation (via h k ) on simple problems (see section 4.1).
Revised: The general approach of eq. (4) allows us to perform prediction (via h ) and system identiﬁcation (via h k ) on simple problems (see section 4.1).

Original: Handling complex dynamics requires additional physical information since training relying on eq. (4) seems insufﬁcient as illustrated in section 4.2. 
Revised: The learning of real-world complex dynamics, via data-driven hybrid models, often fails at yielding a physically sound estimation, as illustrated in section 4.2. This suggests that learning complex dynamics requires additional information.

Original: In many real-world cases, auxiliary information is available in the form of measurements providing complementary information on f k .
Revised: In many real-world cases, auxiliary information is available in the form of measurements providing complementary information on f k .

Original: Indeed, a common issue in physics is to infer an unobserved variable of interest (in our case f k parameters θ (cid:63) ) from indirect or noisy measurements that we refer to as proxy data.
Revised: Indeed, a common issue in physics is to infer an unobserved variable of interest (in our case f k parameters θ (cid:63) ) from indirect or noisy measurements that we refer to as proxy data.

Original: For instance, one can access a physical quantity but only at a coarse resolution, as in (Um et al., 2020; Belbute-Peres et al., 2020) and in the real world example detailed in section 4.2.
Revised: For instance, one can access a physical quantity but only at a coarse resolution, as in (Um et al., 2020; Belbute-Peres et al., 2020) and in the real world example detailed in section 4.2.

Original: We show in the next subsection how to incorporate such an information in order to approximate d( h k , fk ) .
Revised: We show in the next subsection how to incorporate such an information in order to approximate d( h k , f

Original: Link to the Literature The least action principle on the ML component i.e. constraining d( h u , 0) is invoked for a geometric argument in (Yin et al., 2021), and appears as a co-product of the introduction of d( h k , f ) in eq. Optimizing d( h k , f ) to match the physical model with observations is investigated in (Forssell & Lindskog, 1997).
Revised: [Equation]

Original: 3.1. M ATCHING THE P HYSICAL H YPOTHESES : I NTRODUCING A UXILIARY D ATA
Revised: 3.1. M ATCHING THE P HYSICAL H YPOTHESES : I NTRODUCING A UXILIARY D ATA

Original: We here assume one accesses a proxy of f k , denoted f prk ∈ H k .
Revised: We here assume one accesses a proxy of f k , denoted f prk ∈ H k .

Original: Our goal is to adapt our framework to incorporate such auxiliary information, bringing the regularization induced by f pr k within the scope of the control of an upper bound.
Revised: Our goal is to adapt our framework to incorporate such auxiliary information, bringing the regularization induced by f prk within the scope of the control of an upper bound.

Original: This enables us to extend our proposition towards the solving of real world physical problems, still largely unexplored by the ML community.
Revised: This enables us to extend our proposition towards the solving of real world physical problems, still largely unexplored by the ML community.

Original: We have:
Revised: We have:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Γ is a constant of the problem that cannot be optimized (see appendix C.2).
Revised: where Γ is a constant of the problem that cannot be optimized (see appendix C.2).

Original: In that context, we can beneﬁt from auxiliary information providing us with coarse estimates of θ (cid:63) , denoted θ pr ,
Revised: In that context, we can beneﬁt from auxiliary information providing us with coarse estimates of θ (cid:63) , denoted θ pr ,

Original: such that f prk = h k ( ., θ pr ) ≈ f k .
Revised: such that f prk = h k ( ., θ pr ) ≈ f k .

Original: To use the available θ pr to guide our estimation towards the true parameters θ (cid:63) of f k , a simple solution is to directly enforce the minimization of d( h k , f prk ) in the parameter space by minimizing (cid:107) θ k − θ pr (cid:107) 2 , where θ k are the parameters of h k .
Revised: To use the available θ pr to guide our estimation towards the true parameters θ (cid:63) of f k , a simple solution is to directly enforce the minimization of d( h k , f prk ) in the parameter space by minimizing (cid:107) θ k − θ pr (cid:107) 2 , where θ k are the parameters of h k .

Original: Indeed, because f k and f prk have identical parametric forms (as both belong to the same functional space
Revised: Indeed, because f k and f prk have identical parametric forms (as both belong to the same functional space

Original: H k ), minimizing (cid:107) θ k − θ pr (cid:107) 2 will bring h k closer to f prk and thus to f k .
Revised: H k ), minimizing (cid:107) θ k − θ pr (cid:107) 2 will bring h k closer to f prk and thus to f k .

Original: As above, we propose to minimize d( h, f ) while controlling both d( h u , 0) and d( h k , f prk ) , as described in section 3.2.
Revised: As above, we propose to minimize d( h, f ) while controlling both d( h u , 0) and d( h k , f prk ) , as described in section 3.2.

Original: Link to the Literature
Revised: Link to the Literature

Original: In (Linial et al., 2021)
Revised: In (Linial et al., 2021)

Original: f prk stands for true observations used to constrain a learned latent space, minimizing d( h k , f prk ) .
Revised: f prk stands for true observations used to constrain a learned latent space, minimizing d( h k , f prk ) .

Original: Jia et al.
Revised: Jia et al.

Original: (2019) uses synthetic data as f prk to pre-train their model which amounts to the control an upper bound, see appendix C.3.
Revised: (2019) uses synthetic data as f prk to pre-train their model which amounts to the control an upper bound, see appendix C.3.

Original: Finally, this setting ﬁnds an extension, when the model f prk is a learned model, for example trained using eq.
Revised: Finally, this setting ﬁnds an extension, when the model f prk is a learned model, for example trained using eq.

Original: (4), leading to a self-supervision approach described in appendix C.4.
Revised: (4), leading to a self-supervision approach described in appendix C.4.

Original: 3. L EARNING A LGORITHM AND O PTIMIZATION P ROBLEM
Revised: 3. L EARNING A LGORITHM AND O PTIMIZATION P ROBLEM

Original: From the upper bounds, we ﬁrst recover the well-posedness of the optimization and derive a theoretical learning scheme (section 3.2.1).
Revised: From the upper bounds, we ﬁrst recover the well-posedness of the optimization and derive a theoretical learning scheme (section 3.2.1).

Original: We then discuss its practical implementation (section 3.2.2).
Revised: We then discuss its practical implementation (section 3.2.2).

Original: 3.2. W ELL -P OSEDNESS AND A LTERNATE O PTIMIZATION A LGORITHM
Revised: 3.2. W ELL -P OSEDNESS AND A LTERNATE O PTIMIZATION A LGORITHM

Original: Recovering Well-Posedness
Revised: Recovering Well-Posedness

Original: We reformulate the ill-posed learning of min h k ,h u ∈H k ×H u d( h, f ) , by instead optimizing d( h, f ) while constraining the upper bounds.
Revised: We reformulate the ill-posed learning of min h k ,h u ∈H k ×H u d( h, f ) , by instead optimizing d( h, f ) while constraining the upper bounds.

Original: Let us deﬁne S k and S u as
Revised: Let us deﬁne S k and S u as

Original: [Equation]
Revised: [Equation]

Original: where µ k , µ u are two positive scalars and (cid:96) ( h k )
Revised: where µ k , µ u are two positive scalars and (cid:96) ( h k )

Original: =
Revised: =

Original: d( h k , f ) in the case of section 3.1.1 and (cid:96) ( h k ) =
Revised: d( h k , f ) in the case of section 3.1.1 and (cid:96) ( h k ) =

Original: d( h k , f prk ) in the case of section 3.1.2.
Revised: d( h k , f prk ) in the case of section 3.1.2.

Original: Our proposition then amounts to optimizing d( h, f ) over the Minkowski-sum S k + S
Revised: Our proposition then amounts to optimizing d( h, f ) over the Minkowski-sum S k + [Equation] u

Original: [Equation]
Revised: [Equation]

Original: This constrained optimization setting enables us to recover the well-posedness of the optimization problem under the relative compactness of the family of function H k
Revised: This constrained optimization setting enables us to recover the well-posedness of the optimization problem under the relative compactness of the family of function H k

Original: (proof in appendix D.3).
Revised: (proof in appendix D.3).

Original: Proposition 1 (Well-posedness) .
Revised: Proposition 1 (Well-posedness) .

Original: Under the relative compactness of S k , eq.
Revised: Under the relative compactness of S k , eq.

Original: (7) ﬁnds a solution h that writes as h = h k
Revised: (7) ﬁnds a solution h that writes as h = h k

Original: + h u ∈ S k + S u .
Revised: + h u ∈ S k + S u .

Original: Moreover, this solution is unique.
Revised: Moreover, this solution is unique.

Original: Alternate Optimization AlgorithmSolving eq.
Revised: Alternate Optimization Algorithm

Original: (7) with a Lagrangian formulation leads to optimizing jointly all parameters of h k and h u .

Original: Because the terms in both upper bounds of eqs.
Revised: As the terms in both upper bounds of eqs.

Original: (4) and (5) speciﬁcally address either h k or h u , weinstead isolate losses relative to h k and h u and alternate projections of h k on S k and h u on S u , as described in Algorithm 1.
Revised: (4) and (5) specifically address either h k or h u , we isolate losses relative to h k and h u and alternate projections of h k on S k and h u on S u , as described in Algorithm 1.

Original: Said otherwise, we learn h by alternately optimizing h k ( h u being ﬁxed) and h u ( h k being ﬁxed) according to:
Revised: Said otherwise, we learn h by alternately optimizing h k ( h u being ﬁxed) and h u ( h k being ﬁxed).

Revised: In practice, we rely on a dual formulation (see section 3.2.2 and the SGD version of Algorithm 1 in Appendix F).

Original: Result:
Revised: Result:

Original: Converged h k and h u
Revised: Converged h k and h u

Original: Set h 0 u = 0 , h 0 k = min h k ∈H k d( h k , f ) , tol ∈ R + while d( h, f ) > tol do h n +1 k =arg min 
Revised: Set h 0 u = 0 , [Equation] = min [Equation]

Revised: d

Original: [Equation]
Revised: [Equation]

Original: k ∈S k d( h 
Revised: h [Equation] , f ) , tol [Equation]

Revised: R + while d( h, f ) > tol do

Original: [Equation]
Revised: [Equation]

Original: n ← n + 1 end
Revised: n ← n + 1 end

Original: The convergence of the alternate projections is well studied for the intersection of convex sets or smooth manifolds (von Neumann, 1950; Lewis & Malick, 2008) and has been extended in our setting of Minkowski-sum of convex sets (Lange et al., 2019).
Revised: The convergence of the alternate projections is well studied for the intersection of convex sets or smooth manifolds (von Neumann, 1950; Lewis & Malick, 2008) and has been extended in our setting of Minkowski-sum of convex sets (Lange et al., 2019).

Original: Because d as deﬁned in eq.
Revised: Because d as deﬁned in eq.

Original: (3) is convex, S u and S k are convex sets as soon as H k and H u are convex (Appendix A).
Revised: (3) is convex, S u and S k are convex sets as soon as H k and H u are convex (Appendix A).

Original: Thus, if d( ., f ) is strongly convex, eq.
Revised: Thus, if d( ., f ) is strongly convex, eq.

Original: (8) ﬁnds one and only one solution (Boyd et al., 2004).
Revised: (8) ﬁnds one and only one solution (Boyd et al., 2004).

Original: However, neither the convexity of H u nor of H k is practically ensured.
Revised: However, neither the convexity of H u nor of H k is practically ensured.

Original: Nonetheless, we recover the well-posedness of eq.
Revised: Nonetheless, we recover the well-posedness of eq.

Original: (7) and show the convergence of Algorithm 1 in the simpliﬁed case where h is an afﬁne function of X t (see section 3.3).
Revised: (7) and show the convergence of Algorithm 1 in the simpliﬁed case where h is an afﬁne function of X t (see section 3.3).

Original: For complex PDE where convexity may not hold, we validate our approach experimentally and we evidence in section 4 that this formulation enables us to recover both an interpretable decomposition
Revised: For complex PDE where convexity may not hold, we validate our approach experimentally and we evidence in section 4 that this formulation enables us to recover both an interpretable decomposition

Original: h
Revised: h

Original: = h k + h u and improved prediction and identiﬁcation performances.
Revised: = h k + h u and improved prediction and identiﬁcation performances.

Original: 3.2. P RACTICAL O PTIMIZATION
Revised: 3.2. P RACTICAL O PTIMIZATION

Original: Equation (6) involves the choice of µ k and µ u .
Revised: Equation (6) involves the choice of µ k and µ u .

Original: In practice, we implement the projection algorithm by descending gradients on the parameters of h k and h u , with respect to the following losses:
Revised: In practice, we implement the projection algorithm by descending gradients on the parameters of h k and h u , with respect to the following losses:

Original: [Equation]
Revised: [Equation]

Original: where λ h , λ h k , λ h u are positive real values, dynamically increased/decreased during training (see
Revised: where λ h , λ h k , λ h u are positive real values, dynamically increased/decreased during training.

Original: SGD version of Algorithm 1, Appendix F).
Revised: Indeed, d( h u , 0) can be interpreted as a stability loss , preventing the neural networks to trump the physical component. On the other hand, d( h k , f ) can be interpreted has an initialization loss yield a ﬁrst estimate of θ k explaining the dynamics.

Original: However, f being unknown, d( h, f ) is not tractable.
Revised: Yet, f being unknown: d( h, f ) is not tractable.

Original: To estimate d( h, f ) , we rely on the trajectories associated to the dynamics.
Revised: To estimate d( h, f ) , we rely on the trajectories associated to the dynamics.

Original: We minimize the distance between the ODE ﬂows φ h and φ f deﬁned by h and f , d φ ( φ h , φ f ) , over all initial conditions X 0 :
Revised: We minimize the distance between the ODE ﬂows φ h and φ f deﬁned by h and f , d φ ( φ h , φ f ) , over all initial conditions X 0 :

Original: [Equation]
Revised: [Equation]

Original: We have: d φ ( φ h , φ f ) = 0 ⇔ d( h, f ) = 0 .
Revised: We have: [Equation]

Original: Deﬁnitions of ﬂows for ODE and in depth consideration on these distances are available in appendix A.
Revised: Deﬁnitions of ﬂows for ODE and in depth consideration on these distances are available in appendix A.

Original: The gradients of d φ ( φ h , φ f ) with respect to the parameters of h k or h u can be either estimated analytically using the adjoint method(optimizethen-discretize approach) (Chen et al., 2018) or using explicit solvers, e.g. Rk45, and computing the gradients thanks to the backpropagation (discretize-then-optimize), see (Onken & Ruthotto, 2020).
Revised: The gradients of d φ ( φ h , φ f ) with respect to the parameters of h k or h u can be either estimated analytically using the adjoint method (Chen et al., 2018) or using explicit solvers, e.g. Rk45, and computing the gradients thanks to the backpropagation, see (Onken & Ruthotto, 2020).

Original: To compute eq.
Revised: To compute eq.

Original: (10), we rely on a temporal sampling of X : our datasets are composed of n sequences of observations of length N , X i =
Revised: (10), we rely on a temporal sampling of X : our datasets are composed of n sequences of observations of length N , X i =

Original: ( X it 0 , . . .
Revised: ( X it 0 , . . .

Original: , X it 

Revised: , X it

Original: + N ∆ t ) , where each sequence X i follows eq.
Revised: + N ∆ t ) , where each sequence X i follows eq.

Original: (2) and corresponds to one initial condition X it 0 .
Revised: (2) and corresponds to one initial condition X it 0 .

Original: We then sample the space of initial conditions X it 0 to compute a Monte-Carlo approximation to d φ ( φ h , φ f ).
Revised: We then sample the space of initial conditions X it 0 to compute a Monte-Carlo approximation to d φ ( φ h , φ f ).

Original: Let ODESolve be the function integrating any arbitrary initial state x t 0 up to time t with dynamics h , so that x t = ODESolve ( x t 0 , h, t ) .
Revised: Let ODESolve be the function integrating any arbitrary initial state x t 0 up to time t with dynamics h , so that x t = ODESolve ( x t 0 , h, t ) .

Original: The estimate of d φ ( φ h , φ f ) then writes as:
Revised: The estimate of d φ ( φ h , φ f ) then writes as:

Original: [Equation]
Revised: [Equation]

Original: Note that the way to compute ODEsolve differs across the experiments (see section 4).
Revised: Note that the way to compute ODEsolve differs across the experiments (see section 4).

Original: 3. T HEORETICAL A NALYSIS FOR A L INEAR A PPROXIMATION
Revised: 3. T HEORETICAL A NALYSIS FOR A L INEAR A PPROXIMATION

Original: A common strategy to approximate unknown dynamics is to use afﬁne functions as ﬁrst guess approximators.
Revised: We investigate the validity of our proposition when approximating an unknown derivative with an afﬁne function (interpretable ﬁrst guess approximators). We here consider h k as a linear function. We do not assume any information on f , thus relieving this section from the need of an accurate prior knowledge f k .

Original: We show the convergence of the learning scheme introduced in Algorithm 1 in this common scenario, hence demonstrating the validity of our framework in this simpliﬁed setting.
Revised: In this context, we show the convergence of the learning scheme introduced in Algorithm 1 with (cid:96) = d( h k , f ) , hence demonstrating the validity of our framework in this simpliﬁed setting.

Original: For more complex cases, for which theoretical analysis cannot be conducted, our framework is validated experimentally in section 4.
Revised: For more complex cases, for which theoretical analysis cannot be conducted, our framework is validated experimentally in section 4.

Original: All proofs of this section are conducted using the distance d φ .
Revised: All proofs of this section are conducted using the distance d φ .

Original: Let X s be the unique solution to the initial value problem:
Revised: Let X s be the unique solution to the initial value problem:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: With h k ( X ) =
Revised: With h k ( X ) =

Original: AX and h u ( X )
Revised: AX and h u ( X )

Original: =
Revised: =

Original: D A , the afﬁne approximation of f writes as:
Revised: D A , the afﬁne approximation of f writes as:

Original: [Equation]
Revised: [Equation]

Original: (12) where A ∈ M p,p ( R ) , D A ∈ R p .
Revised: where A ∈ M p,p ( R ) , D A ∈ R p .

Original: We write X D the solution to eq.
Revised: We write X D the solution to eq.

Original: (12) and X A the solution to eq.
Revised: (12) and X A the solution to eq.

Original: (12) when D A = 0 .
Revised: (12) when D A = 0 .

Original: The alternate projection algorithm with the distance d φ writes as:
Revised: The alternate projection algorithm with the distance d φ writes as:

Original: [Equation]
Revised: [Equation]

Original: where λ D , λ A > 0 .
Revised: where λ D , λ A > 0 .

Original: As the optimization of eq.
Revised: As the optimization of eq.

Original: (13) is not convex on A , the solution existence and uniqueness is not ensured.
Revised: (13) is not convex on A , the solution existence and uniqueness is not ensured.

Original: However, we can recover well-posedness  by discretizing eq. (12) with a simple numerical scheme, e.g. X t +1{++
+} ≈
Revised: The well-posedness w.r.t A can be recovered by instead considering a simple discretization scheme, e.g. X t {++
+} ≈

Original: ( AX t + D A )
Revised: ( AX t + D A )

Original: ∆ t + X t and solving the associated least square regression,  see  appendix D.2.
Revised: ∆ t + X t and solving the associated least square regression, which well-posedness is guaranteed, see details in appendix D.2.

Original: Such strategy is common practice in system identiﬁcation.
Revised: Such strategy is common practice in system identiﬁcation.

Original: Generic theoretical considerations on existence and uniqueness of solutions to eqs.
Revised: Generic theoretical considerations on existence and uniqueness of solutions to eqs.

Original: (13) and (14) are hard to retrieve.
Revised: (13) and (14) are hard to retrieve.

Original: Nonetheless, if A is an invertible matrix, we prove in appendix D.4:
Revised: Nonetheless, if A is an invertible matrix, we prove in appendix D.4:

Original: Proposition 2 (Existence and Uniqueness) .
Revised: Proposition 2 (Existence and Uniqueness) .

Original: If ˆ A is invertible, There exists a unique D A , hence a unique X D , solving eq.
Revised: If ˆ A is invertible, There exists a unique D A , hence a unique X D , solving eq.

Original: Finally, formulating Algorithm 1 as a least square problem in an afﬁne setting (see appendix D.5), we prove the convergence of the alternate projection algorithm (appendix D.6) :
Revised: Finally, formulating Algorithm 1 as a least square problem in an afﬁne setting (see appendix D.5), we prove the convergence of the alternate projection algorithm (appendix D.6) :

Original: Proposition 3.
Revised: Proposition 3.

Original: For λ D and λ A sufﬁciently high, the algorithm that alternates between the estimation of A and the estimation of D A following eqs.
Revised: For λ D and λ A sufﬁciently high, the algorithm that alternates between the estimation of A and the estimation of D A following eqs.

Original: (13) and (14) converges.
Revised: (13) and (14) converges.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Weexperimentally validate Algorithm 1 onphysical datasets of increasing difﬁculty (see appendix E), where the system state is either fully or partially observed (resp.
Revised: We validate Algorithm 1 on datasets of increasing difﬁculty (see appendix E), where the system state is either fully or partially observed (resp.

Original: section 4.1 and section 4.2).
Revised: section 4.1 and section 4.2).

Original: We no longer rely on an afﬁne prior and explicit h k and h u for each dataset.
Revised: We no longer rely on an afﬁne prior and explicit h k and h u for each dataset.

Original: Performances are evaluated via standard metrics: Mean Squared Error (MSE, lower is better) and relative Mean Absolute Error (rMAE, lower is better).
Revised: Performances are evaluated via standard metrics: MSE (lower is better) and relative Mean Absolute Error (rMAE, lower is better).

Original: We assess the relevance of our proposition based on eqs.
Revised: We assess the relevance of our proposition based on eqs.

Original: (4) and (5), against NeuralODE (Chen et al., 2018) and state of the art Aphynity
Revised: (4) and (5), against NeuralODE (Chen et al., 2018) and state of the art Aphynity

Original: Yin et al., 2021) and ablation studies.
Revised: Yin et al., 2021) and ablation studies.

Revised: We denote Ours eq.

Revised: (4) (resp.

Revised: Ours eq.

Revised: the results when (cid:96) =

Original: When d( h k , f )  (resp.
Revised: d( h k , f ) i.e eq. (4), (resp.

Original:  d( h u , 0) ) is not considered in the optimization, we refer to the results as d( h, f ) + d( h u , 0) (resp.
Revised: (cid:96) = d( h k , f prk ) i.e. eq. When d( h k , f ) (resp. d( h u , 0) ) is not considered in the optimization, we refer to the results as d( h, f )+d( h u , 0) (resp.

Original: d( h, f ) + d( h k , f ) ).
Revised: d( h, f )+d( h k , f ) ).

Original: When h is trained by only minimizing the discrepancy between actual and predicted trajectories the results are denoted «Only d( h, f ) ».
Revised: When h is trained by only minimizing the discrepancy between actual and predicted trajectories the results are denoted «Only d( h, f ) ».

Original: We report between brackets the standard deviation of the metrics over 5 runs and refer to Appendices F and G for training information and additional results.
Revised: We report between brackets the standard deviation of the metrics over 5 runs and refer to Appendices F and G for training information and additional results.

Original: Code and datasets will be provided upon acceptance.
Revised: Code and datasets will be provided upon acceptance.

Original: 4. F ULLY O BSERVABLE D YNAMICS
Revised: 4. F ULLY O BSERVABLE D YNAMICS

Original: To illustrate the learning scheme induced by eq.
Revised: To illustrate the learning scheme induced by eq.

Original: (4), we focus on fully observed low dimensional dynamics: a simple example emerging from Newtonian mechanics and a population dynamics model.
Revised: (4), we focus on fully observed low dimensional dynamics: a simple example emerging from Newtonian mechanics and a population dynamics model.

Original: Damped Pendulum (DPL)
Revised: Damped Pendulum (DPL)

Original: Now a standard benchmark for hybrid models, we consider the motion of a pendulum of length L damped due to viscous friction (Greydanus et al., 2019;
Revised: Now a standard benchmark for hybrid models, we consider the motion of a pendulum of length L damped due to viscous friction (Greydanus et al., 2019;

Original: Yin et al., 2021).
Revised: Yin et al., 2021).

Original: Newtonian mechanics provide an ODE describing the evolution of the angle x of the pendulum:
Revised: Newtonian mechanics provide an ODE describing the evolution of the angle x of the pendulum:

Original: [Equation]
Revised: [Equation]

Original: We suppose access to observations of the system state Z =
Revised: We suppose access to observations of the system state Z =

Original: ( x, ˙ x ) .
Revised: ( x, ˙ x ) .

Original: We consider as physical motion hypothesis h k ( x, θ k )
Revised: We consider as physical motion hypothesis h k ( x, θ k )

Original: =
Revised: =

Original: θ k sin( x ) .
Revised: θ k sin( x ) .

Original: The true pulsation θ ∗ =
Revised: The true pulsation θ ∗ =

Original: g/L of the pendulum has to be estimated with θ k .
Revised: g/L of the pendulum has to be estimated with θ k .

Original: The viscous friction term k ˙ x remains to be estimated by h u .
Revised: The viscous friction term k ˙ x remains to be estimated by h u .

Original: Population Dynamics (LV) Lotka-Volterra ODE system models a prey/predator population dynamics describing the growth of the preys ( x ) without predators ( y ), and the extinction of predators without preys, the non linear terms expressing the encounters between both species:
Revised: Population Dynamics (LV) Lotka-Volterra ODE system models a prey/predator population dynamics describing the growth of the preys ( x ) without predators ( y ), and the extinction of predators without preys, the non linear terms expressing the encounters between both species:

Original: [Equation]
Revised: [Equation]

Original: and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We observe the system state Z =
Revised: We observe the system state Z =

Original: ( x, y ) and set as prior knowledge: h k ( x, y )
Revised: ( x, y ) and set as prior knowledge: h k ( x, y )

Original: = ( θ 1 k x, − θ 2 k y ) .
Revised: = ( θ 1 k x, − θ 2 k y ) .

Original: θ (cid:63) =
Revised: θ (cid:63) =

Original: ( α, β ) has to be estimated by θ
Revised: ( α, γ ) has to be estimated by θ

Original: k
Revised: k

Original: =
Revised: =

Original: θ 1 k , θ 2 k ) .
Revised: θ 1 k , θ 2 k ) .

Original: h u accounts for the non linear terms ( βxy, δxy ) .
Revised: h u accounts for the non linear terms ( βxy, δxy ) .

Original: Experimental Setting For both DPL and LV experiments, we consider the following setting: we sample the space of initial conditions building 100/50/50 trajectories for the train, validation and test sets.
Revised: Experimental Setting For both DPL and LV experiments, we consider the following setting: we sample the space of initial conditions building 100/50/50 trajectories for the train, validation and test sets.

Original: The sequences share the same parameters; respectively ( gL , k ) , for DPL, and ( α, β, γ, δ ) for
Revised: The sequences share the same parameters; respectively ( gL , k ) , for DPL, and ( α, β, γ, δ ) for

Original: LV.
Revised: LV.

Original: The parameter θ k is set to a neuron (of dimension 1 in the pendulum and 2 for LV) and h u is a 2-layer MLP.
Revised: The parameter θ k is set to a neuron (of dimension 1 in the pendulum and 2 for LV) and h u is a 2-layer MLP.

Original: Further experimental details are available in appendices E.1, E.2 and F.
Revised: Further experimental details are available in appendices E.1, E.2 and F.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Only d( h, f )
Revised: Only d( h, f )

Original: [Table] [Table]
Revised: [Table] [Table]

Original: u
Revised: u

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Identiﬁcation and Prediction Results Tableshows that despite accurate trajectory forecasting, the unconstrained setting «Only d( h, f ) » fails ataccurately estimating the models parameters, showing the need for regularization forparameters identiﬁcation.
Revised: Identiﬁcation and Prediction Results Tableshows that despite accurate trajectory forecasting, the unconstrained setting «Only d( h, f ) » fails at estimating the models parameters, showing the need for regularization for identiﬁcation.

Original: Constraining the norm of the ML component can be insufﬁcient: for LV data, bothAphynity and d( h, f )
Revised: Constraining the norm of the ML component can be insufﬁcient: for LV data, both

Original:  + d( h u , 0) do not accurately estimate the model parameters.
Revised: Aphynity and d( h, f ) + d( h u , 0) do not accurately estimate the model parameters.

Original: However, the control of d( h k , f ) , following eq.
Revised: However, the control of d( h k , f ) , following eq.

Original: (4), significantly improves the parameter identiﬁcation for both 
Revised: (4), signiﬁcantly improves the parameter identiﬁcation for both datasets.

Original: PDL and LV datasets.

Original: Indeed, in the PDL case, h k and f are (pseudo)-periodic of the same period, hence the gain in the performances.
Revised: Indeed, in the PDL case, h k and f are (pseudo)-periodic of the same period, hence the gain in the performances.

Original: Finally, our proposition based on eq.
Revised: Finally, our proposition based on eq.

Original: (4) is able to identify the parameters of DPL and LV equation with a precision of respectively 1 . 56% and 7 . 8%
Revised: (4) is able to identify the parameters of DPL and LV equation with a precision of respectively 1 . 56% and 7 . 8%

Original: beating all considered baselines.
Revised: beating all considered baselines.

Original: Regarding prediction performances, in underconstrained settings ( «Only d( h, f ) » in Table 1), h u learns to corrects the inaccurate h k .
Revised: Regarding prediction performances, in under-constrained settings ( «Only d( h, f ) » in Table 1), h u learns to corrects the inaccurate h k .

Original: Tableand ﬁgs.
Revised: Table 1 and ﬁgs.

Original: 4 and 5 (appendix G.1) show that our proposition provides more consistent prediction performances.
Revised: 4 and 5 (appendix G.1) show that our proposition provides more consistent prediction performances.

Original: These experiments conﬁrm that the constraints on h k and h u arising from the control of the upper bound of eq.
Revised: These experiments conﬁrm that the constraints on h k and h u arising from the control of the upper bound of eq.

Original: (4) increase interpretability and maintainhigh prediction performances.
Revised: (4) increase interpretability and maintain prediction performances.

Original: Throwback to the Afﬁne Case We verify the convergence proved in section 3.3 using the damped pendulum (eq.
Revised: Throwback to the Afﬁne Case We verify the convergence proved in section 3.3 using the damped pendulum (eq.

Original: (15)) linearized in the small oscillations regime (see appendix E.1).
Revised: (15)) linearized in the small oscillations regime (see appendix E.1).

Original: Making an afﬁne hypothesis following eq.
Revised: Making an afﬁne hypothesis following eq.

Original: (12), we apply our alternate projection algorithm and optimize A and D
Revised: (12), we apply our alternate projection algorithm and optimize A and D

Original: A alternately using SGD.
Revised: A alternately using SGD.

Original: Figure 1 shows that we are able to accurately estimate A and D using our proposition, recovering both the oscillation pulsation and the damping coefﬁcient.
Revised: Figure 1 shows that we are able to accurately estimate A and D using our proposition, recovering both the oscillation pulsation and the damping coefﬁcient.

Original: 4. H IGH D IMENSIONAL D YNAMICS
Revised: 4. H IGH D IMENSIONAL D YNAMICS

Original: We now address the learning of transport equations, describing a wide range of physical phenomena such as chemical concentration, ﬂuid dynamics or material properties.
Revised: We now address the learning of transport equations, describing a wide range of physical phenomena such as chemical concentration, ﬂuid dynamics or material properties.

Original: We evaluate the learning setting induced by eq.
Revised: We evaluate the learning setting induced by eq.

Original: (4) and (5) on two physical datasets depicting the evolution of the temperature
Revised: (4) and (5) on two physical datasets depicting the evolution of the temperature

Original: T advected by a time-dependent velocity ﬁeld U and subject to forcing S , following:
Revised: T advected by a time-dependent velocity ﬁeld U and subject to forcing S , following:

Original: [Equation]
Revised: [Equation]

Original: The system state Z =
Revised: The system state Z =

Original: ( T, U, S ) is partially observed, we only access T .
Revised: ( T, U, S ) is partially observed, we only access T .

Original: Every quantities, observed or to estimate, are regularly sampled on a spatiotemporal grid: at each timestep t , the time varying velocity ﬁeld U t writes as U t =
Revised: Every quantities, observed or to estimate, are regularly sampled on a spatiotemporal grid: at each timestep t , the time varying velocity ﬁeld U t writes as U t =

Original: ( u t , v t ) and u t , v t , T t and the forcing term S t are all of size 64 × 64 .
Revised: ( u t , v t ) and u t , v t , T t and the forcing term S t are all of size 64 × 64 .

Original: Experimental Setting We consider as physical prior the advectionphenomenon i.e h k ( T, θ k ) = −∇ .
Revised: Experimental Setting We consider as physical prior the advection i.e h k ( T, θ k ) = −∇ .

Original: ( Tθ k ) .
Revised: ( Tθ k ) .

Original: We learn θ k  to approximate θ (cid:63) =
Revised: Thus, θ k is time-dependent, as we learn it to approximate θ (cid:63) =

Original: U .
Revised: U .

Original: To do so, we identify the velocity ﬁeld θ k from observations of T , which is common practice in oceanography (Béréziat & Herlin, 2015): we learn a mapping between T and U parameterized by a neural network G ψ , so that θ k = G ψ ( T t − k , ..., T t ) ≈ U t .
Revised: We identify the velocity ﬁeld θ k from observations of T , learning a mapping between T and U parameterized by a neural network G ψ , so that θ k = G ψ ( T t − l , ..., T t ) ≈ U t , which is common practice in oceanography (Béréziat & Herlin, 2015).

Original: G ψ is optimized following eq.
Revised: G ψ is optimized following eq.

Original: S remains to be learned by h u .
Revised: S remains to be learned by h u .

Original: h k implements a differentiable semi-Lagrangian scheme (Jaderberg et al., 2015) (see appendix E.3) and h u is a ResNet.
Revised: h k implements a differentiable semi-Lagrangian scheme (Jaderberg et al., 2015) (see appendix E.3) and h u is a ResNet.

Original: G ψ is a UNet.
Revised: G ψ is a UNet.

Original: Training details and a schema of our model are to be found in appendix F.
Revised: Training details and a schema of our model are to be found in appendix F.

Original: [Table]
Revised: [Table]

Original: Synthetic Advection and Source (Adv+S)
Revised: Synthetic Advection and Source (Adv+S)

Original: To test the applicability of the learning setting induced by eq.
Revised: To test the applicability of the learning setting induced by eq.

Original: (4) on partially observed settings, we ﬁrst study a synthetic setting (denoted Adv+S) of eq.
Revised: (4) on partially observed settings, we ﬁrst study a synthetic setting (denoted Adv+S) of eq.

Original: (17) by generating velocity ﬁelds U , simulated following (Boffetta et al., 2001) and adding a source term S inspired by (Frankignoul, 1985).
Revised: (17) by generating velocity ﬁelds U , simulated following (Boffetta et al., 2001) and adding a source term S inspired by (Frankignoul, 1985).

Original: The simulation details are given in appendix E.3.
Revised: The simulation details are given in appendix E.3.

Original: Real Ocean Dynamics (Natl)
Revised: Real Ocean Dynamics (Natl)

Original: We consider a complex dataset emulating real world observations of the North ATLantic ocean (denoted Natl)
Revised: We consider a complex dataset emulating real world observations of the North ATLantic ocean (denoted Natl)

Original: (Ajayi et al., 2019).
Revised: (Ajayi et al., 2019).

Original: Modeling the evolution of T in
Revised: Modeling the evolution of T in

Original: Natl is particularly challenging as its dynamics is chaotic and highly non-linear.
Revised: Natl is particularly challenging as its dynamics is chaotic and highly non-linear.

Original: This simulation is representative of the complexity encountered in real world data.
Revised: This simulation is representative of the complexity encountered in real world data.

Original: The principled approach corresponding to eq.
Revised: The principled approach corresponding to eq.

Original: (4) is insufﬁcient here and one must resort to additional physical information.
Revised: (4) is insufﬁcient here and one must resort to additional physical information.

Original: To illustrate the extension developed in section 4.2, we make use of available auxiliary data: measurements from satellite observations provide a coarse estimate of the surface ocean currents velocity ﬁelds (see appendix E).
Revised: To illustrate the extension developed in section 4.2, we make use of available auxiliary data: measurements from satellite observations provide a coarse estimate of the surface ocean currents velocity ﬁelds (see appendix E).

Original: Then, the learning goal is to reﬁne the approximated velocity ﬁelds to ﬁt the ocean dynamics.
Revised: Then, the learning goal is to reﬁne the approximated velocity ﬁelds to ﬁt the ocean dynamics.

Original: Therefore we proceed as described in eq.
Revised: Therefore we proceed as described in eq.

Original: (5) and enforce d( h k , f prk ) by supervising G ψ with the proxy data (see appendix E.3).
Revised: (5) and enforce d( h k , f prk ) by supervising G ψ with the proxy data (see appendix E.3).

Original: Identiﬁcation and Prediction Results Table 2 indicates that for Adv+S dataset, we estimate accurately the unobserved velocity ﬁelds.
Revised: Identiﬁcation and Prediction Results Table 2 indicates that for Adv+S dataset, we estimate accurately the unobserved velocity ﬁelds.

Original: Qualitatively, Figure 2 shows that controlling our proposed upper bound eq.
Revised: Qualitatively, Figure 2 shows that controlling our proposed upper bound eq.

Original: (4) facilitates the recovery of truthful velocity ﬁelds U along with an accurate prediction of T .
Revised: (4) facilitates the recovery of truthful velocity ﬁelds U along with an accurate prediction of T .

Original: For the highly complex Natl, Table 2 shows that the introduction of auxiliary data following the formulation in eq.
Revised: For the highly complex Natl, Table 2 shows that the introduction of auxiliary data following the formulation in eq.

Original: (5) signiﬁcantly helps identiﬁcation, as the dynamics is too complex to be able to recover physically interpretable velocity ﬁelds using the bound of eq.
Revised: (5) signiﬁcantly helps identiﬁcation, as the dynamics is too complex to be able to recover physically interpretable velocity ﬁelds using the bound of eq.

Original: Regarding prediction performances on the Adv+S data, Table 2 shows that thanks to our truthful estimates of U , our model provides more precise prediction than NODE and Aphynity.
Revised: Regarding prediction performances on the Adv+S data, Table 2 shows that thanks to our truthful estimates of U , our model provides more precise prediction than NODE and Aphynity.

Original: For real world data, thanks to the proxy data our model recovers better velocity ﬁelds terms while providing a better estimate for T .
Revised: For real world data, thanks to the proxy data our model recovers better velocity ﬁelds terms while providing a better estimate for T .

Original: Besides, adding prior knowledge in the prediction systems improves prediction performances: appendix G shows that NODE minimizes d( h, f ) by predicting average and blurred frames.
Revised: Besides, adding prior knowledge in the prediction systems improves prediction performances: appendix G shows that NODE minimizes d( h, f ) by predicting average and blurred frames.

Original: This shows the need for regularization when learning on structured physical data.
Revised: This shows the need for regularization when learning on structured physical data.

Original: Ablation Study We present in Table 3 an ablation study on the Adv+S dataset evidencing the inﬂuence of our learning choices on the resolution of both identiﬁcation and prediction tasks (see appendix G for detailed results).
Revised: Ablation Study We present in Table 3 an ablation study on the Adv+S dataset evidencing the inﬂuence of our learning choices on the resolution of both identiﬁcation and prediction tasks (see appendix G for detailed results).

Original: “Joint” rows of Table 3 indicate that the learning of h u and h k is done simultaneously.
Revised: “Joint” rows of Table 3 indicate that the learning of h u and h k is done simultaneously.

Original: As shown in Table 3, the sole optimization of d( h, f ) fails at estimating physically sounded U .
Revised: As shown in Table 3, the sole optimization of d( h, f ) fails at estimating physically sounded U .

Original: This evidences the ill-posedness in such unconstrained optimization.
Revised: This evidences the ill-posedness in such unconstrained optimization.

Original: Tableindicates that all introduced regularizations improve the recovery of U w.r.t.
Revised: Tableindicates that all introduced regularizations improve the recovery of U w.r.t.

Original: the «Only d( h, f ) » baseline, while adding d( h u , 0) signiﬁcantly improves both prediction performances and velocity ﬁelds estimation.
Revised: the «Only d( h, f ) » baseline, while adding d( h u , 0) signiﬁcantly improves both prediction performances and velocity ﬁelds estimation.

Revised: We highlight that the alternate optimization performs better compared to optimiz-

Revised: ing jointly all parameters of h k and h u .

Original: Notably, our proposition to optimize h k and h u alternately beats all baselines on both T prediction and U identiﬁcation (Table 3, Joint rows).
Revised: Notably, our proposition to optimize h k and h u alternately beats all baselines on both T prediction and U identiﬁcation (Table 3, Joint rows).

Original: Finally, jointly trained models fail at estimating U in Table 3, forcing h u to capture the whole dynamics.
Revised: Finally, jointly trained models fail at estimating U in Table 3, forcing h u to capture the whole dynamics.

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Grey-box or hybrid modeling, combining ODE/PDE and data based models, has received an increasing focus in the machine learning community (Rico-Martinez et al., 1994; Thompson & Kramer, 1994;Brunton et al., 2016; Raissi et al., 2020b).
Revised: Grey-box or hybrid modeling, combining ODE/PDE and data based models, has received an increasing focus in the machine learning community (Rico-Martinez et al., 1994; Thompson & Kramer, 1994; Raissi et al., 2020b).

Original: Hybrid approaches allow for alleviated computational costs for ﬂuid simulation (Tompson et al., 2017; De Avila Belbute-Peres et al., 2020;
Revised: Hybrid approaches allow for alleviated computational costs for ﬂuid simulation (Tompson et al., 2017; De Avila Belbute-Peres et al., 2020;

Original: Wandel et al., 2021), and show better prediction performances through data speciﬁc constraints that preserve physics (Raissi et al., 2020a;
Revised: Wandel et al., 2021), and show better prediction performances through data speciﬁc constraints that preserve physics (Raissi et al., 2020a;

Original: Jia et al., 2019).
Revised: Jia et al., 2019).

Original: They offer increased interpretability via constraints on convolutional ﬁlters (Long et al., 2018; 2019) or on learned residual (Geneva & Zabaras, 2020).
Revised: They offer increased interpretability via constraints on convolutional ﬁlters (Long et al., 2018; 2019) or on learned residual (Geneva & Zabaras, 2020).

Original: Physical knowledge, introduced through ODE/PDE regularization (Psichogios & Ungar, 1992; Bongard & Lipson, 2007; de Bézenac et al., 2018) or Hamiltonian priors (Greydanus et al., 2019; Lee et al., 2021), increases generalization power w.r.t pure ML approaches.
Revised: Physical knowledge, introduced through ODE/PDE regularization (Psichogios & Ungar, 1992; Bongard & Lipson, 2007; de Bézenac et al., 2018) or Hamiltonian priors (Greydanus et al., 2019; Lee et al., 2021), increases generalization power w.r.t pure ML approaches.

Original: Closer to our work, (Mehta et al., 2020; San & Maulik, 2018; Young et al., 2017; Saha et al., 2020) study the learning of a physical model augmented with a statistical component.
Revised: Closer to our work, (Mehta et al., 2020; San & Maulik, 2018; Young et al., 2017; Saha et al., 2020) study the learning of a physical model augmented with a statistical component.

Original: Yin et al.
Revised: Yin et al.

Original: (2021) tackle the same task, ensuring the uniqueness in the decomposition by constraining the norm of the ML component.
Revised: (2021) tackle the same task, ensuring the uniqueness in the decomposition by constraining the norm of the ML component.

Original: We generalize latter approaches and address the well-posedness in the learning of hybrid ML/MB models through additional regularization on the estimated parameters of the physical part.
Revised: We generalize latter approaches and address the well-posedness in the learning of hybrid ML/MB models through additional regularization on the estimated parameters of the physical part.

Original: Indeed, to describe natural phenomena relying on hybrid MB/ML models, one major task lies in the estimation of the MB part parameters.
Revised: Indeed, to describe natural phenomena relying on hybrid MB/ML models, one major task lies in the estimation of the MB part parameters.

Original: This can be done using neural networks (Raissi et al., 2019;2020b; Mehta et al., 2020).
Revised: This can be done using neural networks (Raissi et al., 2019; Mehta et al., 2020).

Original: However, identiﬁcation tasks being intrinsically ill-posed (Sabatier, 2000), imposing prior knowledge or regularization is necessary to ensurephysically sound estimations (Stewart & Ermon, 2017).
Revised: However, identiﬁcation tasks being intrinsically ill-posed (Sabatier, 2000), imposing prior knowledge or regularization is necessary to ensure sound estimations (Stewart & Ermon, 2017).

Original: Yet, using only prediction as supervision, the recovered parameters are not physically interpretable (de Bézenac et al., 2018;
Revised: Yet, using only prediction as supervision, the recovered parameters are not physically interpretable (de Bézenac et al., 2018;

Original: Ayed et al., 2020).
Revised: Ayed et al., 2020).

Original: To ensure uniqueness of the estimation solution, Ardizzone et al.
Revised: To ensure uniqueness of the estimation solution, Ardizzone et al.

Original: (2018) use invertible neural networks.
Revised: (2018) use invertible neural networks.

Original: Linial et al.
Revised: Linial et al.

Original: ; Tait & Damoulas (2020); Saemundsson et al.
Revised: ; Tait & Damoulas (2020); Saemundsson et al.

Original: (2020) combine variational encoding (Kingma & Welling, 2013) and a PDE model, sampling the space of initial conditions and parameters to solve both identiﬁcation and prediction.
Revised: (2020) combine variational encoding (Kingma & Welling, 2013) and a PDE model, sampling the space of initial conditions and parameters to solve both identiﬁcation and prediction.

Original: However, such methods only deal with low-dimensional dynamics.
Revised: However, such methods only deal with low-dimensional dynamics.

Original: Besides low dimensional systems, we also show the soundness of our approach on complex high dimensional and partially observed dynamics.
Revised: Besides low dimensional systems, we also show the soundness of our approach on complex high dimensional and partially observed dynamics.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We propose in this work an algorithm to learn hybrid MB/ML models.
Revised: We propose in this work an algorithm to learn hybrid MB/ML models.

Original: For interpretability purposes, we impose constraints ﬂowing from an upper bound of the prediction error and derive a learning algorithm in a general setting.
Revised: For interpretability purposes, we impose constraints ﬂowing from an upper bound of the prediction error and derive a learning algorithm in a general setting.

Original: We prove its well posedness and its convergence in a linear approximation setting.
Revised: We prove its well posedness and its convergence in a linear approximation setting.

Original: Besides theoretical considerations, we empirically evidence the soundness of our approach thanks to ablation studies and comparison with recent baselines on several low and high dimensional datasets.
Revised: Besides theoretical considerations, we empirically evidence the soundness of our approach thanks to ablation studies and comparison with recent baselines on several low and high dimensional datasets.

Original: This work can see several extensions, e.g. considering non uniform 3-D grid for climate models, further considerations on the investigated upper bounds, or less restrictive decomposition hypothesis.
Revised: This work can see several extensions, e.g. considering non uniform 3-D grid for climate models, further considerations on the investigated upper bounds, or less restrictive decomposition hypothesis.