Paper: Impact of Computation in Integral Reinforcement Learning for Continuous-Time Control (ICLR-2024)

Reviewer comment (Theory):
While CT formulation helps in some analysis or for some applications, it was a bit unclear what the motivations behind studying CTRL if the task can be done with DT formulations. (Especially when the time interval is even; in which case DT can well manage.) In particular, for DT and CT systems, there should be different conditions for the solutions to exist. For contact-rich dynamics for example, this kind of analysis becomes harder for example. (Does PIM ensures the existence of solutions throughout the whole process?) Also, for stochastic systems, CT requires more conditions for certain analysis.

Author response:
**[Q1: The motivations behind studying CTRL are unclear]**

**Response:**

Many physical and biological systems are inherently continuous in time, governed by ordinary differential equations (ODEs). Compared to discretising time and then applying discrete-time (DT) RL algorithms, directly employing CTRL offers these advantages [1]:

1. **Smoother Control Law**: Direct application of CTRL typically results in smoother control outputs. This contrasts with the outcomes of coarse discretization, where control is less smooth and can lead to suboptimal performance [2].
2. **Time Partitioning**: When addressing CTRL directly, there's no need to predefine time partitioning. Instead, it is efficiently managed by numerical integration algorithms, which find the appropriate granularity [3]. Thus, CTRL is often a better choice when the time interval is uneven.


In light of these factors, we believe that studying CTRL is crucial. 

**[Q2: For DT and CT systems, there should be different conditions for the solutions to exist. Does PIM ensure the existence of solutions throughout the whole process?]**

**Response:**

We are not sure about what the word "solution" refers to. So, we answer this question based on two circumstances:

**(1) If "solution" refers to the solution of ODE, i.e., the trajectory of the state**

This is indeed an important question. 

First, we want to highlight that our algorithm doesn't directly solve the ODE. Instead, it approximates the value function using *data* (state samples) directly in the policy evaluation (PEV) step in each iteration.

Second, for continuous time-invariant systems, the Lipschitz continuity of the closed-loop systems can guarantee the existence of the solution. For the control affine system (4) in our paper, if the functions $f$ and $g$ are Lipschitz continuous, and PIM in each iteration yields a continuous control policy $u^{(i)}$, then $f(x)+g(x)u^{(i)}$ will also be Lipschitz continuous. In this case, the existence of solution of the closed-loop system can be guaranteed.

**(2) If "solution" refers to the solution of the algorithm, i.e., the optimal controller**

DT and CT systems have different conditions for the optimality of solutions. DT systems focus on the Bellman optimality equation, while CT systems center on the Hamilton–Jacobi–Bellman (HJB) equation. The table below summarizes these conditions:

| System |                     Optimality Condition                     |          Condition of PEV           |
| :----: | :----------------------------------------------------------: | :---------------------------------: |
|   DT   | Bellman optimality equation: $V^*(k)=\min_u\\left\\{l(x(k),u(k))+V^*(x(k+1))\\right\\}$ |       Bellman equation: (1a）       |
|   CT   | HJB equation: $\min_u\\left\\{\dot{V}^*(x(t))+l(x(t), u(t))\\right\\}=0$; Equal to (7) for control affine systems | (Nonlinear) Lyapunov function: (1b) |

In ideal conditions without learning errors or computational errors, iterative policy iteration in CT systems will converge to the optimal control solution, given an initial admissible policy [4]. In the presence of computational error, the convergence analysis of the policy iteration is addressed in our paper's Theorem 2.

**References**

[1] Yu Jiang and Zhong-Ping Jiang. Robust adaptive dynamic programming. John Wiley & Sons, 2017.

[2] Kenji Doya. Reinforcement learning in continuous time and space. Neural Computation, 12(1): 219–245, 2000

[3] Cagatay Yildiz, Markus Heinonen, and Harri L ̈ahdesm ̈aki. Continuous-time model-based reinforcement learning. In International Conference on Machine Learning, pp. 12009–12018. PMLR, 2021.6

[4] Draguna Vrabie and Frank Lewis. Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Networks, 22(3):237–246, 2009



**Due to space constraints, we have divided the comment into 2 parts. Please see Part 2.**

Reviewer comment (Theory):
About approximation error of value functions; if we know the utility lives in certain RKHS, can we say anything about the value function which may validate the assumptions? At least, there should be a trivial case for this assumption: if you know the value function exactly, that becomes a single basis function; when is it continuously differentiable? A bit more discussions needed. (also there is a type (additional “]”) for the interval for the integral value.)

Author response:
**[Q5: If we know the utility lives in certain RKHS, can we say anything about the value function which may validate the assumptions]**

**Response:**

This is an important question. The value function in a control system is determined not only by the utility function but also by the underlying dynamical system. This relationship can be verified when we combine the continuous-time (CT) Bellman equation (1a) with the system dynamics (4), leading to the equation:
$$
\nabla_x{V}(x)(f(x)+g(x)u) = -l(x(t), u(x(t)).
$$
However, the precise formulation of the value function in a general nonlinear system is more complex and challenging to analyze. As of now, we cannot provide a definitive answer.

But, we can restrict our analysis to linear quadratic systems with system dynamics $\dot{x} = Ax + Bu$ and utility function $l(x,u)=x^{\top}Qx+u^{\top}Ru$. In this case, if $[A,B]$ is controllable, the value function is known to be quadratic [1], represented as $V(x)=x^{\top}Px$, where $P\succeq0$. Hence, the value function is continuously differentiable, satisfying the conditions outlined in Assumption 1.

Besides, the conditions for the basis function in Assumption 1 can also be verified. The key idea is to choose the basis function $\phi(x)$ as the non-redundant elements of $x\otimes x$, with $\otimes$ denoting the Kronecker product. Therefore, in this framework, $V(x)$ can be represented as $V(x) = \omega^{\top}\phi(x)$, with $\omega,\phi \in \mathbb{R}^{\frac{n_x(n_x+1)}{2}}$ and $\omega$ represents the non-redundant components of $\mathrm{vec}(P)$. For example, in a two-dimensional system, $\phi(x) := \begin{bmatrix}x_1^2 & x_1x_2 & x_2^2\end{bmatrix}^{\top}$. In such a configuration, $\phi(x)$ fulfils Assumption 1, which encompasses continuity, differentiability, and linear independence.

We also appreciate your pointing out the typo regarding the integral value interval, which has been corrected in the updated version of our paper.

Reviewer comment (Theory):
In Appendix H; no approach to find a suitable T? There may require some discussions on how rare the independence property fails for a random T.

Author response:
**[Q6: In Appendix H; no approach to find a suitable T? How rare the independence property fails for a random T?]**

In Appendix H, we highlight that a suitable integration interval $T$ is essential to ensure that the matrix $\Theta^{(i)}$ achieves full column rank, which is necessary for the least squares method to produce a unique solution. The full rank of  $\Theta^{(i)}$ is primarily determined by two factors:

1. **Number of Data Points $m$**: It's advisable to have the number of rows in $\Theta^{(i)}$ be at least twice the dimension of the basis function. Practically, this means ensuring that $m$, the number of data points, is greater than twice $n_{\phi}$. This approach aligns with the guidance provided in [1] (Remark 2.3.2).
2. **Integration Interval $T**: In our experiments, we found that achieving full rank for $\Theta^{(i)}$ in nonlinear systems is generally straightforward. We did not come across any instances where this condition was not met. For linear systems, an integration interval of $T>0.1$ is typically sufficient to satisfy the full rank condition.

Reviewer comment (Evaluation):
For Appendix G, not only for the figure for the utility itself, a plot for the integral of the utility and the worst case error would be informative too.

Author response:
**[Q7: For Appendix G, not only for the figure for the utility itself, a plot for the integral of the utility and the worst case error would be informative too.]**

Thank you for your valuable suggestion! We have included the requested figures in Appendix G.

Reviewer comment (Experiments):
The paper provides theoretical claims about the impact of computational methods on control performance in IntRL. However, the experimental validation seems to be limited in scope.

Author response:
Thank you for this important question. To apply to high-dimensional and real-world scenarios will definitely make a huge impact. In fact, we did try some complicated tasks such as drones, autonomous vehicles, etc. Unfortunately, we can't train the algorithm to converge; thus, it wasn't shown in this paper.

We analyze the failure and recognize that this is mainly due to the inherent limitation of the integral RL (IntRL) algorithm. IntRL is a canonical control task in CTRL, just like the Q learning in DTRL. However, IntRL is still underdeveloped for high-dimensional systems compared to Q learning. As shown in [1], [2], and [3], IntRL is typically applied to systems with no more than 4 dimensions. Additionally, we wish to emphasize that our paper primarily focuses on analyzing the phenomenon of "computation impacts control" rather than on applying IntRL in complex systems. 

However, since we recognized the importance of high-dimensional applications, we have considered the challenges associated with this. Specifically, the difficulties in achieving convergence in high-dimensional systems can be attributed to several factors:

1. **Limited Approximation Capability**: IntRL uses a linear combination of basis functions to approximate the value function, which is less powerful than the neural network-based approaches prevalent in DTRL.
2. **Lack of Effective Exploration Mechanisms**: IntRL struggles with poor data distribution due to inadequate exploration strategies. Common DTRL methods improve exploration by adding noise during data collection, but in continuous systems, this leads to complex stochastic differential equations (SDEs) that are challenging to manage.
3. **Absence of Advanced Training Techniques**: Techniques like replay buffers, parallel exploration, delayed policy updates, etc., which enhance sample efficiency and training in DTRL, are lacking in IntRL.
4. **No Discount Factor**: The absence of a discount factor in IntRL makes it difficult to ensure that policy iteration acts as a contraction mapping, which can affect the stability of the algorithm's training process. 
5. **Non-existence of Q function**: In CT systems, the concept of a Q function is not directly applicable. The absence of Q function impedes the direct translation of many DTRL methodologies and insights to IntRL.

Considering these challenges, we will investigate and develop CTRL algorithms for complex and high-dimensional scenarios in future research. 

Since this is an important question, we add Appendix K in our paper to discuss the limitations of IntRL.

Reviewer comment (Experiments):
The authors only consider canonical control tasks to validate their findings. The authors could consider a broader set of experiments, including more complex and real-world scenarios, to showcase the practical implications of their findings.

Author response:
Thank you for this important question. To apply to high-dimensional and real-world scenarios will definitely make a huge impact. In fact, we did try some complicated tasks such as drones, autonomous vehicles, etc. Unfortunately, we can't train the algorithm to converge; thus, it wasn't shown in this paper.

We analyze the failure and recognize that this is mainly due to the inherent limitation of the integral RL (IntRL) algorithm. IntRL is a canonical control task in CTRL, just like the Q learning in DTRL. However, IntRL is still underdeveloped for high-dimensional systems compared to Q learning. As shown in [1], [2], and [3], IntRL is typically applied to systems with no more than 4 dimensions. Additionally, we wish to emphasize that our paper primarily focuses on analyzing the phenomenon of "computation impacts control" rather than on applying IntRL in complex systems. 

However, since we recognized the importance of high-dimensional applications, we have considered the challenges associated with this. Specifically, the difficulties in achieving convergence in high-dimensional systems can be attributed to several factors:

1. **Limited Approximation Capability**: IntRL uses a linear combination of basis functions to approximate the value function, which is less powerful than the neural network-based approaches prevalent in DTRL.
2. **Lack of Effective Exploration Mechanisms**: IntRL struggles with poor data distribution due to inadequate exploration strategies. Common DTRL methods improve exploration by adding noise during data collection, but in continuous systems, this leads to complex stochastic differential equations (SDEs) that are challenging to manage.
3. **Absence of Advanced Training Techniques**: Techniques like replay buffers, parallel exploration, delayed policy updates, etc., which enhance sample efficiency and training in DTRL, are lacking in IntRL.
4. **No Discount Factor**: The absence of a discount factor in IntRL makes it difficult to ensure that policy iteration acts as a contraction mapping, which can affect the stability of the algorithm's training process. 
5. **Non-existence of Q function**: In CT systems, the concept of a Q function is not directly applicable. The absence of Q function impedes the direct translation of many DTRL methodologies and insights to IntRL.

Considering these challenges, we will investigate and develop CTRL algorithms for complex and high-dimensional scenarios in future research. 

Since this is an important question, we add Appendix K in our paper to discuss the limitations of IntRL.

Reviewer comment (Experiments):
Could you provide more details on the choice of the canonical control tasks used for experimental validation? Were any real-world scenarios or more complex tasks considered?

Author response:
Thank you for your insightful question. Our selection of canonical control tasks was intentionally based on seminal works within the field of IntRL. Example 1 (linear system) in our paper is adapted from Example 1 in Chapter 2 of [4], and Example 2 (nonlinear system) is based on Example 1 in [1]. These specific examples were chosen for their significance and representational value in the established IntRL literature.

Our research is primarily centered on introducing and demonstrating the "computation impacts control" phenomenon using integral reinforcement learning (IntRL) algorithms. We acknowledge the challenges in extending IntRL to high-dimensional systems, and this is a limitation of our current approach. We hope that in the future, together with the community, we can further broaden its application to more complex systems.

Reviewer comment (Theory):
Corollary 1 assumes that \( i \to +\infty \). By using the decay rate of the computation error in Theorem 1, can the authors comment on a more realistic convergence rate based on different number of iterations \( i \)? Further, can the theoretical results provide some guidelines on how to choose the number of iterations to terminate at?

Author response:
Yes, of course! According to Theorem 3, the convergence rate considering the number of iterations can be described as $O(2^{-i})+O(N^{-b})$, where $i$ represents the algorithm's iteration steps, and $N$ is the sample size. The term $O(2^{-i})$ indicates the error due to insufficient iterations for convergence. Since this error decays exponentially with increasing $i$, it rapidly approaches zero. 

In simulations, especially for low-dimensional systems, we generally find that $i\geq10$ ensures convergence.

Reviewer comment (Experiments):
How does this result compare with LSPI, which is also based on a linear combination of basis functions?

Author response:
Thank you for this intriguing question. Indeed, when referring to least-squares policy iteration (LSPI) [1], there are similarities between LSPI and Integral RL (IntRL) in terms of approximating the value function using linear basis functions. From this perspective, IntRL can be viewed as a continuous-time (CT) counterpart of LSPI.

LSPI employs the least squares method to solve for the Q function, while IntRL focuses on the V function.  However, the concept of a Q function can not be applicable to CT systems [2]. This is because in the context of CT systems, if we were to define the Q function as it is in discrete-time systems, then the control input can be instantaneously switched to an optimal control without impacting the overall cost.

Therefore, LSPI is not directly applicable to CT systems, and the IntRL algorithm itself can be seen as the CT version of LSPI.

Reviewer comment (Theory):
Are the affine nonlinear systems necessary? Can you consider a more general class of systems?

Author response:
The original research on IntRL indeed focused primarily on affine nonlinear systems. In fact, most continuous-time reinforcement learning (CTRL) papers, such as [2], [3], and [7], concentrate on control affine systems with quadratic costs. This focus is largely due to the fact that, in such scenarios, policy improvement method (PIM) does not require knowledge of the system's internal dynamics and provides an explicit solution (as detailed in equation (8b) of our paper).

However, when the internal dynamics of a system is known, IntRL algorithms can indeed be extended to a broader class of systems and cost functions. Consider a general nonlinear system with dynamics $\dot{x}=f(x,u)$ with a cost function $l(x,u)$.   In this case, the policy iteration (PI) process in CTRL involves:

**Policy evaluation (PEV):** Evaluate the value of the control policy using
$$
V^{(i)}\left(x(t)\right) = \int_{t}^{t+\Delta{T}} l(x(s), u^{(i)}(x(s)) \\,\mathrm{d}s + V^{(i)}\left(x(t+\Delta{T})\right).
$$
**Policy improvement (PIM):** Determine an improved policy
$$
u^{(i+1)}(x) = \underset{u}{\arg\min} \\left\\{l(x,u) + (\nabla_x {V^{(i)})^{\top}f(x,u)}\\right\\}.
$$
The corresponding Hamilton-Jacobi-Bellman (HJB) equation is
$$
V^{\star}(x(t)) = \min_{u(t:t+T)} \\left\\{ \int_t^{t+T}l(x(s),u(s)) \\,\\mathrm{d}s+ V^{\star}(x(t+T))\\right\\}.
$$

Technically, the convergence analysis in our paper is also limited to control affine systems with quadratic costs due to our approach of treating the HJB equation as an iteration of Newton's method. For such systems, the HJB equation simplifies to a function of $V$, i.e., $G(V)=0$  (as shown in equation (7)). However, for general nonlinear systems, their corresponding HJB equations are more complex, making the transformation of PI iteration into Newton's method iteration unfeasible.

In summary, to better elucidate our theory, we restrict our analysis to control affine systems. The convergence analysis of general nonlinear systems involves more intricate mathematical tools and is a subject for future exploration.

Reviewer comment (Theory):
The assumptions of Theorem 2 need more explanations. To guarantee those assumptions, what properties do you need for the systems functions \( f \), \( g \), and cost function \( J \)?

Author response:
Thank you for highlighting the need for more detailed explanations of Theorem 2's assumptions. 

Regarding the first part of the question, the central assumption of Theorem 2 is the selection of an appropriate initial control policy $u^{(0)}$, which ensures that the mapping  $G(V)$ meets specific criteria in the vicinity of the value function $V^{(0)}$. This vicinity corresponds to the range covered by the first iteration of Newton's method. We aim for the mapping $G(V)$ to maintain second-order continuity (Assumption i), have invertible derivatives (Assumption iii), and ensure that these derivatives are sufficiently large (Assumptions ii and iv).  Furthermore, when the computational error $\bar{\epsilon}$ is small enough, Assumptions (iv) and (v) are more easily satisfied.

Regarding the second part of the question, we have to admit that we don't have a definite answer to show the properties yet. Now we try to explain it qualitatively.

Determining the quantitative relationship between the system's properties and the assumptions in Theorem 2 is a challenging question. Current convergence conclusions predominantly build on the findings in [8], which state that convergence is achievable if $f$ and $g$ are Lipschitz continuous and the initial control policy is admissible. However, the framework in [8] does not account for computational errors, this is precisely why we did not directly use their conclusions in our analysis. A deeper exploration into this, potentially building upon [8]'s analysis to outline necessary system assumptions, would be a significant challenge at now. We think we require more advanced mathematical tools to analyze this question. We admit that establishing a quantitative relationship would significantly improve our understanding of the relationship between policy iteration, dynamic systems, and Newton's method. So we plan to tackle this complex task in future work.

However, it's important to note that even if the conditions of Theorem 2 cannot be directly translated into checkable system characteristics, our proof remains rigorous and the conclusions valid. 

**References**

[7] Hamidreza Modares, Frank L Lewis, and Mohammad-Bagher Naghibi-Sistani. Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems. Automatica, 50(1):193–202, 2014.

[8] Randal W Beard, George N Saridis, and John T Wen. Galerkin approximations of the generalized hamilton-jacobi-bellman equation. Automatica, 33(12):2159–2177, 1997.

Reviewer comment (Experiments):
The experimental examples are basically toy examples (3D linear system and 2D nonlinear system). These are far from high-dimensional real-time applications. How the computation error and convergence will behave in higher dimensional cases need to be examined.

Author response:
This is the same question as commented by reviewer jx6Y. So we just copy the answer.

To apply to high-dimensional and real-world scenarios will definitely make a huge impact. In fact, we did try some complicated tasks such as drones, autonomous vehicles, etc. Unfortunately, we can't train the algorithm to converge; thus, it wasn't shown in this paper.



We analyze the failure and recognize that this is mainly due to the inherent limitation of the integral RL (IntRL) algorithm. IntRL is a canonical control task in CTRL, just like the Q learning in DTRL. However, IntRL is still underdeveloped for high-dimensional systems compared to Q learning. As shown in [1], [9], and [10], IntRL is typically applied to systems with no more than 4 dimensions. Additionally, we wish to emphasize that our paper primarily focuses on analyzing the phenomenon of "computation impacts control" rather than on applying IntRL in complex systems. 

However, since we recognized the importance of high-dimensional applications, we have considered the challenges associated with this. Specifically, the difficulties in achieving convergence in high-dimensional systems can be attributed to several factors:

1. **Limited Approximation Capability**: IntRL uses a linear combination of basis functions to approximate the value function, which is less powerful than the neural network-based approaches prevalent in DTRL.
2. **Lack of Effective Exploration Mechanisms**: IntRL struggles with poor data distribution due to inadequate exploration strategies. Common DTRL methods improve exploration by adding noise during data collection, but in continuous systems, this leads to complex stochastic differential equations (SDEs) that are challenging to manage.
3. **Absence of Advanced Training Techniques**: Techniques like replay buffers, parallel exploration, delayed policy updates, etc., which enhance sample efficiency and training in DTRL, are lacking in IntRL.
4. **No Discount Factor**: The absence of a discount factor in IntRL makes it difficult to ensure that policy iteration acts as a contraction mapping, which can affect the stability of the algorithm's training process. 
5. **Non-existence of Q function**: In CT systems, the concept of a Q function is not directly applicable. The absence of Q function impedes the direct translation of many DTRL methodologies and insights to IntRL.

Considering these challenges, we will investigate and develop CTRL algorithms for complex and high-dimensional scenarios in future research. 

Since this is an important question, we add Appendix K to our paper to discuss the limitations of IntRL.

**References**

[9] Hamidreza Modares, Frank L Lewis, and Mohammad-Bagher Naghibi-Sistani. Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems. Automatica, 50(1):193–202, 2014.

[10] Brent A Wallace and Jennie Si. Continuous-time reinforcement learning control: A review of theoretical results, insights on performance, and needs for new designs. IEEE Transactions on Neural Networks and Learning Systems, 2023.

In closing, we hope that our responses to the comments have addressed your questions and concerns. We deeply appreciate the opportunity to engage in this insightful dialogue, and we are open to any further suggestions or clarifications that may enhance the quality and understanding of our work. Thank you for your valuable input, which has been instrumental in enriching our research.