Paper: A New, Physics-Based Continuous-Time Reinforcement Learning Algorithm with Performance Guarantees (ICLR-2024)

Reviewer comment (Novelty):
The proposed methodology presupposes a comprehensive understanding of system dynamics. However, with known system dynamics, one could conduct policy iteration directly using a "differential" formulation as opposed to the "integral" formulation, which seems unnecessarily convoluted. For instance, a comparison could be made with the "Relaxed Actor-Critic" method detailed in [1], which offers a solution to the HJB equation through policy iteration in the context of fully understood system dynamics. Reference: [1] J. Duan et al., "Relaxed Actor-Critic With Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems," in IEEE Transactions on Intelligent Vehicles, vol. 8, no. 5, pp. 3299-3311, May 2023, doi: 10.1109/TIV.2023.3255264.

Author response:
> Thank you very much for the explanation! It helps me a lot to understand the paper in depth. I can see my comments are similar to those of Reviewer QT7M.

* You are welcome, we are glad to provide some necessary backdrops on basic RL/ADP/control concepts. 

> However, I still find it confusing and struggle to accept the claimed contributions, because compared to methods like RADP and Integral RL, the approach in this paper requires knowledge of system dynamics.

* Firstly: We encourage the Reviewer to GLANCE at how the IRL and RADP methods you argue for fare empirically in prior studies (Wallace \& Si, 2022). These both fail to synthesize for the toy problem:

    \begin{align}
        f(x)
        =
        \left[ 
        \begin{array}{c}
             - x_{1} + x_{2} + 2 x_{2}^{3}
            \\\\
             - \frac{1}{2} (x_{1} + x_{2}) + \frac{1}{2} x_{2} (1 + 2 x_{2}^{2}) \sin^{2}(x_{1})
        \end{array}
        \right],
        \qquad
        g(x)
        =
        \left[ 
        \begin{array}{c}
             0
             \\\\
             \sin(x_{1})
        \end{array}
        \right].      
    \end{align}

* Secondly: As we have now stated in:

   * Table 1 of the manuscript
   * Remark 2.1 of the manuscript
   * Our General Points 2 of this rebuttal
 
  our proposed RCI work requires SIGNIFICANTLY LESS system dynamics knowledge than the leading FVI works in deep RL.

* Allow us to copy and paste here again the "system dynamics requirement":

  * **RCI:** $f, g$
  * **SOTA Deep RL FVIs:** $f, g$, $\partial f/\partial x$, $\partial g/\partial x$, $\partial f/\partial \theta$, $\partial g/\partial \theta$

* Should we tell the greatly-successful deep RL FVI works that their results aren’t SOTA, then?

> From my understanding, when system dynamics are known, we can directly solve ADP using differential forms, utilizing the system's derivative information. Additional derivative information can make differential-form ADP more stable and can be directly solved using neural networks.

* Yes, and No -- 1) we are glad that the reviewer is aware that the Hamilton-Jacobi-Bellman equation (HJB) PDE offers an exact solution to optimal control problems when the dynamics are known. 2) However, except for toy problems, "directly" solving the HJB as you propose is numerically intractable. Hence the inception of **approximate** dynamic programming (ADP). By the way, there is a quick timeline in the development of ADP/RL for optimal control in (Wallace \& Si, 2023).

* We would like to point out to this reviewer that the above toy problem was constructed such that the closed-form HJB solution is known *a priori* (because it is an unrealistic toy problem) in exact polynomial closed-form:

    \begin{align}
        V^{*}(x) = \frac{1}{2} x_{1}^{2} + x_{2}^{2} + x_{2}^{4}
    \end{align}

   Yet still, IRL and RADP fail to synthesize this $V^{*}$.

* **This is a venue for SOTA learning results, not already-published findings.** The "experimental comparisons" you seek for ADP are already well documented in the literature.

* **More on the “differential form” approach:** Please note that, contrary to the Reviewer’s opinion, we are not sure if the work of Duan et al. can be considered SOTA yet for the following reasons: 

   1. Evaluations: Its evaluations are far less than what is typically required and presented in leading venues such as ICLR.
   2. Solution Method: As with all ADP/RL algorithms, these methods still solve the HJB approximately. There need to be more comprehensive and substantial evaluations to solidify its performance characteristics.

* **More on why we consider FVI SOTA:** Also contrary to the reviewer’s belief, we consider FVI results SOTA for the following reasons:

   1. Evaluations: This work presents substantial evaluations on three SOTA CT-RL environments, including generalization studies with respect to system initial conditions and modeling error.
   2. Solution Method: FVI is a time-tested method in the DT setting, and (see Lutter et al., 2023a, 2022) this CT formulation inherits many of the favorable properties in DT.

* **In short about this SOTA method discussion:** We understand the Reviewer’s skepticism, and we also understand the Reviewer’s endorsement of "differential form" approach. We would like to bring to the Reviewer’s attention that our "integral" reinforcement method has been a standard ADP learning principle for almost 15 years (Vrabie \& Lewis, 2009). Instead, the "differential" method (J. Duan et al., 2023) has yet to be proven by substantial evaluations to the standard of highest quality venues such as the ICLR.

Reviewer comment (Writing):
I don’t get what “Thus, RCI can improve learning of existing CT-RL algorithms” mean from the paragraph. Can you elaborate on this?

Author response:
* In regards to this question, the associated Equation (13) (pp. 4) provides a direct means to accommodate reference command inputs for standard-formulation CT-RL algorithms. Thus, the empirical issues observed for CT-RL algorithms which only insert probing noise (Wallace \& Si, 2022) can be overcome by applying reference command input via (13).

Reviewer comment (Novelty):
This work failed to mention other similar works in continuous-time LQR setting where different exploitation of the same linear-quadratic structure (as Kleinman's method) leads to different theoretical guarantees and efficient algorithms. The authors may want to conduct a thorough survey on existing works and compare their approaches with other model-based continuous-time LQR methods. A few examples can be found like:

[1] Jeongho Kim, Jaeuk Shin, and Insoon Yang. Hamilton-jacobi deep q-learning for deterministic continuous-time systems with lipschitz continuous controls. The Journal of Machine Learning Research, 22(1):9363–9396, 2021.

[2] Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. The Journal of Machine Learning Research, 21(1): 8145–8178, 2020.

Author response:
> This work failed to mention other similar works in continuous-time LQR [...]

* Again, we emphasize that RCI is a full nonlinear learning algorithm, not an LQR learning algorithm. Thus, the respective LQR subsections of the two works referenced do not constitute "similar works" to this one. 

* However, even the nonlinear control sections do not furnish direct comparability to the proposed method, for the following reasons:

  * Regarding J. Kim, et al. (2021): This algorithm studies a discounted cost formulation, which requires an entirely different theoretical framework to prove convergence, optimality, and closed-loop stability and does not furnish direct numerical comparability to the undiscounted problem studied. Furthermore, the algorithm requires discretizing the system in order to apply its Q-learning framework. Thus, this is not a properly *continous-time* learning algorithm.

  * Regarding H. Wang, et al. (2020): As is stated in the title, this algorithm requires a stochastic control formulation, which also requires an entirely separate theoretical structure and machinery than the deterministic optimal control problem studied. As with J. Kim, et al. (2021), this work also studies a discounted problem. Furthermore, this algorithm also requires applying limiting arguments to a discretized system and control problem. Thus, this is not a properly *continous-time* learning algorithm.

Reviewer comment (Experiments):
It is questionable if the method in this work can be fairly compared to other general purpose RL methods or ADP methods, since the latter typically won't consider the specific underlying structure of the control problem. The authors may want to proceed more carefully when utilizing FVI as the benchmark and perform comparison for tasks like pendulum for which model-based LQR-type algorithm can easily excel.

Author response:
> The authors may want to proceed more carefully when utilizing FVI as the benchmark and perform comparison for tasks like pendulum for which model-based LQR-type algorithm can easily excel.

* Respectfully, we struggle to fathom how more carefully we could choose a system to benchmark FVI than to select the **identical** pendulum system chosen by the FVI authors for benchmarking in **both** the original cFVI study (Lutter et al., 2023a) and rFVI study (Lutter et al., 2022).

* We explicitly discuss this point in Table 2 of the manuscript.

* We would like to point out that our work studies swing-up of the pendulum, where the system nonlinearities are at their strongest -- far from chosing a comparison for which our algorithm can "easily excel."