Paper: Investigating the chaotic dynamics produced by deep reinforcement learning controllers (ICLR-2024)

Reviewer comment (Writing):
The paper introduces several unfamiliar terms such as "TRAJECTORY STABILITY," "Sensitivity," and "Chaotic," without providing sufficient context or clarification. It is imperative for the author to offer clear definitions of these terms and elaborate on how they relate to Lyapunov stability, which readers might be more acquainted with.

Author response:
**The paper introduces several unfamiliar terms such as "TRAJECTORY STABILITY," "Sensitivity," and "Chaotic," without providing sufficient context or clarification.**
In Section 2 of the revised version, we provide a more detailed explanation of each term.

Reviewer comment (Novelty):
The paper repeatedly emphasizes that the MDP framework can be reinterpreted as a controllable dynamical system. However, this assertion is widely acknowledged and lacks the novelty; therefore, continually highlighting it does not contribute significantly to the paper's academic value.

Author response:
**The paper repeatedly emphasizes that the MDP framework can be reinterpreted as a controllable dynamical system.**
This was included to provide clarity in case the reader was unfamiliar with the relation between the MDP and dynamical systems. This definition is fundamental for measuring the stability of the control system so has been included to provide clarity.

Reviewer comment (Evaluation):
The authors quantify the instability of the system by using the sensitivity of a dynamical system to the initial condition. However, a sensitive controller does not necessarily mean unstable from a control performance perspective. For example, Fig 1 shows sensitive dynamics, but it is hardly to be recognized as unstable.

Author response:
Our experiments show that instability in RL policies also causes instability in the rewards of those policies. To clarify this in the manuscript, we introduce a method for calculating the MLE of the reward over a trajectory and use this to identify if the reward function is stable, unstable or chaotic. We show that for simple low-dimensional environments, reward is stable with exponentially converging reward trajectories. Conversely, for high-dimensional environments, this reward is highly sensitive to initial conditions. As a result, policies which produce unstable state dynamics also produce unstable reward trajectories.

Reviewer comment (Experiments):
The modification of the reward function (2) is weak to me. In order to obtain reference value $\bar{s}$, one has to run the standard problem first, this is rather inefficient. I believe such a method is not practical.

Author response:
This is perhaps something which we did not make clear in the original draft. In section 5 each agent is initially trained for 5 Million (SAC, TD3) or 10 Million (PPO) environment steps. This allows each policy to find a solution to the task and optimize this to attain a high reward. Having completed this training a reference value can be calculated for each policy and used for the constrained reward function. Each policy is then trained for a further 1 Million environment steps using this updated policy in order to improve the stability of the solution. In this sense, the proposed approach can be seen as fine-tuning the policy for stability

Reviewer comment (Reproducibility):
In Fig 5, how to compute the MLE during the training phase?

Author response:
During training a frozen copy of the control policy is combined with the control environment to produce a single, deterministic function which maps states to states. Using this update rule the spectrum of Lyapunov exponents is then calculated using the method outlined by Benettin et al and the MLE is taken as the largest value in the spectrum of Lyapunov Exponents. We have clarified this in Section 4.

Reviewer comment (Evaluation):
The MLE of the walker walk problem shown in Fig 5 seems inconsistent with the ones in Table 2, are they different sets of experiments?

Author response:
This inconsistency has been updated to reflect the results from Section 4.

Reviewer comment (Experiments):
Another weakness is that while a range of RL algorithms are considered, they are not as varied as they could be. The authors should consider other algorithms types and show their dynamics. For example dynamics in off vs on-policy algorithms, or value vs policy based algorithms.

Author response:
**The authors should consider other algorithm types and show their dynamics.**  
This is something we would like to achieve but was not finalized at the time of submission. However, note that the selection of algorithms tested covers both on- and off-policy. In Section 4 we have clarified the definition of each model and specified if they are On Policy (SAC, TD3) or Off Policy (PPO).

Reviewer comment (Writing):
Please define the co-domain A of the function/policy \pi in section 3.

Author response:
**Please define the co-domain A of the function/policy \pi in section 3.**  
This definition has been added to Section 3.

Reviewer comment (Evaluation):
From figure 2, it seems like PPO has difficulties solving CR. Is this why in figure 4 we see PPO having an unusually high MLE for CR? If so, this needs to be addressed in the paper.

Author response:
**From figure 2, it seems like PPO has difficulties solving CR. Is this why in figure 4 we see PPO having an unusually high MLE for CR?**  
While the lower performance of PPO does have an impact on the dynamics produced by the Cheetah Run environment, we have also found that the sum of all the Lyapunov exponents is positive which indicates diverging dynamics. As a result, the trajectories for this system are not bound by a chaotic attractor and can diverge indefinitely in the phase space. This lack of converging dynamics allows for a larger MLE.

Reviewer comment (Presentation):
Please expand figure 5 to include the other tasks as well.

Author response:
**Please expand figure 5 to include the other tasks as well.**  
This figure has been updated to include the other environments. Furthermore, the raw values have been replaced with the mean and standard deviation at each time step to provide clarity.