Paper: A Simple Open-Loop Baseline for Reinforcement Learning Locomotion Tasks (ICLR-2024)

Reviewer comment (Experiments):
Although the method requires minimal parameters, a lot of trial-and-error is needed in determining the number of oscillators to use for each environment, which is in direct contrast to RL methods which are general across multiple environments. The need for fine-tuning in each context may limit the method's scalability.

Author response:
Thank you for your remark.
It seems that our initial description may have led to a misunderstanding.
To clarify, the number of oscillators used in our method is the same as the number of joints, since they operate in joint space.
In practice, extending the method to additional environments is relatively straightforward; for example, it took the authors only 20 minutes to modify the code from Hopper to Walker.

Reviewer comment (Novelty):
While this result is really nice, this is also very expected. When maximizing the default reward of these environments, the resulting policy always corresponds to cyclical behavior which suggests that a simple cyclical controller could solve the task. Furthermore, these environments are very simple, and this work further intensifies the importance of moving on from these simple environments and focusing on harder tasks.

Author response:
While the success of the open-loop oscillators in the Swimmer environment is anticipated, its effectiveness in the Walker, Hopper or quadruped environments is more unexpected, as one might assume that feedback control or inverse kinematics would be necessary to balance the robots or to learn a meaningful open-loop policy.
Another surprising fact is the tiny number of parameters needed compared to DRL algorithms like DDPG, which have almost a million of parameters.
The results with RBF policies or ARS with linear policy were already unexpected when they came out [1, 2].

It is true that studies of nature and experiments have shown that periodic control is at the heart of locomotion [3], but it is unclear and surprising how simple a controller can be.
As discussed in the last section of the paper, the fact that a single global frequency (per phase for hopper/walker) shared by each joint oscillator is sufficient is also an interesting finding related to another field that studies the intrinsic oscillation of robotic systems (nonlinear modes).

The surprise expressed by R2:vPWP and R3:w6U7 at the failure of DRL in a seemingly straightforward sim-to-real transfer scenario underscores the importance of the insights presented in our paper.
By challenging common assumptions and exploring alternative solutions, we hope to contribute to the application of RL to robotics. 

We updated the discussion section to underscore more the unexpected results from our paper.

[1] Rajeswaran, A. et al, Neurips 2017. Towards generalization and simplicity in continuous control. \
[2] Mania, H. et al. Neurips 2018. Simple random search of static linear policies is competitive for reinforcement learning.
[3] Ijspeert, A (2008) "Central pattern generators for locomotion control in animals and robots: a review."

Reviewer comment (Novelty):
In terms of novelty of the work, central pattern generators have been applied to legged locomotion in different ways in much more realistic and complex setting, for example it’s been used to build a better action space for RL as described in Bellegarda, et al [1]. In this case, it’s hard to see the value of applying CPG to many simple locomotion tasks (Mujoco-v4 in gymnasium). Though Gym contains simple locomotion tasks, no matter whether it’s hard to solve or not, it’s mainly designed for evaluating different RL methods; providing a specific manually designed controller for these tasks can hardly address issues in current RL methods.

Author response:
Thanks for your comment.

Our goal is not to replace DRL methods, nor to claim to be state of the art, but as the reviewer noted, to provide insight into "common pitfalls of existing RL techniques" (see Sections 4.3 and 4.4 for instance), as recommended in section 4 of [1].
The MuJoCo locomotion tasks are indeed simple, but they still require hundreds of thousands of parameters and thousands of trials to solve, and are widely used in the RL community.
The complexity of DRL algorithm makes it hard to reproduce, and the computation power required prevent them from being used on an on-board computer.
Being aware of the failures of DRL on simple problems like the elastic quadruped, which illustrates the challenges of sim2real, and understanding the cost of complexity is crucial for improving application of RL to robotics.

To clarify the position of the paper, we have updated the abstract, the introduction and discussion sections.

[1] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."

Reviewer comment (Experiments):
Though this work is compared with several RL methods trying to study the robustness of current RL methods, no recent works are compared. The most recent work compared in this work is SAC.

Author response:
Thank you for your remark.

We use SAC v3 (with TD3 double critics and automatic entropy tuning [2]), which is an algorithm that provides strong baselines for MuJoCo tasks (similar performance to AWR), is widely used for continuous control problems, and is the basis for recent sample-efficient variants like TQC, REDQ or DroQ [3, 4, 5].

We could replace SAC with its distributional variant TQC [5], but this won't change the findings or the discussion of the paper.
TQC has slightly better performance in simulation (still much better than a simple open-loop baseline) and a worse runtime compared to SAC [6].
However, because it shares the same policy structure (among other elements), it has the same shortcomings as SAC for the sim2real experiment (e.g., learning high-frequency control), and that's exactly why we use SAC as a baseline.
We believe that SAC is already complex enough (especially with respect to an open-loop model-free baseline), and since SAC and its variants are widely used by the RL community in robotics, providing insight into SAC's pitfalls provides insight into the variants that share the same components.

It is for a similar reason that we use DDPG, to provide better context for the reader (following [1]), despite it being outdated.

We updated the experiments' section to reflect our choice.

[1] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning." \
[2] Haarnoja, T, et al. (2019) "Learning to walk via deep reinforcement learning." \
[3] Chen, X, et al. (2020) "Randomized ensembled double q-learning: Learning fast without a model." \
[4] Hiraoka, T, et al. (2022) "Dropout q-functions for doubly efficient reinforcement learning." \
[5] Kuznetsov, A, et al. (2020) "Controlling overestimation bias with truncated mixture of continuous distributional quantile critics." \
[6] https://github.com/openrlbenchmark/openrlbenchmark and https://wandb.ai/openrlbenchmark/sb3

Reviewer comment (Experiments):
When it comes to sim2real, many rewards like minimizing the energy consumption, penalizing the high frequency action have been well-studied, as well as the domain randomization techniques. When studying sim2real performance, these techniques should be applied to see the actual performance difference. It would be great to apply these techniques if this work is trying to claim the performance of the proposed method in sim2real transfer.

Author response:
Thank you for your comment.

We intentionally simplified the environment and removed task-specific heuristics in our sim2real experiments to understand why RL fails in this case and to observe the resulting behavior.
Our goal is to provide insight and stimulate discussion about the trade-offs between complexity, generality, and performance in current deep RL algorithms.

While we could improve our results by adding complex reward engineering, randomization, or low-pass filtering, our intention is to propose the simplest model-free baseline possible.
It would also be unfair to compare our simple open-loop baseline to a highly tuned DRL algorithm that incorporates many implementation details, uses the state as input, and has 10,000 more parameters.

We have updated the sim2real section to make that point clearer.

Reviewer comment (Novelty):
The method is not innovative. In fact, the baseline of many previous RL work was a simple open-loop controller. However, whether those works only used RL [1] or considered the combination of RL and classical control algorithms [2][3], they all produced much more impressive locomotion control results.

Author response:
Thank you for your feedback.

The authors assume that the reviewer is referring to Kumar et al. (2021), Iscen et al. (2019) and Miki et al. (2022) when mentioning that "the baseline of many previous RL work was a simple open-loop controller".
(If we are mistaken, we kindly ask the reviewer to provide us with specific references. We will gladly incorporate them into our related work section.)

We believe that our work has a different scope than these three papers.

In Kumar et al. (2021), the focus is on proposing a new method for sim2real transfer, not understanding RL failures through a comparison with an open-loop baseline, which we were not able to find in that work.
Iscen et al. (2019) propose a new approach to combine learning and trajectory generation. The trajectory generator is tailored to the quadruped problem (defined in task space) and is not compared to the learned policy (only briefly mentioned in the experiment section).
In Miki et al. (2022), the goal is to learn robust locomotion policies using a height scan in addition to proprioceptive data.
The observation and action spaces are inspired by CPG, but no open-loop baseline is used.

In summary, the three papers do not propose a simple baseline for RL tasks but rather focus on bootstrapping learning with robot knowledge, and provide results only for quadrupeds.
In contrast, we present a simple baseline that is easy to study, easy to adapt to new problems and evaluate it on a popular benchmark using six different robot morphologies.

We would also like to clarify that the MuJoCo locomotion tasks, which we use in our experiments, were originally designed to evaluate model-based control methods [1], and were later adopted for continuous control RL.
To the best of our knowledge, the simplest baseline previously employed for these tasks was a linear policy [2], while open-loop control has not been extensively explored in this context.

Since the MuJoCo locomotion tasks are widely used in the RL community (for both online and offline RL), we believe that providing a very simple baseline would provide valuable insights and better understanding of RL algorithms and their limitations (see Section 4.3 for instance).

[1] Todorov E. et al. (2012) "Mujoco: A physics engine for model-based control." \
[2] Mania H. et al. (2018) "Simple random search provides a competitive approach to reinforcement learning"

Reviewer comment (Evaluation):
I think the training results of RL methods in this paper are much worse than the level of other RL for locomotion papers, and they have not considered clearly what problems RL for control policy solves (like robustness in unseen environments).

Author response:
Thank you for your remark.
It appears we had not yet made the paper’s stance clear enough.
Our intention is not to replace DRL algorithms or to advance SOTA in performance, but to shed light on their limitations (such as failure on simple problems [3, 4, 5]) and to open discussion on their current complexity and "common pitfalls" (as noted by R1:nvZX, R3:w6U7, and R4:bfuq).

Following the recommendations of Section 4 of [6], we designed the experiments to provide "insights rather than state-of-the-art claims".
For example, we show that only ten parameters are sufficient for a model-free approach that doesn't depend on the current state in several locomotion tasks, compared to the hundreds of thousands used by current DRL algorithms.
Similarly, our results suggest that DRL algorithms are very brittle when exposed to sensor noise or failure, even on an uninformative sensor (this was confirmed by the experiment we added where SAC is trained with noise on the first sensor).

Current DRL algorithms can indeed solve more challenging locomotion tasks, but despite their generality, they require numerous implementation details and careful task design such as reward engineering, low-pass filtering or domain randomization (mentioned in the "discussion" section).
Without these heuristics, DRL succeeds in simulation but fails to transfer on simple tasks like the one presented in the paper.
On the same task, a very simple baseline that uses a single frequency for all joints succeeds.
We believe that understanding the failure case on simple problems that are commonly used in the RL community is important for advancing the field.

To clarify the position of the paper, we have updated the abstract, the introduction and discussion sections.
If the reviewer has other suggestions to make our intentions clearer, we would be happy to incorporate them.

[3] Colas, C et al. (2018) "Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms." \
[4] Franceschetti, Maël, et al. (2022) "Making Reinforcement Learning Work on Swimmer." \
[5] Matheron, G et al. (2019) "The problem with DDPG: understanding failures in deterministic environments with sparse rewards." \
[6] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."

Reviewer comment (Novelty):
The paper has limited inspiration for the field of machine learning, which is the main topic of this conference.

Author response:
We acknowledge that our paper provides a robotics perspective for reinforcement learning practitioners.
However, we also believe that the insights provided in the paper (and demonstrated by the real robot experiment) are valuable for applications of RL in robotics (which is one of the topics mentioned in the ICLR 2024 paper call [7], see "applications to robotics, autonomy, planning").

[7] https://iclr.cc/Conferences/2024/CallForPapers

Reviewer comment (Evaluation):
In experiments, why the authers choose speed as the metric of the locomotion task? Can you consider more realistic metrics like stability under the noises or some uneven terrains, as well as show more videos beyond walking in a straight line?

Author response:
We chose this metric because it was the simplest to define the task (only one reward term) and we could compare it to other controllers developed for the elastic quadruped robot.

Reviewer comment (Experiments):
The method works because the tasks only depend on cyclic movements for joints. Can the proposed method still be used if there are more obstacles in the environment, which need some complex behavior (turn left/right)?

Author response:
As mentioned in the "limitation" section of the paper and in the reply above, open-loop control cannot, by design, address tasks that require responding to variations in the environment.
However, the aim of the paper is not to propose a method that can replace RL, but provide a point of reference that is easy to study and that doesn't fail where "vanilla" DRL can be brittle (e.g. with sim2real transfer or when exposed to sensor noise).

Reviewer comment (Experiments):
Is it possible to try more to adjust the parameters of the pd controller and the range of the joint target position generated by the RL policy to improve the results of the RL method? The demo shown by the authors is indeed worse than other RL-based locomotion works recently.

Author response:
Thank you for your remark.
The result of the RL controller on the real robot is indeed worse than the recently published RL based locomotion, because we intentionally removed all the task specific heuristics and tried to make the environment as simple as possible.
This is one of the main points of the paper, which is to understand why RL fails in this case and what is the observed behavior.

We could improve the results by adding complex reward engineering, randomization, or low-pass filtering (or equivalently tuning the PD controller until it works at the cost of slowness).
However, our intention is to provide insight and food for thought about the cost of complexity and generality of the current DRL algorithm by proposing the simplest model-free baseline possible.
It would also be unfair to compare the simple open-loop baseline to a highly tuned DRL algorithm, that includes already many implementation details, uses the state as an input and has 10 000 more parameters.

We have updated the sim2real section to make that point clearer.

Reviewer comment (Evaluation):
The paper is missing a discussion on how RL might still play a role when applied to robotics, despite presenting a strong argument for simplicity and demonstrating that the baseline requires far fewer parameters and less computation to train.

Author response:
We agree that RL has a role to play in robotics (especially for locomotion tasks).
Our intention was to convey this in the earlier version.
We have updated the discussion section to make this statement clearer.

Reviewer comment (Evaluation):
As the paper argues that domain specific knowledge could help improve algorithm design, it is possible that RL algorithms could also benefit from such knowledge. More generally, the results might suggest that if final performance is the key optimization goal, RL may still be the preferred tool.

Author response:
RL is the tool of choice for achieving top performance in simulation, as demonstrated by our experiment and discussed in the paper (see Table 2).
However, applying RL in real-world scenarios requires numerous additional techniques, such as reward engineering, low-pass filtering, domain randomization, and leveraging privileged information.

Our goal is not to replace DRL algorithms but to highlight their limitations, including their struggles with simple problems.
In doing so, we aim to raise awareness within the community about the tradeoffs between the generality of these algorithms and the effectiveness of a much simpler baseline, which has "natural advantages" as noted by the reviewer.

To clarify this point, we have updated the abstract, the introduction and discussion sections.

Reviewer comment (Experiments):
The paper does not include any pattern generating baselines for comparison. Central pattern generators have been studied extensively, so it would be important to know how the proposed open-loop baseline compares against existing ideas in this field.

Author response:
The authors interpret the reviewer's suggestion as an endorsement for the inclusion of a CPG baseline.

Our proposed baseline can be seen as a simplified version of [1, 2], which has similar properties (smooth by design, open-loop, periodic), but is much easier to implement and adapt to new problems.
In fact, the CPG presented in these works was our starting point, but it was tailored for quadruped locomotion (using IK and fixed coupling matrix).
To be able to adapt it to all MuJoCo locomotion tasks, we simplified it, removed the constraint of a fixed coupling and converged to the simple oscillators presented in the paper.

Additionally, our intention is to provide the simplest baseline for RL locomotion tasks and provide insights for RL algorithms (following [3]), not to advance SOTA in the CPG domain.

We updated the introduction and changed the order of the related work to emphasize the main message of the paper.

[1] Bellegarda, G., & Ijspeert, A. (2022). CPG-RL: Learning central pattern generators for quadruped locomotion. \
[2] Raffin, A, et al. (2022) "Learning to Exploit Elastic Actuators for Quadruped Locomotion." \
[3] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."

Reviewer comment (Experiments):
The `Contributions` list states that the baseline can handle sparse rewards, but none of the environments used for evaluation appear to use sparse rewards. Clarification on this point is needed.

Author response:
Yes, the baseline we propose can handle sparse rewards, which is a benefit of black-box optimization (BBO).
Because BBO uses only episodic returns, rather than immediate rewards, the baseline and ARS can effectively tackle problems with sparse or delayed rewards.
Since this is a known pitfall of DRL algorithms, we wanted to mention this advantage without making it the main focus of our work.
For this reason we did not focus on this aspect during the evaluation but chose to highlight it textually instead.

We added a sentence when presenting the method to make that point clearer.

Reviewer comment (Experiments):
The authors could consider including another non-learning baseline to better understand the performance gains from the specific implementation proposed.

Author response:
The authors interpret the reviewer's suggestion as an endorsement for the inclusion of a CPG baseline.

Our proposed baseline can be seen as a simplified version of [1, 2], which has similar properties (smooth by design, open-loop, periodic), but is much easier to implement and adapt to new problems.
In fact, the CPG presented in these works was our starting point, but it was tailored for quadruped locomotion (using IK and fixed coupling matrix).
To be able to adapt it to all MuJoCo locomotion tasks, we simplified it, removed the constraint of a fixed coupling and converged to the simple oscillators presented in the paper.

Additionally, our intention is to provide the simplest baseline for RL locomotion tasks and provide insights for RL algorithms (following [3]), not to advance SOTA in the CPG domain.

We updated the introduction and changed the order of the related work to emphasize the main message of the paper.

[1] Bellegarda, G., & Ijspeert, A. (2022). CPG-RL: Learning central pattern generators for quadruped locomotion. \
[2] Raffin, A, et al. (2022) "Learning to Exploit Elastic Actuators for Quadruped Locomotion." \
[3] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."

Reviewer comment (Experiments):
For robustness to sensor noise, the authors could include a baseline where SAC is trained with a subset of the noisy parameters. For instance, if SAC were trained to withstand random noise of 3N, it might be more likely to handle the 5N noise tested in the paper.

Author response:
Thanks for the suggestion, we added an experiment where SAC is trained with a noisy sensor to the paper.

Reviewer comment (Presentation):
In Figure 3, the dot for `Open Loop` appears slightly below 1 but should theoretically sit exactly at 1. This discrepancy seems unusual and needs clarification.

Author response:
The plot is indeed slightly below one.
This is because we normalize with respect to the maximum performance, and since there is some variance between runs (see performance profiles), the normalized performance of the oscillators is not exactly one, but close to one.

Reviewer comment (Presentation):
Table 6 in the Appendix, which shows the actual scores, is informative and could be added to the main text since there is still space, provided the authors feel it would not disrupt the narrative flow.

Author response:
This is indeed a conscious choice.
The authors follow the best practice recommendations from [4 - NeurIPS 21 outstanding paper, 5] and provide a better statistical representation of performance through the use of IQM, stratified bootstrap, performance profiles, and probability of improvement.

We would like to emphasize that while performance is a critical aspect, runtime and parameter efficiency are another.
To maintain the clarity of the paper's narrative, we have chosen not to include the table in the main text.
Instead, it is provided in the appendix for readers who are interested in exploring this aspect further.

[4] Agarwal, R, et al. (2021) "Deep reinforcement learning at the edge of the statistical precipice." \
[5] Patterson, A, et al. (2023) "Empirical Design in Reinforcement Learning."