Paper: Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies (ICLR-2024)

Reviewer comment (Novelty):
The discussion of related work is rather limited and should be expanded by a control theory angle, particularly relating to natural emergence of bang-bang-type controllers [1] or pulse-width modulation. Established benchmarks such as the DeepMind Control Suite can furthermore often be solved with high-frequency bang-bang control as studied in [2] or general discretization in [3] – these works provide further motivation for the need of well-designed benchmarks as this paper aims to provide.

Author response:
> The discussion of related work is rather limited and should be expanded by a control theory angle.

The related works section of our paper lists other benchmark papers and focuses mainly on categorizing and describing past methods that learns policies that produce smooth control behaviors, with few oscillations. Some of the methods investigated in our work do have more of a control theory perspective (esp. [1] and [2]), though not all of them. For example, Liu-Lipschitz [3] is an approach originally presented in the context of 3D imaging. 

Because in our work we are interested in measuring the effectiveness of these approaches in a performance x smoothness analysis more so than their mechanisms, we believe an extended discussion from a control theory angle is a bit out of scope. We would invite an interested reader to check the original papers for details, specially for methods proposed from a control perspective.

> Established benchmarks such as the DeepMind Control Suite can furthermore often be solved with high-frequency bang-bang control as studied in [...] or general discretization in [...]

Thank you for the reference suggestions. Although these works demonstrate that bang-bang control and discretization can solve the tasks, we did not include these types of controllers in our benchmark since they are the opposite of the type of smooth control we wish to achieve. A major part of our benchmark is done with real-world hardware and bang-bang/discrete type controllers cannot be safely deployed.

Nonetheless, the works listed are interesting benchmarks that evaluate performance in many environments, and we'll include them in the discussion of related works in the final paper.

Reviewer comment (Novelty):
A very relevant related work that automatically learns regularization trade-offs was presented in [4].

Author response:
> A very relevant related work that automatically learns regularization trade-offs was presented in [4]

We apologize for missing the related work. The work cited by the reviewer is indeed very relevant. We'll include it in the related works section discussion in the final paper version.

Reviewer comment (Evaluation):
The majority of cumulative returns as well as smoothness scores in Table 1 have overlapping error bands making performance differences difficult to judge.

Author response:
> The majority of cumulative returns as well as smoothness scores in Table 1 have overlapping error bands making performance differences difficult to judge.

We argue that our benchmark results show a significant improvement over a "Vanilla" policy versus regularization methods, showing that regularization is indeed effective. It is true that in a single scenario the smoothness between two regularization techniques might not be significant (see Hybrid methods in "Reacher", "ShadowHand", and "Velocity"). However, analyzing the smoothness x performance  across all scenarios we can observe that the "Hybrid" methods outperformed others more often than not. While other approaches have significant deviations scenario to scenario, it is clear that the "Hybrids" are the most consistent method across the board.

Reviewer comment (Experiments):
Pendulum/Reacher/Lunar Lander are rather toy-ish examples that are not representative of the challenges the paper aims to analyze, while they could provide good illustrative examples if analyzed more in-depth.

Author response:
> Pendulum/Reacher/Lunar Lander are rather toy-ish examples that are not representative of the challenges the paper aims to analyze, while they could provide good illustrative examples if analyzed more in-depth.

We've added two additional scenarios: "ShadowHand" and "Handstand". These are challenging tasks that require agile and dexterous movements. The Handstand task is also deployed and evaluated with real hardware.

Additionally, we have also measured the performance of every single method in the real-world and included an ablation case of a Vanilla policy trained without domain randomization. Please refer to the comment "Extended Experiments and Results -- Updated Table 1" for the updated version of our results.

Reviewer comment (Experiments):
The analysis would profit from extension to a broader set of baseline algorithms other than PPO to identify trends in order to serve as an extensive benchmark.

Author response:
> The analysis would profit from extension to a broader set of baseline algorithms other than PPO to identify trends in order to serve as an extensive benchmark

We agree with the reviewer that more extensive and broader experiments are better. Still, PPO is one of the most popular algorithm used by the community, specially for sim2real deployment. We believe the scope of the work is wide enough to be relevant to the research community.

Reviewer comment (Experiments):
Why did the “Locomotion – Velocity Controller” task not use the action rate penalization of Rudin (2021)?

Author response:
>Why did the “Locomotion – Velocity Controller” task not use the action rate penalization of Rudin (2021)?

As the reviewer mentioned, adding an action penalty term to the reward function is a popular practice but it is not "free". Reward functions in RL are known to be difficult to tune, and the more terms added to the function the more challenging of a balancing act it becomes. In an idealized setting, we envision a practitioner designing a reward function with no concern for smoothness, only performance. Once the desired task reward is complete, a smoothness regularization method -- such as the ones investigated in this work -- can be added on top. This process would facilitate reward design and reduce complexity, while guaranteeing smooth control.

As such, in the original scenarios evaluated in our work we opted to not use a direct action penalty term in the reward function. Note that "Velocity" and "Handstand" tasks do include reward terms for joint acceleration and joint torques, which can be seen as analogous to action penalty. 

To address the concerns of the reviewer and also consider the case of a policy with action penalty, we included it for the "Handstand" scenario that was added during the rebuttal process.  Even in this case, the results show that the regularization methods further improve the smoothness compared to a Vanilla policy. (Please refer to the updated Table and Real World experiments in the summary comment.) In conclusion, our results show that whether a direct action penalty term or "indirect" terms (joint acceleration and torque) are employed, regularization is still beneficial and significantly improves smoothness compared to no regularization.

Reviewer comment (Evaluation):
What do state and action trajectories of trained policies look like? How do they compare across agents?

Author response:
> What do state and action trajectories of trained policies look like? How do they compare across agents?

Here are trajectory samples of actions and joint angles for "Vanilla" and "Hybrid - LipsNet + L2C2" for the Handstand scenario.

* Handstand | Vanilla | Actions: https://i.imgur.com/4GNZmLv.png
* Handstand | Hybrid - LipsNet + L2C2 | Actions: https://i.imgur.com/8GI9Wjh.png
* Handstand | Vanilla | Joint Angles: https://i.imgur.com/c2JERaq.png
* Handstand | Hybrid - LipsNet + L2C2 | Joint Angles: https://i.imgur.com/iqWGMmb.png

There is an observable difference between vanilla and the regularized methods. The difference between any two regularized methods is less significant, except in the cases where the method failed to learn a good policy.

We'll include an extended comparison of joint angles and actions for every method in a dedicated appendix section in the final paper.

Reviewer comment (Theory):
The authors (and title even) mix the concepts of policy smoothness and reducing high-frequency oscillations, but how these concepts relate and why you get oscillations is not thoroughly defined. The paper also talks about sim2real where oscillations are indeed a common problem, but the authors do not make a convincing argument that this is due to non-smooth policies. Sometimes it might be but this has not been well explored as far as I am aware. Overall the paper does a good job of benchmarking existing approaches to increase policy smoothness, but the sim2real / oscillation connection seems weak.

Author response:
The reviewer raises a good point that our language in the paper is ambiguous. When we use the term "policy smoothness" or a "smooth policy" we simply mean a policy that exhibits/produces smooth control behaviors in the form of reduced oscillations, and not necessarily a policy that produces a smooth mapping function. However, and interestingly, the two do seem to be correlated as most methods that we investigated/refer in our paper do use the latter definition of smooth policy and aim to achieve smoothness, as in "a smooth mapping function", which in turns reduces oscillations, evidenced by our results.

We will clarify the meaning of these terms in the paper and better define "smoothness".

Reviewer comment (Experiments):
"... but in the open-source code a linear activation is used. In our implementation, we used a softplus activation as in the original paper." - This seems like an arbitrary choice that might degrade the performance of a defacto available option. Maybe the implementation is more up to date, or there was a typo in the paper? Can you test and confirm that it performs as well, or why not include both versions?

Author response:
We've confirmed with the original authors of LipsNet that the version we implemented is indeed the correct and intended way. The authors had incorrectly used a linear activation as a default when open-sourcing their code (The output $K$ of LipsNet must be $ > 0 $. We've confirmed that the activation function used by the authors in their original experiments matches the version we've used in our work. Additionally, the original authors have since updated their open-source code to match the version we've used after our inquiry.

Reviewer comment (Evaluation):
In Figure 2, in the disturbance imitation task test, why does the combination of LipsNet + L2C2 underperform compared to the vanilla policy regarding smoothness?

Author response:
Every policy deployed to the real-world is trained with domain randomization, including environment physics, terrain and actuator properties. The addition of domain randomization by itself results in smoother control behaviors. For a given task, regular execution (i.e. no disturbances) might be smooth enough that there are no significant differences between a regularized policy and a vanilla DR policy. However, when the policy needs to execute a sort of recovery behavior from an unstable state (which could be out of distribution), for example from an external disturbance, we can observe that a policy trained with smoothness regularization is indeed smoother. 

For supporting evidence to our point above, we have included an ablation of a "Vanilla (No DR)" vs "Vanilla" which demonstrates this behavior.

Reviewer comment (Evaluation):
As you have exclusively provided the measure of policy smoothness in real-world situations, I'm curious as to how the real-world performance (reward) measures up against other baselines?

Author response:
We've followed up on the reviewer's suggestions and reran all of our real-world experiments with every method reported on the paper. Additionally, we've also measured the cumulative return in the real-world to enable a performance vs smoothness analysis. (See below).

---
**Updated Real-World Results** - Comparison of All Methods + No DR Ablation + Cumulative Return Computation:

Imitation: https://i.imgur.com/zuWvjqq.png

Velocity: https://i.imgur.com/UG2dvYF.png

Handstand: https://i.imgur.com/vmjOltq.png
* We did not deploy "No DR" ablation for handstand out of concern for robot and actuator damage.
---

Reviewer comment (Evaluation):
Could you share training plots at least for Ant and quadruped robot for the reward vs wall-clock time for LipsNet + CAPS and LipsNet + L2C2 vs vanilla MLP? Are there losses in training (wall-clock time) performance when using these more advanced methods vs MLP or they are computationally comparable to the vanilla MLP?

Author response:
> Could you share training plots at least for Ant and quadruped robot for the reward vs wall-clock time for LipsNet + CAPS and LipsNet + L2C2 vs vanilla MLP?

Unfortunately, because we ran our experiments in parallel, sharing computing load with other processes and experiments, the wall-clock vs reward graph is not truly representative of performance efficiency. We'll consider running a separate experiment sequentially to accurately measure reward vs (real) time.

> Are there losses in training (wall-clock time) performance when using these more advanced methods vs MLP or they are computationally comparable to the vanilla MLP?

For pure loss-based methods we have not observed any difference in training time. In regards to the architectural methods, Spectral Normalization (SN-Local) and Liu-Lipschitz are both similar to a vanilla MLP in training time. Because our policy networks are not large, the training bottleneck ends up in the sample collection time in this case. 

However, it should be noted that the LipsNet method comes with a significant slowdown. We observe an iteration time approximately 10 times longer than other methods, which is line with what the original authors reported in Appendix D. in [1]. By extension, this also affects the training time for the Hybrid methods that we proposed in our paper. 

Overcoming such a slowdown and achieving the same smoothness and performance metrics we measured here could be a fruitful path for future research.

We'll incorporate the points above in the discussion in Section 4 of the paper.

Reviewer comment (Experiments):
Could you run experiments for more challenging control problems - humanoid, or one of the Allegro (Shadow) Hand dexterous manipulation tasks?

Author response:
> Could you run experiments for more challenging control problems - humanoid, or on of the Allegro (Shadow) Hand dexterous manipulation tasks?

Thank you for your suggestion. We have run our experiments with the ShadowHand task in Isaac Gym. Please refer to the full updated table and comments in our top official comment "Extended Experiments and Results -- Updated Table 1".

We also invite the reviewer to check the "Summary of Changes" comment for a more extensive list of major and minor changes we've done thanks to your feedback.