Characterising the Robustness of Reinforcement Learning for Continuous Control using Disturbance Injection
Abstract
In this study, we leverage the deliberate and systematic fault-injection capabilities of an open-source benchmark suite to perform a series of experiments on state-ofthe-art deep and robust reinforcement learning algorithms.
We aim to benchmark robustness in the context of continuous action spaces—crucial for deployment in robot control.
We find that robustness is more prominent for action disturbances than it is for disturbances to observations and dynamics.
We also observe that state-of-the-art approaches that are not explicitly designed to improve robustness perform at a level comparable to that achieved by those that are.
Our study and results are intended to provide insight into the current state of safe and robust reinforcement learning and a foundation for the advancement of the field, in particular, for deployment in robotic systems.
Introduction
Reinforcement learning (RL) has become a promising approach for robotic control, showing how robotic agents can learn to perform a variety of tasks, such as trajectory tracking and goal-reaching, on several robotic systems, from robotic manipulators to self-driving vehicles [9, 21, 24].
While many of these results have been achieved in highly controlled simulated environments [12], the next wave of artificial intelligence (AI) research is now faced with the challenge to deploy in the real world.
When using reinforcement learning to solve real-world problems safety must be paramount [25, 4, 2, 27, 11, 3].
Unsafe interaction with the environment and/or people in that environment can have very serious consequences, ranging from the destruction of the robot itself to, most importantly, harm to humans.
For safety to be guaranteed, an embodied RL agent (i e., the robot) must be robust to variations in the environment, its dynamics and unseen situations that can emerge in the real world.
In this paper, we quantitatively study and report on the performance of a set of state-of-the-art reinforcement learning approaches in the context of continuous control.
We systematically evaluate
RL agents (or “controllers”) on their performance (i.e., the ability to accomplish the task specified by the environment’s reward signal) as well as their robustness [30, 6, 13, 15, 17].
To do so, we use an open-source RL safety benchmarking suite [29].
First, we empirically compare the control policies produced by both traditional and robust RL agents at baseline and then when a variety of disturbances are injected into the environment.
We observe that both the traditional and robust RL agents are more robust to disturbances injected through the actions of the agent, while disturbances injected in the observations and dynamics can cause much more rapid destabilisation.
We also note that “vanilla” agents show similar performance to the robust RL agents even when disturbances are injected, despite not being explicitly designed with this purpose in mind.
By leveraging open-source simulation, we hope that this work and our insights can provide a basis for further research into safe and robust RL, especially for robot control.
Evaluation Setup
Our objective then is to train vanilla and robust RL agents (see additional detail in the Supplementary
Material) to perform a task (cart-pole stabilisation) in ideal conditions (i.e., without disturbances) and then assess the robustness of the resulting policies in environments that include injected disturbances in actions, observations, or dynamics (see additional detail in the Supplementary Material).
Each RL agent was trained by randomising the initial state across episodes to improve performance [29] while at test/evaluation time, a unique initial state was used for fairness and consistency.
The range of disturbances used in each experiment was selected to include (low) values, at which all or most agents still succeeded in completing the tasks, up until (high) values at which the robustness of all agents eventually fails.
In the case of the cart-pole, the goal of the controller is to stabilise the system at a pose of 0 m, or centre, in 𝑥 and 0 rads in 𝜃 , when the pole is upright.
Evaluation Metrics To measure the performance of the control policies, the exponentiated negated quadratic return is averaged over the length of each episode, over 25 evaluation episodes.
The same metric was used for training and evaluation.
Cost
[Equation]
Ep.
Return
[Equation]
Avg.
Norm.
Return ∶ 𝐽
[Equation]
[Equation] Equation (1) shows the task’s cost computed at each episode’s step 𝑖 , where 𝑥 and 𝑥 𝑔𝑜𝑎𝑙 are the actual and goal states of the system, 𝑢 and 𝑢 𝑔𝑜𝑎𝑙 the actual and goal inputs, and 𝑊 𝑥 and 𝑊 𝑢 are constant weight matrices.
𝐿 is the total number of steps in a given episode.
Equation (2) shows how to compute the return of an episode 𝑗 of length 𝐿 𝑗 from the cost function.
𝐿 𝑗 is equal (or lower) than the maximum episode duration of 250 steps.
Equation (3) shows the average return for 𝑁 (25) evaluation runs normalised by the length of the run.
Results
In the Supplementary Material (Fig. 5), we report the training results when no additional disturbances are applied, showing the reference performance of each controller at baseline.
The three algorithms which reach convergence fastest were SAC, PPO, and RAP.
SAC and PPO benefit from the stochastic characteristics of their updates.
RARL trains more slowly which is what we expect as RARL is also learning to counteract the adversary.
However, the same behaviour is not observed for the other robust approach RAP, which also converges quickly, suggesting RAP can be trained more efficiently.
[Figure]
A vg . N o r m . R e t u r n 𝐽 𝑅 𝑒 𝑣 𝑎 𝑙 White Noise Disturbance on External Dynamics
[Figure]
[Figure]
White Noise Disturbance on
[Figure]
RARL RAP
SAC
[Figure]
rads
[Figure]
rads
[Figure]
White Noise Disturbance on Observations -periodic Disturbances We want to assess the robustness of the trained policies for two vanilla (PPO, SAC) and two robust (RARL, RAP) RL agents. In Figure 1, we introduce three types of disturbances, two non-periodic and a periodic one (see the Supplementary Material for more).
White Noise Disturbances We first look at white noise disturbances (Figure 1a), often used to mimic the natural stochastic noise that an agent encounters in the real world.
The noise is applied, from zero, at increasing values of standard deviation.
In Figure 2, we see very similar low robustness across all control approaches for disturbances on external dynamics, with, as expected, a linear decrease in performance as the noise increases.
However, the robust approaches, RARL and RAP, show no significant difference in performance w.r.t.
PPO.
For action disturbances, RAP consistently has the highest average normalised return.
For observation disturbances, PPO has the highest average normalised return at high levels of disturbances.
Overall, the difference across the four approaches is small and they all demonstrate similarly good robustness when white noise is applied to observations or actions.
Step Disturbances Step disturbances (Figure 1b) allow us to see the system’s response to a sudden and sustained change.
As in all experiments, the disturbance is applied at varying levels, here representing the magnitude of the step.
The step occurs two steps into the episode for all runs.
As expected, compared to white noise disturbances, step disturbances have a much greater effect and even low magnitudes result in a large decrease in performance.
There are especially steep decreases in average normalised return when the agent can no longer stabilise the cart-pole, e.g., in the second and third plots of Figure 3.
For the step disturbance on external dynamics, there is no unique better controller, with RARL marginally outperforming the others.
For actions and observations, PPO again achieves the best overall performance, yielding almost ideal average normalised return up until the step magnitude reaches, for action disturbances, as high as 5 N. SAC’s performance is technically higher at low levels of disturbances but fails quickly as the magnitude of the step increases.
[Figure]
[Figure] [Figure] [Figure] N)
[Figure]
𝐽 𝑅 𝑒 𝑣 𝑎 𝑙
[Figure] [Figure]
Magnitude (N) on Actions
PPO
RARL
RAP
SAC
[Figure]
[Figure]
[Figure]
Magnitude (m, rads
[Figure]
m/s, rads/s)
Step Disturbance on Observations
[Figure]
Saw Wave Disturbance on External Dynamics
[Figure]
Actions
[Figure] RARL [Figure] Saw Wave Disturbance on Observations
3.2 Periodic Disturbances
Beyond episodic disturbances, we also want to explore the ability of the controllers to deal with periodic disturbances.
Such disturbances further challenge the agents as they introduce long-lasting perturbations, that force them to seek robustness in an environment that never behaves as ideal.
Saw Wave Disturbances A saw (or sawtooth) wave (Figure 1c) is a cyclic wave that increases linearly to a set magnitude and instantaneously drops back to a starting point before repeating the cycle.
Thus, this disturbance type includes aspects of the step and impulse disturbances, yet it is applied periodically throughout the evaluation episodes.
In Figure 4, the difference in performance between the approaches is less marked (in comparison to the disturbances applied in previous experiments).
For disturbances in the dynamics, there is little difference in performance (and low overall robustness) for all control approaches.
RARL performs better than the other approaches at low amplitude disturbances.
For action disturbances, PPO is the agent that best preserves its average normalised return, while the other approaches, in particular SAC, display lower robustness.
When the policy behaviour of the controllers was re-played, it was evident that RAP and RARL failed more often than PPO and SAC, resulting in a lower average normalised return.
When the saw wave disturbance is applied to observations, all approaches have great difficulty stabilising and the average normalised return quickly drops to zero.
Conclusions and Outlook
In this article, we presented results that provide insight into the robustness of reinforcement learning, in particular in the context of continuous control.
One of our main findings for roboticists is that the
RL agents are more susceptible to disturbances injected into dynamics and observations.
On the other hand, all agents under test, both vanilla RL agents and robust ones, display some inherent robustness to action disturbances.
As the field of robust reinforcement learning develops, our results indicate that particular care should be dedicated to improving robustness to observation and dynamics disturbances.
Nonetheless, building on traditional RL approaches that already demonstrate to generalise well against disturbances may be a promising path for robust robot control using reinforcement learning
Supplementary Material
In RL, an agent, in our case, a robot, performs an action and receives feedback (reward) from the environment on how well it is doing at the environment’s task, perceives the updated state of the environment resulting from the action taken and repeats the process, learning over time to improve the actions it takes to maximise reward collection (and this to correctly perform the task).
The resulting behaviour is called the agent’s policy and maps the environment’s state to actions [26].
While early
RL research was demonstrated in the context of grid worlds and games, in recent years, we have seen a growing interest in physics-based simulation for robot learning [7, 10, 18, 5].
For simplicity and reproducibility reasons, however, many of these simulators are still fully deterministic (and prone to be exploited by the agents).
In this study, we deliberately inject disturbances at different points of the
RL learning and control interaction loop to emulate the conditions an agent might encounter in the real world.
Injecting Disturbances in Robotic Environments
We systematically inject each of the disturbances in Figures 1 and 6 in one of three possible sites: observations, actions, and dynamics of the environment that the RL agent interacts with.
Observation/state Disturbances Observation/state disturbances occur when the robot’s sensors cannot perceive the exact state of the robot.
This is a very common problem in robotics and is tackled with state estimation methods [1].
In the case of the cart-pole, this disturbance is four-dimensional— as is the state—and is measured in metres in the first dimension, radians in the second, metres per second in the third, and radians per second in the fourth.
This disturbance is implemented by directly modifying the state observed by the system.
Action Disturbances Action disturbances occur when the actuation of the robot’s motors is not exactly as the control output specifies, resulting in a difference between the actual and expected action.
For example, action delays are often neglected or coarsely modeled in simple simulations.
In the case of the cart-pole, this disturbance is a one-dimensional force (in Newtons) in the 𝑥 -direction directly applied to the slider-to-cart joint.
External Dynamics Disturbances External dynamics disturbances are disturbances directly applied to the robot that can be thought of as environmental factors such as wind or other external forces.
In the case of the cart-pole, this disturbance is two-dimensional and implemented as a tapping force (in
Newtons) applied to the top of the pole.
Reinforcement Learning Agents for Continuous and Robust Control
While some of the most notable results of deep RL control [14] were achieved in the context of discrete action spaces, we focus on actor-critic agents capable of dealing with the continuous actions spaces needed for embodied AI and robotics [20, 16].
Here, we summarise the agent whose results we reported in Section 3.
Proximal Policy Optimisation (PPO) PPO [23] is a state-of-the-art policy gradient method proposed for the tasks of robot locomotion and Atari game playing.
It improves upon previous policy optimisation methods such as ACER (Actor-Critic with Experience Replay) and TRPO (Trust Region
Policy Optimisation)
PPO reduces the complexity of implementation, sampling, and parameter tuning using a novel objective function that performs a trust-region update that is compatible with stochastic gradient descent.
Soft Actor-Critic (SAC) SAC [8] is an off-policy actor-critic deep RL algorithm proposed for continuous control tasks.
The algorithm merges stochastic policy optimisation and off-policy methods like DDPG (Deep Deterministic Policy Gradient).
This allows it to better tackle the explorationexploitation trade-off pervasive in all reinforcement learning problems by having the actor maximise both the reward and the entropy of the policy.
This helps to increase exploration and prevent the policy from getting stuck in local optima. ⋅ 100 .0 .0 .0 . [Figure] [Figure]
Robust Adversarial Reinforcement Learning (RARL)
Unlike the previous two approaches,
RARL [19], as well as the following approach, RAP, are designed to be robust and bridge the gap between simulated results for control and performance in the real world.
To achieve this, an adversary is introduced that learns an optimal destabilisation policy and applies these destabilising forces to the agent, increasing its robustness to real disturbances.
Robust Adversarial Reinforcement Learning with Adversarial Populations (RAP) RAP [28] extends RARL by introducing a population of adversaries that are sampled from and trained against.
This algorithm hopes to reduce the vulnerability that previous adversarial formulations had to new adversaries by increasing the kinds of adversaries and therefore adversarial behaviours seen in training.
Similar to RARL, RAP was originally evaluated on continuous control problems. [Figure]
t u r n 𝐽 𝑅 𝑒 𝑣 𝑎 𝑙 [Figure] [Figure] N o
[Figure]
m .
R e t u r n 𝐽 𝑅 𝑒 𝑣 𝑎 𝑙
[Figure]
[Figure]
Impulse Disturbances
Impulse disturbances (Figure 6a) allow us to see the system’s response to a sudden, but temporary change.
Again, we look at varying levels of the impulse’s magnitude to test the controllers’ robustness.
The width of the impulse is two steps and it is applied two steps into the run for all runs.
In the case of dynamic and action impulse disturbances, the dramatic decrease in performance seen in the previous experiment (with the step disturbances) is just as pronounced.
We expected impulse disturbances to be more easily handled than step disturbances, as step responses may require the system to adapt to a new baseline whereas the impulse disturbances’ change is only temporary.
However, the first two plots in Figure 7 show a dramatic change in average normalised return as the sharp disturbance causes the agent to fail to stabilise.
PPO is the most robust to impulse disturbances on external dynamics while RARL displays more robust performance than it did with step disturbances.
For disturbances applied to actions, SAC, PPO, and RARL are able to handle higher magnitudes of impulse disturbance than RAP.
On the other hand, the short-lived impulse disturbance on observations does not significantly affect any of the control approaches, even at very high values.
Triangle Wave Disturbance
A triangle wave (Figure 6b) is a cyclic wave that increases linearly to a set magnitude and decreases at the same rate to a starting point before repeating.
This disturbance type is very similar to the saw wave disturbance but also acts more similarly to a sinusoidal wave.
Not surprisingly, the results for triangle wave disturbances (Figure 8) are similar to those of the sawtooth wave disturbances.
The triangle wave disturbance results in a slightly lower average normalised return than the sawtooth wave disturbances but the relative performance of the control approaches remains the same.
SAC performs slightly worse in the case of disturbances applied to dynamics.
For disturbances applied to observations, the drop in performance occurs even earlier for all controllers, showing the increased sensitivity to the triangle wave disturbance compared to the sawtooth wave.
Training with Disturbances
In the results presented so far, no additional disturbances were introduced during training.
It is natural to wonder whether including disturbances during training (and reducing the distributional shift between train and test scenario) can improve the evaluation performance of the controllers.
Disturbances during training—akin to how the RARL and RAP use adversaries to increase their robustness—can potentially lead to the learning of more generalisable policies.
In Figure 9, we look at two of the control approaches, PPO, the best performing vanilla RL approach, and RAP, the best performing robust approach, trained with varying levels of white noise (for 1,000,000 steps).
The evaluation/test performance with higher levels of noise is almost always still better when training with low levels of noise, and achieving the best performance when trained with no disturbances.
For
[Figure]
T r a i n i ng N o
i s e
( S t a nd a r d
D e v i a ti on ( N ))
Training w/ Disturbances
[Figure]
[Table]
[Figure] [Table]
a i n i ng N o
i s e
( S t a nd [Table] r d D
[Table] v i a [Table] N ))
[Figure]
T r a i n i ng N o
i s e
( S t a nd a r
[Figure]
v i a ti [Figure] ( N )
[Figure]
external dynamics disturbances, the average normalised return gradually decreases as the training noise is increased.
At higher values of training noise, the performance when the levels of testing noise are also higher improves slightly, suggesting there are small improvements.
This phenomenon, however, is only visible for disturbances in the dynamics (first row of Figure 9).
For action disturbances, the average normalised return is not affected by increased training noise or testing noise, except at specific, high values where the average normalised return decreases dramatically, showing no obvious performance improvement.
For noise added to observations, there is a sudden decrease to nearly zero average normalised return when noise is introduced during training.