Smooth Exploration for Robotic Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
Reinforcement learning (RL) enables robots to learn skills from interactions withthe real world.
In practice, the unstructured step-based exploration used in Deep
RL – often very successful in simulation – leads to jerky motion patterns on realrobots.
Consequences of the resulting shaky behavior are poor exploration, oreven damage to the robot.
We address these issues by adapting state-dependent exploration (SDE) [1] to current Deep RL algorithms.
To enable this adaptation, wepropose two extensions to the original SDE, using more general features and resampling the noise periodically, which leads to a new exploration method generalized state-dependent exploration ( g SDE).
We evaluate g SDE both in simulation,on PyBullet continuous control tasks, and directly on three different real robots:a tendon-driven elastic robot, a quadruped and an RC car.
The noise samplinginterval of g SDE enables a compromise between performance and smoothness,which allows training directly on the real robots without loss of performance.
Introduction
One of the ﬁrst robots that used artiﬁcial intelligence methods was called “Shakey” because itwould shake a lot during operation [2].
Shaking has now again become quite prevalent in robotics,but for a different reason.
When learning robotic skills with deep reinforcement learning (DeepRL) the de facto standard for exploration is to sample a noise vector (cid:15) t from a Gaussian distributionindependently at each time step t and then adding it to the policy output.
This approach leads to thetype of noise illustrated to the left in Fig.
and it can be very effective in simulation [3, 4, 5, 6, 7]
[Figure]
Unstructured exploration has also been applied to robotics [8, 9].
But for experiments on real robots,it has many drawbacks, which have been repeatedly pointed out [1, 10, 11, 12, 13]: 1) Samplingindependently at each step leads to shaky behavior [14], and noisy, jittery trajectories.
The jerkymotion patterns can damage the motors on a real robot, and lead to increased wear-and-tear.
Inthe real world, the system acts as a low pass ﬁlter.
Thus, consecutive perturbations may cancel each other, leading to poor exploration.
This is particularly true for high control frequency [15].
4) Itcauses a large variance which grows with the number of time-steps [10, 11, 12]
In practice, we have observed all of these drawbacks on three real robots, including the tendondriven robot David, depicted in Fig. 4a, which is the main experimental platform used in this work.
For all practical purposes, Deep RL with unstructured noise cannot be applied to David.
In robotics, multiple solutions have been proposed to counteract the inefﬁciency of unstructurednoise.
These include correlated noise [8, 15], low-pass ﬁlters [16, 17], action repeat [18] or lowerlevel controllers [16, 9].
A more principled solution is to perform exploration in parameter space,rather than in action space [19, 20].
This approach usually requires fundamental changes in thealgorithm, and is harder to tune when the number of parameters is high.
State-Dependent Exploration (SDE) [1, 11] was proposed as a compromise between exploring inparameter and action space.
SDE replaces the sampled noise with a state-dependent explorationfunction, which during an episode returns the same action for a given state.
This results in smootherexploration and less variance per episode.
To the best of our knowledge, no Deep RL algorithm has yet been successfully combined with
SDE.
We surmise that this is because the problem that it solves – shaky, jerky movement – is not asnoticeable in simulation, which is the current focus of the community.
In this paper, we aim at reviving interest in SDE as an effective method for addressing explorationissues that arise from using independently sampled Gaussian noise on real robots.
Our concretecontributions, which also determine the structure of the paper, are:
Highlighting the issues with unstructured Gaussian exploration (Sect. 1).
Adapting SDE to recent Deep RL algorithms, and addressing some issues of the original
formulation (Sects. 2.2 and 3).
Evaluate the different approaches with respect to the compromise between smoothness andperformance, and show the impact of the noise sampling interval (Sects. .1 and 4.2).4.
Successfully applying RL directly on three real robots: a tendon-driven robot, a quadrupedand an RC car, without need of a simulator or ﬁlters (Sect. 4.3).
Background
In reinforcement learning, an agent interacts with its environment, usually modeled as a Markov
Decision Process (MDP) ( S , A , p, r ) where S is the state space, A the action space and p ( s (cid:48) | s , a ) 56 the transition function.
At every step t , the agent performs an action a in state s following its policyπ : S (cid:55)→ A .
It then receives a feedback signal in the next state s (cid:48) : the reward r ( s , a ) .
The objectiveof the agent is to maximize the long-term reward.
More formally, the goal is to maximize theexpectation of the sum of discounted reward, over the trajectories
ρ π generated using its policy π :
[Equation]
where γ ∈ [0 , 1) is the discount factor and represents a trade-off between maximizing short-termand long-term rewards.
The agent-environment interactions are often broken down into sequencescalled episodes , that end when the agent reaches a terminal state.
2.1 Exploration in Action or Policy Parameter Space
In the case of continuous actions, the exploration is commonly done in the action space [21, 22, 23,24, 25, 5].
At each time-step, a noise vector (cid:15) t is independently sampled from a Gaussian distributionand then added to the controller output:
[Equation]
[Equation]
where µ ( s t ) is the deterministic policy and π ( a t | s t ) ∼ N ( µ ( s t ) , σ 2 ) is the resulting stochasticpolicy, used for exploration.
θ µ denotes
Alternatively, the exploration can also be done in the parameter space [11, 19, 20]:
[Equation]
at the beginning of an episode, the perturbation (cid:15) is sampled and added to the policy parametersθ µ .
This usually results in more consistent exploration but becomes challenging with an increasingnumber of parameters [19].
2.2 State-Dependent Exploration
State-Dependent Exploration ( SDE ) [1, 11] is an intermediate solution that consists in adding noiseas a function of the state s t , to the deterministic action µ ( s t ) .
At the beginning of an episode, theparameters θ (cid:15) of that exploration function are drawn from a Gaussian distribution.
The resultingaction a t is as follows:
[Equation]
This episode-based exploration is smoother and more consistent than the unstructured step-basedexploration.
Thus, during one episode, instead of oscillating around a mean value, the action a for agiven state s will be the same.
SDE should not be confused with unstructured noise where the variance can be state-dependent butthe noise is still sampled at every step, as it is the case for SAC.
In the remainder of this paper, to avoid overloading notation, we drop the time subscript t , i. e. wenow write s instead of s t .
s j or a j now refer to an element of the state or action vector.
In the case of a linear exploration function (cid:15) ( s ; θ (cid:15) )
= θ (cid:15) s , by operation on Gaussian distributions,
R ¨ uckstieß et al.
[1] show that the action element a j is normally distributed:
[Equation]
(5) where ˆ σ is a diagonal matrix with elements ˆ σ j = (cid:112)(cid:80) i ( σ ij s i ) 2 .
Because we know the policy distribution, we can obtain the derivative of the log-likelihoodlog π ( a | s ) with respect to the variance σ :
[Equation]
This can be easily plugged into the likelihood ratio gradient estimator [26], which enables adaptationof σ during training.
SDE is therefore compatible with standard policy gradient methods, whileaddressing most shortcomings of the unstructured exploration.
For a non-linear exploration function, the resulting distribution π ( a | s ) is most of the time unknown.
Thus, computing the exact derivative w.r.t.
the variance is not trivial and may require approximateinference.
As we focus on simplicity, we leave this extension for future work.
Generalized State-Dependent Exploration
Considering Eqs.
(5) and (6), some limitations of the original formulation are apparent:
i The noise does not change during one episode, which is problematic if the episode length is long, because the exploration will be limited [ ii The variance of the policy ˆ σ j = (cid:112)(cid:80) i ( σ ij s i ) 2 depends on the state space dimension (itgrows with it), which means that the initial σ must be tuned for each problem.
iii There is only a linear dependency between the state and the exploration noise, which limitsthe possibilities.
iv The state must be normalized, as the gradient and the noise magnitude depend on the statemagnitude.
To mitigate the mentioned issues and adapt it to Deep RL algorithms, we propose two improvements:
We sample the parameters θ (cid:15) of the exploration function every n steps instead of everyepisode.2.
Instead of the state s , we can in fact use any features.
We chose policy features z µ ( s ; θ z µ )(last layer before the deterministic output µ ( s )
= θ µ z µ ( s ; θ z µ ) ) as input to the noise function (cid:15) ( s ; θ (cid:15) )
= θ (cid:15)
z µ ( s ) .
Sampling the parameters θ (cid:15) every n steps tackles the issue i. and yields a unifying framework [27]which encompasses both unstructured exploration ( n = 1 ) and original SDE ( n = episode length ).
Although this formulation follows the description of Deep RL algorithms that update their parameters every m steps, the inﬂuence of this crucial parameter on smoothness and performance was untilnow overlooked.
Using policy features allows mitigating issues ii, iii and iv: the relationship between the state s andthe noise (cid:15) is non-linear and the variance of the policy only depends on the network architecture.
This makes g SDE more task-independent (as the network architecture is usually kept constant) andsaves a lot of parameters and computation when working with large state space such as images: thenumber of parameters and operations is only a function of the last layer size and action dimensionand no more of the state space size.
This formulation is therefore more general and includes theoriginal SDE description, when using state as input to the noise function or when the policy islinear.
We call the resulting approach generalized State-Dependent Exploration ( g SDE).
Deep RL algorithms Integrating this updated version of SDE into recent Deep RL algorithms,such as those listed in the appendix, is straightforward.
For those that rely on a probability distribution, such as SAC or PPO, we can replace the original Gaussian distribution by the one from Eq.
(5),where the analytical form of the log-likelihood is known (cf.
Eq.
Experiments
In this section, we study g SDE to answer the following questions:
• How does g SDE compares to the original SDE?
What is the impact of each proposed modiﬁcation?
• How does g SDE compares to other type of exploration noise in terms of compromise be-
tween smoothness and performance?
• How does g SDE performs on a real system?
4.1 Compromise Between Smoothness and Performance
Experiment setup In order to compare g SDE to other type of exploration in terms of compromisebetween performance and smoothness, we chose 4 locomotion tasks from the PyBullet [28] environments: H ALF C HEETAH , A NT , H OPPER and W ALKER 2D.
They are similar to the one found in
OpenAI Gym [29] but the simulator is open source and they are harder to solve 1 .
In this section, wefocus on the SAC algorithm as it will be the one used on the real robot, although we report resultsfor additional algorithms such as PPO in the appendix.
To evaluate smoothness, we deﬁne a continuity cost C = 100 × E t (cid:104) ( a t +1 − a t ∆ a max ) 2 (cid:105) which yields valuesbetween 0 (constant output) and 100 (action jumping from one limit to another at every step).
Thecontinuity cost of the training C train is a proxy for the wear-and-tear of the robot.
We compare the performance of the following conﬁgurations: (a) no exploration noise, (b) unstructured Gaussian noise (original SAC implementation), (c) correlated noise (Ornstein–Uhlenbeck process [30] with σ =0 .
2 , OU noise in the ﬁgure), (d) adaptive parameter noise [19] ( σ =0 . 2 ), (e) g SDE.
To decorrelate the exploration noise from the one due to parameter update, and to be closer to a realrobot setting, we apply the gradient updates only at the end of each episode.
We ﬁx the budget to 1 million steps and report the average score over 10 runs together with theaverage continuity cost during training and their standard error.
For each run, we test the learnedpolicy on 20 evaluation episodes every 10000 steps, using the deterministic controller µ ( s t ) .
Regarding the implementation, we use a modiﬁed version of Stable-Baselines3 [31] together with the
RL Zoo training framework [32].
The methodology we follow to tune the hyperparameters andtheir details can be found in the appendix.
The code we used to run the experiments and tune thehyperparameters can be found in the supplementary material
[Table] [Table]
[Figure]
Results Table 1 and Fig.
2 shows the resultson the PyBullet tasks and the compromise between continuity and performance.
Withoutany noise (“No Noise” in the ﬁgure), SAC isstill able to solve partially those tasks thanks toa shaped reward, but it has the highest variancein the results.
Although the correlated and parameter noise yield lower continuity cost during training, it comes at a cost of performance.g SDE is able to achieve a good compromise between unstructured exploration and correlatednoise by making use of the noise repeat parameter.
g SDE-8 (sampling the noise every 8 steps)even achieves better performance with a lowercontinuity cost at train time.
Such behavior iswhat is desirable for training on a real robot: wemust minimize wear-and-tear at training whilestill obtaining good performance at test time.
4.2 Comparison to the Original SDE
In this section, we investigate the contribution of the proposed modiﬁcations to the original SDE:sampling the exploration function parameters every n steps and using policy features as input to thenoise function.
Sampling Interval g SDE is a n -step version of SDE, where n allows interpolating between theunstructured exploration n = 1 and the original SDE per-episode formulation.
This interpolationpermits a compromise between performance and smoothness at train time (cf.
Table 1 and Fig. 2).
Fig.
3b shows the importance of that parameter for PPO on the W ALKER 2D task.
If the samplinginterval is too large, the agent would not explore enough during long episodes.
On the other hand,with a high sampling frequency n ≈ 1 , the issues mentioned in Sect.
1 arise.
Policy features as input
Fig.
3a shows the effect of changing the exploration function input for
SAC and PPO.
Although it varies from task to task, using policy features (“latent” in the ﬁgure) isusually beneﬁcial, especially for PPO.
It also requires less tuning and no normalization as it dependsonly on the policy network architecture.
Here, the PyBullet tasks are low dimensional and the state
[Figure]
Inﬂuence of the exploration function input
[Figure]
1 Sampling interval
[Figure]
S c o r e
Inﬂuence of the sampling interval on Walker2D
[Figure]
space size is of the same order, so no careful per-task tuning is needed.
Relying on features alsoallows learning directly from pixels, which is not possible in the original formulation.
Compared to the original SDE, the two proposed modiﬁcations are beneﬁcial to the performance,with the noise sampling interval n having the most impact.
Fortunately, as shown in Table 1and Fig. 2, it can be chosen quite freely for SAC.
In the appendix, we provide an additional ablation study that shows g SDE is robust to the choice of the initial exploration variance.
4.3 Learning to Control a Tendon-Driven Elastic Robot
[Figure]
Experiment setup To assess the usefulness of g SDE, we apply it on a real system.
The task isto control a tendon-driven elastic continuum neck [33] (see Fig. 4a) to a given target pose.
Controlling such a soft robot is challenging because of the nonlinear tendon coupling, together with thedeformation of the structure that needs to be modeled accurately.
This modeling is computationallyexpensive [34, 35] and requires assumptions that may not hold in the physical system.
The system is under-actuated (there are only 4 tendons), hence, the desired pose is a 4D vector: 3angles for the rotation θ x , θ y , θ z and one for the position x .
The input is a 16D vector composedof: the measured tendon lengths (4D), the current tendon forces (4D), the current pose (4D) and thetarget pose (4D).
The reward is a weighted sum between the negative geodesic distance to the desiredorientation and the negative Euclidean distance to the desired position.
The weights are chosen such that the two components have the same magnitude.
We also add a small continuity cost to reduce theoscillations in the ﬁnal policy.
The action space consists in desired delta in tendon forces, limitedto 5 N .
For safety reasons, the tendon forces are clipped below 10 N and above 40 N .
An episodeterminates either when the agent reaches the desired pose or after a timeout of 5 s .
The episode isconsidered successful if the desired pose is reached within a threshold of 10 mm for the positionand 5 ◦ for the orientation.
The agent controls the tendons forces at 30 Hz , while a PD controllermonitors the motor current at 3 kHz on the robot.
Gradient updates are directly done on a 4-corelaptop, after each episode.
Results We ﬁrst ran the unstructured exploration on the robot, but had to stop the experiment early:the high-frequency noise in the command was damaging the tendons and would have broken themdue to their friction on the bearings.
Therefore, as a baseline, we trained a policy using SAC with ahand-crafted action smoothing ( 2 Hz cutoff Butterworth low-pass ﬁlter) for two hours.
Then, we trained a controller using SAC with g SDE for the same duration.
We compare bothlearned controllers to an existing model-based controller (passivity-based approach) presentedin [34, 35] using a pre-deﬁned trajectory (cf. Fig. 4b).
On the evaluation trajectory, the controllersare equally precise (cf.
Table 2): the mean error in orientation is below 3 ◦ and the one in positionbelow 3 mm .
However, the policy trained with the low-pass ﬁlter is much more jittery than the twoothers.
We quantify this jitter as the mean absolute difference between two timesteps, denoted ascontinuity cost in Table 2.
[Table]
[Figure]
[Table]
[Figure]
Additional Real Robot Experiments We also successfully applied SAC with g SDE on two additional real robotics tasks (see Fig.
5a): training an elastic quadruped robot to walk (learning curvein Fig.
5b) and learning to drive around a track using an RC car.
Both experiments are fully done onthe real robot, without the use of simulation nor ﬁlter.
We provide in the supplementary material thevideos of the trained controllers.
Related Work
Exploration is a key topic in reinforcement learning [36].
It has been extensively studied in thediscrete case and most recent papers still focus on discrete actions [37, 38].
Several works tackle the issues of unstructured exploration for continuous control by replacing itwith correlated noise.
Korenkevych et al.
[15] use an autoregressive process and introduce twovariables that allows to control the smoothness of the exploration.
In the same vein, van Hoofet al.
[27] rely on a temporal coherence parameter to interpolate between the step- or episode-basedexploration, making use of a Markov chain to correlate the noise.
This smoothed noise comes at acost: it requires an history, which changes the problem deﬁnition.
Exploring in parameter space [10, 39, 11, 12, 40] is an orthogonal approach that also solves someissues of the unstructured exploration.
It was successfully applied to real robot but relied on motorprimitives [41, 12], which requires expert knowledge.
Plappert et al.
[19] adapt parameter exploration to Deep RL by deﬁning a distance in the action space and applying layer normalization tohandle high-dimensional space.
Population based algorithms, such as Evolution strategies (ES) or Genetic Algorithms (GA), also explore in parameter space.
Thanks to massive parallelization, they were shown to be competitive [42]with RL in terms of training time, at the cost of being sample inefﬁcient.
To address this problem,recent works [20] proposed to combine ES exploration with RL gradient update.
This combination,although powerful, unfortunately adds numerous hyperparameters and a non-negligible computational overhead.
Obtaining smooth control is essential for real robot, but it is usually overlooked by the DeepRLcommunity.
Recently, Mysore et al.
[14] integrated a continuity and smoothing loss inside RLalgorithms.
Their approach is effective to obtain a smooth controller that reduces the energy used attest time on the real robot.
However, it does not solve the issue of smooth exploration at train time,limiting their training to simulation only.
Conclusion
In this work, we highlighted several issues that arise from the unstructured exploration in Deep RLalgorithms for continuous control.
Due to those issues, these algorithms cannot be directly appliedto learning on real-world robots.
To address these issues, we adapt State-Dependent Exploration to Deep RL algorithms by extendingthe original formulation: we sample the noise every n steps and replace the exploration function input by learned features.
This generalized version ( g SDE), provides a simple and efﬁcient alternativeto unstructured Gaussian exploration.g SDE achieves very competitive results on several continuous control benchmarks, while reducingwear-and-tear at train time.
We also investigate the contribution of each modiﬁcation by performingan ablation study: the noise sampling interval has the most impact and allows a compromise betweenperformance and smoothness.
Our proposed exploration strategy, combined with SAC, is robust tohyperparameter choice, which makes it suitable for robotics applications.
To demonstrate it, wesuccessfully apply SAC with g SDE directly on three different robots.
Although much progress is being made in sim2real approaches, we believe more effort should beinvested in learning directly on real systems, even if this poses challenges in terms of safety andduration of learning.
This paper is meant as a step towards this goal, and we hope that it will reviveinterest in developing exploration methods that can be directly applied to real robots.