D EEP S EQUENCED L INEAR D YNAMICAL S YSTEMS FOR M ANIPULATION P OLICY L EARNING
A BSTRACT
In policy learning for robotic manipulation tasks, action parameterization can have a major impact on the final performance and sample efficiency of a policy.
Unlike highly-dynamic continuous-control tasks, many manipulation tasks can be efficiently performed by a sequence of simple, smooth end-effector motions.
Building on this intuition, we present a new class of policies built on top of differentiable Linear Dynamical System (dLDS) units, our differentiable formulation of the classical LDS.
Constructing policies using dLDS units yields several advantageous properties, including trajectory coherence across timesteps, stability, and invariance under translation and scaling.
Inspired by the sequenced
LDS approach proposed by Dixon & Khosla (2004), we propose a deep neuralnetwork policy parameterization based on sequenced dLDS units, and we integrate this policy class into standard on-policy reinforcement learning settings.
We conduct extensive experiments on Metaworld environments and show a notable improvement in performance and sample efficiency compared to other state-of-the-art algorithms.
Additional visualizations and code can be found at https://sites.google.com/view/deep-sequenced-lds .
I NTRODUCTION
Imagine a robot trying to pour water into a glass from a bottle.
First, it has to reach the bottle’s cap, open the cap, move the bottle towards the glass and then pour the water into the glass.
The complex trajectory that the robot follows to accomplish this task can be decomposed into a sequence of simpler trajectories.
Even when we change the positions of glass and bottle, the underlying simpler trajectories follow approximately the same shape, but are transformed accordingly.
In this paper, we explore whether we can parameterize a learned policy that is biased towards producing smooth trajectories with a similar decomposable, transformable structure.
In order to achieve smooth, coherent trajectories when the underlying action space does not implicitly provide them, a policy must jointly reason about an agent’s actions across time steps.
One popular approach to achieve this is to predict sequences of actions together at a lower frequency than the actions are executed - for instance, by directly repeating a single action (“frame skip” Braylan et al. (2015)), or jointly predicting low-level actions for the next k timesteps Bahl et al.
However, repeating the same action for k steps achieves smoothness without local expressiveness; and jointly predicting k low-level actions achieves local expressiveness but produces actions that are not necessarily coherent or smooth.
Recently, Bahl et al.
(2020) proposed a policy representation (NDP) that is guaranteed to produce smooth, coherent trajectories by reparametrizing the action space with parameters of a non-linear dynamical system based on Dynamic Motion Primitives (DMPs), which consist of a set of forcing function weights and a goal state Ijspeert et al.
They then use a standard trajectory-following controller to follow the trajectory for a fixed number of time steps before re-evaluating the policy.
While the underlying Dynamic Motion Primitives used in NDPs can express very complex trajectories, their performance is sensitive to the choice of basis function and number of basis functions, and they aren’t inherently invariant to translation or rotation of the state space Saveriano et al.
With these limitations in mind, outputting DMP parameters to provide a wide variety of trajectories for different environmental variations – particularly in goal-conditioned manipulation tasks – presents a challenging optimization problem.
To address these limitations, we propose a different action parameterization based on a Linear Dynamical System (LDS).
This choice of dynamical system achieves guaranteed scale and translational invariance (proof in Appendix B), structurally improving the generalization of goal-conditioned policies.
Based on the intuition that many manipulation tasks may be accomplished by a sequence of simple skills, we decompose an episode into a sequence of LDS units, in which we learn a neuralnetwork based policy to predict the goal for each unit.
This action space parameterization is heavily inspired by the classical sequential LDS formulation proposed by Dixon & Khosla (2004), who observed that a complex trajectory can be decomposed into a sequence of simpler linear dynamical systems.
However, their policy formulation is not differentiable, making it non-trivial to adapt to a deep reinforcement learning setting.
Therefore, we replace the LDS unit with a differentiable version (dLDS), and we propose a corresponding sequenced dLDS policy learning algorithm that can be trained using any policy-gradient algorithm.
We conduct thorough experiments on Metaworld Yu et al.
(2019) environments and show a notable improvement in sample efficiency in comparison to other state-of-the-art algorithms.
To summarize our contributions:
• We introduce a differentiable formulation of training Linear Dynamical Systems which can be integrated into any gradient-based learning algorithm.
• We propose a novel RL policy composed from a sequence of dLDS units.
• We conduct thorough experiments in numerous Metaworld environments and show a notable improvement in sample efficiency compared to baseline policy-learning algorithms.
R ELATED W ORK
Dynamical systems for manipulation Dynamic motion primitives have been classically used for solving complex manipulation tasks Ijspeert et al.
; Tamosiunaite et al.
; Ijspeert et al.
; Thota et al.
More recently, methods have employed DMPs with Neural Networks in order to learn policies that could output the parameters of the underlying dynamical system (DMP), which is then use to sample trajectories Bahl et al.
Pahi ˇ c et al.
(2018; 2020); Bahl et al.
Hierarchial NDPs Bahl et al.
(2021) builds upon NDPs with a hierarchial framework which can work on unseen task configurations.
Reinforcement learning for manipulation tasks Along with recent progress in deep reinforcement learning, there has been increased interest in applying reinforcement learning to manipulation tasks.
Several popular benchmark tasks widely used in RL literature feature challenging goal-conditioned manipulation tasks such as Fetch and MetaWorld, Andrychowicz et al.
Yu et al.
Several methods have leveraged fully on-robot training to accomplish generalized manipulation tasks using robot arms equipped with parallel-jaw grippers Gu et al.
; Kalashnikov et al.
RL has also been used for dynamic manipulation tasks Zeng et al.
and manipulation tasks involving high-dimensional control OpenAI et al.
Several methods have also sought to encode explicit inductive biases for manipulation tasks directly into the policy structure, including graph neural networks Li et al.
motion planners Yamada et al.
(2020), and through the use of action primitives Dalal et al.
Residual Reinforcement Learning :
Recent papers Johannink et al.
; Silver et al.
(2018) proposed combining a hand-engineered control policy with a reinforcement learning policy.
This combination can solve tricky tasks that can be solved approximately by a hand-engineered policy but can only be solved completely using the help of a residual RL policy applied on top.
However, coming up with such an underlying hand-engineered policy for each task is complex and labor-intensive.
Our sequenced linear dynamical system policy can also be viewed as coming up with a trainable hard-coded structure accompanied by a residual reinforcement learning policy.
Residual RL has been explored for manipulation tasks Johannink et al.
; Silver et al.
Zeng et al.
Garcia-Hernando et al.
Ranjbar et al.
; Schoettler et al.
(2019) and in a Learning from
Demonstration setting Alakuijala et al. ; Schoettler et al.
P RELIMINARIES
Reinforcement Learning :
The aim of applying reinforcement learning (RL) is to learn to generate actions that maximize the cumulative reward on a given task.
More specifically, RL aims to find optimal parameters θ for the policy π θ which can maximise the average discounted return J :
[Equation]
where s t represents current state, a t represents action sample from policy π θ , r( s t , a t ) represents the reward function and T is the maximum episode length.
Policy Gradient algorithms approximate the gradient of J with respect to policy parameters θ as
[Equation]
where V ( s ) represents value function and G t
= r t +1 + γV
( s t +1 ) .
Most modern algorithms parametrize the policy π θ as a neural network, and hence θ represents the network parameters Schulman et al.
; Mnih et al. (2016).
Linear Dynamical Systems : Dixon & Khosla (2004) present a way to represent trajectories using
Linear Dynamical Systems.
Consider a trajectory Y =
[ y 1 , y 2 , . . .
, y T ] where each state y t is an m -dimensional vector and T ≥ m .
Dixon & Khosla (2004) provide a least squares solution for extracting these LDS representations from a given demonstration.
An LDS can be represented using the difference equation:
[Equation]
where R is a matrix that parameterizes the LDS.
For a complete trajectory Y, We can rewrite Eq.
3 in a stacked matrix equation as:
[Equation]
or equivalently as
[Equation]
where Γ T =
[ y T , ..., y T ] .
The least-squares solution for is:
[Equation]
where, for any matrix M , M P = M T · ( M · M T ) − 1 .
Here y T represents the goal state or “attractor” of the trajectory.
The estimated LDS matrix ˆ R captures important properties of the provided trajectory Y .
A simple trajectory can be represented using a single LDS.
A complex trajectory can be represented using a sequence of LDS’s.
The LDS induces a control law which maps the current state to the desired state of the robot.
Consider a robot with end-effector position y t and end effector velocity ˙ y t .
An LDS matrix R induces a control field which maps each current end-effector position y t to a desired velocity ˙ y t .
Given a goal state g ∗ , an LDS matrix R can be used to predict ˙ y t .
[Equation]
M ETHOD 4. D IFFERENTIABLE LDS
For the LDS system in equations 7 and 8 to converge towards state g ∗ , the eigenvalues of R have to lie in the range ( − 1 , 0) .
We utilize this property to reparametrize the R matrix.
Consider Q , an ndimensional full rank matrix, and Λ , an n-dimensional diagonal matrix.
We can use the eigen-structure to construct the LDS matrix R from its eigenvalues ( Λ ) and eigenvectors ( Q ) as :
[Equation] [Equation]
Here Λ and Q are the learnable parameters used to construct an LDS matrix R .
Defined in this way, the eigenvalues of R , given by Λ s , are guaranteed to be in the range ( − 1 , 0) as required.
This re-parametrization of the LDS matrix in terms of its eigenvalues and eigenvectors can help us integrate it into any end-to-end gradient-based method.
[Figure]
Every k timesteps, the neural network ϕ θ estimates a goal g t and a residual δ t , which are used to parameterize one of the n dLDS units.
This dLDS unit is then integrated for k steps using Euler integration to generate a sequence of k actions, which are applied in sequence to the environment.
4. D LDS E XTRACTION FROM A R EFERENCE T RAJECTORY
In this subsection, we demonstrate how we can extract dLDS representations in a learning from demonstration (LfD) setting using a batched gradient descent algorithm.
Given a reference trajectory
Y =
[ y 1 , y 2 , ... y T ], we randomly initialize the dLDS parameters Q, Λ .
Given the current position y t , we can get the next state estimate ˆ y t as
[Equation]
[Equation]
Here, the goal g ∗ is the same as y T , the trajectory’s final position.
We can compute the loss as the mean squared error between predicted next state ˆ y t +
and the actual next state y t +1 .
[Equation] [Equation]
We can calculate the gradients for parameters Q, Λ according to the above-defined loss function.
Algorithm 1 in the Appendix summarizes our dLDS learning formulation when a reference trajectory is available.
4. D EEP S EQUENCED LDS P OLICIES
In this section we explain how dLDS can be integrated with a standard on-policy reinforcement learning algorithm like PPO Schulman et al.
We decompose an episode with length T , into a sequence of n dLDS units which can be represented by R 1 , R 2 , ..., R n .
dLDS parameters (Λ i , Q i ) can be used to construct R i using Equations 9 and 10.
Our policy also contains a goal estimator ϕ θ , which takes state s t as input and predicts the goal g ∗ t and residual δ t .
In this work, we select a deep neural network as the model class for ϕ θ , although any estimator suitable for policy gradient methods is compatible with our method.
[Equation]
Once we get the goal g ∗ t , we need an dLDS matrix R j to generate a trajectory from the current end effector position y t to g t .
We use the current time-step within the episode to choose a dLDS from n dLDS’s; specifically, at time-step t , dLDS index j is calculated as
[Equation]
where T is maximum episode length.
Now using goal g ∗ t , end effector position y t , and LDS matrix R j , we can perform Euler Integration to estimate the next state:
[Equation]
We perform this integration for k discrete time steps and use the trajectory obtained to calculate the actions from our policy.
We assume that our action can be represented as a target velocity ˙
y or position ˙ y ∆ t .
A key difference between dLDS policies and NDPs Bahl et al.
(2020) is that in NDPs, a neural network predicts ϕ ndp ( s t ) outputs the parameters w t of an underlying dynamical system (DMP) at each inference step:
[Equation]
In contrast, our estimation module ϕ lds ( s t ) only outputs some of the LDS parameters, namely the goal and residual (Eq 14).
The LDS matrices ( Λ 1 ..n and Q 1 ..n ) of our sequenced dLDS system (Eqs 15, 16, 17) are higher-level parameters that are a function of the “segment” that the trajectory is in; they do not need to be predicted at each timestep.
These parameters are still updated during training using gradient descent:
[Equation] [Equation]
This presents our policy with a simpler optimization problem of only learning the mapping at each timestep from state to goals and residuals rather than outputting the entire non-linear dynamical system at each inference-step.
To further verify this design choice, we compare our method against a policy that outputs the LDS parameters at each timestep from the deep network, rather than learning them at training time and then keeping them fixed per task.
4. G OAL G ENERATION FOR D LDS
As described in Equation 14, our learned policy generates the goals for the dLDS at each timestep.
Rather than outputting just a single goal, we found it more effective to have the network output n goals at each timestep ( g 1 , g 2 , ..., g n ), where n is the number of dLDS’s.
We then select the goal g j according to the index of the current LDS (Equation 15).
We found out that this works much better than outputting just a single goal at each timestep.
This is expected since when the network predicts a single goal, it has to figure out which dLDS it is predicting the goal for.
However, when we predict n goals, the policy has to solve a more straightforward problem of predicting goal states for each dLDS (without needing to reason about exactly which dlDS the robot is in at the moment).
4. R ESIDUAL P REDICTION
Apart from predicting goal states for each LDS, we also predict a residual δ t for each step in the policy rollout.
We take inspiration from Johannink et al.
(2018), where the idea is to combine a hand-engineered control structure with a residual RL policy.
Similarly, we treat our underlying dLDS structure as the basis on top of which a residual can be learned.
However, parameters of our underlying dLDS structure are learnable, unlike that of Johannink et al.
(2018) where one has to come up with a hand-engineered policy for each task involved.
We empirically verify that combining the action output ˙
y t of dLDS with a residual δ t gives a decent boost in its performance.
4. T RAINING DETAILS
To train our dLDS policy
[Figure]
we generate k actions to apply to the environment for each of the next k timesteps.
We use PPO Schulman et al.
(2017) to optimise the policy parameters (neural network parameters and dLDS parameters).
However
[Figure]
any other policy gradient algorithm could be used.
Algorithm 2 in the Appendix summarises our approach.
Hyperparameters and other details can be found in the appendix.
[Figure]
performs at least as well as the baselines on every task.
Here, the y -axis is the success rate, while the x -axis represents the number of environment episodes.
We compare our algorithm against PPO Schulman et al.
(2017), Multi-action PPO, NDP Bahl et al.
(2020), and NDP with residual.
We use 3 dLDS’s for our dLDS policy in each of these tasks.
R ESULTS
The motivation behind our work is to reparameterize a reinforcement learning policy with a decomposable, transformable structure so as to simplify the underlying policy learning problem.
We conduct an extensive experimental evaluation of our method on seven Metaworld-v2 Yu et al.
(2019) environments, namely ‘faucet-open-v2’, ‘reach-v2’, ‘soccer-v2‘, ‘assembly-v2’, ‘push-v2’, ‘box-close-v2’ and ‘bin-picking-v2’.
Our policy’s action space is the same as that of Metaworld-v2 environments, which consists of end-effector position change and “grab-effort” which is the normalized torque the fingers should apply.
We compare the performance of our dLDS Policy with other State of the art on-policy algorithms:
• PPO (raw) :
Vanilla Proximal Policy Optimisation Schulman et al.
(2017) without
any
modifications.
PPO-Multi
A multi-action version of PPO, which outputs actions for the next ‘k’ timesteps.
• NDP : Neural Dynamic Policy Bahl et al.
(2020) which builds on Dynamic Motion Primitives.
• NDP-Residual : A modified version of NDP with a learned residual.
5. P OLICY L EARNING R ESULTS
Figure 2 shows the comparison of our dLDS Policy with the various other baselines.
We obtained a significant improvement in sample efficiency compared to the other baseline algorithms.
For the tasks ‘faucet-open-v2’ and ‘reach-v2’, both NDP and dLDS Policy have similar performance, while for other tasks, dLDS policy outperforms other algorithms.
dLDS Policy outperforms the other baselines consistently across all tasks.
PPO is competitive for task ‘push-v2’ and ‘soccer-v2’.
However, its performance does not scale to other tasks.
Multi-PPO is comparatively better than PPO but fails to learn the tasks ‘faucet-open-v2’ and ‘assembly-v2’.
NDP is competitive in ‘reach-v2’ and ‘faucet-open-v2’.
However, its performance in harder tasks (‘assembly-v2’, ‘box-close-v2’ and ‘bin-picking-v2’) is significantly lower than that of the dLDS Policy.
The Residual version of NDP (NDP-Residual) shows comparative performance to NDP and dLDS in tasks ‘faucet-open-v2’ and ‘reach-v2’.
NDP-Residual) outperforms the vanilla-NDP algorithm in other tasks except ‘assemblyv2’, where both the algorithms fail to learn anything meaningful.
We provide our algorithm with a decomposable, transformable structure which results in an easy optmisation problem.
5. A BLATIONS
We now study the effect of various design choices on the performance our policy.
5.2. D IFFERENTIABLE LDS VS . D YNAMICALLY PREDICTED LDS
Similar to NDP, we train a version of our policy in which the deep part of policy outputs the parameters of the underlying dynamical system (in our case, parameters Q and Λ ) at each inference-step instead of learning them globally for each task.
We call this ablation “O-LDS policy.”
This is an inherently harder optimization problem since the network has to determine the goal and shape the trajectory concurrently.
We can verify our intuition from Figure 3 that learning the dLDS parameters (Q, Λ ) globally for each task instead of outputting them at each timestep from a deep network simplifies the learning problem
[Figure]
The above figure compares the performance of the two versions.
We can see that dLDS policy is easily able to outperform the O-LDS policy
5.2. N UMBER OF D LDS
The number of dLDS’s n is a key hyperparameter of dLDS policies.
Figure 4 compares the performance of dLDS Policy as we vary the number of dLDS’s.
The number of LDS’s controls the expressiveness of the policy.
For example, when n = 1, the policy is constrained to express very
[Figure]
simple trajectories.
This dLDS ( n =1) policy might be sufficient for a simpler task like ‘reach-v2’ and ‘faucet-open-v2’ but cannot learn to perform a complex task like ‘bin-picking-v2’ or ‘assembly-v2’, where policy needs to figure out a sequence of meaningful skills to accomplish the task.
Similarly, having a very high number of dLDS’s (e.g., 20) makes the policy too complex, and hence its performance is affected (Figure 4(b)).
We find out that n=3 is sufficient for most metaworld tasks.
[Figure]
We can see that including a residual term significantly improves the sample efficiency and performance of the dLDS policy.
How does adding a residual term affect the performance of a dynamical-system-based policy?
We find that adding a residual term to the output of a dynamical system-based policy – for both dLDS and or NDP – can improve performance substantially on a range of tasks.
Figure 5 compares the effect of residual on the performance of our dLDS Policy.
We observe that incorporating a residual improves sample-efficiency and performance on several environments, and on others is as least as good as the dLDS policy without a residual.
This result also holds when we train a residual version of NDPs, showing similar improvements over NDP on the ‘boxclose-v2’.
and ‘bin-picking-v2‘ tasks.
C ONCLUSION
Our paper introduces a novel reinforcement learning policy with a decomposible and transformable structure, which can generate smooth robot trajectories.
At the heart of our approach lies the novel differentiable LDS (dLDS) formulation.
We re-parametrize the LDS in terms of eigen-properties, which helps us to learn the LDS parameters directly in an end-to-end fashion.
We then propose a novel dLDS based reinforcement learning policy, which integrates these dLDS’s with a standard on-policy algorithm (PPO).
We perform extensive experiments and show a notable improvement in sample efficiency of our algorithm compared to other policy parameterizations.
Last
, we perform various ablations to verify several design choices of our algorithm
Choosing simpler models (i.e. LDS units) for parts of tasks where complexity is lower is a first step towards encoding meaningful inductive biases into manipulation policies in reinforcement learning settings.
We hope that our use of dynamical systems for policy representation inspires further investigation into this promising direction.
Specifically, we hope to investigate a more flexible tradeoff between learning a fixed skill bank of dynamical policies and predicting a new dynamical system at each time-step.
Additionally, we hope to explore how explicitly sequencing dynamical systems may improve policy transfer, generalization, and learning from demonstrations.
A A LGORITHMS
Require: Reference Trajectory Y =
[ y 1 , y 2 , ..., y T ] , learning rate α 1: Initialise Λ , Q ▷ Initialise LDS Matrix parameters.
2: while ∥ y t − g ∗ ∥
≥ ϵ do ▷ Convergence criterion 3: y t := get-current-obs()
▷ Obtain the current State information 4: Λ s = − sigmoid (Λ) ▷ Normalize Eigen values to be negative and less than5:
R := Q · Λ s · Q −▷ Construct R using its Eigen vecturs and Eigen values 6: ˆ Y :=
R · ( g ∗ − Y [: − 1])
+ Y
[: − 1] ▷ Predict y t +1 from y t 7: loss =
MSE ( ˆ Y , Y [1 :])
▷ Compute loss wrt reference trajectory.
8: loss.backward() ▷ Compute Gradients 9:
Q = Q − α dlossdQ ▷ Update parameters 10: Λ = Λ − α dlossd Λ
Require: Policy π , number of lds n, max episode len T
Require: rollout length k, discount factor γ 1: for 1, 2 ... episodes do 2: for t = 0, k, ..., T do 3:
Get goal g ∗ t , δ t using Eq.4:
Select dLDS j using Eq.5:
Create dLDS R j from Λ j , Q j (Eq. 16 and 17) 6: for i = 0, 1, ..., k do 7: ˙
y t
+ i =
R j · ( y t
+ i − g ∗ t )
+ δ i 8: y t
+ i +1 = ˙
y t
+ i dt + y t
+ i 9:
Set Action a t,...,t + k = ˙
y
t,...,t + k 10:
Apply a t,..,t + k on Environment 11:
Get s t +1 ,..,t + k +1 , r t,..,t
[Equation]
[Equation]
Store Transition ( s t , a t,..,t + k , s t
+ k +1 , R t ) 14: end for 15:
Compute Policy Gradients ∇ θ J 16:
Update Network and dLDS parameters 17: end for
B T RANSLATIONAL I NVARIANCE P ROOF
We present a proof to prove translational variance in dLDS’s.
Let’s assume our LDS matrix is represented by R j .
At any discrete time-step t , the position is y t and goal position is g .
The position at time t + 1 is given by equation
[Equation]
Now assuming we shift the origin of our coordinate frame by ∇ o .
The right hand side of Eq.becomes
[Equation]
Hence translational invariance is proved.
A proof of scale invariance follows in the same way.
C I MPLEMENTATION D ETAILS
We modified the
RL
Garage repository from garage contributors (2019) (https://github.com/rlworkgroup/garage) to support multi-action policies.
dLDS’s paramters learn at a higher learning rate than the neural network paramters of the policy.
We do not change rest of the parameters given in garage’s examples.
We use Metaworld environments to test the performance of our algorithms.
We use ‘faucet-open-v2’, ‘reach-v2’, ‘soccer-v2‘, ‘assembly-v2’, ‘push-v2’, ‘box-close-v2’ and ‘bin-picking-v2’ from the larger suite of environments available in
Metaworld.
These environments were chosen randomly.
We use a maximum episode length offor each of these environments.
Hyper-parameters Details
Hypyerparameter
Value
Learning Rate 3 x 10 −
Discount Factor 0.
Use GAE
True
Entropy Coefficient
Normalized Observation
True
Maximum Gradient
Norm 0.
PPO Mini-Batches
PPO Epochs
Clip Parameters 0.
Optimizer
Adam
Batch Size
Max Episode Length
Learning Rate for
LDS parameters 1 x 10 −
D A DDITIONAL A BALATIONS
In this section we present several additional ablations we performed to analyze the impact of various design decision on our policy formulation.
Descriptions can be found in the captions of Figures 6, 7, 8, and 9.
[Figure]
S u cc e ss
R a t e faucet-open-v
[Figure]
cc e ss
R a t e faucet-open-v
[Figure] [Figure]
improves performance.
[Figure]