Paper:
LEARNING LOCOMOTION SKILLS USING DEEPRL: DOES
The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gaitcycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.
The use of deep reinforcement learning allows for high-dimensional state descriptors, but little is known about how the choice of action representation impacts the learning difficulty and the resulting performance. We compare the impact of four different action parameterizations (torques, muscle-activations, target joint angles, and target joint-angle velocities) in terms of learning time, policy robustness, motion quality, and policy query rates. Our results are evaluated on a gaitcycle imitation task for multiple planar articulated figures and multiple gaits. We demonstrate that the local feedback provided by higher-level action parameterizations can significantly impact the learning, robustness, and quality of the resulting policies.
1 INTRODUCTION
The introduction of deep learning models to reinforcement learning (RL) has enabled policies to operate directly on high-dimensional, low-level state features. As a result, deep reinforcement learning (DeepRL) has demonstrated impressive capabilities, such as developing control policies that can map from input image pixels to output joint torques (Lillicrap et al., 2015). However, the quality and robustness often falls short of what has been achieved with hand-crafted action abstractions, e.g., Coros et al. (2011); Geijtenbeek et al. (2013). While much is known about the learning of state representations, the choice of action parameterization is a design decision whose impact is not yet well understood.
Joint torques can be thought of as the most basic and generic representation for driving the movement of articulated figures, given that muscles and other actuation models eventually result in joint torques. However this ignores the intrinsic embodied nature of biological systems, particularly the synergy between control and biomechanics. Passive-dynamics, such as elasticity and damping from muscles and tendons, play an integral role in shaping motions: they provide mechanisms for energy storage, and mechanical impedance which generates instantaneous feedback without requiring any explicit computation. Loeb coins the term preflexes (Loeb, 1995) to describe these effects, and their impact on motion control has been described as providing intelligence by mechanics (Blickhan et al., 2007). This can also be thought of as a kind of partitioning of the computations between the control and physical system.
In this paper we explore the impact of four different actuation models on learning to control dynamic articulated figure locomotion: (1) torques (Tor); (2) activations for musculotendon units (MTU); (3) target joint angles for proportional-derivative controllers (PD); and (4) target joint velocities (Vel). Because Deep RL methods are capable of learning control policies for all these models, it now becomes possible to directly assess how the choice of actuation model affects the learning difficulty. We also assess the learned policies with respect to robustness, motion quality, and policy query rates. We show that action spaces which incorporate local feedback can significantly improve learning speed and performance, while still preserving the generality afforded by torque-level control. Such parameterizations also allow for more complex body structures and subjective improvements in motion quality.
Our specific contributions are: (1) We introduce a DeepRL framework for motion imitation tasks; (2) We evaluate the impact of four different actuation models on learned control policies according to four criteria; and (3) We propose an optimization approach that combines policy learning and actuator optimization, allowing neural networks to effective control complex muscle models.
2 BACKGROUND
Our task will be structured as a standard reinforcement problem where an agent interacts with its environment according to a policy in order to maximize a reward signal. The policy π(s, a) = p(a|s) represents the conditional probability density function of selecting action a ∈ A in state s ∈ S. At each control step t, the agent observes a state st and samples an action at from π. The environment in turn responds with a scalar reward rt, and a new state s′t = st+1 sampled from its dynamics p(s′|s, a). For a parameterized policy πθ(s, a), the goal of the agent is learn the parameters θ which maximizes the expected cumulative reward
J(πθ) = E [ T∑ t=0 γtrt ∣∣∣∣∣πθ ]
with γ ∈ [0, 1] as the discount factor, and T as the horizon. The gradient of the expected reward OθJ(πθ) can be determined according to the policy gradient theorem (Sutton et al., 2001), which provides a direction of improvement to adjust the policy parameters θ.
OθJ(πθ) = ∫ S dθ(s) ∫ A Oθlog(πθ(s, a))A(s, a)da ds
where dθ(s) = ∫ S ∑T t=0 γ
tp0(s0)p(s0 → s|t, πθ)ds0 is the discounted state distribution, where p0(s) represents the initial state distribution, and p(s0 → s|t, πθ) models the likelihood of reaching state s by starting at s0 and following the policy πθ(s, a) for t steps (Silver et al., 2014). A(s, a) represents a generalized advantage function. The choice of advantage function gives rise to a family of policy gradient algorithms, but in this work, we will focus on the one-step temporal difference advantage function (Schulman et al., 2015)
A(st, at) = rt + γV (s ′ t)− V (st) where V (s) = E [∑T
t=0 γ trt ∣∣∣s0 = s, πθ] is the state-value function, and can be defined recursively via the Bellman equation
V (st) = E rt,s′t
[rt + γV (s ′ t)|st, πθ]
A parameterized value function Vφ(s), with parameters φ, can be learned iteratively in a manner similar to Q-Learning by minimizing the Bellman loss,
L(φ) = E st,rt,s′t
[ 1
2 (yt − Vφ(st))2
] , yt = rt + γVφ(s ′ t)
πθ and Vφ can be trained in tandem using an actor-critic framework (Konda & Tsitsiklis, 2000).
In this work, each policy will be represented as a gaussian distribution with a parameterized mean µθ(s) and fixed covariance matrix Σ = diag{σ2i }, where σi is manually specified for each action parameter. Actions can be sampled from the distribution by applying gaussian noise to the mean action at = µθ(st) +N (0,Σ) The corresponding policy gradient will assume the form
OθJ(πθ) = ∫ S dθ(s) ∫ A Oθµθ(s)Σ −1 (a− µθ(s))A(s, a)da ds
which can be interpreted as shifting the mean of the action distribution towards actions that lead to higher than expected rewards, while moving away from actions that lead to lower than expected rewards.
3 TASK REPRESENTATION
3.1 REFERENCE MOTION
In our task, the goal of a policy is to imitate a given reference motion {q∗t } which consists of a sequence of kinematic poses q∗t in reduced coordinates. The reference velocity q̇∗t at a given time t is approximated by finite-difference q̇∗t ≈ q∗t+4t−q ∗ t
4t . Reference motions are generated via either using a recorded simulation result from a preexisting controller (“Sim”), or via manually-authored keyframes. Since hand-crafted reference motions may not be physically realizable, the goal is to closely reproduce a motion while satisfying physical constraints.
3.2 STATES
To define the state of the agent, a feature transformation Φ(q, q̇) is used to extract a set of features from the reduced-coordinate pose q and velocity q̇. The features consist of the height of the root (pelvis) from the ground, the position of each link with respect to the root, and the center of mass velocity of each link. When training a policy to imitate a cyclic reference motion {q∗t }, knowledge of the motion phase can help simplify learning. Therefore, we augment the state features with a set of target features Φ(q∗t , q̇t ∗), resulting in a combined state represented by st = (Φ(qt, q̇t),Φ(q∗t , q̇t ∗)). Similar results can also be achieved by providing a single motion phase variable as a state feature, as we show in Figure 15 (supplemental material).
3.3 ACTIONS
We train separate policies for each of the four actuation models, as described below. Each actuation model also has related actuation parameters, such as feedback gains for PD-controllers and musculotendon properties for MTUs. These parameters can be manually specified, as we do for the PD and Vel models, or they can be optimized for the task at hand, as for the MTU models. Table 1 provides a list of actuator parameters for each actuation model.
Target Joint Angles (PD): Each action represents a set of target angles q̂, where q̂i specifies the target angles for joint i. q̂ is applied to PD-controllers which compute torques according to τ i = kip(q̂ i − qi) + kid(ˆ̇qi − q̇i), where ˆ̇qi = 0, and kip and kid are manually-specified gains.
Target Joint Velocities (Vel): Each action specifies a set of target velocities ˆ̇q which are used to compute torques according to τ i = kid(ˆ̇q
i − q̇i), where the gains kid are specified to be the same as those used for target angles.
Torques (Tor): Each action directly specifies torques for every joint, and constant torques are applied for the duration of a control step. Due to torque limits, actions are bounded by manually specified limits for each joint. Unlike the other actuation models, the torque model does not require additional actuator parameters, and can thus be regarded as requiring the least amount of domain knowledge. Torque limits are excluded from the actuator parameter set as they are common for all parameterizations.
Muscle Activations (MTU): Each action specifies activations for a set of musculotendon units (MTU). Detailed modeling and implementation information are available in Wang et al. (2012). Each MTU is modeled as a contractile element (CE) attached to a serial elastic element (SE) and parallel elastic element (PE). The force exerted by the MTU can be calculated according to FMTU = FSE = FCE + FPE . Both FSE and FPE are modeled as passive springs, while FCE is actively controlled according to FCE = aMTUF0fl(lCE)fv(vCE), with aMTU being the muscle activation, F0 the maximum isometric force, lCE and vCE being the length and velocity of the contractile element. The functions fl(lCE) and fv(vCE) represent the force-length and force-velocity relationships, modeling the variations in the maximum force that can be exerted by a muscle as a function of its length and contraction velocity. Analytic forms are available in Geyer et al. (2003). Activations are bounded between [0, 1]. The length of each contractile element lCE are included as state features. To simplify control and reduce the number of internal state parameters per MTU, the policies directly control muscle activations instead of indirectly through excitations (Wang et al., 2012).
3.4 REWARD
The reward function consists of a weighted sum of terms that encourage the policy to track a reference motion.
r = wposerpose + wvelrvel + wendrend + wrootrroot + wcomrcom
wpose = 0.5, wvel = 0.05, wend = 0.15, wroot = 0.1, wcom = 0.2
Details of each term are available in the supplemental material. rpose penalizes deviation of the character pose from the reference pose, and rvel penalizes deviation of the joint velocities. rend and rroot accounts for the position error of the end-effectors and root. rcom penalizes deviations in the center of mass velocity from that of the reference motion.
3.5 INITIAL STATE DISTRIBUTION
We design the initial state distribution, p0(s), to sample states uniformly along the reference trajectory. At the start of each episode, q∗ and q̇∗ are sampled from the reference trajectory, and used to initialize the pose and velocity of the agent. This helps guide the agent to explore states near the target trajectory.
4 ACTOR-CRITIC LEARNING ALGORITHM
Instead of directly using the temporal difference advantage function, we adapt a positive temporal difference (PTD) update as proposed by Van Hasselt (2012).
A(s, a) = I [δ > 0] =
{ 1, δ > 0
0, otherwise
δ = r + γV (s′)− V (s) Unlike more conventional policy gradient methods, PTD is less sensitive to the scale of the advantage function and avoids instabilities that can result from negative TD updates. For a Gaussian policy, a negative TD update moves the mean of the distribution away from an observed action, effectively shifting the mean towards an unknown action that may be no better than the current mean action (Van Hasselt, 2012). In expectation, these updates converges to the true policy gradient, but for stochastic estimates of the policy gradient, these updates can cause the agent to adopt undesirable behaviours which affect subsequent experiences collected by the agent. Furthermore, we incorporate experience replay, which has been demonstrated to improve stability when training neural network policies with Q-learning in discrete action spaces. Experience replay often requires off-policy methods, such as importance weighting, to account for differences between the policy being trained and the behavior policy used to generate experiences (WawrzyńSki & Tanwani, 2013). However, we have not found importance weighting to be beneficial for PTD.
Stochastic policies are used during training for exploration, while deterministic policy are deployed for evaluation at runtime. The choice between a stochastic and deterministic policy can be specified by the addition of a binary indicator variable λ ∈ [0, 1]
at = µθ(st) + λN (0,Σ)
where λ = 1 corresponds to a stochastic policy with exploration noise, and λ = 0 corresponds to a deterministic policy that always selects the mean of the distribution. Noise from a stochastic policy will result in a state distribution that differs from that of the deterministic policy at runtime. To imitate this discrepancy, we incorporate -greedy exploration in addition to the original Gaussian exploration. During training, λ is determined by a Bernoulli random variable λ ∼ Ber( ), where λ = 1 with probability ∈ [0, 1]. The exploration rate is annealed linearly from 1 to 0.2 over 500k iterations, which slowly adjusts the state distribution encountered during training to better resemble the distribution at runtime. Since the policy gradient is defined for stochastic policies, only tuples recorded with exploration noise (i.e. λ = 1) can be used to update the actor, while the critic can be updated using all tuples.
Training proceeds episodically, where the initial state of each episode is sampled from p0(s), and the episode duration is drawn from an exponential distribution with a mean of 2s. To discourage falling, an episode will also terminate if any part of the character’s trunk makes contact with the ground for an extended period of time, leaving the agent with zero reward for all subsequent steps. Algorithm 1 in the supplemental material summarizes the complete learning process.
MTU Actuator Optimization: Actuation models such as MTUs are defined by further parameters whose values impact performance (Geijtenbeek et al., 2013). Geyer et al. (2003) uses existing anatomical estimates for humans to determine MTU parameters, but such data is not be available for more arbitrary creatures. Alternatively, Geijtenbeek et al. (2013) uses covariance matrix adaptation (CMA), a derivative-free evolutionary search strategy, to simultaneously optimize MTU and policy parameters. This approach is limited to policies with reasonably low dimensional parameter spaces, and is thus ill-suited for neural network models with hundreds of thousands of parameters. To avoid manual-tuning of actuator parameters, we propose a heuristic approach that alternates between policy learning and actuator optimization, as detailed in the supplemental material.
5 RESULTS
The motions are best seen in the supplemental video https://youtu.be/L3vDo3nLI98. We evaluate the action parameterizations by training policies for a simulated 2D biped, dog, and raptor as shown in Figure 1. Depending on the agent and the actuation model, our systems have 58–214 state dimensions, 6–44 action dimensions, and 0–282 actuator parameters, as summarized in Table 3 (supplemental materials). The MTU models have at least double the number of action parameters because they come in antagonistic pairs. As well, additional MTUs are used for the legs to more accurately reflect bipedal biomechanics. This includes MTUs that span multiple joints.
Each policy is represented by a three layer neural network, as illustrated in Figure 8 (supplemental material) with 512 and 256 fully-connected units, followed by a linear output layer where the number of output units vary according to the number of action parameters for each character and actuation model. ReLU activation functions are used for both hidden layers. Each network has approximately 200k parameters. The value function is represented by a similar network, except having a single linear output unit. The policies are queried at 60Hz for a control step of about 0.0167s. Each network is randomly initialized and trained for about 1 million iterations, requiring 32 million tuples, the equivalent of approximately 6 days of simulated time. Each policy requires about 10 hours for the biped, and 20 hours for the raptor and dog on an 8-core Intel Xeon E5-2687W.
Only the actuator parameters for MTUs are optimized with Algorithm 2, since the parameters for the other actuation models are few and reasonably intuitive to determine. The initial actuator parameters ψ0 are manually specified, while the initial policy parameters θ0 are randomly initialized. Each pass optimizes ψ using CMA for 250 generations with 16 samples per generation, and θ is trained for 250k iterations. Parameters are initialized with values from the previous pass. The expected value of each CMA sample of ψ is estimated using the average cumulative reward over 16 rollouts with a duration of 10s each. Separate MTU parameters are optimized for each character and motion. Each set of parameters is optimized for 6 passes following Algorithm 2, requiring approximately 50 hours. Figure 5 illustrates the performance improvement per pass. Figure 6 compares the performance of MTUs before and after optimization. For most examples, the optimized actuator parameters significantly improve learning speed and final performance. For the sake of comparison, after a set of actuator parameters has been optimized, a new policy is retrained with the new actuator parameters and its performance compared to the other actuation models.
Policy Performance and Learning Speed: Figure 2 shows learning curves for the policies and the performance of the final policies are summarized in Table 4. Performance is evaluated using the normalized cumulative reward (NCR), calculated from the average cumulative reward over 32 episodes with lengths of 10s, and normalized by the maximum and minimum cumulative reward possible for each episode. No discounting is applied when calculating the NCR. The initial state of each episode is sampled from the reference motion according to p(s0). To compare learning speeds, we use the normalized area under each learning curve (AUC) as a proxy for the learning speed of a particular actuation model, where 0 represents the worst possible performance and no progress during training, and 1 represents the best possible performance without requiring training.
PD performs well across all examples, achieving comparable-to-the-best performance for all motions. PD also learns faster than the other parameterizations for 5 of the 7 motions. The final performance of Tor is among the poorest for all the motions. Differences in performance appear more pronounced as characters become more complex. For the simple 7-link biped, most parameterizations achieve similar performance. However, for the more complex dog and raptor, the performance of Tor policies deteriorate with respect to other policies such as PD and Vel. MTU policies often exhibited the slowest learning speed, which may be a consequence of the higher dimensional action spaces, i.e., requiring antagonistic muscle pairs, and complex muscle dynamics. Nonetheless, once optimized, the MTU policies produce more natural motions and responsive behaviors as compared to other parameterizations. We note that the naturalness of motions is not well captured by the reward, since it primarily gauges similarity to the reference motion, which may not be representative of natural responses when perturbed from the nominal trajectory. A sensitivity analysis of the policies’ performance to variations in network architecture and hyperparameters are available in the supplemental material.
Policy Robustness: To evaluate robustness, we recorded the NCR achieved by each policy when subjected to external perturbations. The perturbations assume the form of random forces applied
to the trunk of the characters. Figure 3 illustrates the performance of the policies when subjected to perturbations of different magnitudes. The magnitude of the forces are constant, but direction varies randomly. Each force is applied for 0.1 to 0.4s, with 1 to 4s between each perturbation. Performance is estimated using the average over 128 episodes of length 20s each. For the biped walk, the Tor policy is significantly less robust than those for the other types of actions, while the MTU policy is the least robust for the raptor run. Overall, the PD policies are among the most robust for all the motions. In addition to external forces, we also evaluate robustness over randomly generated terrain consisting of bumps with varying heights and slopes with varying steepness. We evaluate the performance on irregular terrain (Figure 12, supplemental material). There are few discernible patterns for this test. The Vel and MTU policies are significantly worse than the Tor and PD policies for the dog bound on the bumpy terrain. The unnatural jittery behavior of the dog Tor policy proves to be surprisingly robust for this scenario. We suspect that the behavior prevents the trunk from contacting the ground for extended periods for time, and thereby escaping our system’s fall detection.
Query Rate: Figure 4 compares the performance of different parameterizations for different policy query rates. Separate policies are trained with queries of 15Hz, 30Hz, 60Hz, and 120Hz. Actuation models that incorporate low-level feedback such as PD and Vel, appear to cope more effectively to lower query rates, while the Tor degrades more rapidly at lower query rates. It is not yet obvious to us why MTU policies appear to perform better at lower query rates and worse at higher rates. Lastly, Figure 14 shows the policy outputs as a function of time for the four actuation models, for a particular joint, as well as showing the resulting joint torque. Interestingly, the MTU action is visibly smoother than the other actions and results in joint torques profiles that are smoother than those seen for PD and Vel.
6 RELATED WORK
DeepRL has driven impressive recent advances in learning motion control, i.e., solving for continuous-action control problems using reinforcement learning. All four of the actions types that we explore have seen previous use in the machine learning literature. WawrzyńSki & Tanwani (2013) use an actor-critic approach with experience replay to learn skills for an octopus arm (actuated by a simple muscle model) and a planar half cheetah (actuated by joint-based PD-controllers).
Recent work on deterministic policy gradients (Lillicrap et al., 2015) and on RL benchmarks, e.g., OpenAI Gym, generally use joint torques as the action space, as do the test suites in recent work (Schulman et al., 2015) on using generalized advantage estimation. Other recent work uses: the PR2 effort control interface as a proxy for torque control (Levine et al., 2015); joint velocities (Gu et al., 2016); velocities under an implicit control policy (Mordatch et al., 2015); or provide abstract actions (Hausknecht & Stone, 2015). Our learning procedures are based on prior work using actorcritic approaches with positive temporal difference updates (Van Hasselt, 2012).
Work in biomechanics has long recognized the embodied nature of the control problem and the view that musculotendon systems provide “preflexes” (Loeb, 1995) that effectively provide a form intelligence by mechanics (Blickhan et al., 2007), as well as allowing for energy storage. The control strategies for physics-based character simulations in computer animation also use all the forms of actuation that we evaluate in this paper. Representative examples include quadratic programs that solve for joint torques (de Lasa et al., 2010), joint velocities for skilled bicycle stunts (Tan et al., 2014), muscle models for locomotion (Wang et al., 2012; Geijtenbeek et al., 2013), mixed use of feed-forward torques and joint target angles (Coros et al., 2011), and joint target angles computed by learned linear (time-indexed) feedback strategies (Liu et al., 2016). Lastly, control methods in robotics use a mix of actuation types, including direct-drive torques (or their virtualized equivalents), series elastic actuators, PD control, and velocity control. These methods often rely heavily on modelbased solutions and thus we do not describe these in further detail here.
7 CONCLUSIONS
Our experiments suggest that action parameterizations that include basic local feedback, such as PD target angles, MTU activations, or target velocities, can improve policy performance and learning speed across different motions and character morphologies. Such models more accurately reflect the embodied nature of control in biomechanical systems, and the role of mechanical components in shaping the overall dynamics of motions and their control. The difference between low-level and high-level action parameterizations grow with the complexity of the characters, with high-level parameterizations scaling more gracefully to complex characters. As a caveat, there may well be tasks, such as impedance control, where lower-level action parameterizations such as Tor may prove advantageous. We believe that no single action parameterization will be the best for all problems. However, since objectives for motion control problems are often naturally expressed in terms of kinematic properties, higher-level actions such as target joint angles and velocities may be effective for a wide variety of motion control problems. We hope that our work will help open discussions around the choice of action parameterizations.
Our results have only been demonstrated on planar articulated figure simulations; the extension to 3D currently remains as future work. Furthermore, our current torque limits are still large as compared to what might be physically realizable. Tuning actuator parameters for complex actuation models such as MTUs remains challenging. Though our actuator optimization technique is able to improve performance as compared to manual tuning, the resulting parameters may still not be optimal for the desired task. Therefore, our comparisons of MTUs to other action parameterizations may not be reflective of the full potential of MTUs with more optimal actuator parameters. Furthermore, our actuator optimization currently tunes parameters for a specific motion, rather than a larger suite of motions, as might be expected in nature.
Since the reward terms are mainly calculated according to joint positions and velocities, it may seem that it is inherently biased in favour of PD and Vel. However, the real challenges for the control policies lie elsewhere, such as learning to compensate for gravity and ground-reaction forces, and learning foot-placement strategies that are needed to maintain balance for the locomotion gaits. The reference pose terms provide little information on how to achieve these hidden aspects of motion control that will ultimately determine the success of the locomotion policy. While we have yet to provide a concrete answer for the generalization of our results to different reward functions, we believe that the choice of action parameterization is a design decision that deserves greater attention regardless of the choice of reward function.
Finally, it is reasonable to expect that evolutionary processes would result in the effective co-design of actuation mechanics and control capabilities. Developing optimization and learning algorithms to allow for this kind of co-design is a fascinating possibility for future work.

Review:
Paper studies deep reinforcement learning paradigm for controlling high dimensional characters. Experiments compare the effect different control parameterizations (torques, muscle-activations, PD control with target joint positions and target joint velocities) have on the performance of reinforcement learning and optimized control policies. Evaluated are different planer gate cycle trajectories. It is illustrated that more abstract parameterizations are in fact better and result in more robust and higher quality policies. 
> Significance & Originality:
The explored parameterizations are relatively standard in humanoid control. The real novelty is systematic evaluation of the various parameterizations. I think this type of study is important and insightful. However, the findings are very specific to the problem and specific tested architecture. Its not clear that findings will transferable to other networks on other control problems/domains. As such for the ICLR community, this may have limited breadth and perhaps would have broader appeal in robotics / graphics community. 
> Clarity:
The paper is well written and is pretty easy to understand for someone who has some background with constrained multi-body simulation and control. 
> Experiments:
Experimental validation is lacking somewhat in my opinion. Given that this is a fundamentally experimental paper, I would have liked to see more analysis of sensitivity to various parameters and analysis of variance of performance when policy is optimized multiple times.