Walk These Ways : Learning with Parametric Auxiliary Rewards for an Uncertain Future
Abstract: Learned locomotion policies can rapidly adapt to diverse environments similar to the ones the agent experienced during training, but lack a mechanism for fast tuning when they fail in an out-of-distribution test environment. This necessitates a slow, iterative cycle of reward and environment redesign to achieve good performance in sim-to-real transfer. As an alternative, we propose training a single policy with parametric auxiliary rewards to represent a structured family of locomotion strategies. The auxiliary reward parameters can be adjusted in real-time to improve performance in out-of-distribution environments without time-consuming retraining.
We release a fast, robust open-source locomotion controller, Walk These Ways , that can trot, pronk, pace, and bound with variable footswing, posture, and speed, unlocking human demonstration of diverse downstream tasks including crouching, hopping, high-speed running, stair traversal, bracing against shoves, and rhythmic dance.
Video and code release: https://sites.google.com/view/gait-conditioned-rl/ .
Introduction
Consider training a locomotion policy to be applied in all of the environments and tasks in Figure 1: climbing over stairs and curbs, sprinting fast on ﬂat ground, bracing against shoves, crawling under obstacles, manipulating payloads and executing choreographed dance. Online system identiﬁcation approaches have been shown to rapidly adapt to in-distribution variations in the environment [1,
[Figure]
[Figure]
However, they have no fast mechanism for adjusting their behavior if they fail in an out-ofdistribution environment or task. In practice, all of the environments we care about in sim-to-real reinforcement learning are out-of-distribution due to imperfect simulation. Therefore, sim-to-real practitioners implement application-speciﬁc biases towards generalization, such as constraints on contact schedule, body posture, energy consumption, or footswing height [1, 2, 3, 4, 5, 6]. These biases are commonly encoded through auxiliary rewards and tuned as hyperparameters of RL.
This tuning process is slow because it requires training and deploying the policy repeatedly, and it results in a single tuned bias or preference among behaviors. Such a monolithic bias can be insufﬁcient to yield success across diverse tasks. For example, consider sprinting, climbing stairs, and resisting shoves: each can be thought of as requiring a different tradeoff between robustness and agility. The tuning of the auxiliary reward impacts this tradeoff.
We illustrate this in the nested red boxes in the top row of Figure 1; each shows an instance where a ﬁxed choice of auxiliary reward would lead to failure for one task, despite working well for another. Towards reducing the labor of tuning policies for open-world sim-to-real transfer, we propose and implement a task speciﬁcation for learned quadrupedal locomotion based on parametric auxiliary rewards . Here, the auxiliary reward is deﬁned by a set of parameters, which are sampled randomly at regular intervals during training and also directly observed by the policy. This incentivizes learning locomotion with different gaits, postures, and speeds that will be controllable by changing the input to the policy at deployment time.
A language of gait parameters is common in the setting of analytical control, where it is usually necessary to deﬁne constraints, objectives, and models [7, 8, 9].
It is more often avoided in reinforcement learning, where it is possible for motion styles to emerge from a reward function without direct speciﬁcation [1, 3, 4, 6].
We ﬁnd that a single policy with parametric auxiliary rewards can execute diverse and controllable motion styles while retaining a high degree of robustness against variations in robot and terrain dynamics.
Crucially, our method not only learns to locomote with diverse contact schedules, body postures, and foot postures, but also learns to transition between them online.
This enables real-time tuning of the locomotion behavior or gait for application to new downstream tasks.
Related Work
Most previous work on learning locomotion addresses the task of achieving a target body velocity [1, 2, 4, 5, 6, 10, 11, 12, 13].
In these works, the operator provides the target velocity as input to a control policy, and the policy coordinates leg motions to track that velocity as closely as possible.
There are many ways a legged robot can move in order to achieve the same target velocity; it may select different foot contact patterns or carry its body in different poses while progressing forward at the same speed.
For this reason, we claim that the velocity tracking task is generally underspeciﬁed .
In response, several forms of speciﬁcation have been used in the prior literature:
Auxiliary Rewards.
Some prior works overcome underspeciﬁcation by tuning auxiliary rewards.
For example, penalties on energy expenditure [3, 14], body motion [4, 15], or footswing height [5] can serve to break ties in the simulated training environment, in favor of gaits that generalize to a target environment.
In theory, an objective such as energy minimization may cause substantially different gaits to be preferred at different speeds.
In practice, all works that learn joint-space policies with auxiliary rewards have ended up learning relatively static contact patterns and body motion [5, 6, 10, 13].
Fu et al.
[3] notably found that different gaits emerge at different speeds when applying an auxiliary energy minimization reward, but they achieved this by training a separate policy at each speed and combining them through a manual distillation step.
Imitation Learning.
Another line of work uses reference trajectories to specify robot motions [16, 17, 18, 19, 20, 21].
Some approaches use an imitation objective to reward adherence to the reference motion [16, 20, 21], while others use adversarial techniques to reward stylistically similar movements [17, 18].
Most imitation learning approaches learn only one style of behavior per policy [16, 17].
Vollenweider et al.
[18] provided a recent demonstration that a small number of discrete motion styles can be learned simultaneously.
Shao et al.
[21] notably generated a library of reference trajectories and trained a goal-conditioned policy to imitate them.
However, using a library of full reference trajectories can penalize the agent for exploring behaviors that are difﬁcult to procedurally generate.
We prefer to specify the desired behavior more softly as an auxiliary reward, hoping this will help retain complex strategies such as dynamic body motion or slip recovery. Generalized Diversity Objectives.
Several prior methods explicitly optimize for a generalized metric of diversity during training.
Quality diversity methods [22, 23, 24] learn a repertoire of highperforming discrete skills by maximizing a novelty objective deﬁned among trajectories.
Other work uses diversity as an unsupervised objective function for RL [25].
In contrast, parameterized auxilary reward is good for learning diverse behaviors corresponding to a set of structured parameters that are predeﬁned using domain knowledge.
Parameterized-Reward Task Speciﬁcation.
The prior works most closely related to ours explicitly include gait parameters in the task speciﬁcation through additional reward terms, including tracking rewards for contact patterns [26] and target foot placements [27].
Siekmann et al.
[26] used gaitdependent reward terms to penalize foot motion during stance and contact force during swing phases.
Compared to quadrupeds, a biped has fewer feet and its body has fewer practical degrees of freedom.
As such, the multi-gait policy in [26] has few parameters: the timing offset between the two feet and the duration of the swing phase.
This smaller space of command parameters was not shown to be useful for compositional tasks or out-of-distribution environments, but it inspires our work.
Hierarchical Control Using Gait Parameters.
Several works have learned a high-level policy to modulate the gait parameters of a low-level model-based controller to accomplish downstream tasks. This approach has been applied to energy minimization [14] and vision-guided foot placement [15, 28]. All opted for a model-predictive low-level controller to execute the different gaits; a learned controller with suitable gait parameters did not yet exist. The tradeoffs between model-based control and reinforcement learning are well discussed in prior literature [6, 13].
Our work opens the door to revisiting hierarchical methods with learning at both levels.
Method 3.1 Gait Speciﬁcation
We parameterize the common quadrupedal gaits by an 11 -dimensional command vector,
[Equation]
[Equation]
v [Equation] [Equation] z are the linear velocities in the body-frame x- and y- axes, and the angular velocity in the yaw axis
• θ cmd =
( θ cmd 1 , θ cmd 2 , θ cmd 3 ) are the timing offsets between pairs of feet.
These express gaits including pronking ( θ cmd =
trotting ( θ cmd =
(0 . 5 , 0 , 0) ), bounding, ( θ cmd = (0 , 0 . 5 , 0) ), pacing ( θ cmd = (0 , 0 , 0 . 5) ), as well as their continuous interpolations such as galloping ( θ cmd =
Taken together, the parameters θ cmd can express all two-beat quadrupedal contact patterns.
Figure 2 provides a visual illustration.
• f cmd is the stepping frequency expressed in Hz .
Commanding f cmd =
Hz will result in each foot making contact three times per second.
[Figure]
[Table] structure [Table] • h cmd z is the body height command; φ cmd is the body pitch command.
• s cmd y is the foot stance width command; h fz cmd is the footswing height command.
Reward function.
All reward terms are given in Table 1.
Task rewards are deﬁned as functions of the command vector c t for body velocity tracking, body posture tracking, foot posture tracking, and contact schedule tracking.
Body velocity, body posture, and foot position rewards are simple functions of the squared error, weighted to account for the relative error magnitude.
The desired foot position in the ground plane p fx,y cmdfoot ( s cmd y ) is computed from stance width, body velocity and contact schedule using the Raibert heuristic [7].
To deﬁne the desired contact schedule, function
C cmdfoot ( θ cmd , t ) computes the desired contact state of each foot from the phase and timing variable, as described in [26], with details given in the appendix.
Stability and smoothness rewards express auxiliary locomotion objectives that are desired across all gaits.
As in [5], we force the total reward to be positive by computing it as r pos exp (0 .
02 r neg ) where r pos is the sum of all positive reward terms and r neg is the sum of all negative reward terms.
To demonstrate that our method is not detrimentally sensitive to reward tuning, we ﬁrst tune the hyperparameters (reward weights, optimizer parameters) for a policy with no rewards for body pitch, stance width, or footswing height control ( φ cmd , s cmd y , h fz cmd ).
Then we introduce the rewards for ( φ cmd , s cmd y , h fz cmd ) and tune only their weights.
The original reward terms and weights do not need any re-tuning to add new gait parameters and capabilities to our proposed framework.
3.2 Learning Quadrupedal Gaits
We implement our training environment in the Isaac Gym simulator [4, 29] and train our locomotion policy using Proximal Policy Optimization [30].
Observation Space.
The policy observes history
o t − H...t where o t consists of command c t , previous action a t − 1 , sensor data s t , and timing reference variables t t .
The sensor data s t includes joint positions and velocities q t , ˙ q t (measured by joint encoders) and the gravity vector in the body frame g t (measured by accelerometer).
Timing reference variables t t are sine functions aligned to the contact offset of each foot:
[Equation]
[Table]
where t is a counter variable that advances from 0 to 1 during each gait cycle and FR , FL , RR , RL are the four feet.
These timing variables are adapted from [26] to represent the quadrupedal gaits.
Action Space.
The action a t consists of position targets for each of the twelve joints.
A zero action corresponds to the nominal joint position, ˆ q .
The position targets are tracked using a proportionalderivative controller with k p = 20 , k d = 0 .
Domain Randomization.
We randomize the robot’s body mass, motor strength, and joint position calibration, the friction and restitution of the terrain, and the orientation and magnitude of gravity.
To simplify training and provide a case study in generalization, we only train on ﬂat ground and do not randomize the terrain geometry.
All randomization ranges are given in Appendix A.
Latency and Actuator Modeling.
We perform system identiﬁcation to reduce the sim-to-real gap in the robot dynamics.
Following [13], we train an actuator network to capture the non-ideal relationship between PD error and realized torque.
Separately, we identify a consistent latency of around 20 ms in our system and model this as a constant action delay during simulation.
Policy Architecture.
We apply the Concurrent State Estimation architecture proposed by [5].
The “estimator module”, a feedforward neural network, is trained using supervised learning to predict the mass, velocity, and height of the robot body, the friction and restitution of the ground, and the gravity vector.
The input to this estimator module is a 15 -step history of observations.
The estimator output is concatenated with the state history as input to the policy body.
The policy body is an MLP with hidden layer sizes [512 , 256 , 128] ; the estimator module is an MLP with hidden layer sizes [256 , 128] ; both use ELU activations.
Task Curriculum.
To enable the robot to both run and spin fast, we sample velocity commands using the grid adaptive curriculum strategy from [6].
A discrete grid tabulates successful combinations of linear and angular velocity thus far.
Velocity commands for the policy are sampled from a small neighborhood of the successful region.
The procedure for sampling a command c t is as follows:
First, one of the four major contact patterns (pronking, trotting, bounding, or pacing) is selected with equal probability.
Then, a candidate velocity command ( v cmd x , ω cmd z ) for the chosen gait is sampled following [6].
To let the policy learn to interpolate between major contact patterns, ( θ cmd 1 , θ cmd 2 , θ cmd 3 ) are sampled from a Gaussian distribution that is centered at the chosen major gait.
The remaining command parameters ( v cmd y , f cmd , h cmd z , φ cmd , h f z cmd , s cmd y ) are sampled independently and uniformly.
Their ranges are given in Table 6.
3.3 Deployment
We deploy our controller in the real world on the Unitree Go1 Edu robot [31].
[Table] We the fraction
Computing Architecture: An onboard Jetson TX2 NX computer runs our trained policy.
We implement an interface based on Lightweight Communications and Marshalling (LCM) [32] to pass sensor data, motor commands [Table] and joystick state between our code and the low-level control SDK provided by Unitree.
For both training and deployment, the control frequency is 50Hz.
Sensors:
In addition to joint position encoders and an inertial measurement unit, the robot is equipped with foot contact sensors.
We log the detected contact states for our analysis, but they are not provided to our controller as we did not ﬁnd that they improve performance.
Experimental Results 4.1 Sim-to-Real Transfer and Gait Switching
We deploy our controller in zero-shot sim-to-real manner using the robot and software architecture described in Section 3.3.
We ﬁnd that all gait parameters are consistently tracked after sim-to-real transfer.
Figure 2 shows torques and contact states during transition between trotting, pronking, bounding, and pacing in place while alternating f cmd between 2 Hz and 4 Hz .
In the bottom subﬁgure, good alignment between the colored rectangles, representing true contacts, and the dark shaded regions, representing desired contacts, reﬂects successful tracking across gaits and frequencies.
Videos (i)-(iv) on the project website demonstrate the modulation of each parameter.
4.2 Impact of Auxiliary Reward Modulation on Locomotion Performance
We probe the reward parameters of our trained model to quantify their role in quadrupedal locomotion performance.
We compare our controller with various auxiliary reward settings to a baseline velocity-tracking controller (the “gait-free baseline”).
The gait-free baseline is trained by the same method as our controller, but without the parameterized auxiliary reward terms for contact timing, body posture, and foot placement.
We aim to illustrate that varied styles of locomotion can have different performance characteristics on new tasks, and that by training a model that can achieve multiple styles, we grant the user a useful degree of freedom to exploit for any new task without task-speciﬁc retraining.
Energy Efﬁciency.
We measure the power consumption ( J / s ) of symmetric quadrupedal gaits and the gait-free baseline.
We consider mechanical power consumption measured by summing the product of joint velocity and torque at each of the 12 motors: (cid:80) i max( τ i ˙
q i , 0) .
Table 3 reports the power consumption proﬁle for each gait.
The robot expends less energy while trotting or pacing than while pronking or bounding.
Trotting and pacing also consume less energy than the gait-free baseline at low and medium speeds.
This indicates that tuning the auxiliary reward parameters can facilitate optimizing a new objective (energy efﬁciency) in the original training environment.
Robustness.
We evaluate learned behaviors on a non-ﬂat test terrain consisting of randomly arranged platforms up to 8cm in height (the “platform terrain”, pictured beside Table 4).
All policies in this work are trained only on ﬂat ground (Section 3.2), so the platform terrain is out-of-distribution.
In Table 4, we report the mean task reward on the platform terrain for each of the major gaits and
[Figure]
the gait-free baseline.
We also report the survival time as the mean fraction of the maximum episode length ( 10 s ) before failure.
The episode was terminated if the agent did not fall after 10 seconds.
In this experiment, velocity commands were limited to v cmd x ∈ [ − 2 , 2] , ω cmd z ∈ [ − 2 , 2] .
Different gaits yield different robustness characteristics on the platform terrain.
Pacing has the highest survival time, higher than the gait-free policy, which indicates that it was able to remain stable on some terrains where the emergent gait on ﬂat ground falls.
However, pacing tracked the target velocity less accurately than other gaits.
Pronking has the closest characteristics to the gait-free policy in terms of velocity tracking, and with a similar survival time.
Figure 9 in the appendix shows a similar analysis for footswing height. These results suggest that tuning parameterized auxiliary rewards is a viable mechanism for out-of-distribution adaptation.
Velocity Tracking and Top Speed.
We report velocity tracking performance of multiple gaits and the gait-free baseline across a wide range of speeds including high-speed sprinting in Table 5.
Removing gait constraints results in improved velocity tracking task performance on ﬂat ground.
Heat maps break down the mean task reward for each velocity, revealing that the parameterized auxiliary rewards do not hinder performance at low speeds, but are restrictive for combinations of high linear and angular velocity.
We conclude that improved generalization via parametric auxiliary rewards can come with some cost to in-distribution task performance.
4.3 Case Studies: Achieving New Tasks with Auxiliary Reward Modulation
Navigating conﬁned spaces.
Different body postures and foot stances are useful for navigating conﬁned spaces with limited width or height.
We demonstrate crawling beneath an increasingly lowered bar.
The robot was able to crawl under a 22 cm bar, and the robot body thickness is 13 cm , leaving maximum 9 cm of clearance between the robot and the ground.
Agile forward leap.
Modulation of contact schedule, velocity, and gait frequency at high speed encode an agile forward leap (Figure 3).
The robot ﬁrst accelerates to a target speed of 3 m / s at a trot while increasing its step frequency from 2 Hz to 4 Hz , then switches to pronking at 2 Hz for one second, then decelerates to a standstill while trotting.
During the leap phase, the distance from the location of the robot’s front feet at takeoff to the location of the hind feet upon landing is 60 cm .
Choreographed dance.
We program a sequence of gait parameters to generate a dance routine synchronized to a jazz song with a tempo of 90 bpm.
At this tempo, combinations of phases 0 , 0 . 25 , and 0 .
5 with frequencies of 1 .
Hz and 3 Hz yield eighth, quarter, half, and full beat gaps between consecutive footsteps.
We also modulate body height and velocity in time with the music.
An assistant script procedurally generates gait parameters, fed into the controller in open-loop fashion.
Climbing over curbs.
Modulating footswing height enables our controller to climb smoothly across stairs and curbs (Figure 1).
Previous works demonstrating blind obstacle traversal on a quadruped [1, 2] learned a foot-trapping reﬂex where the robot ﬁrst trips, then raises its foot over the obstacle.
In contrast, our gait-conditioned policy with high footswing command enables fast and smooth obstacle traversal without tripping, and is trained on a simpler ﬂat-ground terrain. [Table]
Payload manipulation.
Body posture control enables manipulation of objects.
In one example, the robot carries a ball in a payload box tied to its back.
The operator pilots the robot to the delivery location, then pitches the body backward to dump its payload.
Next, the robot runs to a resting ball and moves it into a goal by ducking and pushing with its shoulders (Figure 1, second left).
In-motion posture control is also useful when a mounted camera needs to be aimed at a high or low target.
Bracing against shoves.
Learned locomotion controllers often keep the feet nominally below the hips as incentivized by auxiliary rewards [2, 5].
Widening the stance can make a quadruped more robust to perturbations, while narrowing it can facilitate better running efﬁciency or navigation of tight areas.
We shove the robot while modulating between wide and narrow stance.
The robot remains upright in both cases, but its body motion is signiﬁcantly less in the wide stance.
Limitations
Our experiments show that the beneﬁts of adding parametric auxiliary rewards can come at a cost to in-distribution task performance, speciﬁcally limiting the robot’s ﬂat-ground sprinting performance (Section 4.2; Table 5).
To quantify and control the tradeoff between task performance and reward shaping, some prior methods have been proposed [33, 34].
Some gaits suffer from performance degradation in the real world.
For example, in the agile leap sequence (Figure 3), the robot’s body sinks lower than the command height during the leap.
Also, for some less stable gaits, such as low-frequency pacing with narrow stance and high body, the robot tends to drift to one side or stumble.
All behaviors we demonstrate rely on a human pilot to tune the auxiliary reward parameters during deployment.
In the future, the parameter adaptation could be automated using behavioral cloning on real-world human demonstrations, or by learning a high-level policy with RL as in [14, 15, 28].
Conclusion
We have implemented and analyzed a novel method for learning quadrupedal locomotion with parametric auxiliary rewards.
The resulting controller can accomplish a number of previously inaccessible downstream tasks and can be rapidly tuned in out-of-distribution environments without any retraining.
Quantitative experiments suggest that modulating parametric auxiliary rewards is a practical mechanism to quickly improve RL policy performance in out-of-distribution environments.
Acknowledgments
We thank the members of the Improbable AI lab for the helpful discussions and feedback on the paper.
We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources.
This research was supported by the DARPA Machine Common Sense
Program, the MIT-IBM Watson AI Lab, and the National Science Foundation under Cooperative
Agreement PHY-2019786 (The NSF AI Institute for Artiﬁcial Intelligence and Fundamental Interactions, http://iaiﬁ.org/).
This research was also sponsored by the United States Air Force Research
Laboratory and the United States Air Force Artiﬁcial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the United States Air Force or the U
S. Government.
The U
S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
Author Contributions
• Gabriel B. Margolis concieved, designed, and implemented the controller, ran all experiments, and played the primary role in paper writing.
• Pulkit Agrawal advised the project and contributed to its conceptual development, experimental design, positioning, and writing.