Walk These Ways: Learning Diverse Gaits for an Uncertain Future
Abstract: We investigate the utility of structuring learned quadrupedal gaits with dynamically specified gait parameters.
We present a fast and robust locomotion controller that can trot, pronk, pace, and bound with variable footswing, posture, and speed.
Modulating these parameters at runtime enables a human operator or high-level planner to execute varied sequences of agile behavior useful for downstream tasks.
A single policy realizes crouching, hopping, highspeed running, agile leaps, and rhythmic dance.
Our analyses suggest new perspectives on the role of gait in learned locomotion.
Video is available at https://sites.google.com/view/gait-conditioned-rl/ .
Introduction
Much work on learning quadrupedal locomotion targets a static locomotion style objective, such as robustness to disturbances [1, 2, 3], energy efficiency [4] or similarity to a particular reference motion [5, 6, 7].
While successful, these approaches offer the user meager control over the robot’s locomotion style during deployment.
In practical use cases, the end user may prefer to have a choice among a large number of locomotion styles in the hope that some will suit their needs at any given moment.
The user’s needs might range from interaction with unanticipated environments to the generation of stylish motions for entertainment. [Figure]
[Figure] One way to place more control in the hands of the user is to provide a set of flexible and intuitive motion primitives that they can compose to produce the locomotion behavior of their preferred style or which suits their downstream task.
To this end, we implement and evaluate a task specification language for learned quadrupedal locomotion which enables the online composition of motion sequences combining diverse gaits, postures, and speeds.
Such language is common in the setting of analytical control, where it is often necessary to define constraints, objectives, and models [8, 9, 10].
It is often avoided in reinforcement learning, where it is possible for motion styles to emerge from a reward function without direct specification [1, 4, 11, 12].
We find that a single gait-conditioned policy can execute diverse and controllable motion styles while retaining a high degree of robustness against variations in robot and terrain dynamics.
Crucially, our method not only learns to locomote with diverse contact schedules, body postures, and foot postures, but also learns to transition between them online.
This enables diverse real-time adaptive behaviors useful for diverse downstream tasks, including navigation through confined spaces, agile forward leaps longer than the robot’s body length, and the execution of dynamic choreographed dance. Additional case studies show how dynamic gait adaptation can enable fast obstacle traversal, object manipulation, and robustness to shoves.
The combinatorial diversity of behaviors we demonstrate in a single locomotion policy goes beyond previous results in multi-task learned locomotion. This strong base policy invites future work to build high-level controllers for diverse tasks.
Our method employs shaped reward, but it is simple and extensible compared to other works employing discrete task-specific heuristics or reference trajectory libraries [7, 13, 14, 15].
Practitioners of learning-based legged locomotion have reported limits on the behavioral expressiveness of a single policy, and hypothesized that specialized architectures or learning procedures would be necessary [1, 4, 14, 16]. We demonstrate that given appropriate design choices, RL yields extremely diverse controllable quadrupedal gaits in a small, fully-connected policy network. We show comprehensive real-world validation of our claims in the setting of robust high-speed locomotion.
Related Work
Most previous work on learning locomotion addresses the task of achieving a target body velocity [1, 2, 3, 11, 13, 17, 16, 12, 18].
In these works, the operator provides the target velocity as input to a control policy, and the policy coordinates leg motions to track that velocity as closely as possible.
There are many ways a legged robot can move in order to achieve the same target velocity; it may select different foot contact patterns or carry its body in different poses while progressing forward at the same speed.
For this reason, we claim that the velocity tracking task is generally underspecified .
In response, several forms of specification have been used in the prior literature:
Auxiliary Rewards.
Some prior works overcome underspecification by defining auxiliary rewards.
For example, penalties on energy expenditure [4, 19], body motion [20], or deviation from a nominal pose [18] may distinguish one locomotion gait as preferred over another.
In theory, an objective such as energy minimization may cause substantially different gaits to be preferred at different speeds.
In practice, all works that learn joint-space policies with auxiliary rewards have ended up learning relatively static contact patterns and body motion [3, 16, 12, 18].
Fu et al.
[4] notably reported that different gaits emerge at different speeds when applying an auxiliary energy minimization reward, but they achieved this by training a separate policy at each speed and combining them through a manual distillation step. Other works learned a high-level policy to modulate the gait of a low-level controller to achieve energy minimization [19] or vision-guided gap crossing [20, 21].
All opted for a model-predictive low-level controller to execute the different gaits; a learned controller with suitable gait parameters did not yet exist.
Our work demonstrates such a controller and will enable revisiting these hierarchical frameworks with learning at both levels.
Imitation Learning.
Another line of work uses reference trajectories to specify robot motions [5, 6, 7, 22, 23, 15].
Some approaches use an imitation objective to reward states similar to the reference motion [5, 15, 23], while others use adversarial techniques to reward stylistically similar movements [6, 7]. Most imitation learning approaches learn only one style of behavior per policy [5, 6]. Vollenweider et al.
[7] provided a recent demonstration that a small number of discrete motion styles can be learned simultaneously.
Shao et al.
[15] notably generated a library of reference trajectories and trained a goal-conditioned policy to imitate them.
However, using a library of detailed reference trajectories penalizes the agent for exploring behaviors that are difficult to procedurally generate.
We prefer to specify the gait in a less explicit way, to allow the policy to discover complex strategies such as dynamic body motion, slip recovery, and resistance to perturbation.
Reward-based Task Specification.
The prior works most closely related to ours explicitly include gait parameters in the task specification through additional reward terms, including tracking rewards for contact patterns [24] and target foot placements [25].
Siekmann et al.
[24] used gait-dependent reward terms to penalize foot motion during stance and contact force during swing phases.
Compared to quadrupeds, a biped has fewer feet and its body has fewer practical degrees of freedom.
As such, the multi-gait policy in [24] has few parameters: the timing offset between the two feet and the duration of the swing phase.
The authors also reported difficulty eliciting multiple well-formed gaits from one policy, and implemented additional reward shaping to achieve specific gaits.
Method 3.1 Gait Specification
We parameterize the common quadrupedal gaits by an 11 -dimensional command vector,
[Equation]
• v cmd x , v cmd y , ω cmd z are the linear velocities in the body-frame x- and y- axes, and the angular velocity in the yaw axis.
• θ cmd =
( θ cmd 1 , θ cmd 2 , θ cmd 3 ) are the timing offsets between pairs of feet.
These express gaits including pronking ( θ cmd =
trotting ( θ cmd =
(0 . 5 , 0 , 0) ), bounding, ( θ cmd = (0 , 0 . 5 , 0) ), pacing ( θ cmd = (0 , 0 , 0 . 5) ), as well as their continuous interpolations such as galloping ( θ cmd =
Taken together, the parameters θ cmd can express all two-beat quadrupedal contact patterns.
Figure
2 provides
a visual illustration
• f cmd is the stepping frequency expressed in Hz .
Commanding f cmd =
Hz will result in each foot making contact three times per second.
• h cmd z is the body height command; ϕ cmd is the body pitch command.
• s cmd y is the foot stance width command; h fz cmd is the footswing height command.
Reward function.
All reward terms are given in Table 1.
Task rewards are defined as functions of the command vector c t for body velocity tracking, body posture tracking, foot posture tracking, and contact schedule tracking. Body velocity, body posture, and foot posture rewards are simple [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Table] functions of the squared error, weighted to account for the relative error magnitude. To define the desired contact schedule, function C cmdfoot ( θ cmd , t ) computes the desired contact state of each foot from the phase and timing variable, as described in [24], with details given in the appendix.
Stability and smoothness rewards express auxiliary locomotion objectives that are desired across all gaits.
As in [18], we force the total reward to be positive by computing it as r pos exp (0 .
02 r neg ) where r pos is the sum of all positive reward terms and r neg is the sum of all negative reward terms.
To demonstrate that our method is not detrimentally sensitive to reward tuning, we first tune the hyperparameters (reward weights, optimizer parameters) for a policy with no rewards for body pitch, stance width, or footswing height control ( ϕ cmd , s cmd y , h fz cmd ).
Then we introduce the rewards for ( ϕ cmd , s cmd y , h fz cmd ) and tune only their weights.
The original reward terms and weights do not need any re-tuning to add new gait parameters and capabilities to our proposed framework.
3.2 Learning Quadrupedal Gaits
We implement our training environment in the Isaac Gym simulator [26] and train our locomotion policy using Proximal Policy Optimization [27].
Observation Space.
The policy observes history o t − H...t where o t consists of command c t , previous action a t − 1 , sensor data s t , and timing reference variables t t .
The sensor data s t includes joint positions and velocities q t , ˙ q t (measured by joint encoders) and the gravity vector in the body frame g t (measured by accelerometer).
Timing reference variables t t are sine functions aligned to the contact offset of each foot: [Equation] [Equation] [Equation]
[Equation]
] = [Equation] )] where t is a counter variable that advances from 0 to 1 during each gait cycle and FR , FL , RR , RL are the four feet.
These timing variables are adapted from [24] to represent the quadrupedal gaits.
Action Space.
The action a t consists of position targets for each of the twelve joints.
A zero action corresponds to the nominal joint position, ˆ q . The position targets are tracked using a proportionalderivative controller with k p = 20 , k d = 0 . [Table]
Domain Randomization.
We randomize the robot’s body mass, motor strength [Table] and joint position calibration, the friction and restitution of the terrain, and the orientation and magnitude of gravity.
This randomization facilitates sim-to-real transfer and makes the controller robust to different terrain geometries and material properties.
All randomization ranges are given in Appendix A.
Policy Architecture.
We apply the Concurrent State Estimation architecture proposed by [18].
The “estimator module”, a feedforward neural network, is trained using supervised learning to predict the mass, velocity, and height of the robot body, the friction and restitution of the ground, and the gravity vector.
The input to this estimator module is a 15 -step history of observations.
The estimator output is concatenated with the state history as input to the policy body.
The policy body is an MLP with hidden layer sizes [512 , 256 , 128] ; the estimator module is an MLP with hidden layer sizes [256 , 128] ; both use ELU activations.
Task Curriculum.
To enable the robot to both run and spin fast, we sample velocity commands using the grid adaptive curriculum strategy from [12].
A discrete grid tabulates successful combinations of linear and angular velocity thus far.
Velocity commands for the policy are sampled from a small neighborhood of the successful region.
We grow a separate grid adaptive curriculum for each of the four major gaits: pronking, trotting, bounding, and pacing.
The procedure for sampling a command c t is as follows:
First, one of the four major gaits (pronking, trotting, bounding, or pacing) is selected with equal probability.
Then, a candidate velocity command ( v cmd x , ω cmd z ) for the chosen gait is sampled following [12].
To let the policy learn to interpolate between major gaits, ( θ cmd 1 , θ cmd 2 , θ cmd 3 ) are sampled from a Gaussian distribution that is centered at the chosen major gait.
The remaining command parameters ( v cmd y , f cmd , h cmd z , ϕ cmd , h fz cmd , s cmd y ) are sampled independently and uniformly.
Their ranges are given in Table 6.
3.3 Deployment
We deploy our controller in the real world on the Unitree Go1 Edu robot [28].
The Go1 is a commercially available and relatively low-cost robot quadruped.
Computing Architecture: An onboard Jetson TX2 NX computer runs our trained policy.
We implement an interface based on Lightweight Communications and Marshalling (LCM) [29] to pass sensor data, motor commands, and joystick state between our code and the low-level control SDK provided by Unitree. For both training and deployment, the control frequency is 50Hz.
Sensors:
In addition to joint position encoders and an inertial measurement unit, the robot is equipped with foot contact sensors.
We log the detected contact states for our analysis, but they are not provided to our controller as we did not find that they improve performance. [Table] [Table]
Experimental Results 4.1 Learning High-Speed Quadrupedal Gaits
Our method enables a single learned controller to run and spin using all common quadrupedal gaits.
Trotting is learned faster, followed by pronking, then pacing and bounding.
The high learnability of trotting aligns with its tendency to emerge from a variety of non-gait-specific auxiliary rewards [2, 11, 18, 20].
However, because our task sampling strategy provides explicit incentive to achieve all gaits, we find that fast and robust control strategies for pacing, bounding, and pronking emerge as well. Details about the relative learning progression of gaits can be found in the appendix.
4.2 Sim-to-Real Transfer and Gait Switching
We deploy our controller in zero-shot sim-to-real manner using the robot and software architecture described in Section 3.3.
We find that all gait parameters are correctly tracked after sim-to-real transfer.
Figure 2 shows torques and contact states during transition between trotting, pronking, bounding, and pacing in place while alternating f cmd between 2 Hz and 4 Hz .
In the bottom subfigure, good alignment between the colored rectangles, representing true contacts, and the dark shaded regions, representing desired contacts, reflects successful tracking across gaits and frequencies.
4.3 Impact of Gait Specification on Locomotion Performance
We probe the gait parameters of our trained model to quantify their role in quadrupedal locomotion performance.
We compare our gait-conditioned controller to a baseline velocity-tracking controller (the “gait-free baseline”).
The gait-free baseline is trained by the same method as our gaitconditioned controller, but without task reward terms for contact timing or body posture.
We aim to illustrate that varied styles of locomotion can have different performance characteristics on new tasks, and that by training a model that can achieve multiple styles, we grant the user a useful degree of freedom to exploit for any new task without task-specific retraining.
Energy Efficiency.
We measure the power consumption ( J / s ) of symmetric quadrupedal gaits and the gait-free baseline.
We consider mechanical power consumption measured by summing the product of joint velocity and torque at each of the 12 motors: (cid:80) i max( τ i ˙
q i , 0) . Table 3 reports the power consumption profile for each gait.
The robot expends less energy while trotting or pacing than while pronking or bounding.
Fixed trotting and pacing achieve equal or lower energy consumption compared to the gait-free baseline at low and medium speeds.
This suggests that policies trained using standard reward structure on flat ground are not achieving a gain in energy efficiency via uncommon adaptation of contact pattern, gait frequency, or body posture.
Robustness.
We evaluate quadrupedal gaits on a non-flat test terrain consisting of randomly arranged platforms up to 8cm in height (the “platform terrain”, pictured beside Table 4).
All policies in this work are trained only on flat ground, so the platform terrain is out-of-distribution.
In Table [Figure] 4, we report the mean task reward on the platform terrain for each of the major gaits and the gaitfree baseline.
We also report the survival time as the mean fraction of the maximum episode length ( 10 s ) before failure.
The episode was terminated if the agent did not fall after 10 seconds.
In this experiment, velocity commands were limited to v cmd x ∈ [ − 2 , 2] , ω cmd z ∈ [ − 2 , 2] .
[-C o n t a c t S t a t e s 
-]
Different gaits yield different robustness characteristics on the platform terrain.
Pacing has the highest survival time, higher than the gait-free policy, which indicates that it was able to remain stable on some terrains where the emergent gait on flat ground falls.
However, pacing tracked the target velocity less accurately than other gaits.
Pronking has the closest characteristics to the gaitfree policy in terms of velocity tracking, and with a similar survival time.
In the real world, we demonstrate robustness with different gaits while traversing uneven, slippery, and granular terrain as well as stairs.
The robot can traverse down stairs with a variety of gaits, but not up.
Video showing a number of real world deployment scenarios is at the project website.
Velocity Tracking and Top Speed.
We report velocity tracking performance of multiple 3Hz gaits and the gait-free baseline across a wide range of speeds including high-speed sprinting in Table 5.
Removing gait constraints results in improved velocity tracking task performance on flat ground.
Heat maps break down the mean task reward for each velocity, revealing that the gait-free approach does not help at low speeds, but is better for combinations of high linear and angular velocity.
4.4 Case Studies: Downstream Utility of Gait Modulation
Navigating confined spaces.
Different body postures and foot stances are useful for navigating confined spaces with limited width or height.
We demonstrate crawling beneath an increasingly lowered bar.
The robot was able to crawl under a 22 cm bar, and the robot body thickness is 13 cm , leaving maximum 9 cm of clearance between the robot and the ground.
Agile forward leap.
Modulation of contact schedule, velocity, and gait frequency at high speed encode an agile forward leap (Figure 3).
The robot first accelerates to a target speed of 3 m / s at a trot while increasing its step frequency from 2 Hz to 4 Hz , then switches to pronking at 2 Hz for one second, then decelerates to a standstill while trotting.
During the leap phase, the distance from the location of the robot’s front feet at takeoff to the location of the hind feet upon landing is 60 cm .
Choreographed dance.
We program a sequence of gait parameters to generate a dance routine synchronized to a jazz song with a tempo of 90 bpm.
At this tempo, combinations of phases 0 , 0 . 25 , and 0 .
5 with frequencies of 1 .
Hz and 3 Hz yield eighth, quarter, half, and full beat gaps between consecutive footsteps.
We also modulate body height and velocity in time with the music.
An assistant script procedurally generates gait parameters, fed into controller in open-loop fashion.
Climbing over curbs.
Modulating footswing height enables our controller to climb smoothly across 12-cm elevated platforms (Figure 1, left).
Previous works demonstrating blind obstacle traversal on
[Table]
a quadruped [1, 2] learned a foot-trapping reflex where the robot first trips, then raises its foot over the obstacle. In contrast, our gait-conditioned policy with high footswing command enables fast and smooth obstacle traversal without tripping, and is trained on a simpler flat-ground terrain.
Payload manipulation. Body posture control enables manipulation of objects. In one example, the robot carries a ball in a payload box tied to its back. The operator pilots the robot to the delivery location, then pitches the body backward to dump its payload.
Next, the robot runs to a resting ball and moves it into a goal by ducking and pushing with its shoulders (Figure 1, second left). In-motion posture control is also useful when a mounted camera needs to be aimed at a high or low target.
Bracing against shoves.
Learned locomotion controllers often keep the feet nominally below the hips as incentivized by auxiliary rewards [2, 18]. Widening the stance can make a quadruped more robust to perturbations, while narrowing it can facilitate better running efficiency or navigation of tight areas.
We shove the robot while modulating between wide and narrow stance. The robot remains upright in both cases, but its body motion is significantly less in the wide stance.
Limitations
Specifying the locomotion task reduces the system’s degree of autonomy, sacrificing some task performance.
Our experiments show this hinders the robot’s flat-ground sprinting performance (Section 4.3; Table 5), suggesting the best sprinting departs from our parameterized family of gaits. Constraining the gait may also lead to suboptimality in desiderata besides those we analyzed, such as wear on the robot, footstep noise, or foot dragging.
To further understand the effect of locomotion task parameterization on implicit or explicit task criteria is an open direction for future work.
Some gaits suffer from performance degradation in the real world. For example, in the agile leap sequence, the robot’s body sinks lower than the command height during the leap.
We also note that when assuming an unstable gait, such as low-frequency pacing with a narrow stance and high body, the robot tends to drift to one side or stumble. Our work relies on a human or high-level policy to compose behaviors. Future work can integrate gait planning methods such as [19, 20, 21] to give the system high-level autonomous behaviors.
Conclusion
We have implemented and analyzed a novel combination of flexible gait specification and robust training procedure for learning quadrupedal locomotion.
The resulting controller was modulated to accomplish a number of previously inaccessible downstream tasks without task-specific retraining.
This controller also exhibits a high degree of robustness with multiple gaits on uneven, slippery, and granular terrain, as well as stairs.
Quantitative experiments suggest that online modulation of gait parameters can benefit RL policies in out-of-distribution environments. Acknowledgments We thank the members of the Improbable AI lab for the helpful discussions and feedback on the paper. We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources.
This research was supported by the DARPA Machine Common Sense
Program, the MIT-IBM Watson AI Lab, and the National Science Foundation under Cooperative
Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/).
This research was also sponsored by the United States Air Force Research
Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
A Training Details
The ranges used for domain and command randomization are provided in Table 6.
The hyperparameters used for PPO are provided in Table 7.
The hyperparameters used in the curriculum are provided in Table 8.
Figure 6 illustrates data flow during training.
The curriculum engine first samples from a Gaussian distribution centered at one of the four main gaits (trotting, pronking, bounding, pacing); then it samples velocity commands from a grid distribution according to the method of [12]; then finally it samples the stepping frequency and body height uniformly.
The policy and simulator are rolled out, and the reward is computed as a function of the gait parameters.
Then the curriculum is updated if the episodic reward meets the thresholds given in Table 8.
B Teleoperation
Figure 5 illustrates the mapping from remote control inputs to gait parameters used during teleoperation.
The front bumpers on the top toggle between control modes to accommodate mixing our large number of gait parameters.
Preprogrammed sequences such as dancing and leaping can be assigned to the rear bumpers.
C Extended Performance Analysis
Impact of Gait Frequency on High-speed Running.
We evaluate them impact of gait frequency on performance of the robot at high speeds.
Figure 10 reports our result that higher gait frequency is necessary to yield good tracking performance for higher-speed running.
Impact of Footswing Height on Platform Terrain Performance.
We evaluate them impact of footswing height on performance of the robot on the out-of-distribution platform terrain.
Figurereports our result that higher swing heights yield improved platform traversal, outperforming the gait-free policy.
Flat Ground Velocity Heatmaps for More Gaits.
We provide velocity heatmaps as in Table 5 for pronking, pacing, and bounding gaits to supplement the trotting and gait-free heatmaps provided in
Table 5.
[Figure]
Forward and Backward Locomotion.
During evaluation in the random platforms environment, we found that walking backward leads to fewer failures than walking forward.
Figure 8 illustrates this phenomenon by plotting the mean failure rate of each gait at each test velocity.
Possible explanations include (i) recovery strategies that are dependent on knee orientation and (ii) the weight distribution of the robot.
Real-world Robustness Demonstrations.
We conducted several hours of real-world testing of different gaits across a variety of laboratory and outdoor environments.
A selection of this footage is available on the project website.
The robot was able to traverse down stairs, up and down granular and slippery terrain, and to respond to external perturbations.
To provide insight into the robot’s disturbance response, we plot the joint torques, contact states, and learned state estimate of the robot in Figure 7.
The robot trots at two different frequencies and is shoved twice by the operator, once from each side.
The state estimator correctly predicts the direction of the lateral velocity increase (orange), and adapts the joint torques and contact schedule to correct.
D Contact Schedule Parameterization
At each control timestep, we compute the desired contact from the gait parameters θ cmd as follows.
First, we increment the global timing variable t by f cmd f π where f cmd is the commanded stepping frequency and f π is the control frequency.
Then, we compute separate timing variables for each foot, clipped between 0 and 1 :
[Equation]
From these, we can directly compute the desired contact states:
[Equation]
where Φ( x ; σ ) is the cumulative density function of the normal distribution:
[Equation]
[Figure]
J o i n t T
o r q u e s ( N - m )
[Figure]
C o n t a c t S t a t
[Figure]
B o d y V e l o c
i t y
E s t i m a t e
[Figure]
[Figure]
zero and one.
The gait frequency is 3Hz.
The pacing gait boasts the longest mean survival time, possibly due to higher footswings or more practice with recovery strategies during training.
Interestingly, almost all failures occur during forward locomotion, and the robot is much more robust when moving backward.
Possible explanations include (i) recovery strategies that are dependent on knee orientation and (ii) the weight distribution of the robot.
which is an approximation to the Von Mises distribution used in [13] to form a smooth transition between stance and swing.
Hyperparameter
Value discount factor
[Table]
[Table]
[Table]
coefficient ( α
[Table]
[Table]
[Table]
[Figure]
li z e d T i m e t
o F a il u r e
[Figure]
on the platform terrain (Section 4.3).
Increased footswing height yields better generalization to uneven terrain from flat terrain compared to the gait-free policy
[Figure]
reward across speeds (Section 4.3).
Enforcing low frequency (2Hz) makes high speeds less attainable.
The gait-free policy offers slightly better performance at the lowest and highest speeds.