Walk These Ways : Tuning Robot Control for Generalization with Multiplicity of Behavior
{ gmargo, pulkitag } @mit.edu Abstract: Learned locomotion policies can rapidly adapt to diverse environments similar to those experienced during training but lack a mechanism for fast tuning when they fail in an out-of-distribution test environment.
This necessitates a slow and iterative cycle of reward and environment redesign to achieve good performance on a new task.
As an alternative, we propose learning a single policy that encodes a structured family of locomotion strategies that solve training tasks in different ways, resulting in Multiplicity of Behavior (MoB).
Different strategies generalize differently and can be chosen in real-time for new tasks or environments, bypassing the need for time-consuming retraining.
We release a fast, robust open-source MoB locomotion controller, Walk These Ways , that can execute diverse gaits with variable footswing, posture, and speed, unlocking diverse downstream tasks: crouching, hopping, high-speed running, stair traversal, bracing against shoves, rhythmic dance, and more.
Video and code release: https://gmargo11.github.io/walk-these-ways/
Introduction
Recent works have established that quadruped locomotion controllers trained with reinforcement learning in simulation can successfully be transferred to Adaptation to diverse terrains is accomplished by estimating terrain properties from sensory observations that are then used by the controller (i.e., online system identiﬁcation ). The success of this paradigm relies on two assumptions: a priori modeling of environment parameters that can vary during deployment and the ability to estimate these parameters from sensory observations.
To bypass the ﬁrst assumption, one possibility is to widely vary a large set of environment parameters during training. However, this creates a hard learning problem due to creation of challenging or infeasible locomotion scenarios. To simplify learning, typically the designer chooses a subset of parameters that are randomized in a carefully restricted range.
Even in this setup, additional measures such as a learning curriculum and reward shaping are necessary for successful learning in simulation.
As a result of these practical restrictions on the expressiveness of the simulation, quite often the robot encounters scenarios during deployment that were not modeled during training. For instance, if the robot is only presented with ﬂat ground and terrain geometry is not varied during training, it may fail to traverse non-ﬂat terrains such as stairs.
In such a case, it is common to tweak the training environments or the reward functions and re-train the policy. This iterative loop of retraining and real-world testing is tedious. To make things worse, in some scenarios such iteration is insufﬁcient because it is not possible to accurately model or sense important environment properties.
For example, thick bushes are both hard to simulate due to compliance and hard to sense because depth sensors do not distinguish them from walls. Thus, the robot may attempt to climb over thick bushes like a rock or move through them with an overly conservative gait that leaves the robot stuck.
The examples above illustrate that even for the most advanced sim-to-real systems, the real world offers new challenges. We broadly refer to scenarios that can be simulated but are not anticipated during training and the situations which cannot be simulated or identiﬁed from sensory observations as out-of-distribution cases. We present a framework for policy learning that enables improved performance in out-of-distribution scenarios under some assumptions detailed below.
Our key insight is that given a task, there are multiple equally good solutions (i.e., under-speciﬁcation [4]) that have equivalent training performance but can generalize in different ways.
For instance, the task of walking on ﬂat ground only imposes a constraint on the velocity of robot’s body, but not on how the legs should move, or high should the torso be above the ground, etc. Consider two different walking behaviors: crouch where the robot keeps its torso close to the ground and stomp where the torso is high and also the legs have a high foot swing.
While both crouch and stomp succeed at walking on ﬂat ground, their generalization to out-of-distribution scenarios is different: with crouch the robot can traverse under obstacles but not stairs, whereas with stomp it can climb over curbs/stairs but not move under obstacles (Figure 1). Out of the many possible locomotion behaviors that succeed in the training environment, typical reinforcement learning formulations result in a policy that only embodies one solution and therefore expresses a single generalizing bias.
To facilitate generalization to diverse scenarios, we propose a technique, Multiplicity of Behavior (MoB), that given the same observation history and a small set of behavior parameters outputs different walking behaviors.
When faced with an unseen scenario, one can test different behaviors by varying these parameters, which affords much quicker iteration than re-training a new policy and facilitates collection of online demonstrations by a human pilot. The utility of MoB depends on the assumption that some subset of behaviors successful in the restricted training environment will also succeed in the out-of-distribution target environment.
To demonstrate that this is true in a meaningful sense, we chose an extreme case: train a single policy for quadrupedal walking only on ﬂat ground and evaluate on non-ﬂat terrains and new tasks. We show that a human operator can tune behaviors in real-time to enable successful locomotion in the presence of diverse unseen terrains and dynamics including uneven ground, stairs, external shoves, and constrained spaces (Figure 1). The same tuning mechanisms can be used to compose behaviors to perform new tasks such as payload manipulation and rhythmic dance (Figure 1). This work contributes a robust low-level quadruped controller that can execute diverse structured behaviors, which we we hope will be a useful building block for future locomotion applications.
It can serve as a platform for collecting quadruped demonstrations for diverse tasks which is enabled by an interpretable high-level control interface.
Furthermore, our controller showcases MoB as a practical tool for out-of-distribution generalization. While our implementation of MoB leverages expert knowledge in the domain of locomotion, in general, the technique of learning multiple methods of achieving goals to facilitate generalization is a promising approach with potential for broad application in sim-to-real reinforcement learning for robotics. [Figure]  Background
Auxiliary Rewards.
Locomotion gaits learned using only the task reward (e.g. velocity tracking) have not been shown to successfully transfer to the real world. It is necessary to train using auxiliary rewards that bias the robot to maintain a particular contact schedule, action smoothness, energy consumption, or foot clearance [1, 2, 3, 5, 6, 7], to compensate for the sim-to-real gap. Such auxiliary rewards can be interpreted as biases for generalization.
For example, foot clearance enables the robot to be robust when the terrain in the real world is more uneven, or the robot’s body sinks more in the real world than in simulation [7]. If an agent fails on a real-world task, it is common practice to manually tune auxiliary rewards to encourage the emergence of successful real-world behavior. However, such tuning is tedious because it requires repeated iterations of training and deployment.
Furthermore, such tuning is task-speciﬁc and must be repeated for new tasks commanded to the agent.
The difﬁculty of designing a single set of auxiliary rewards that promote generalization in diverse set of downstream tasks is illustrated in the top row insets of Figure 1: each shows an instance where a ﬁxed auxiliary reward would lead to failure for one task, despite working well for another.
Learning Locomotion with Parameterized Style.
Closely related recent work on bipeds has explicitly included gait parameters in the task speciﬁcation through parameter-augmented auxiliary reward terms, including tracking rewards for contact patterns [8] and target foot placements [9].
[8] used gait-dependent reward terms in bipedal locomotion to control the timing offset between the two feet and the duration of the swing phase.
The reward structure of [8] inspired ours, but due to its small space of command parameters, did not propose and would not support application to compositional tasks or out-of-distribution environments. To similar effect, other works have imposed parameterized style through libraries of reference gaits.
[10, 11] generated a library of reference trajectories and trained a goal-conditioned policy to imitate them.
[12] demonstrated that a small discrete set of motion styles can be learned simultaneously from a reference trajectory library.
Learning with Diversity Objectives.
Several prior methods aim to automatically learn a collection of high-performing and diverse behaviors.
Quality diversity (QD) methods [13, 14, 15] learn a library of diverse policies by enforcing a novelty objective deﬁned among trajectories.
They typically perform this optimization using evolutionary strategies.
QD has demonstrated beneﬁts including improved optimization performance and reuse of skills for online adaptation [13, 14].
Another line of work uses unsupervised objectives for skill discovery in RL, towards improving optimization [16] or out-of-distribution generalization [17, 18]. Unsupervised diversity approaches hold promise, but they have not yet scaled to real robotic platforms and do not produce an grounded interface of parameters for guiding behavior.
Hierarchical Control Using Gait Parameters.
Several works have learned a high-level policy to accomplish downstream tasks by modulating the gait parameters of a low-level model-based controller.
This approach has been previously applied to energy minimization [19] and visionguided foot placement [20, 21].
These works relied on a model-predictive low-level controller to execute the different gaits in absence of a gait-conditioned learned controller.
The tradeoffs between model-based control and reinforcement learning are well discussed in prior literature [3, 22].
Our work enables revisiting hierarchical approaches with learning at both the high and low levels.
[Figure]
Reward structure: task rewards, augmented auxiliary rewards, and ﬁxed auxiliary rewards
Task Augmented Auxiliary Fixed Auxiliary  Method
To obtain MoB, we train a conditional policy π ( ·| c t , b t ) that achieves tasks speciﬁed by the command ( c t ) in multiple ways that result from different choices of behavior parameters, b t . The question arises of how to deﬁne b t . We could learn behaviors using an unsupervised diversity metric, but these behaviors might not be useful [4] and are not human tunable. To overcome these issues, we leverage human intuition about useful behavior parameters ( b t ) corresponding to gait properties like foot swing motion, body posture, and contact schedule [6, 7, 8, 23].
During training, the agent receives a combination of task rewards (for velocity tracking), ﬁxed auxiliary rewards (to promote sim-to-real transfer and stable motion), and ﬁnally augmented auxiliary rewards (that encourage locomotion in the desired style). During deployment in a novel environment, a human operator can tune behavior of the policy by changing its input b t . 3.1 Task Structure for MoB Task Speciﬁcation.
We consider the task of omnidirectional velocity tracking.
This task is speciﬁed by a 3 -dimensional command vector c t =
[ v cmd x , v cmd y , ω cmd z ] where v cmd x , v cmd y are the desired linear velocities in the body-frame x- and y- axes, and ω cmd z is the desired angular velocity in the yaw axis. Behavior Speciﬁcation. We parameterize the style of task completion by an 8 -dimensional vector of behavior parameters, b t :
[Equation]
θ cmd =
( θ cmd 1 , θ cmd 2 , θ cmd 3 ) are the timing offsets between pairs of feet.
These express gaits including pronking ( θ cmd =
trotting ( θ cmd =
(0 . 5 , 0 , 0) ), bounding, ( θ cmd = (0 , 0 . 5 , 0) ), pacing ( θ cmd = (0 , 0 , 0 . 5) ), as well as their continuous interpolations such as galloping ( θ cmd =
Taken together, the parameters θ cmd can express all two-beat quadrupedal contact patterns; Figure 2 provides a visual illustration.
f cmd is the stepping frequency expressed in Hz .
As an example, commanding f cmd = [-
-]3 Hz will result in each foot making contact three times per second.
h cmd z is the body height command; φ cmd is the body pitch command.
s cmd y is the foot stance width command; h fz cmd is the footswing height command.
Reward function.
All reward terms are listed in Table 1.
Task rewards for body velocity tracking are deﬁned as functions of the command vector c t . Auxiliary rewards are used constrain the quadruped’s motion for various reasons.
“Fixed” auxiliary rewards are independent of behavior parameters ( b t ) and encourage stability and smoothness across all gaits for better sim-to-real transfer. During training, one concern is that the robot might abandon its task or choose an early termination when the task reward is overwhelmed by penalties from the auxiliary objectives.
To resolve this, as in [7], we force the total reward to be a positive linear function of the task reward by computing it as r task exp ( c aux r aux ) where r task is the sum of (positive) task reward terms and r aux is the sum of (negative) auxiliary reward terms (we use c aux = 0 . 02 ).
This way, the agent is always rewarded for progress towards the task, more when auxiliary objectives are satisﬁed and less when they are not. For MoB, we deﬁne augmented auxiliary rewards as functions of the behavior vector b t . We designed these rewards to increase when the realized behavior matches b t , and to not conﬂict with the task reward. This required some careful design of the reward structure.
For example, when implementing stance width as a behavior parameter, a naive approach would be to simply reward a constant desired distance between left and right feet.
However, this penalizes the robot during fast turning tasks requiring relative lateral motion of the feet.
To avoid this, we implement the Raibert
Heuristic, which suggests the necessary kinematic motion of the feet to achieve a particular body velocity and contact schedule [23, 24]. The Raibert Heuristic computes the desired foot position in the ground plane, p fx,y cmdfoot ( s cmd y ) , as an adjustment to the baseline stance width to make it consistent with the desired contact schedule and body velocity. To deﬁne the desired contact schedule, function
C cmdfoot ( θ cmd , t ) computes the desired contact state of each foot from the phase and timing variable, as described in [8], with details given in the appendix.
3.2 Learning Diversiﬁed Locomotion
Task and Behavior Sampling In order to learn graceful online transitions between behaviors, we resample the desired task and behavior within each training episode. To enable the robot to both run and spin fast, we sample task c t = ( v cmd x , v cmd y , ω cmd z ) using the grid adaptive curriculum strategy from [3]. Then, we need to sample a target behavior b t .
First, we sample ( θ cmd 1 , θ cmd 2 , θ cmd 3 ) as one of the symmetric quadrupedal contact patterns (pronking, trotting, bounding, or pacing) which are known as more stable and which we found a sufﬁcient basis for diverse useful gaits. Then, the remaining command parameters ( v cmd y , f cmd , h cmd z , φ cmd , h fz cmd , s cmd y ) are sampled independently and uniformly. Their ranges are given in Table 6. Policy Input.
The input to the policy is a 30 -step history of observations o t − H...
t , commands c t − H... t , behaviors b t − H... t , previous actions a t − H − 1 ...t − 1 , and timing reference variables t t − H...t .
The observation space o t consists of joint positions and velocities q t , ˙ q t (measured by joint encoders) and the gravity vector in the body frame g t (measured by accelerometer).
The timing reference variables t t = [sin(2 πt FR ) , sin(2 πt FL ) , sin(2 πt RR ) , sin(2 πt RL )] are computed from the offset timings of each foot:
[ t FR , t FL , t RR , t RL ] =
[ t + θ cmd+ θ cmd 3 , t + θ cmd+ θ cmd 3 , t + θ cmd 1 , t + θ cmd 2 ] , where t is a counter variable that advances from 0 to 1 during each gait cycle and FR , FL , RR , RL are the four feet.
This form is adapted from [8] to express quadrupedal gaits. Policy Architecture.
Our policy body is an MLP with hidden layer sizes [512 , 256 , 128] and ELU activations.
Besides the above, the policy input also includes estimated domain parameters: the velocity of the robot body and the ground friction, which are predicted from the observation history using supervised learning in the manner of [7].
The estimator module is an MLP with hidden layer sizes [256 , 128] and ELU activations.
We did not analyze the impact of this estimation on performance but found it useful for visualizing deployments.
Action Space.
The action a t consists of position targets for each of the twelve joints.
A zero action corresponds to the nominal joint position, ˆ q .
The position targets are tracked using a proportionalderivative controller with k p = 20 , k d = 0 .
3.3 Design Choices for Sim-to-Real Transfer
Domain Randomization.
For better sim-to-real transfer, we train a policy that is robust to a range of robot’s body mass, motor strength, joint position calibration, the ground friction and restitution, and the orientation and magnitude of gravity.
As we are interested in studying out-of-distribution generalization, we only train on ﬂat ground without any randomization of terrain geometry. This choice also simpliﬁed training.
The randomization ranges of all parameters are in Appendix A.
[Table]
Latency and Actuator Modeling.
Directly invariant properties avoids overly conservative behavior from unnecessary domain randomization.
We perform system identiﬁcation to reduce the sim-to-real gap in the robot dynamics.
Following [22], we train an actuator network to capture the non-ideal relationship between PD error and realized torque.
Separately, we identify a latency of around 20 ms in our system and model this as a constant action delay during simulation.
3.4 Materials
Simulator and Learning Algorithm.
We deﬁne our training environment in the Isaac Gym simulator [6, 25].
We train policies using Proximal Policy Optimization [26]; details in Appendix A.
Hardware.
We deploy our controller in the real world on the Unitree Go1 Edu robot [27].
An onboard Jetson TX2 NX computer runs our trained policy. We implement an interface based on
Lightweight Communications and Marshalling (LCM) [28] to pass sensor data, motor commands, and joystick state between our code and the low-level control SDK provided by Unitree.
For both training and deployment, the control frequency is 50Hz.
Gait-free Baseline.
To understand the impact of MoB on performance, we compare our controller to a baseline velocity-tracking controller (the “gait-free baseline”). The gait-free baseline is trained by the method above, but excludes all augmented auxiliary rewards (Table 1). Therefore, it only learns one solution to the training environment and its actions are independent of behavior parameters b t .
Experimental Results 4.1 Sim-to-Real Transfer and Gait Switching
We deploy the controller learned in simulation in the real world and ﬁrst evaluate its performance on ﬂat ground similar to the training environment.
To start, we demonstrate generating and switching between structured gaits that are well-known in the locomotion community.
Figure 2 shows torques and contact states during transition between trotting, pronking, bounding, and pacing while alternating f cmd between 2 Hz and 4 Hz .
We ﬁnd that all gait parameters are consistently tracked after sim-to-real transfer.
Videos (i)-(iv) on the project website visualize the different gaits obtained by modulating each parameter in b t individually.
4.2 Leveraging MoB for Generalization
Tuning for New Tasks.
After training using a generic locomotion objective, one might wish to tune a controller’s behavior to optimize a new metric in the original environment. MoB facilitates this if some subset of learned behaviors outperform the gait-free policy by the new task metric.
[Table]
Energy efﬁciency (simulated) : We consider the task of minimizing the mechanical power consumption ( J / s ) measured by summing the product of joint velocity and torque at each of the 12 motors: (cid:80) i max( τ i ˙
q i , 0) .
As shown in Figure 3, several choices of the contact schedule θ cmd , height h cmd z , and frequency f cmd outperform the gait-free policy in this unseen metric, consuming less energy across all speeds.
Therefore, tuning the behavior parameters can facilitate tuning performance on a new objective (energy efﬁciency) in the original training environment.
Payload manipulation : We experiment with another task where the robot is required to transport a ball from one place to another, then bend its body so the ball is deposited on the ground. A gaitfree policy couldn’t do this; while many possible body posture proﬁles are valid to solve the training environment, the gait-free policy will simply converge to one at random.
In a real-world experiment, we demonstrate how MoB-enabled body posture control can be repurposed for teleoperated payload manipulation. The operator pilots the robot to the delivery location with the body level, then pitches the body backward, modulating φ cmd , to dump the payload (Figure 1, bottom row, second from left). Tuning for New Environments.
In the previous section, we showed that MoB can support repurposement to novel tasks in the training environment.
In the real world, there is also always a long tail of environments that are not modeled in training.
We demonstrate the behavior of our controller trained only on ﬂat ground in the presence of challenging non-ﬂat terrains and disturbances such as curbs, bushes, hanging obstacles, and shoves.
Successes suggest that the space of policies learned by MoB contains some behaviors that transfer better than the baseline to unseen environments. Platform terrain (simulated) : We evaluate the performance of our robot in traversing randomly positioned platforms with height up to 16 cm (Table 4).
We report two metrics: mean reward and mean survival time as a fraction of the maximum episode length ( 10 s ). For each metric, we ﬁnd that by modulating the contact schedule θ cmd or footswing height h fz cmd , we can outperform the gait-free policy.
(Figure 4, Appendix Figure 9).
Therefore, it is possible to improve performance in an out-of-distribution terrain by modulating the parameters of the MoB policy.
Climbing over curbs : Previous works demonstrating blind obstacle traversal on a quadruped [1, 2] learned a foot-trapping reﬂex where the robot ﬁrst trips, then raises its foot over the obstacle. In contrast, with the help of a human pilot, our gait-conditioned policy with high footswing command enables fast and smooth obstacle traversal without tripping, despite training on a simpler ﬂat-ground terrain.
In the real world, we demonstrate that modulating footswing height h fz cmd enables our controller to climb smoothly across stairs and curbs (Figure 1, bottom left).
Hacking through thick bushes : Extremely thick bushes pose a methodological challenge for state-ofthe-art perceptive locomotion controllers.
They are hard to simulate, because they comply against the whole body of the robot, and they are hard to sense as they are indistinguishable from solid obstacles to depth sensors. Therefore, prior works would either attempt to climb over bushes as obstacles or fall back on a robust proprioceptive controller that is unaware of the semantic context. In contrast, by modulating the gait parameters for gait frequency f cmd and footswing height h fz cmd in situ , a human operator can guide our system quickly through challenging brush.
Navigating conﬁned spaces : Consider the scenario where the robot needs to go under a bar.
The gait-free baseline cannot accomplish this; in the absence of such constraints during training,
it will [Figure] converge to a ﬁxed body height proﬁle.
Our system with MoB can navigate conﬁned spaces through modulation of the body height h cmd z .
In a real-world example, the robot was able to crawl under a 22 cm bar; the robot body thickness is 13 cm , leaving 9 cm of clearance beneath the robot.
Anticipative bracing against shoves : Widening the stance can make a quadruped more robust to hard shoves, but absent a perception module to anticipate a human kick, the robot would always need to walk in a widened stance to be prepared for a shove. This interferes with performance in other tasks like running efﬁciently, so learned locomotion controllers without MoB often provide incentive to keep the feet nominally below the hips [2, 7]. With MoB, widening the stance width s cmd y in the pilot’s anticipation of a shove enables the robot to remain upright.
Exploring the Speed and Range of Behavior Adaptation. If we want to use our low-level controller for high-level tasks, one thing we might desire is to switch between gaits even at high speeds, which might be useful for applications such as parkour. Another is to transition between diverse gaits with precise timing for a synchronous task like dancing. Agile forward leap : As a demonstration of gait transitions at high speed, we modulate contact schedule, velocity, and gait frequency at to encode an agile forward leap (Figure 3).
The robot ﬁrst accelerates to a target speed of 3 m / s at a trot while increasing its step frequency from 2 Hz to 4 Hz , then switches to pronking at 2 Hz for one second, then decelerates to a standstill while trotting.
During the leap phase, the distance from the location of the robot’s front feet at takeoff to the location of the hind feet upon landing is 60 cm .
Choreographed dance : To demonstrate precisely timed transition between diverse gaits, we program a sequence of gait parameters to generate a dance routine synchronized to a jazz song with a tempo of 90 bpm.
At this tempo, combinations of phases 0 , 0 . 25 , and 0 .
5 with frequencies of 1 .
Hz and 3 Hz yield eighth, quarter, half, and full beat gaps between consecutive footsteps.
We also modulate body height and velocity in time with the music.
An assistant script procedurally generates gait parameters, fed into the controller in open-loop fashion.
Discussion and Limitations
Our experiments show that the beneﬁts of adding MoB can come at a cost to in-distribution task performance, speciﬁcally limiting the robot’s ﬂat-ground sprinting performance (Table 5, appendix).
Heat maps reveal that our behavior parameterization is restrictive for combinations of high linear and angular velocity.
To quantify and control the tradeoff between task performance and reward shaping is an interesting future direction, for which some prior methods have been proposed [29].
MoB confers a single learned policy a structured and controllable space of diverse locomotion behaviors for each state and task in the training distribution.
This yields a set of ‘knobs’ to tune the performance of motor skills in unseen test environments.
The system currently requires a human pilot to manually tune its behavior.
In the future, the autonomy of our system could be extended by automating behavior selection using imitation from real-world human demonstrations, or by using a hierarchical learning approach to automatically self-tune the controller during deployment.
Acknowledgments
We thank the members of the Improbable AI lab for the helpful discussions and feedback on the paper.
We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources.
This research was supported by the DARPA Machine Common Sense
Program, the MIT-IBM Watson AI Lab, and the National Science Foundation under Cooperative
Agreement PHY-2019786 (The NSF AI Institute for Artiﬁcial Intelligence and Fundamental Interactions, http://iaiﬁ.org/).
This research was also sponsored by the United States Air Force Research
Laboratory and the United States Air Force Artiﬁcial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the United States Air Force or the U
S. Government.
The U
S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
Author Contributions
• Gabriel B. Margolis concieved, designed, and implemented the controller, ran all experiments, and played the primary role in paper writing.
• Pulkit Agrawal advised the project and contributed to its conceptual development, experimental design, positioning, and writing.