Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots
Abstract:
Legged locomotion is commonly studied and expressed as a discreteset of gait patterns, like walk, trot, gallop, which are usually treated as givenand pre-programmed in legged robots for efﬁcient locomotion at different speeds.
However, ﬁxing a set of pre-programmed gaits limits the generality of locomotion.
Recent animal motor studies show that these conventional gaits are only prevalentin ideal ﬂat terrain conditions while real-world locomotion is unstructured and morelike bouts of intermittent steps.
What principles could lead to both structured andunstructured patterns across mammals and how to synthesize them in robots?
In thiswork, we take an analysis-by-synthesis approach and learn to move by minimizingmechanical energy.
We demonstrate that learning to minimize energy consumptionplays a key role in the emergence of natural locomotion gaits at different speeds inreal quadruped robots.
The emergent gaits are structured in ideal terrains and looksimilar to that of horses and sheep.
The same approach leads to unstructured gaitsin rough terrains which is consistent with the ﬁndings in animal motor control.
Wevalidate our hypothesis in both simulation and real hardware across natural terrains.
Videos at https://sites.google.com/view/energy loco/ .
Introduction
A common approach to understanding mammalian locomotion is to study it in terms of a set of discretegaits.
Seminal works include Muybridge [1]’s motion video of a galloping horse and Hildebrand[2]’s study of different gaits observed at different speeds.
This has inspired research on replicatingthese gaits in robots by pre-programming them to achieve effective locomotion.
This approach ofpre-programming ﬁxed gait patterns has lead to immense progress in robotic locomotion [3, 4, 5, 6].
However, pre-programmed gaits limit the ability of legged systems to perform general locomotion indiverse terrains and at different speeds.
Studies of motor control in both animals and human infants[7]in the last few decades have shown that general locomotion in natural settings is irregular and doesnot ﬁt predeﬁned ﬁxed gaits.
It is better deﬁned in terms of bouts of steps than periodic cycles [8, 9].
That being said, we do see consistent regular patterns in ideal conditions when an animal is movingin a straight line over ﬂat terrain at different speeds, e.g., walk, trot, canter, gallop, etc.
[2, 10, 11, 12]and
these patterns are shared across species of diverse animals [13].
How do we reconcile theseseemingly contradictory arguments about unstructured locomotion patterns on complex terrains withthat of regular patterns seen in ﬂat terrains under the same rubric?
One way to make sense of these differences is to see them from the lens of minimizing energyconsumption.
Locomotion consumes a signiﬁcant fraction of an animal’s metabolic energy budgetwhich suggests that there would be evolutionary selection pressure for locomotion strategies thatminimize energy consumption.
Indeed, this has been a focus of several studies in biomechanics andenergetics.
Hoyt and Taylor [12] used energy minimization to explain why animals switch fromone gait to another as their speed increases.
Similar studies explain gaits across different animalsincluding humans [14, 15].
However, most of the biomechanics studies are limited to the straight-linemotion of animals in ideal ﬂat terrains.
Furthermore, these studies take gaits as given and offer anexplanation for them from an energy perspective without any statement about their synthesis.
In this work, we take an analysis-by-synthesis approach to show how energy minimization leads tothe emergence of structured locomotion gait patterns in ﬂat terrains as well as unstructured gaits incomplex terrains.
We use energetics to design an end-to-end learning framework and display theresulting gait patterns in a real quadruped robot.
We employ model-free reinforcement learning as theoptimizer to learn controllers to make the quadruped move forward while simultaneously minimizingthe bio-energetics constraints of minimizing work [16].
We train our policies on simple fractalterrains with varying frequency of terrain heights instead of perfectly ﬂat terrain.
This mimics thereal world more closely and achieves efﬁcient and robust gaits without the need for artiﬁcial rewardsfor foot clearance or periodic external push during training.
At low frequency, it resembles the simpleﬂat terrain settings and at high frequency, it simulates complex terrains.
In addition, we present alearning pipeline that enables smooth gait transition given changing target speeds by combining RLand knowledge distillation from trained expert gait policies.
We then transfer the policies onto thereal robot by performing online adaptation [17].
We empirically demonstrate that minimizing energy consumption plays a key role in the emergence ofnatural gait patterns in a quadruped legged robot.
Our quadruped robot is closest to sheep and horse interms of physicality as determined by the Froude number [18] which is a scalar metric characterizinggaits of quadrupeds (discussed more in Section 4.2).
Therefore, energetics-based training leads tosimilar three gaits in our robot as found in horses and sheep, namely, walk, trot and gallop as thespeed is gradually increased in ﬂat terrains Figure 1.
Interestingly, we show that the gait selection isinherently linked with the speed of the robot.
For instance, trotting is only energy efﬁcient around themedium speed of 0.9m/s.
If we increase or decrease the speed of the robot, it takes more energy fortrot than to gallop (similar to bouncing gait) or walk as shown in Figure 2.
Finally, when the samepipeline is run in uneven complex terrains, it leads to unstructured gaits as consistent with animallocomotion in usual diverse conditions [9, 8].
Prior works have used energy penalty to obtain energy-efﬁcient gaits [19, 20] but have not shownthe correspondence to different gaits at different speeds.
Furthermore, out of these works, the oneswhich deploy these gaits on a real robot either use a predeﬁned hand-coded gait library, use referencemotions [19], or only learn high-level contact sequences for a predeﬁned swing leg motion [21].
These approaches have some fundamental limitations in contrast to what we propose.
Firstly, theuse of predeﬁned foot motions never optimize for efﬁcient swing leg motions, and using referencetrajectories gives gaits that look realistic, but are not necessarily energy-efﬁcient for the speciﬁc robotwe have.
Second, learning controllers end to end without the use of any priors (reference motions orpredeﬁned motions) allows for the possibility of learning behaviors that don’t follow any predeﬁnedgaits.
We show the robustness of this complex terrain policy by successfully deploying it in complexterrains in the real world.
The main contributions of this paper include:
• Show that minimizing energy consumption plays a key role in the emergence of natural locomotion patterns in both ﬂat as well as complex terrains at different speeds without relying ondemonstrations or predeﬁned motion heuristics.•
Show that the emergent gaits at different target speeds correspond to conventional animals in thesimilar Froude number range (sheep/horse) without any sort of pre-programming.• Present a distillation-based learning pipeline to obtain velocity-conditioned policy that displayssmooth gait transition as the target speed is changed.• Demonstrate the emergent behaviors, robustness analysis, and gait patterns in simulation as wellas a real-world budget quadruped robot.
Method
2.1 Claim: Energy Minimization leads to Emergence of Natural Gaits
We would like to validate that indeed certaingaits will emerge at certain speeds if we minimize the energy consumption.
To do so, onthe right, we plot the energy consumption vsspeed of the A1 robot in simulation for walk,trot and bounce (modiﬁcation of gallop/canter)gaits.
We use [6] to achieve the different gaitsby hard-coding the contact sequences for walk,trot and bounce.
This plot is qualitatively similar to the plot in [12], which shows the energyconsumption of different horse gaits across different speeds.
We additionally plot the energyconsumption of the three policies we get at different speeds (0.375 m/s, 0.9 m/s, 1.5 m/s).
Thisplot shows two things.
First, it is observed thatcertain gaits are only efﬁcient at certain speedsand using them for other speeds is suboptimal.
For instance, walk is optimal at low speeds withtransition to trot and bounce.
Secondly, we observe that our learning framework converges tothe optimal gait at each of the speeds, and beatsthe efﬁciency of MPC gaits from [6], which doesnot optimize the swing foot trajectory for energy.
Details of MPC method are in the supplementary.
2.2 Learning Locomotion Policy
We learn the locomotion policy π which takes as input the current state x t ∈ R 30 , previous actiona t − 1 ∈ R 12 to predict the next action a t (Equation 1).
The predicted action a t is the desired jointposition for the 12 robot joints which is converted to torque using a PD controller.
[Equation]
116 We implement π as MLPs (details in Section 3) and train the base policy π end to end using model-freereinforcement learning.
At time step t , π takes the current state x t , previous action a t − 1 to predict anaction a t .
RL maximizes the following expected return of the policy π :
[Equation]
where τ = { ( x 0 , a 0 , r 0 ) , ( x 1 , a 1 , r 1 ) ... } is the trajectory of the agent when executing policy π , andp ( τ | π ) represents the likelihood of the trajectory under π .
2.3 Stable Gait through Natural Constraints
Instead of adding artiﬁcial simulation noise, we train our agent under the following natural constraints.
First, the reward function is motivated from bio-energetic constraints of minimizing work [16].
Wefound these reward functions to be critical for learning realistic gaits in simulation.
Second, we trainour policies on uneven terrain (Figure 1) as a substitute for additional rewards used by [22] for footclearance and robustness to external push.
We see that these constraints are enough to achieve all thegaits and results we demonstrate in this paper.
2.4 Energy Consumption-Based Reward
Let’s denote the linear velocity as v and the angular velocity as ω , both in the robot’s base frame.
Weadditionally deﬁne joint torques as τ and joint velocities as ˙ q .
We deﬁne our reward as sum of thefollowing three terms:
[Equation]
where,
[Equation]
) 134 r forward rewards the agent for walking straight at the speciﬁed speed, r energy penalizes energyconsumption and r alive is the survival bonus.
We simple rules to set the hyper-parameters.
At a given target linear speed v target x , the survival bonusc is set to 20 ∗ v target x .
We set α 1 = 0 . 04 , and α 2 = 20 across all settings.
Notice that the actual energy consumption on the robot depending on the low-level hardware designis not directly measurable.
We estimate the unit energy consumption per time step by summing theinstantaneous power of the 12 motors by multiplying the torque and the joint velocity at each motor.
Also notice that this reward is much more concise than those used in prior works.
We discuss the roleof minimizing energy consumption in Section 4.4.
2.5 Sim to Real Transfer
For simulation to real transfer, we use RMA [17] which consists of the an adaptation module on topof the base policy.
During deployment, the adaptation module uses the state history to estimate thevector of extrinsics (which contains environment information) online.
This adaptation module istrained in simulation using supervised learning after the base policy is learned using RL with the trueextrinsics parameters.
The environment perturbations we train on is listed in Table 1.
Experimental Setup and Training Details
Hardware and Simulation: We use Unitree’s A1 robot as our hardware platform [23] and use its
A1 URDF [23] to simulate the A1 robot in RaiSim simulator [24].
The full details of terrain andenvironment setup are provided in supplementary due to space constraints.
State-Action Space:
The state is 30 dimensional containing the joint positions ( 12 values), jointvelocities ( 12 values), roll and pitch of the torso and binary foot contact indicators ( 4 values).
Theaction space is 12 dimensional corresponding to the target joint position for the 12 robot joints.
Thepredicted target joint angles a = ˆ q ∈ R 12 is converted to torques τ using a PD controller with targetjoint velocities set to 0.
[Table]
Environmental Variations: All environmental variations with their ranges are listed in Table 1.
Policies withsimple terrain and mild environment variations achievewalking, trotting and bouncing gaits.
To deploy our robotin the complex terrain, we use aggressive environmentrandomization with sharper fractal terrains which gives usthe unstructured walking terrain.
Policy Learning:
The policy is a multi-layer perceptronwith 3 layers which takes in the current state x t ∈ R 30 ,previous action a t − 1 ∈ R 12 and outputs 12-dim targetjoint angles.
The hidden layers have 128 units.
We trainthe policy and the environment encoder network using
PPO [25].
The training runs for 15 , 000 iterations with a batch size of 100 , 000 split into 4 minibatches and learning rate =
5e − 4 .
We train for a total of 1.5 billion samples which takes roughly 24hours on a desktop machine with 1 GPU.
Results and Analysis
4.1 Emergence of Locomotion Patterns in Complex Terrain
We analyze the performance of the walking policy on complex terrain with aggressive perturbationsin the real world (videos on website).
We show two uneven terrain deployments of our method in
Figure 3.
During the deployment of the robot in the rocky terrain, we see that the robot roughlyfollows the walking gait which repeatedly gets obstructed by the rocks and has to step over rocks ofdifferent heights, prematurely terminating the swing motion of the leg.
Consequently, the gait weobserve is an unstructured walking gait.
We further analyse this in another deployment on unstableplanks which move as the robot tries to cross it (see video).
The foot contact plot for this setup isnot periodic and changes depending on the complexity of the terrain the policy needs to react to.
Inthe video, this can been seen when the front right foot of the robot steps on an unstable plank whichmoves as a result, but the robot maintains stability and successfully crosses it.
4.2 Emergence of Walking, Trotting and Bouncing
[Table]
Simulation Gaits: We train 3separate policies for 3 targetspeeds - 0.375 m/s (low), 0.9 m/s(median) and 1.5 m/s (high) insimulation.
We use the samehyper-parameters across all target speeds.
We observe thatthe walking gait emerges at target speed 0.375 m/s, trottingemerges at 0.9 m/s and hoppingemerges at 1.5 m/s in simulation.
Note that we use RMA [17] (Section 2.5) to transfer our policies tothe real world without any ﬁne-tuning.
We observe the same foot contact sequence in the real worldas in simulation.
In Table 2, we compare the performance of our policies with MPC [6] and observethat our walking gait can achieve accurate speed tracking while being 50% more energy efﬁcient thanthe MPC baseline.
Real-World Behaviors: At the low speed (0.375 m/s), the robot demonstrates a quarter-off walkinggait with the following foot contact sequence: Right-Front (RF), Left-Rear (LR), Left-Front (LF) and
[Figure] [Figure]
Right-Rear (RR).
At the median speed (0.9 m/s), two beat trot gait emerges where diagonal legs (RF& LR or LF & RR) are synchronized and hit the ground at the same time.
At the high speed (1.5m/s), the policy converges to bouncing with approximately half the gait cycle as ﬂight phase.
Keyframes of the 3 gaits are shown in Supplementary and the foot contact plots in Figure 4.
We use thefoot contact sensor reading from the A1 robot to generate these plots.
We also measured the actual speed of the robot in the real world by taking the average of 3 trialsper target speed.
In each trial, we log the total distance divided by total time taken to traverse.
Theaverage real-world speeds are 0.396 m/s at target speed 0.375 m/s, 0.914 m/s at target speed 0.9 m/s,and 1.714 m/s at target speed 1.5 m/s.
The average real-world speeds closely match the target speedsin all settings.
The standard deviation of speeds across 3 trials is negligible.
We test the robustness of the walking policies by adding a 1 Kg payload on the robot by strappingtwo bottles with 500 ml water.
In Figure 5, we show 4 key frames of trotting gait.
Note that the 1 kgpayload is outside the range of training perturbations (Table 1), but our policies with the Sim-to-Real
Adaptation in Section 2.5 learn to compensate the increase in weight by implicitly predicting changesin dynamics via latent extrinsics.
We didn’t test bouncing with payload, because the extra 1 kgprevents the human operator from swiftly preempting the robot at high speed by lifting.
[Table] [Table]
Analysis:
In
Biomechanics, Froude number Fis a scalar metric characterizinggaits of quadrupeds and bipeds [18].
Concretely, F = v 2 /gh , where v isthe linear speed, g is the gravitationalacceleration, and h is the height of thehip joint from the ground.
Existingresearch shows that animals with asimilar morphology but with differentsizes tend to use the same gait when walking with equal Froude numbers [26, 10].
We calculate the
[Figure]
Froude numbers of our robot in the real world for the low, median and high speeds, and comparethem with the Froude numbers of quadrupeds in nature – dogs, horses and sheep [12, 10].
Wecompare the Froude number of animals at the gait transition boundaries and observe that our robot isclosest to the sheep in Table 3.
4.3 Gait Transition via a Velocity-Conditioned Policy
We have shown walking, trotting and bouncing emerged at three different target speeds.
To demonstrate foot patterns at a continuous range of velocities, we propose a learning scheme (Figure 6) for avelocity-conditioned policy that enables smooth gait transition when the command velocity changes.
Naive Multi-Task Training Fails: We ﬁrst tried the naive approach to train the velocityconditioned policy in a multi-task fashion by randomly sampling desired velocities and using thecorresponding velocity-conditioned reward (Section 2.4).
However, this did not work and the resultingemerged gaits collapse to only two modes: walking and trotting, but trotting at high speeds is notenergy-efﬁcient (Figure 7).
We believe the reason for failure is difﬁculty in optimization as the robotis now tasked not only to learn to move forward but also do it by learning different gaits which causesit to collapse.
Learning via Distillation and RL: We sidestep this issue via stage-wise distillation approach.
Weﬁrst train our ﬁxed-velocity policies as described in the paper which lead to walking, trotting and [Figure] bouncing (galloping) gaits.
Wethen treat these policies as experts and collect demonstrationdata from them to self-superviseand bootstrap the initial trainingphase of the velocity-conditionedpolicy.
These three policies serveas experts at three velocities atlow (0.375 m/s), median (0.9m/s) and high (1.5 m/s), represented as three-dimensional onehot vectors, which are fed intothe velocity-conditioned policyas input.
Expert supervisionsguide the velocity-conditionedpolicy to follow the energyefﬁcient gaits at the three expert velocities.
To learn motor skills and smooth gait transition atintermediate velocities in the continuous range of 0.375 m/s to 1.5 m/s, the learning relies on thevelocity-conditioned RL rewards only without expert supervisions. To represent a randomly sampled target velocity to feed into the policy, we use an interpolated vector based on the closest twoexperts’ one-hot velocity representations (e.g. 1.2 m/s is represented as [0, 0.5, 0.5] and 0.5 m/s as[0.238, 0.762, 0]).
We linearly anneal the L2 supervision loss from the expert supervision (for thethree velocity modes) as the training progresses. Towards the end, the learning relies only on thevelocity-conditioned RL loss for the entire velocity range.
Details are in the supplementary.
Real-World Smooth Gait Transition: We test our velocity-conditioned policy in the wild and wantto highlight the smooth gait transition happening at different speeds.
Video is in the supplementary.
4.4 Ablation Studies
The two key components we use during policy training to get reliable walking gaits is an energybased reward function, and a fractal terrain.
We posit that these ingredients are minimal, and wedemonstrate this by ablating each of them below.
[Figure]
Minimizing Energy Consumption Also Minimizes Artiﬁcial Penalties: To demonstrate the sufﬁciency of minimizing energy consumption, we plot the decreasing trend of the artiﬁcial penaltiesduring training in Figure 7, even though these artiﬁcial penalties are not included in our rewardfunction.
These include Torque penalty, delta torque penalty, foot slip penalty, joint speed penalty, andthe action regularization [21, 27, 22, 28, 29].
A high positive correlation between energy consumptionand artiﬁcial penalties can be observed.
All these artiﬁcial penalties are minimized in our trainingframework as a byproduct of minimizing energy consumption.
During training, the robot ﬁrst learnsto stand still with a minimal energy consumption (around epoch 500), and then starts moving forwardwith a slightly higher energy consumption.
Minimizing Energy Consumption:
In Figure 8, we show the the resulting gaits of policies trainedwithout the unit energy consumption term in the reward function.
All policies show high-frequencytapping behaviors that are not natural and inefﬁcient.
The high-frequency joint movements can burnthe joint motors in the robot and are not transferable to the real hardware.
Fractal Terrain during Training: We train the policies with energy minimization (Section 2.4)but on a ﬂat terrain without fractal perturbations (Section 2.3).
All training trials converge to unnaturaland unstable gaits.
We found that adding fractal terrain also facilitates a larger foot clearance androbustness which improves real-world hardware deployment.
We show the key frames of 2 exampleunnatural gaits in the supplementary.
Related Works
Model-Based Optimization for Gaits: Model-based controllers to generate gaits [6] optimize forthe stance legs for a given foot contact sequence.
They use predeﬁned swing leg motion and showstable gait generation on a real robot system.
One class of method is contact-implicit optimization [30,31, 32], optimizing contact forces along with sequences.
However, these gaits do not emerge as anenergy-minimizing solution to moving at different speeds.
[33] analyzes biped gaits by optimizingenergy consumption, but uses a minimal mechanical model assuming point-mass body and masslesslegs.
In contrast, our swing leg motion, stance leg force and contact sequence, all emerge fromminimizing energy consumption of moving of a real robot.
We take a functional approach to gaitswhere we achieve them not as predeﬁned movements, but as a consequence of energy minimization.
Learning for Legged Locomotion and Gaits: Data-driven learning for legged locomotion hasshown robust controllers for quadrupedal robots [22, 34, 35, 36, 37].
Of these, [34, 22] focus ona controller on complex terrains but do not synthesize and analyze leg patterns of the robot acrossdiverse gaits at different target speeds.
[35] demonstrates a learning method to select gait controllers,but the low-level gait controllers are predeﬁned primitives.
Concurrent work [21] shows the learnedgaits and gait transition, but their method relies on many human priors (e.g. predeﬁned cyclic motionpriors), shows trotting at high speed with short periods of ﬂight phase, and requires powerful on-boardcomputation power (Apple M1 chip) to solve online MPC optimization.
In graphics, [38] shows lowenergy also plays a role in generating natural locomotion animations.
Conclusion
This work demonstrates that energy consumption plays a key role in the emergence of naturallocomotion patterns in animals by following an analysis-by-synthesis approach of showing the resultin real quadruped robots.
The allows generation of the straight line lab gaits, as well as unstructuredcomplex terrain gaits in animals.
We perform thorough analysis of gait patterns, show their robustnessunder disturbances, and propose a learning pipeline for smooth gait transition .
Interestingly, the gaitsobtained by our robot are most similar to those in horses and sheep – the animals which are close to itin terms of the Froude number metric space.