Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract:
In this work, we present and study a training set-up that achieves fastpolicy generation for real-world robotic tasks by using massive parallelism on asingle workstation GPU.
We analyze and discuss the impact of different trainingalgorithm components in the massively parallel regime on the ﬁnal policy performance and training times.
In addition, we present a novel game-inspired curriculum that is well suited for training with thousands of simulated robots in parallel.
We evaluate the approach by training the quadrupedal robot ANYmal to walk onchallenging terrain.
The parallel approach allows training policies for ﬂat terrainin under four minutes, and in twenty minutes for uneven terrain.
This representsa speedup of multiple orders of magnitude compared to previous work.
Finally,we transfer the policies to the real robot to validate the approach.
We open-sourceour training code to help accelerate further research in the ﬁeld of learned leggedlocomotion.
[Figure]
Introduction
Deep reinforcement learning (DRL) is proving to be a powerful tool for robotics.
Tasks such aslegged locomotion [1], manipulation [2], and navigation [3], have been solved using these newtools, and research continues to keep adding more and more challenging tasks to the list.
Theamount of data required to train a policy increases with the task complexity.
For this reason, mostwork focuses on training in simulation before transferring to real robots.
We have reached a pointwhere multiple days or even weeks are needed to fully train an agent with current simulators.
Forexample, OpenAI’s block reorientation task was trained for up to 14 days and their Rubik’s cubesolving policy took several months to train [4].
The problem is exacerbated by the fact that deepreinforcement learning requires hyper-parameter tuning to obtain a suitable solution which requiressequentially rerunning time-consuming training.
Reducing training times using massively parallel Submitted to the 5th Conference approaches such as presented here can therefore help improve the quality and time-to-deploymentof DRL policies, as a training setup can be iterated on more often in the same time frame.
In this paper, we examine the effects of massive parallelism for on-policy DRL algorithms andpresent considerations in how the standard RL formulation and the most commonly used hyperparameters should be adapted to learn efﬁciently in the highly parallel regime.
Additionally, wepresent a novel game-inspired curriculum which automatically adapts the task difﬁculty to the performance of the policy.
The proposed curriculum architecture is straightforward to implement, doesnot require tuning, and is well suited for the massively parallel regime.
Common robotic simulatorssuch as Mujoco [5], Bullet [6], or Raisim [7] feature efﬁcient multi-body dynamics implementations.
However, they have been developed to run on CPUs with only a reduced amount of parallelism.
Inthis work, we use NVIDIA’s Isaac Gym simulation environment [8], which runs both the simulationand training on the GPU and is capable of simulating thousands of robots in parallel.
The massively parallel training regime has been explored before [4, 9] in the context of distributedsystems with a network of thousands of CPUs each running a separate instance of the simulation.
The parallelization was achieved by averaging the gradients between the different workers withoutreducing the number of samples provided by each agent.
This results in large batch sizes of millionsof samples for each policy update which improves the learning dynamics, but does not optimizethe overall training time.
In parallel, recent works have aimed to increase the simulation throughput and reduce training times of standard DRL benchmark tasks.
A framework combining parallelsimulation with multi-GPU training [10] was proposed to achieve fast training using hundreds ofparallel agents.
In the context of visual navigation, large batch simulation has been used to increasethe training throughput [11].
Furthermore, GPU accelerated physics simulation has been shown tosigniﬁcantly improve the training time of the Humanoid running task [12].
A differentiable simulator running on Google’s TPUs has also been shown to greatly accelerate the training of multipletasks [13].
We build upon [10, 12] by pushing the parallelization further, optimizing the trainingalgorithm, and applying the approach to a challenging real-world robotics task.
Perceptive and dynamic locomotion for legged robots in unstructured environments is a demandingtask that, until recently, had only been partially demonstrated with complex model-based approaches[14, 15].
Learning-based approaches are emerging as a promising alternative.
For quadrupeds, DRLhas been used to train blind policies robust to highly uneven ground [16] (12 hours of training).
Perceptive locomotion over challenging terrain has been achieved by combining learning with optimalcontrol techniques [17, 18] (82 and 88 hours of training) and recently, a fully learned approach hasshown great robustness in this setting [19] (120 hours of training).
Similarly, bipedal robots havealso been trained to walk blindly on stairs [20] (training time not reported).
With our approach wecan train a perceptive policy in under 20 minutes on a single GPU, with the complexity of simto-real transfer to the hardware, which increases the performance and robustness requirements andprovides clear validation of the overall approach.
Training such behaviors in minutes opens up newexciting possibilities ranging from automatic tuning to customized training using scans of particularenvironments.
Massively Parallel Reinforcement Learning
Current (on-policy) reinforcement learning algorithms are divided into two parts: data collection andpolicy update.
The policy update, which corresponds to back-propagation for neural networks, iseasily performed in parallel on the GPU.
Parallelizing data collection is not as straightforward.
Eachstep consists of policy inference, simulation, reward, and observation calculation.
Current popularpipelines have the simulation and reward/observation calculation computed on the CPU, making the
GPU unsuitable for policy inference because of communication bottle-necks.
Data transfer over
PCIe is known to be the weakest link of GPU acceleration, and can be as much as 50 times slowerthan the GPU processing time alone [21].
Furthermore, with CPU data collection, a large amountof data must be sent to the GPU for each policy update, slowing down the overall process.
Limitedparallelization can be achieved by using multiple CPU cores and spawning many processes, eachrunning the simulation for one agent.
However, the number of agents is quickly limited by the number of cores and other issues such as memory usage.
We explore the potential of massive parallelismwith Isaac Gym’s end-to-end data collection and policy updates on the GPU, signiﬁcantly reducingdata copying and improving simulation throughput.
2.1 Simulation Throughput
The main factor affecting the total simulation throughput is the number of robots simulated in parallel.
Modern GPUs can handle tens of thousands of parallel instructions.
Similarly, IsaacGym’s
PhysX engine can process thousands of robots in a single simulation and all other computations ofour pipeline are vectorized to scale favorably with the number of robots.
Using a single simulationwith thousands of robots presents some new challenges.
For example, a single common terrain meshmust be used, and it cannot be easily changed at each reset.
We circumvent this problem by creatingthe whole mesh with all terrain types and levels tiled side by side.
We change the terrain level ofthe robots by physically moving them on the mesh.
In supplementary material, we show the computational time of different parts of the pipeline, examine how these times scale with the number ofrobots, and provide other techniques to optimize the simulation throughput.
2.2 DRL Algorithm
We build upon a custom implementation of the Proximal Policy Optimization (PPO) algorithm
Our implementation is designed to perform every operation and store all the data on the GPU.
In order to efﬁciently learn from thousands of robots in parallel, we perform some essential modiﬁcationsto the algorithm and change some of the commonly used hyper-parameter values.
2.2.1 Hyper-Parameters Modiﬁcation
In an on-policy algorithm such as PPO, a ﬁxed policy collects a selected amount of data before doing the next policy update.
This batch size, B , is a crucial hyper-parameter for successful learning.
With too little data, the gradients will be too noisy, and the algorithm will not learn effectively.
Withtoo much data, the samples become repetitive, and the algorithm cannot extract more informationfrom them.
These samples represent wasted simulation time and slow down the overall training.
Wehave B = n robots n steps , where n steps is the number of steps each robot takes per policy updateand n robots the number of robots simulated in parallel.
Since we increase n robots by a few ordersof magnitude, we must choose a small n steps to keep B reasonable and hence optimize trainingtimes, which is a setting that has not been extensively explored for on-policy reinforcement learningalgorithms.
It turns out that we can not choose n steps to be arbitrarily low.
The algorithm requirestrajectories with coherent temporal information to learn effectively.
Even though, in theory, information of single steps could be used, we ﬁnd that the algorithm fails to converge to the optimal solutionbelow a certain threshold.
This can be explained by the fact that we use Generalized Advantage Estimation (GAE) [23], which requires rewards from multiple time steps to be effective.
For ourtask, we ﬁnd that the algorithm struggles when we provide fewer than 25 consecutive steps, corresponding to 0 .
5 s of simulated time.
It is important to distinguish n steps from the maximum episodelength leading to a time-out and a reset, which we deﬁne as 20 s .
The environments are reset whenthey reach this maximum length and not after each iteration, meaning that a single episode can covermany policy updates.
This limits the total number of robots training in parallel, and consequently,prohibits us from using the full computational capabilities of the GPU.
The mini-batch size represents the size of the chunks in which the batch size is split to performback-propagation.
We ﬁnd that having mini-batch sizes much larger than what is usually consideredbest practice [24] is beneﬁcial for our massively parallel use case.
We use mini-batches of tens ofthousands of samples and observe that it stabilizes the learning process without increasing the totaltraining time.
2.2.2 Reset Handling
During training, the robots must be reset whenever they fall, and also after some time to keep themexploring new trajectories and terrains.
The PPO algorithm includes a critic predicting an inﬁnitehorizon sum of future discounted rewards.
Resets break this inﬁnite horizon assumption and canlead to inferior critic performance if not handled carefully.
Resets based on failure or reaching agoal are not a problem because the critic can predict them.
However, a reset based on a time out cannot be predicted (we do not provide episode time in the observations).
The solution is to distinguishthe two termination modes and augment the reward with the expected inﬁnite sum of discountedfuture rewards in a time-out case.
In other words, we bootstrap the target of the critic with itsown prediction.
This solution has been discussed in [25], but interestingly, this distinction is not [Figure] part of the widely used Gym environment interface [26] and is ignored by popular implementationssuch as Stable-Baselines [27] 1 .
After investigating multiple implementations, we conclude that thisimportant detail is often avoided by assuming that the environments either never time out or only onthe very last step of a batch collection.
In our case, with few robot steps per batch, we can not makesuch an assumption since a meaningful episode length covers the collection of many batches.
Wemodify the standard Gym interface to detect time-outs and implement the bootstrapping solution.
Insupplementary material, we show the effect of this solution on the total reward as well as the criticloss.
Task Description
A quadruped robot must learn to walk across challenging terrain, including uneven surfaces, slopes,stairs, and obstacles, while following base-heading and linear-velocity commands.
We conductmost of the simulation and real-world deployment experiments on the ANYbotics
ANYmal C robot.
However, in simulation, we demonstrate the broader applicability of the approach by additionallytraining policies for ANYmal B, ANYmal C with an attached arm, and the Unitree A1 robots.
3.1 Game-Inspired Curriculum
The terrains are selected to be representative of real-world environments.
We create ﬁve types ofprocedurally generated terrains presented in Fig.
2: ﬂat, sloped, randomly rough, discrete obstacles,and stairs.
The terrains are tiled squares with 8m sides.
The robots start at the center of the terrainand are given randomized heading and velocity commands (kept constant for the duration of anepisode) pushing them to walk across the terrain.
Slopes and stairs are organized in pyramids toallow traversability in all directions.
Previous works have shown the beneﬁts of using an automated curriculum of task difﬁculty to learncomplex locomotion policies [29, 30, 16].
Similarly, we ﬁnd that it is essential to ﬁrst train the policyon less challenging terrain before progressively increasing the complexity.
We adopt a solutioninspired by [16], but we replace the particle ﬁlter approach with a new game-inspired automaticcurriculum.
All robots are assigned a terrain type and a level that represents the difﬁculty of thatterrain.
For stairs and randomized obstacles, we gradually increase the step height from 5 cm to20 cm .
Sloped terrain inclination is increased from 0 deg to 25 deg .
If a robot manages to walk pastthe borders of its terrain, its level is increased, and at the next reset, it will start on more difﬁcultterrain.
However, if at the end of an episode it moved by less than half of the distance required byits target velocity, its level is reduced again.
Robots solving the highest level are looped back to arandomly selected level to increase the diversity and avoid catastrophic forgetting.
This approachhas the advantage of training the robots at a level of difﬁculty tailored to their performance withoutrequiring any external tuning.
It adapts the difﬁculty level for each terrain type individually andprovides us with visual and quantitative feedback on the progress of the training.
When the robotshave reached the ﬁnal level and are evenly spread across all terrains due to looping back, we canconclude they have fully learned to solve the task.
The proposed curriculum structure is well suited for the massively parallel regime.
With thousandsof robots we can directly use their current progress in the curriculum as the distribution of thepolicy’s performance, and do not need learn it with a generator network [31].
Furthermore, our method doesn’t require tuning and is straightforward to implement in a parallel manner with nearzero processing cost.
We remove the computational overhead of re-sampling and re-generating newterrains needed for the particle ﬁlter approach.
Fig.
3 shows robots progressing through the terrains at two different stages of the training process.
On complex terrain types, the robots require more training iterations to reach the highest levels.
Thedistribution of robots after 500 iterations shows that while the policy is able to cross sloped terrainsand to go down stairs, climbing stairs and traversing obstacles requires more training iterations.
However, after 1000 iterations, the robots have reached the most challenging level for all terraintypes and are spread across the map.
We train for a total for 1500 iterations to let the policy convergeto its highest performance.
3.2 Observations, Actions, and Rewards
The policy receives proprioceptive measurements of the robot as well as terrain information aroundthe robot’s base.
The observations are composed of: base linear and angular velocities, measurementof the gravity vector, joint positions and velocities, the previous actions selected by the policy,and ﬁnally, 108 measurements of the terrain sampled from a grid around the robot’s base.
Eachmeasurement is the distance from the terrain surface to the robot’s base height.
The total reward is a weighted sum of nine terms, detailed in supplementary material.
The mainterms encourage the robot to follow the commanded velocities while avoiding undesired base velocities along other axes.
In order to create a smoother, more natural motion, we also penalize jointtorques, joint accelerations, joint target changes, and collisions.
Contacts with the knees, shanks orbetween the feet and a vertical surface are considered collisions, while contacts with the base areconsidered crashes and lead to resets.
Finally, we add an additional reward term encouraging therobot to take longer steps, which results in a more visually appealing behavior.
We train a singlepolicy with the same rewards for all terrains.
The actions are interpreted as desired joint positions sent to the motors.
There, a PD controllerproduces motor torques.
In contrast to other works [16, 20], neither the reward function nor theaction space has any gait-dependent elements.
3.3 Sim-to-Real Additions
In order to make the trained policies amenable for sim-to-real transfer, we randomize the frictionof the ground, add noise to the observations and randomly push the robots during the episode toteach them a more stable stance.
Each robot has a friction coefﬁcient sampled uniformly in [0.5,1.25].
The pushes happen every 10 s .
The robots’ base is accelerated up to ± 1 m / s in both x and y directions.
The amount of noise is based on real data measured on the robot and is detailed insupplementary material.
The ANYmal robot uses series elastic actuators with fairly complex dynamics, which are hard tomodel in simulation.
For this reason and following the methodology of previous work [1], we use aneural network to compute torques from joint position commands.
However, we simplify the inputsof the model.
Instead of concatenating past measurements at ﬁxed time steps and sending all of thatinformation to a standard feed-forward network, we only provide the current measurements to an
LSTM network.
A potential drawback of this set-up is that the policy does not have the temporalinformation of the actuators as in previous work.
We have experimented with various ways ofproviding that information through memory mechanisms for the policy but found that it does notimprove the ﬁnal performance.
Results
4.1 Effects of Massive Parallelism
In this section, we study the effects of the number of parallel robots on the ﬁnal performance ofthe policy.
In order to use the total reward as a single representative metric, we have to remove thecurriculum, otherwise a more performant policy sees its task difﬁculty increase and consequently adecrease in the total reward.
As such, we simplify the task by reducing the maximum step size ofstairs and obstacles and directly train robots on the full range of difﬁculties.
We begin by setting a baseline with n robots = 20000 and n steps = 50 , resulting in a batch size of1M samples.
Using this very large batch size results in the best policy but at the cost of a relativelylong training time.
We then conduct experiments in which we increase the number of robots while keeping the batchsize constant.
As a result, the number of steps each robot takes per policy update decreases.
In thiscase, the training time decreases with a higher number of robots, but the policy performance dropsif that number is too high.
We start from 128 robots corresponding to the level of parallelizationof previous CPU implementations and increase that number up to 16384, which is close to themaximum amount of robots we could simulate on rough terrain with Isaac Gym running on a singleworkstation GPU.
In Fig.
4, we compare these results with the baseline, which allows us to select the most favorabletrade-off between policy performance and training time.
We see two interesting effects at play.
First, when the number of robots is too high, the performance drops sharply, which can be explainedby the time horizon of each robot becoming too small.
As expected, with larger batch sizes, theoverall reward is higher, and the time horizon effect is shifted, meaning that we can use more robotsbefore seeing the drop.
On the other hand, below a certain threshold, we see a slow decrease inperformance with fewer robots.
We believe this is explained by the fact that the samples are very [Figure] [Figure] similar with many steps per robot because of the relatively small time steps between them.
Thismeans that for the same amount of samples, there is less diversity in the data.
In other words, with alow number of robots, we are further from the standard assumption that the samples are independentand identically distributed, which seems to have a noticeable effect on the training process.
In termsof training time, we see a nearly linear scaling up to 4000 robots, after which simulation throughputgains slow down.
As such, we can conclude that increasing the number of robots is beneﬁcial forboth ﬁnal performance and training time, but there is an upper limit on this number after which anon-policy algorithm cannot learn effectively.
Increasing the batch size to values much larger thanwhat is typically used in similar works seems highly beneﬁcial.
Unfortunately, it also scales thetraining time so it is a trade-off that must be balanced.
From the third plot we can conclude thatusing 2048 to 4096 robots with a batch size of ≈ 100 k or ≈ 200 k provides the best trade-off for thisspeciﬁc task.
4.2 Simulation
For our simulation and deployment experiments, we use a policy trained with 4096 robots and abatch size of 98304, which we train for 1500 policy updates in under 20 minutes 2 .
We begin bymeasuring the performance of our trained policy in simulation.
To that end, we perform robustnessand traversability tests.
For each terrain type, we command the robots to traverse the representativedifﬁculty of the terrain at high forward velocity and measure the success rate.
A success is deﬁnedas managing to cross the terrain while avoiding any contacts on the robot’s base.
Fig.
5 showsthe results for the different terrains.
For stairs, we see a nearly 100 % success rate for steps up to0 .
2 m , which is the hardest stair difﬁculty we train on and close to the kinematic limits of our robot.
Randomized obstacles seem to be more demanding, with the success rate decreasing steadily.
Wemust note that in this case, the largest step is double the reported height since neighboring obstaclescan have positive and negative heights.
In the case of slopes, we can observe that after 25 deg therobots are not able to climb anymore but still learn to slide down with a moderate success rate.
Given our relatively simple rewards and action space, the policy is free to adopt any gait and behavior.
Interestingly, it always converges to a trotting gait, but there are often artifacts in the behavior,such as a dragging leg or unreasonably high or low base heights.
After tuning of the reward weights,we can obtain a policy that respects all our constraints and can be transferred to the physical robot.
To verify the generalizability of the approach, we train policies for multiple robots with the sameset-up.
We use the ANYmal C robot with a ﬁxed robotic arm, which adds about 20 % of additionalweight, and the ANYmal B robot, which has comparable dimensions but modiﬁed kinematic and2 Trained on: i9-11900k, NVIDIA RTX A6000, GPU VRAM requirements are in supplementary material. dynamic properties.
In these two cases, we can retrain a policy without any modiﬁcations to therewards or algorithm hyper-parameters and obtain a very similar performance.
Next, we use the
Unitree A1 robot, which has smaller dimensions, four times lower weight, and a different leg conﬁguration.
In this case, we remove the actuator model of the ANYdrive motors, reduce PD gainsand the torque penalties, and change the default joint conﬁgurations.
We can train a dynamic policythat learns to solve the same terrains even with the reduced size of the robot.
Finally, we apply ourapproach to Agility Robotics’ bipedal robot Cassie.
We ﬁnd that an additional reward encouragingstanding on a single foot is necessary to achieve a walking gait.
With this addition, we are ableto train the robot on the same terrains as its quadrupedal counterparts.
Fig.
6 shows the differentrobots.
4.3 Sim-to-real Transfer
On the physical robot, our policy is ﬁxed.
We compute the observations from the robot’s sensors,feed them to the policy, and directly send the produced actions as target joint positions to the motors.
We do not apply any additional ﬁltering or constraint satisfaction checks.
The terrain heightmeasurements are queried from an elevation map that the robot is building from Lidar scans.
Unfortunately, this height map is far from perfect, which results in a decrease in robustness betweensimulation and reality.
We observe that these issues mainly occur at high velocities and thereforereduce the maximum linear velocity commands to 0 .
6 m / s for policies deployed on the hardware.
The robot can walk up and down stairs and handles obstacles in a dynamic manner.
We show samplesof these experiments in Fig.
7 and in the supplementary video.
To overcome issues with imperfectterrain mapping or state estimation drift, the authors of [19] implemented a teacher-student set-up,which provided outstanding robustness even in adverse conditions.
As part of future work, we planto merge the two approaches.
Conclusion
In this work, we demonstrated that a complex real-world robotics task can be trained in minutes withan on-policy deep reinforcement learning algorithm.
Using an end-to-end GPU pipeline with thousands of robots simulated in parallel, combined with our proposed curriculum structure, we showedthat the training time can be reduced by multiple orders of magnitude compared to previous work.
We discussed multiple modiﬁcations to the learning algorithm and the standard hyper-parametersrequired to use the massively parallel regime effectively.
Using our fast training pipeline, we performed many training runs, simpliﬁed the set-up, and kept only essential components.
We showedthat the task can be solved using simple observation and action spaces as well as relatively straightforward rewards without encouraging particular gaits or providing motion primitives.
The purpose of this work is not to obtain the absolute best-performing policy with the highest robustness.
For that use case, many other techniques can be incorporated into the pipeline.
We aimto show that a policy can be trained in record time with our set-up while still being usable on thereal hardware.
We wish to shift other researchers’ perspective on the required training time for areal-world application, and hope that our work can serve as a reference for future research.
We expect many other tasks to beneﬁt from the massively parallel regime.
By reducing the training timeof these future robotic tasks, we can greatly accelerate the developments in this ﬁeld.