Visual-Locomotion: Learning to Walk on Complex Terrains with Vision
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Vision is one of the essential perception modalities for legged robots tosafely and efﬁciently navigate uneven terrains, such as stairs and stepping stones.
However, training robots to effectively understand high-dimensional visual inputfor locomotion is a challenging problem.
In this work, we propose a framework totrain a vision-based locomotion controller which enables a quadrupedal robot totraverse uneven environments.
The key idea is to introduce a hierarchical structurewith a high-level vision policy and a low-level motion controller.
The high-levelvision policy takes as inputs the perceived vision signals as well as robot statesand outputs the desired footholds and base movement of the robot.
These arethen realized by the low level motion controller composed of a position controllerfor swing legs and a MPC-based torque controller for stance legs.
We train thevision policy using Deep Reinforcement Learning and demonstrate our approachon a variety of uneven environments such as randomly placed stepping stones,quincuncial piles, stairs, and moving platforms.
We also validate our method on areal robot to walk over a series of gaps and climbing up a platform.
Introduction
[Figure]
Reproducing natural vision-based locomotion skills seen in animals on artiﬁcial creatures as legged robots has long been at the forefront of robotics research.
Progress towards developingrobust visual locomotion controllers not only deepens our understanding of how humans and animalsperceive the environment and control our limbs, but also enables us to build autonomous machinesthat can reliably traverse real-world environments.
To tackle this problem, most existing methods adopt a three-stage pipeline [1, 2, 3, 4, 5, 6, 7]: perception, motion planning and control.
In the perception stage, raw sensor data such as RGBD imageand/or LiDAR point clouds are carefully fused with proprioceptive streams such as IMU, motor angles, and wheel odometry.
For mobile robots, a SLAM (simultaneous localization and mapping) isoften used to produce a elevation terrain map centered around the robot [8].
The generated elevationmap is then fed to the downstream motion planning modules to select paths, motion style, and footplacements (on legged robots).
Finally, the planned robot pose or joint angles are tracked by a low- level motion controller: model predictive control (MPC) [9] or whole body control methods (WBC)[4] are popular choices for unstable, highly dynamical platforms such as legged robots.
While there have been great results from previous works that adopt the three-stage control pipeline,it often leads to a system that is overly complex and requires signiﬁcant manual effort to develop.
For instance, SLAM algorithms require careful parameter tuning to achieve a balance between latency and accuracy for mobile robots [10].
On the other hand, recent developments in reinforcementlearning (RL) based methods open an alternative path towards creating vision-based locomotioncontrollers without relying on terrain reconstruction or extensive prior knowledge for foothold selection.
Researchers have proposed learning-based algorithms that teach robot arms to retrieve objectsfrom cluttered environments [11] and teach drones to avoid obstacles [12] using vision input.
Theydemonstrate the great potential in an end-to-end learning approach to achieve a low-latency controlpipeline and reduce the prior knowledge required to design a working controller.
Inspired by this recent progress, we propose a novel control architecture that enables legged robotsto successfully solve various visual-locomotion tasks.
Speciﬁcally, we adopt the philosophy ofend-to-end learning and merge the perception and motion planning modules using a neural network(NN).
Our contribution is a learnable hierarchical system that contains two individual layers: a highlevel vision policy and a low-level motion controller.
The high-level vision policy takes two depthimages and outputs the desired pose of the robot’s base and foothold placements for all swing legs,thereby eliminating the need for a complex SLAM algorithm.
The low-level locomotion controllertakes the high-level vision policy output, and computes the target motor positions and torques toachieve the desired states.
This hierarchical approach allows us to achieve dynamic locomotionin challenging simulated environments including randomly placed stepping stones, staircases, andmoving platforms.
We also demonstrate zero-shot sim-to-real transfer of visual locomotion policieson the real hardware for walking over stepping stones and climbing up a platform.
Related Work
Recently, visual-locomotion researchers have developed various promising approaches that tacklethe problem using on-board sensors [1, 2, 3, 4, 5, 6, 7, 13, 14, 15, 16].
A main idea in many ofthese works is to perform explicit terrain shape reconstruction (i.e. local SLAM).
For example,
Fankhauser and Hutter [13] developed a “Grid Map” stack to construct local elevation maps aroundthe robot and Kim et al.
[14] utilized an Intel RealSense D435 depth sensor and a T265 trackingcamera to obtain an elevation map of the robot’s surroundings.
In contrast, we do not employ anyexplicit mapping mechanism: our learning based visual policy consumes raw depth images fromonboard cameras for decision making.
Because we eliminate the highly complex mapping processwhich requires special expertise to tune, our system has a simpliﬁed and low latency data pipelinewith less chance for manual error accumulation.
Given a representation of the environment provided by a perception module, the controller needsto plan a sequence of leg movements that can guide the robot through the environment.
Unlikeour approach where perception and motion planning are subsumed into a single neural network,previous works utilize heuristic-based approaches or local optimization to plan adequate motions[5, 3, 16, 2, 7, 17, 18, 19].
For instance, Jenelten et al. developed a metric for scoring each point onan elevation map based on its roughness, closeness to an edge, as well as slope degrees.
They thenemployed an online batch optimization to choose the best landing positions for the feet.
In contrast,our learning system directly learns successful foot targets from reward, which greatly simpliﬁesthe system architecture. Similar to our approach, Tsounis et al.
also proposed to learn a high-levelplanner module to plan the gait sequence and a low-level gait controller module to control the robot.
They demonstrated impressive simulated results of traversing a variety of terrains using a height mapas input.
In this work, we focus on demonstrating visual-locomotion skills on a real-robot, whichled us to different design choices such as depth images input, an MPC-based low-level controller,and the development of a sim-to-real transfer pipeline.
Foothold optimization is traditionally performed within a single swing step, however, recent workshave proposed multi-step contact sequence optimization [3, 16, 21] to account for upcoming changesin the terrain. For instance, Nguyen et al. proposed a 2-step gait optimization to plan the footstep aswell as transition between walking gaits. They demonstrated the approach on a biped robot walkingover uneven terrains.
In our work, we train the perception and planning module together using deep reinforcement learning, which endows the policy with implicit long-horizon planning capabilities.
Although neural networks were also employed in prior works [2, 7] to decide optimal footholds, oursystem differs in that it also decides the desired body pose and speed.
By simultaneous adjustingbody pose and footholds, our system can learn challenging visual locomotion tasks such as randomstepping stones, which is deemed as difﬁcult to solve with constant body velocities [5],
In addition to methods that leverages the vision input for controlling the robot, recent works havealso shown remarkable results for training robots to traverse some uneven terrain without relyingon visual perception [22, 23].
By carefully designing the reward function and training procedure,these methods produces policies that can robustly handle moderate uneven terrains by adjustingleg movements.
Despite the impressive results demonstrated in these works, vision inputs are stillbeneﬁcial for efﬁciently and safely traversing general unstructured terrains.
For some tasks, such aswalking over gaps, visual guidance is still necessary. [Figure]
Methodoloy
[Figure]
3.1 Overview
[Figure]
In this work, we adopt a hierarchical architecture for vision-locomotion control, as illustrated in
Figure 2.
The controller consists of two components: a high-level vision policy and a low-levelmotion controller.
The high-level vision policy uses raw depth images from the onboard camerasand robot state to decide the desired center of mass (CoM) pose (and speed) and landing positionsof the swing feet (Section 3.2).
Then the low-level motion controller tracks the desired CoM stateby combining a position-based swing leg controller and an model predictive control (MPC) basedstance leg controller (Section 3.3).
The vision policy runs at 20 Hz, while the motion controllerruns at 250 Hz.
We train our policy entirely in a physics simulation environment and transfer thetrained policy to a real robot platform using techniques described in Section 3.4.
Unless otherwisespeciﬁed, all quantities are deﬁned in the yaw aligned inertial frame (Appendix A.1).
3.2 Learning-Based Vision Policy
We formulate the visual-locomotion task as a Partially-observable Markov Decision Process(POMDP), ( O, S, A, R, P, p 0 ,  ) , where O is the observation space, S is the state space, A is theaction space, R : S ⇥
A 7!
R is the reward function, P : S ⇥
A 7!
S is the dynamics equation, p 0 111 is the initial state distribution and  is the discount factor.
Our goal is to ﬁnd a policy ⇡ : O 7!
A 112 that maximizes the expected accumulated reward: J ( ⇡ ) =
E ⌧ =( s 0 , a 0 ,..., s T )
P
Tt =
t r ( s t , a t ) .
Observation and action spaces.
We design the observation space in our experiments as O =(
I 1 , 2 , q s , ˙ p , ˙  , ˙ ⇥ , r 1 ... 4 , c 1 ... 4 ,  1 ... 4 , a prev ) , i.e the two depth images I 1 , 2 from depth sensorsshown in Figure 1 (to cover terrains near both front and rear legs), a subset of the CoM poseq s =
( p z ,  , ⇥ ) (CoM height, roll, and pitch), estimated CoM velocity ˙ p (Appendix C), gyroscope readings ˙
, ˙ ⇥ , the robot’s feet positions r 1 ... 4 , the feet contact states c 1 ... 4 (one if in contact and zero otherwise), the phase  of each leg in its respective gait cycle (Appendix A.2), and the previous action.
The action space is ( q ds , ˙q d , r d 1 ... 4 ) , i.e. the desired CoM pose, velocity, and i th foot’starget landing position ( r xi , r yi ) .
The z component of the landing location is inferred from the depthreadings: we compute the point cloud from the depth image near the predicted ( x, y ) coordinate andtake the average height of four nearest points.
This enables us to achieve better exploration by eliminating invalid foothold targets (e.g. inside an obstacle or high in the air).
We ﬁnd this to be criticalfor the policy to traverse highly uneven terrains such as stairs.
Reward function.
We design a reward function:
[Equation]
where ˙
p x is the CoM velocity in the forward direction, p y the CoM displacement in the lateraldirection, and the base yaw angle.
The ﬁrst term rewards the robot to move forward with amaximum speed controlled by ˙ p maxx , the second term penalizes the robot from moving sideways, andthe last term encourages the robot to walk straightly; w 1 , w 2 modulates the importance of differentterms.
In our experiments, we chose ˙
p maxx = 0 .
375 m/s, w 1 = 1 . 25 , and w 2 = 0 .
Early termination.
A training episode is terminated if: 1) the robot loses balance (CoM heightp z below 0.15 m, pitch | ⇥ | > 1 rad, or roll |  | > 0 .
3 rad in our experiments), 2) the robot stepswithin 0 .
02 m of the boundary of the stepping stones or stairs, or 3) the robot reaches an invalid jointconﬁguration, e.g. knee bending backwards.
3.3 Motion Controller
The motion controller computes appropriate motor position and torque commands to achieve thedesired landing positions of the swing legs and the target base poses from high level policy.
Ourmotion controller separately controls swing and stance legs.
The swing leg controller computes thefeet positions by interpolating a time based curve ↵ ( t ) between the swing start and target landing positions.
Instantaneous feet positions are converted to desired joint angles through inverse kinematics(IK) and tracked using proportional-derivative (PD) control.
The stance leg controller, on the otherhand, achieves the desired position and velocity of the robot base by computing sequences of contact forces between the feet and the ground.
By approximating the robot dynamics using Centroidal Dynamics Model (CDM), we formulate this problem as a convex model predictive control (MPC)problem similar to [24].
The optimized contact forces are mapped to stance leg joint torques using Jacobian Transpose.
More details regarding the motion controller can be found in Appendix A.
3.4 Sim-to-real Transfer
Sim-to-real transfer is a persisting problem in robotics, and is especially hard for visual locomotionproblems because of distribution shift in image space and discrepancies in robot dynamics.
Images from simulated and real-world sensors can be signiﬁcantly different (Figure 3, A and B).
To bridge the gap, we adopt a post-processing procedure that maps both simulated and real depthimages to a similar domain.
As shown in Figure 3, C and D, we ﬁrst add random Gaussian noiseand randomly paint pixels black in the simulated depth image.
To mimic the noise around the object edges in a stereo depth camera, pixels along an edges in the depth image identiﬁed by a Canny edgedetector have a higher probability of being dropped.
We then apply an in-painting operation for bothsimulated and real images to ﬁll the missing pixels[25], followed by a down-sampling operation.
This transforms the depth images in simulation and real world to a similar distribution.
Discrepancies in dynamics such as mismatch in friction make precise foot-placements difﬁcult.
Forexample, we found the average difference of feet landing positions in the world frame can be 2 ⇠ 4cm between sim and real when the same sequence of actions are executed. This is enough to triggera failure in the step-stone task.
To compensate this issue, we adopt the domain randomization technique [26] to train a robust policy with randomized simulation parameters.
More details regardingthe randomization process can be found in Appendix D. Adding randomization not only allows thepolicy to be more robust, but also enables the vision policy to observe more diverse states.
Experiment and Results
[Figure]
4.1 Experiment Setup
[Figure]
We evaluate our method on the Unitree Laikago [27] quadruped robot, which weighs 24 kg .
Therobot is equipped with 12 actuated joints.
To collect visual data, we install two depth cameras onthe robot: one Intel D435 in the front for a wider ﬁeld of view and one Intel L515 on the belly withbetter depth quality in close range (Figure 1).
At inference time, we process all depth images asdescribed in Section 3.4, which results in two 32 ⇥ 24 depth images as inputs to the policy.
We train our hierarchical policies using the PyBullet physics simulator [28].
We use a distributedimplementation of augmented random search (ARS) [29], which optimizes the policy parametersusing an evolutionary strategy.
We use N = 256 perturbations per ARS iteration and run thealgorithm until convergence with a maximum of 2000 training iterations, amounting to 1 , 024 , 000simulation episodes per trial.
We use a Multi-layer Perceptron (MLP)-based policy architecture with54 , 656 parameters.
For all experiments, we run ARS with a grid search for four hyper-parameters,resulting in 12 trials.
We then report the performance of top-3 policies by testing them on 150randomized environments for each task.
More training details can be found in Appendix E.
For experiments in both simulation and real world, we train policies with perception noise. Wethen ﬁne-tune them with the dynamics randomization scheme as described in Section 3.4 beforedeploying to the real robot.
We empirically ﬁnd this helpful for faster training convergence.
Theﬁne-tuned policy is deployed on the robot without the need of additional hardware data.
Videos ofour trained policies in simulation and real-world can be found in the supplementary video.
4.2 Simulation Results
We ﬁrst evaluate the capability of our learning system on a variety of challenging visual locomotiontasks, including walking over randomly placed stepping stones, quincuncial piles, uneven terrains,and moving platforms.
A subset of the simulation environments are shown in Figure 4.
We trainseparate policies for each environment and show the statistical results in Figure 5.
To measurethe performance of each policy, we design a metric, performance ratio , as p Tx p maxx , where p Tx is thedistance travelled by the policy in the environment, and p maxx is the maximum distance the policycould reach, e.g. where the terrain ends.
Randomly placed stepping stones.
The ﬁrst task is to walk over a series of randomly placedstepping stones, Figure 4A. The widths, lengths and gap sizes of stepping stones are sampled from[0 .
7] , [0 . 5 , 0 . 8] , and [0 . 07 , 0 . 2] meters respectively.
The agent fails the task if the robot stepsoutside of the stone or into the gaps.
Since the stones and their positions are randomly sampled, thistask cannot be accomplished without vision.
To successfully achieve this task, the robot needs toidentify the position and size of the stones and also plan for future footholds.
We demonstrate thatour method can obtain policies that solves this task using both trotting and pacing gaits.
Quincuncial piles.
The random stepping stone task evaluates the robot’s ability to identify andhandle terrain changes in the forward direction.
A natural extension is to also include the lateraldirection.
In the second task we create a grid of 2 dimensional stepping stones (Figure 4B). Eachstone has an area of 0 .
15 ⇥ 0 .
15 m 2 with a standard deviation of 0 .
015 m in height, and is separatedby [0 . 13 , 0 . 17] m from each other in both x and y directions.
At the beginning of each episode, wealso randomly rotate entire stone grid in [  0 . 1 , 0 . 1] rad.
Despite being signiﬁcantly more difﬁcult,using our framework we can obtain policies that can traverse the ﬁeld using trotting or walking gaits.
Uneven terrains.
So far the tasks focus on evaluating the robot’s ability to avoid undesired regions on relatively even terrains.
To test the policy’s performance in handling different heights,we designed a staircase climbing task (Figure 4C).
The depth of each stair is uniformly sampled in[0 .
33] m and the height in [0 . 16 , 0 .
19] m to mimic the dimensions of real-world stairs.
Thistask is quite challenging for Laikago robot because each stair is as tall as the robot’s knee joint.
Figure 4D shows another environment we designed for evaluating the ability of our approach to handleuneven terrains.
In this environment, the height offsets of neighboring stones are uniformly sampledin [  0 . 2] m, and a gap of [0 . 05 , 0 . 1] m is added between the stones.
Moving platforms.
One beneﬁt of using vision input to the policy is that it can potentially handlemoving objects better than methods that relies on explicit terrain reconstruction.
To demonstratethe capability of our learning system, we take the random stepping stone environment (Figure 4A)and allow each piece to move.
Each platform follows a periodic movement whose magnitude andfrequency are randomly sampled in [0 . 10 , 0 . 15] m and [0 . 4 , 1 . 0]
Hz, respectively.
Also, we randomlypick half of the platforms to move horizontally and the rest vertically.
This task requires the robotto infer both the position and velocity of the platforms.
To facilitate learning, we stack a historyof three recent images as input to the policy for this task.
As shown in the accompanying video,our policy learns to identify the moving objects in the scene and will slow-down and wait for theplatforms to reach an ideal location before striding.
4.3 Comparison to baseline methods
We compare our proposed method to two baselines: end-to-end training and heuristics-based footplacement.
For the sake of simplicity yet without loss of generality, all comparisons are done in theuneven terrain environment (Figure 6 Top), which captures the difﬁculty of both the stepping stonesand staircases.
The results can be found in in Figure 7.
4.3.1 End-to-end training
In the ﬁrst baseline, we train an end-to-end neural network policy that takes the images and therobot states as input (same as our hierarchical policy) and outputs desired motor angles.
We choosethe architecture of Policy Modulating Trajectory Generator (PMTG) [30], which has demonstrated high-quality and transferrable locomotion policies [22].
Since PMTG generates the trajectory in thejoint space, it was not able to achieve precise foot placement that is needed for many of our testingenvironments, as shown in Figure 7.
4.3.2 Heuristics-based foot placement
In the second baseline, we compare our method to prior approaches that adjust the foot placement bycomputing a score map of surrounding terrains [2, 4, 18].
In our baseline implementation, we deﬁnea point on the nearby terrain to be valid if 1) it is within reach of the swing leg, 2) the z-componentof the surface normal is larger than 0.9 (i.e. pointing upward), and 3) the standard deviation of thenearby heights (i.e. roughness) is less than 0 .
05 m. For better comparison, we use the same motioncontroller as in our approach and train a high-level policy to output the desired base pose and velocityusing the same observation space and learning procedure.
As shown in Figure 7, using heuristicsalone to determine footplacement got lower score than our proposed approach.
One important reasonis that this heuristics does not take the base movement of the robot into consideration.
As a result, itmay propose landing positions that are incompatible with the robot’s CoM speed.
4.4 Ablation Study
A key hypothesis we make in this work is that, despite that the images have low resolution, theycontain critical information that enables our policy to traverse a large variety of environments.
Tovalidate this hypothesis, we ﬁrst perform an ablation run by removing the vision component fromthe observation space and retraining a vision-less policy.
As seen in Figure 7, without vision input,the policy is not able to accomplish the task, indicating the importance of having vision as input.
Another key component in our method is to use the depth input for inferring the z-component ofthe foothold location.
This creates a more meaningful action space for the policy to explore duringlearning.
As shown in Figure 7, the performance drops signiﬁcantly we do not use depth to infer thez-component of the foot placement.
[Figure]
We deploy the trained policy to a Laikago robot for two tasks, walking over gaps and climbing ontostepping stones (see supplementary video).
Our real-world setup (Figure 8 Top) consists of fourstepping stones that are separated by three gaps with widths between [0 . 12 , 0 . 18] m.
In the ﬁrst task, the performance is measured by the number of gaps that are crossed successfully,summed over legs.
For instance, if all four legs cross one gap and the robot is completely on the nextstepping stone, the score would be 4.
Therefore, the upper bound of the score is 12: three gaps timesfour legs.
An experiment episode is terminated if the robot falls, any leg steps into the gaps or therobot steps outside the stones.
As a baseline, the score is 0 .
9 ± 1 .
for a blind-locomotion policy:it can rarely clear a single gap.
Over eight real-world episodes
, our policy is able to consistentlyachieve a score of 10 . 1 ± 2 . 2 .
There are two episodes in which the robot completes the full coursewith a score of 12; For seven out of eight episodes, the robot reaches the last stepping stone.
Thereal-world failure cases are mainly caused by the robot stepping too close to the edge of the terrainand slipping into the gap.
This is likely due to the discrepancies in the dynamics and vision model.
Despite applying domain randomization techniques to mitigate the model discrepancy, we foundprecisely controlling the landing position of the feet for a moving robot still challenging.
In the second task, the robot needs to climb onto a stepping stone, which is 0.18 m above fromthe ground (Figure 8 Bottom).
Similarly, we also deﬁne the performance score as the number oflegs reached the top of the stone (4 is the max).
The learned policy reaches a score of 3 . 7 ± 0 .
7over seven experiment episodes.
The robot successfully steps onto the stone for six out of the seventimes.
In the only failure case, the robot loses balance because its rear foot hits the edge of the stone.
Conclusion and Future Work
In this work, we present a hierarchical learning system to tackle challenging visual-locomotiontasks.
By using a high-level learned vision policy that consumes raw camera images, we eliminatethe need to explicitly construct 3D terrain maps, and thus reduce the control latency and architecturecomplexities, and more importantly, can handle dynamically moving terrains.
The low-level, modelpredictive control based motion controller greatly reduces the motion tracking error on the hardwareand thus narrows the sim-to-real gap.
We demonstrate that policies trained using our system canreliably walk over challenging terrains such as stepping stones that require ﬁne visual-motor control.
One limitation of this work is that the gaits are manually speciﬁed and ﬁxed for each task.
Incontrast, animals can dynamically modulate their walking pattern to adapt to changes in the terrain.
In the future, we plan to extend our framework to enable the visual policy to adjust the gait patternonline.
Recent works have shown great potential in representation learning for vision-based roboticcontrol problems [31, 32].
For example, Hoeller et al.
trained a VAE to encode the environmentfor a legged robot navigation task.
Incorporating these techniques can further improve the learningperformance of our framework.
Additionally, animals can traverse complex terrains while payingattention to a small area, and can reason about their hind limbs that they cannot see.
This makesadding memory [33] and attention [34] to the policy architecture another promising future direction.