Learning to Jump from Pixels
Anonymous Author(s)
Af ﬁ liation Address email
Abstract: Today’s robotic quadruped systems can robustly walk over a diverserange of natural but continuous terrains involving snow, rain, slip, rubble, etc.
Locomotion on discontinuous terrains such as one with gaps or obstacles presents acomplementary set of challenges.
It becomes necessary to plan ahead using visualinputs and execute agile behaviors such as jumps to cross gaps.
Such dynamic motion results in signi ﬁ cant motion of onboard camera that introduces a new setof challenges for real-time visual processing.
The need for agility and the operation from vision reinforce the need for robust control.
We present a system thatcan, in real-time, process visual observations from an onboard RGBD camera to command a quadruped robot to jump over wide gaps.
The proposed method brings together the ﬂ exibility of model-free learning and the robustness of model-basedcontrol.
We evaluate performance both in simulation and in the real world.
Introduction
One of the grand challenges in robotics is to construct legged systems that can successfully navigatenovel and complex landscapes.
The Spot robot and the ANYmal robot are impressive in their abilityto traverse a wide diversity of natural and man-made terrains [1].
During blind locomotion, suchrobots primarily rely on proprioception and robust control schemes to achieve sturdy walking inchallenging conditions involving snow, thick vegetation, slippery mud etc.
The downside of notusing visual observations is the inability to execute motions that anticipate the land surface in frontof the robot.
This is especially prohibitive on terrains with signi ﬁ cant elevation discontinuities.
Forinstance, crossing a wide gap requires the robot to jump, which cannot be initiated without knowingwhere and how wide the gap is.
Without vision, even the most robust system would either step inthe gap and fall or otherwise treat the gap as an obstacle and stop.
Additionally, the inability to planresults in conservative behavior that is unable to achieve the energy ef ﬁ ciency or the speed that thehardware affords.
Vision-based legged locomotion holds the promise to overcome these challenges, and substantialprogress has been made in this direction [2, 3, 4, 5, 6, 7, 8, 9].
The state-of-the-art systems cantraverse uneven terrain, walk across gaps, and climb over stairs.
Many of these systems assumeaccess to the ground truth height-map of the terrain [2, 3, 5] which is generally not available fornew terrains encountered by the robot.
Several works overcome this limitation by performing onlineconstruction of a terrain heightmap from depth images [4, 7].
These works perform a rule-basedsearch for stable footholds and use model-based controllers to plan safe trajectories.
Often, themodel used in planning makes simplifying assumptions such as ﬁ xed body trajectory [7] or restrictedcontact pattern [5].
These assumptions result in conservative and non-agile locomotion
Planning agile behaviors, such as jumps, on discontinuous terrain offers a different and complementary challenge to traversing continuous and unstructured terrain.
Executing a jump requires planningthe location of the jump, the force required to lift the body, and dealing with severe under-actuationduring the ﬂ ight phase.
Prior work has demonstrated standing jumps in simulation [10, 11], on areal robot [7], and running jumps in simulation [12, 13, 14, 9, 8].
Prior work on the MIT Cheetah 2 achieved running and jumping over a single obstacle [15].
However, this system was heavilyhand-engineered: it assumes straight-line motion, uses a specialized control scheme developed forfour manually segmented phases of the jump, and the vision system was specialized for detectingspeci ﬁ c obstacles.
Further, the robot was constrained to a ﬁ xed gait.
Consequently, this system isspeci ﬁ c to jumping over one obstacle type, and substantial engineering effort would be required toextend agile locomotion to diverse terrains in the wild.
Operation in the wild requires a system architecture that can automatically generate a diverse setof agile behaviors directly from visual observations.
One possibility is to apply deep reinforcementlearning to predict joint torques directly from visual inputs.
Previous work [9, 14, 16] trains locomotion RL algorithms in simulation, since a simulated environment can be easily randomized,requires little human maintenance, and can be conveniently parallelized for large-scale experiencecollection.
However, there is a drawback to training in simulation: the agent learns to exploit inaccuracies in the simulation to achieve high reward, when such behavior fails to transfer to the realsystem.
Some prior work aims to reduce simulator exploitation by encoding locomotion-speci ﬁ c priors into the agent’s action space.
One such line of work uses a trajectory generation method knownas PMTG [17], which combined with domain randomization can successfully transfer simulatedwalking behaviors to the real world.
However, it is well known that while greater randomization increases robustness and improves sim-to-real transfer, it also results in more conservative and therebysuboptimal policies [18].
The sim-to-real problem is severely aggravated by requirements of agility and operation from visual inputs.
This motivates a different style and rigor of controller design that emphasizes transferrobustness.
As a step towards vision-guided agile locomotion, we present a general framework forsynthesizing adaptive, agile behaviors with the support of a low-level robust controller.
To evaluateour proposed system and baselines, we construct a suite of gap-world environments that require aquadruped to cross a sequence of randomly placed gaps of varying widths using observations froman attached depth camera (see Figure A.1).
While this environment is much simpler than “in-thewild”, traversing it successfully requires solving many of the core challenges in vision-guided agilelocomotion.
We use the MIT Mini Cheetah robot [19] as the experimental platform and report resultsboth in simulation and the real world.
We contribute a novel analysis of design choices for integrating ﬂ exible model-free learning withrobust model-based trajectory optimization in the context of visually guided locomotion across discontinuous terrain.
The end result is a system that can (a) cross a sequence of wide gaps in real-timeusing depth observations from a body-mounted camera in the real world (i.e., jumping from pixels; Figure A.1); (b) requires no dynamics randomization for sim-to-real-transfer; (c) does not assumeﬁ xed gait; (d) achieves the theoretical limit of jump width with ﬁ xed gaits and even wider jumpswith variable gaits and (e) outperforms existing state-of-the-art methods (e.g., PMTG [17]).
Ourframework relies on a combination of model-free RL for high-level planning, a hybrid model-basedlow-level controller, and asymmetric behavior cloning [1, 20] that we discuss in Section 2.
Implementation details are provided in Section 3 and results in Section 5.
Method Overview
The mapping from depth images to joint torques is complex.
To simplify the problem we makeuse of a hierarchical scheme where a high-level controller processes visual inputs to predict thedesired trajectory of the robot’s body and a blind low-level controller ensures that the predictedtrajectory is tracked.
This separation eases the task for both the controllers: the high-level is shieldedfrom intricacies of joint-level actuation and the low-level is not required to reason about visualobservations.
Our choice of the action space for high-level controller is guided by the intuition thata wide range of agile behaviors can be generated by using a variable gait schedule and commandingthe body velocity of the quadruped.
A high forward velocity results in running, whereas differentratios of vertical and forward velocity can control the height and the span of a jump.
The variablegait schedule allows the robot to change when its foot contacts the ground and thus further expands s t o t a t Joint Trajectory Generator a 1 a 2 a 3 ... q des Baseline: Joint Trajectory Generator Neural Network s t o t a t
Whole-body Trajectory Generator 𝝉 t:t+H a 1 a 2 a 3 ...
Model-based Optimization q des Ours : Whole-body Trajectory Generator + Model-based Optimization Neural [Figure]
the range of feasible contact locations and applied forces.
We solve the problem of mapping depthobservations to velocity and gait-schedule commands using model-free deep reinforcement learning.
To ensure that the robot tracks these commands, one possibility is to simultaneously train a low-levelcontroller using RL that converts the high-level velocity and gait commands into joint torques.
Sucha scheme has two drawbacks: (i) sim-to-real transfer issues discussed in Section 1 and (ii) copiousdata requirement for training.
Another possibility is to leverage an analytical model of the robotand solve for joint torques using trajectory optimization – a scheme commonly known as wholebody control (WBC).
Whole-body control can promote robustness by optimizing for body stabilityat high frequency [21, 22, 23, 24].
One issue, however, is that a typical WBC tracks the robot’scenter-of-mass (CoM) [21, 22], which is infeasible during the ﬂ ight phase of agile motion due tounder-actuation of the robot’s body.
To overcome this issue, we leverage a prior hybrid controlscheme built on the intuition that the changes in body velocity can be realized by modifying theforces applied by the robot’s feet on the ground.
This frees the controller from the requirement offaithfully tracking the CoM and instead tracks the contact timing and the ground forces applied bythe feet.
This approach, called whole-body impulse controller (WBIC), is well suited for highlydynamic motion [24].
The velocity and gait-schedule targets for WBIC are selected in our system by a learned policy.
Wefound that while it was possible to train this high-level policy from depth images, training using theground truth heightmap is more sample ef ﬁ cient and yields higher ﬁ nal performance (see Section5.2).
However, in real-world environments, such a heightmap is typically either unavailable ormust be constructed in real-time [4], incurring latency, information loss, and engineering effort.
Weeschew heightmap construction using a two-stage asymmetric behavioral cloning [1, 20] approach,in which we ﬁ rst train an expert policy using height-map data and then distill it to a student policythat only uses depth images.
More details are provided in Section 3.1.
The quadruped’s whole-body state at time t is fully de ﬁ ned as T t =
[ p b , ˙ p b , ¨ p b , p f , ˙ p f , ¨ p f , C ] t ∈ R 54 × [0 , 1] 4 where p b =
[ x, y, z, α , β , γ ] ∈ R 6 is the robot body pose (position ( x, y, z ) and eulerangles ( α , β , γ ) ).
The terms p f =
[ p LF x , p LF y , p LF z , p RF x , p RF y , p RF z , p LR x , p LR y p LR z , p RR x , p RR y , p RR z ] ∈ R 12denote the position of the Right (R ), Left (L ) Front ( F) and Rear ( R) feet respectively.
C =[ � LF C , � RF C , � LR C , � RR C ] ∈ [0 , 1] 4 is the binary contact state of each foot, with � f C taking a value of 1 iffoot f is in contact with the ground and a value of zero otherwise.
Training the Jumping Policy
Let the high-level policy be a t = π θ ( s t , o t , a t − 1 ) where a t is the action and s t , o t denote the robot’sinternal state and the terrain observation respectively.
The action at previous time-step is fed as inputto encourage smoothness.
We use a deep neural network to represent π .
Observation Space
The proprioceptive state s t ∈ R 34 consists of the robot body height ( R ),orientation ( R 3 ), linear velocity ( R 3 ), and angular velocity ( R 3 ), as well as the joint positions( R 12 ) and velocities ( R 12 ).
The terrain observation o t is either a body-centered elevation mapo t =
E t ∈ R 48 × 15 or a depth image o t =
I t ∈ R 160 × 120 from a body-mounted camera.
Observations are normalized using the running mean and the standard deviation.
Action Space We train policies with either ﬁ xed or variable gait patterns.
With ﬁ xed gait, a t ∈ Rencodes the target body linear velocity and yaw velocity.
By setting the velocity, we are essentially modulating the target acceleration.
For computational ef ﬁ ciency, our low-level controller assumes that the target pitch and roll are near zero, and consequently, we exclude them from the high-level policy output.
This assumption does not prevent our system from making agile jumps.
For variable gaits, we additionally predict the contact schedule of the legs denoted by C ∈ [0 , 1] 4 .
As an example, the contact schedule for trot and pronk gaits corresponds to:
[Equation]
where d is the gait cycle duration.
In our experiments with ﬁ xed gaits, we set d = 10 .
For variablegaits, the contact schedule C ( a t ) is selected by the policy:
[Equation]
])] at time t where f C maps a discrete policy output to a contact state target.
There are 2 4 = 16 possible contactstates for the four feet, so the associated policy output makes a new choice among 16 categories ateach timestep in the fully gait-free case.
We also train gait-free policies with fewer contact stateoptions, such as the Variable Pronk which synchronizes the contacts of all feet.
Algorithm
Policies Modulating Whole-body Objectives 1: t ← 0 ; a − 1 ← 0 2: observe s 0 , o3: while not IS - TERMINAL ( s t ) do 4: sample a t ∼ π θ ( a t
| s t , o t , a t − 1 ) 5:
T t + H ← T ( a t ) 6:
TRACK - TRAJECTORY ( s t , T t : t + H ) 7: t
=
t
+8: observe s t , o t 9: end while
The action a t is converted into the desired whole-body trajectory for the next H time steps denoted as
T
[Equation]
where the key quantity adapted by the policy is ˙ p b ( a t )
=[ ˙ x
= a t [0] , ˙ y = a t [1] , ˙
z = a t [2] , ˙ α = 0 , ˙ β
= 0 , ˙ γ =a t [3]] , from which p b ( a t ) and ¨ p b ( a t ) are ﬁ xed for dynamic consistency.
p raibertf , ˙ p fraibert , ¨ p fraibert are foot targetssatisfying the Raibert Heuristic (see supplement).
Whole-body Trajectory Tracking is performed usingthe hybrid control scheme described in [24].
It is ahigh-frequency controller that solves a quadratic program(QP): q des = QP ( T des , T ) without access to terrain information.
It is composed of three controllersoperating hierarchically:
• A Model Predictive Controller (MPC) converts the future whole-body trajectory T t : t + H into target
ground reaction forces f t : t
+ H at contact for every foot at each timestep.
MPC operates at 40 Hz .
• A Whole-Body Impulse Controller (WBIC)
ﬁ nds the target position, velocity, and feedforwardtorque commands for all joints required to track the current step of the the whole-body trajectoryT t and desired ground reaction forces f t computed by the MPC.
WBIC operates at 500 Hz .
• A Proportional-Derivative Plus Feedforward Torque Controller takes as input a target position,target velocity, and feedforward torque command for each of the robot joints, as well as an observation of each joint’s current position and velocity.
It computes and actuates a resulting outputtorque for each motor at 40 kHz .
Rollout Procedure The iterative execution routine for our model-free planner and model-basedcontroller is given by Algorithm 1.
The low-level controller performs receding-horizon optimizationof contact forces over horizon H , with the assumption that the future contact and pose targets takeninto account for planning will not change.
This motivates our design choice in the high-level policyto select the trajectory target H timesteps into the future.
In our experiments, H = 10 , the policytimestep is 0 .
036 s, and the low-level controller timestep is 0 .
002 s .
Reward Function
The reward r t at time t is de ﬁ ned as:
[Equation]
The ﬁ rst term rewards the forward progress p bt,x − p bt − 1 ,x , where p bt,x is the projection of the 164 body frame position at time t onto the x -axis in the world frame.
The second term applies asoft safety constraint by penalizing when the body velocity v bt exceeds V thresh .
The third, fourth,and ﬁ fth terms incentivize stability by penalizing the roll, pitch, and yaw of the body, denotedas α bt , β bt .
The sixth term rewards smooth motion by penalizing the joint velocity ˙ q ; in training with adaptive contact schedule, we found this term critical to promote exploration of lowerfrequency gaits.
c 1 , c 2 , c 3 , c 4 , c 5 , c 6 are the coef ﬁ cients of each reward term.
In our experiments,c 1 = 1 . 0 , c 2 = 0 .
5 , c 3 = 0 .
02 , c 4 = 0 .
05 , c 5 = 0 .
15 , c 6 = 0 .
03 , V thresh = 1 .
m/s.
3.1 Neural Network Training
Network Architecture
The policy π θ ( a t | s t , o t , a t − 1 ) is modeled using a deep recurrent neural 173 network that includes a convolutional neural network (CNN) to process the high-dimensional terrainobservation o t .
The output features of the perception module are concatenated with proprioceptiveinputs s t , previous action a t − 1 , and a cyclic timing parameter [17] and passed through a sequence offully connected layers to output a probability distribution over the next action a t .
Figure 3 illustratesthe architecture of the policy or actor network; during training, we also use a critic network, in whichthe ﬁ nal layer of the actor is replaced by a value prediction head.
[Figure]
Initialization and Termination
For eachtraining episode, the robot is initialized in astanding pose on ﬂ at ground.
The locationsof gaps and their widths are randomized.
Anepisode terminates if any of three terminal conditions are met: (1) the body height is less than20 centimeters; (2) body roll or pitch exceeds0 .
7 radians; or (3) a foot is placed in a gap.
Themaximum episode length is 500 steps, equivalent to 25 seconds of simulated locomotion.
Policy Optimization The parameters of the neural network ( θ ) are optimized using Proximal Policy
Optimization (PPO) [25].
We use Adam optimizer [26] with learning rate 0 .
0003 and batch size 256 .
During training, 32 environments are simulated in parallel.
We ﬁ nd that policies converge within6000 training episodes, equivalent to 60 hours of simulated locomotion or 12 hours of computation.
Asymmetric-Information Behavioral Cloning Learning directly from depth images is challenging because a front-facing depth camera can only provide information about the terrain in front ofthe robot, not the terrain underneath its feet, making the contact-relevant terrain partially observed.
Furthermore, the depth image is dependent on the robot egomotion as well as the terrain shape, introducing noise.
To address the challenge of learning a policy from the noisy, partial observationsprovided by a body-mounted depth camera, we use a two-stage approach that ﬁ rst trains an expertpolicy ( π E ) from ground truth height map.
A second deployable policy ( π BC ) is trained from depthinputs to mimic the expert policy.
For this, we use a variant of Behavioral Cloning (BC) knownas DAgger [27] to minimize the KL-divergence between the output action distribution of the imitating agent π BC ( a | s ) and the expert π E ( a | s ) : min D KL � π E ( a | s ) || π BC ( a | s ) � .
Because the depthimages in our setup contain only a portion of the information in the heightmaps, it is necessary tointegrate depth data over time.
Therefore, we represent π BC as a recurrent neural network.
Priorworks have applied similar approaches to blind rough-terrain locomotion [1] and autonomous driving [20].
Evaluation reported in Table 1 indicates that this cloned policy matches the performanceof the expert trained from heightmaps, and substantially outperforms learning directly from depthimages.
Experimental Setup
Hardware : We use the MIT Mini-Cheetah [19] , a 9kg electrically-actuated quadruped that stands28cm tall with a body length of 38cm.
A front-mounted Intel RealSense D435 camera providesreal-time stereo depth data.
The robot is also equipped with an onboard computer [7] that supportsa hierarchical trajectory-tracking controller described in Section 2.
Data from the depth camera isprocessed by an offboard computer that communicates the output of the high-level policy to the robotvia an Ethernet cable.
Treating proprioceptive state estimation as an orthogonal research direction toour work, we use a motion capture system to obtain accurate measurements of the robot body state.
Simulator : We trained our vision-conditioned policy using the PyBullet [28] simulator.
In additionto simulating the robot dynamics, PyBullet simulates the frames of our mounted depth camera, calibrated from an accurate CAD model of our robot and from the sensor’s known intrinsic parameters.
Gap World Environment : To evaluate the ability of our system to dynamically traverse discontinuous terrains, we de ﬁ ne a test environment consisting of variable-width gaps and ﬂ at regions.
The dif ﬁ culty of traversing gap worlds depends on the proximity of gaps as well as gap width, withcloser and wider gaps presenting a greater challenge to the controller.
Our training dataset consists
� � � � �� �� �� �� �� �� �� �� �� �� �� �������������� ��� ��� ��� ��� ��� ��� � � � � � � � �� � � � � � � �� � �� � � � � � ���������� ����������� ����������� ������������ ���������� ����������� (a) � �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �������������� ��� ��� ��� ��� ��� ��� � � � � � � � �� � � � � � � �� � �� � � � � � ���� ����� ��������� (b)
��������������� � � � ������������������� ������������������������� � � � � � � � � �������� ������������� ���������������������������� of randomly generated gaps with uniform between and by
Policies Modulating Trajectory Generators (PMTG) [17] Baseline augments the action spaceof model-free RL using a parametric trajectory generator (TG) capable of producing cyclic legmotions.
Given a timing parameter ( t ) that cycles between 0 and 1 and trajectory parameters ( a )– stride frequency, length etc., TG outputs joint position targets q des = TG ( t, a ) .
The policy alsodirectly predicts residuals ( Δ q des ).
The output command is therefore q des + Δ q des .
Results
5.1 Dynamic Performance
By design, learning with a trajectory generator introduces constraints and biases into the resultingpolicies. This aids learning and enables robust behavior.
However, this yields a concern: are theconstraints and biases of the trajectory generator too rigid to accommodate useful locomotion strategies?
Or do they serve to guide learning effectively without hindering ﬁ nal performance?
Our resultsindicate the latter.
In this section, we evaluate the ﬂ exibility and performance of our integrated perception and control approach.
We ﬁ nd that our system is both high-performing under constraintsand ﬂ exible when constraints are removed.
Optimality Under Constraints We train our framework to cross gaps up to the theoretical limit forconstrained families of trotting and pronking gaits.
Figure 4a reports the performance of our methodfor adaptive ﬁ xed-gait gap crossing in simulation.
While the baseline ﬁ xed gaits without vision arecapable of sometimes crossing gaps by chance, our visually-guided approach succeeds at above 90%of gap crossing attempts up to the theoretical limits derived in the supplementary material.
Unconstrained Range of Motion
We relax all constraints on contact schedule and train a controllerwith a vision-adaptive contact schedule to cross wide gaps.
Figure 4b reports the performance ofour method for gait-adaptive gap crossing in simulation.
When trained with extremely wide (40-
[Table]
to 70-cm gaps), the visually informed policy learns to select a variable-bounding contact schedulewhich achieves superior performance to trotting and pronking for very large gaps.
However, we notethat the low-level controller may truly not support the motions generated in simulation for crossingsuch large gaps.
When we restrict the maximum gap size to 40cm or less, a variable-timing pronkinggait emerges in the gait-free controller.
Figure 5 illustrates the variable contact timings and velocitymodulation of the variable pronking controller in simulation.
Ease of Training Our method is capable of learning successful policies for multiple gaits and gapworld parameters with no specialized modi ﬁ cation.
In contrast, we found that the PMTG baselinewas highly sensitive to the tuning of the reward and trajectory generator.
We ﬁ rst tuned the trajectory generator, residual magnitudes, and reward function of PMTG for forward locomotion on ﬂ atground; details and videos of the baseline can be found in the supplementary material.
We foundthat these tuned parameters on ﬂ at ground were overly conservative for gap crossing tasks.
However, an action space with large residuals can signi ﬁ cantly override the prede ﬁ ned motion primitives,presenting an obstacle to learning realistic gap-crossing behaviors without any curriculum or specialized reward design [8].
Indeed, without any such special inclusions in our training, the baselinefailed to learn any gap-crossing behavior when the maximum gap width W max exceeded 15cm fortrotting or 25cm for pronking as well as when gap separation was reduced to 0.5m in simulation.
5.2 Vision and Behavioral Cloning
Performance Table 1 illustrates that behavioral cloning offers an advantage over learning directlyfrom depth images in many but not all cases.
We ﬁ nd that learning from heightmaps + BC consistently achieves higher reward than learning directly from depth images.
These results also demonstrate that the combination of behavioral cloning with a variable gait schedule is bene ﬁ cial, with thecloned Variable Pronk achieving the highest performance for wide gaps of any depth-image policy.
Recurrent Architecture We ablate the recurrent architecture of the cloned policy, and note thatpolicies with recurrent architecture consistently yield higher ﬁ nal performance than without, particularly for environments with larger gaps which require more dynamic motion (Table ?? ).
Thissuggests that the hidden state is helpful in forming a useful representation of unobserved terrainregions given the observation history. [Figure] � �
[Figure]
��������������������������������
5.3 Sim-to-Real Transfer
We deploy vision-adaptive locomotion controllers trained with Jumping from Pixels on the MIT
Mini Cheetah robot [19].
Figure 6 plots motion capture data from four deployments of the adaptivetrotting controller (left) and three deployments of the adaptive pronking controller (right) usingground-truth state information and heightmap input.
The relevant cross-section of the terrain surfaceis drawn in dark green.
The consistency of foot placements and visible adaptive avoidance of thegap verify that our method trained in simulation produces terrain-appropriate behaviors which cancross the sim-to-real gap.
Further, we successfully deploy student policies in fully real-time fashion,directly making use of depth images and an onboard state estimator. We report and analyze theseresults at https://sites.google.com/view/jumpingfrompixels.
Related Work
Model-free RL for locomotion is shown to bene ﬁ t from acting over low-level control loops ratherthan raw commands [29].
Previous work in simulation [11, 13, 9] has applied model-free reinforcement learning to traversal of discontinuous terrains in simulation.
[9] notably applied modelfree RL to the problem of crossing stepping stones with physically simulated characters, but thismethod did not use realistic perception or take measures to promote sim-to-real transfer.
Recentwork on ANYmal [30] learns a model-free policy to predict joint position targets for a PD controller, and demonstrates better energy ef ﬁ ciency and higher maximum velocity than comparablemodel-optimization-based controllers.
However, joint-space policies learned in simulation can stillbe unrobust and fail to cross the sim-to-real gap.
Reward shaping, system identi ﬁ cation, and domainrandomization were used in [30] to facilitate transfer to the real robot.
Model-based control for locomotion has achieved highly dynamic blind walking [31], running[24], and jumping over obstacles [15] using known quadruped whole-body and centroidal dynamics.
Other works have applied model-based control to terrain-aware navigation of a mapped environment,typically with complete information about the terrain [4, 32].
In general, control strategies based onknown models are high-performing and robust where the state is known and the model is suf ﬁ cientlyaccurate.
In contrast, model-free controllers excel at incorporating unstructured or partially observedstate information when large data is available.
Interfacing Model-based and Model-Free Methods .
A previous line of work has leveragedmodel-free perception for foothold selection.
[33] locally adapted foot placements to safe footholdspredicted by a CNN.
RLOC [6] similarly uses a learning-based online footstep planner in combination with a learning-modulated whole-body controller to perform terrain-aware locomotion.
Unlikeour method, [6] uses a complete terrain heightmap as observation, plans by targeting foot placements, and is limited to relatively conservative ﬁ xed walking and slow trotting gaits.
On the otherhand, concurrent work applies RL to modulate a model-based controller’s target command withoutperception.
[34, 35] demonstrated that using a model-free policy to choose contact schedules for areduced-order model leads to the emergence of ef ﬁ cient gait transitions during blind ﬂ at-ground locomotion.
[36] demonstrates the integration of a model-free high-level controller with a centroidaldynamics model.
This framework deployed with a ﬁ xed trotting gait is demonstrated to achieveﬂ at-ground and conservative terrain-aware locomotion.
Unlike our work, [36] does not demonstrategaits with ﬂ ight phases or plan from realistic terrain observations.
Conclusion and Discussion
We have presented a vision-based hierarchical control framework capable of traversing discontinuous terrain with gaps.
The combination of model-free high-level trajectory prediction and modelbased low-level trajectory tracking enables us to simultaneously achieve high performance and robustness.
Consequently, we demonstrate that the learnt behaviors cross the sim-to-real gap.
One aspect that prevents in-the-wild deployment is that the readings of onboard sensors for estimating the robot’s internal state are noisy and insuf ﬁ cient to plan high-precision foot placement forcrossing gaps.
To focus on transfer of visual inputs and dynamic control in this work, we made useof a motion capture system to measure the robot’s state.
We believe that improving on-board stateestimation by leveraging vision is a worthwhile, but a complementary direction of future research.