Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomization
Abstract
Developing robust vision-guided controllers for quadrupedal robots in complex environments, with various obstacles, dynamical surroundings and uneven terrains, is very challenging.
While Reinforcement Learning (RL) provides a promising paradigm for agile locomotion skills with vision inputs in simulation, it is still very challenging to deploy the RL policy in the real world.
Our key insight is that aside from the discrepancy in the domain gap, in visual appearance between the simulation and the real world, the latency from the control pipeline is also a major cause of difﬁculty.
In this paper, we propose Multi-Modal Delay Randomization (MMDR) to address this issue when training RL agents.
Speciﬁcally, we simulate the latency of real hardware by using past observations, sampled with randomized periods, for both proprioception and vision.
We train the RL policy for end-toend control in a physical simulator without any predeﬁned controller or reference motion, and directly deploy it on the real A1 quadruped robot running in the wild.
We evaluate our method in different outdoor environments with complex terrains and obstacles.
We demonstrate the robot can smoothly maneuver at a high speed, avoid the obstacles, and show signiﬁcant improvement over the baselines.
Our project page with videos is at https://mehooz.github.io/mmdr-wild/ .
Introduction
Developing a robust controller for the quadrupedal robot to traverse complex, wild environments (Carlo et al., 2018; Di Carlo et al., 2018) is a challenging and important problem in robotics.
Recent approaches show its wide range of applications such as delivery in the rugged terrains (Arena et al., 2006).
With the advancement in robot learning for navigation and manipulation (Mirowski et al., 2017; Levine et al., 2016), Reinforcement Learning (RL) and Imitation Learning have also brought crucial improvements in legged locomotion (Peng et al., 2020; Hwangbo et al., 2019; Luo et al., 2020;
Da et al., 2020) with proprioceptive state only.
Speciﬁcally, we have witnessed impressive performance in the wild using RL incorporating robustness and adaptive ability (Lee et al., 2020;
Kumar et al., 2021).
However, only given proprioceptive information, the blind RL controller addresses challenging scenarios by training with large-scale randomized environment parameters (Lee et al., 2020; Xie et al., 2020c).
While this technique delivers promising results for maneuvering on uneven and unknownmaterial ground, it’s insufﬁcient for more complicated tasks like avoiding obstacles that are hard to step over, or estimating accurate foot placement positions for safety.
To overcome these challenges, the robot needs to perceive surroundings and plan the path to move.
Thus recent works (Escontrela et al., 2020; Jain et al., 2020; Yang et al., 2021) introduce visual perception to the robot by using a [Table] depth camera or LiDAR.
Moreover, proprioceptive states only provide information about current or prior terrain, while vision provides information about near-future topography for developing better stepping skills.
By combining the information from both the proprioceptive states and visual inputs, the robot can maneuver in intricate, wild scenarios as Figure 1 demonstrates.
Nevertheless, learning with visual inputs introduces new challenges in Sim2Real transfer for RL.
While previous methods focus on narrowing the domain gap in observations using domain randomization and augmentation techniques (Tobin et al., 2017; Laskin et al., 2020b; Hansen & Wang, 2021), the latency in hardware has not been well studied.
Latency exists in all parts of the real robot control system.
The sensors’ transmission, the control algorithm’s computation, and the robot’s response all introduce latency for perception and control.
This problem is further magniﬁed when the control policy requires both high-dimensional vision and proprioceptive state inputs.
First, the deﬁciency of on-board computing resources in a quadrupedal robot causes severe latency in processing the visual inputs; by the time the action is computed from the visual policy, the robot could have already walked a few steps ahead.
Thus the policy will need to be robust to the visual delay and still perform correct decision making.
Second, the proprioceptive state inputs are still provided to the network at a high frequency, which causes misalignment between the proprioceptive state and the visual observation.
Solving the two problems above is the key to transfer vision-based RL policy trained in simulation to the real robot.
In this paper, we propose Multi-Modal Delay Randomization (MMDR) , an asynchronous delay randomization technique for improving the robustness of RL policies in real robots.
Our RL policy takes both proprioceptive states and observed depth images as inputs, and outputs the target angle for each robot joint in an end-to-end manner.
During training, instead of forwarding both observations with ﬁxed temporal intervals, we provide randomized asynchronous multi-modal inputs to the network.
Speciﬁcally, we maintain two buffers online for two types of inputs (one for visual observation, another one for proprioceptive states).
Each buffer will store a stream from one type of recent observations.
We independently sample a sub-sequence of observations from each buffer, and forward both types of inputs to the policy network.
In this way, we can simulate the time discrepancy between proprioceptive and visual signals in the real robot.
MMDR simulates the accumulated latency of the whole system across all parts at once.
The quadrupedal robot trained with RL using MMDR is more robust to the delayed visual inputs and the misalignment between two types of observations for locomotion control.
We experiment with our proposed method on arduous, in-the-wild maneuvering tasks with a Unitree
A1 quadruped robot (Unitree, 2018) as shown in Figure 1.
These environments include static or moving obstacles of different sizes and shapes, uneven terrains, and changing lighting conditions.
Results show that applying MMDR during training signiﬁcantly improves the performance and robustness for real-world deployment of the learned RL policy.
Our policy can even generalize to environments with unseen or moving obstacles, where the A1 robot plans and maneuvers smoothly through the obstacles at a high moving speed.
We highlight our main contributions as follows:
• We present Multi-Modal Delay Randomization (MMDR), which models the latency of multimodal inputs for Sim2Real RL policy transferred for vision-guided quadrupedal locomotion control.
• To the best of our knowledge, this is the ﬁrst work that allows an end-to-end RL trained quadrupedal robot to maneuver in the wild under visual guidance.
Related Work
Sim-to-real with Domain Randomization.
To address the sim2real gap for policy deployment in the real-world, domain randomization introduces variability to different components of the simulated environments during training, including variant physical parameters (Mordatch et al., 2015; Peng et al., 2018b; Tan et al., 2018; Li et al., 2021), visual attributes (Tobin et al., 2017; Pinto et al., 2018;
Andrychowicz et al., 2020), and perturbations (Andrychowicz et al., 2020).
Thanks to its simplicity and effectiveness, domain randomization has been widely used in learning legged robot control to complete real-world tasks with diverse locomotion skills (Xie et al., 2020b,a; Peng et al., 2020;
Kumar et al., 2021; Lee et al., 2020).
However, how to better apply domain randomization to address the sim2real gap for learning vision-guided legged locomotion in the real world (like the multi-modal latency) has not been well-explored yet.
In this work, we propose multi-modal delay randomization to model the sim2real gap beyond the observation discrepancies.
Learning-based Legged Locomotion.
Controlling a legged robot has been studied by the robotics community for a long time.
Control theory and trajectory optimization approaches have shown great results on legged locomotion control (Gehring et al., 2013;
Carlo et al., 2018; Di Carlo et al., 2018; Carius et al., 2019; Ding et al., 2019; Bledt & Kim, 2020).
However, these methods require in-depth knowledge about the environment and substantial manual efforts for parameter tuning.
As an alternative, reinforcement learning provides an autonomous learning paradigm for legged locomotion skills from self-exploration in complex environments (Kohl & Stone, 2004; Luo et al., 2020; Peng et al., 2018a, 2020; Tan et al., 2018; Hwangbo et al., 2019; Lee et al., 2020;
Xie et al., 2020c; Iscen et al., 2018).
Despite the successful application of RL on legged robots, most RL approaches depend only on proprioceptive input.
To utilize the rich visual information, recent works introduce visual observation to legged locomotion learning in end-to-end (Escontrela et al., 2020; Yang et al., 2021) or hierarchical manners (Jain et al., 2020).
But visual-guided locomotion policies learned by these methods are still limited to simple and indoor environments.
With our method, we can successfully apply the learned policies in the wild directly.
Reinforcement Learning with Vision.
To enrich the perception of the agent, many works have studied RL with visual input in navigation (Sax et al., 2018; Faust et al., 2018;
Wijmans et al., 2020), locomotion control (Laskin et al., 2020a; Hansen & Wang, 2021; Yarats et al., 2021;
Hansen et al., 2021), and manipulation (Jain et al., 2019; Levine et al., 2016, 2018).
To generalize visual perception in variant tasks, Hansen et al.
(2021) extend representation learning for out-of-distribution environments.
Our work is most related to previous work leveraging multi-modal input for locomotion control (Heess et al., 2017; Merel et al., 2020; Jain et al., 2020; Escontrela et al., 2020; Yang et al., 2021).
Instead of running the robot in simulators or labs, we show the vision-guided quadruped robot can run smoothly and safely in the wild.
Learning with Delay.
The latency of sensing and control recently draws attention from robotic learning community.
Li et al.
(2020) introduces the concept of streaming perception into computer vision, unlike previous work focusing on ofﬂine efﬁcient (fast) inference (Redmon et al., 2016;
Liu et al., 2016).
In RL area, continuous-time RL (Doya, 2000) and MDP with delay (Ramstedt & Pal, 2019; Katsikopoulos & Engelbrecht, 2003) has been proposed to address constant system latency in simulation.
Beyond simulation, the continuous-time algorithms are developed for concurrent control (Andersen et al., 2015;
Xiao et al., 2020; Lutter et al., 2021) in real-world control tasks.
For modeling delay, previous works (Tan et al., 2018;
Yang et al., 2020) randomize the observation latency for proprioceptive state-only quadrupedal locomotion policies.
When using information from different sources, multi-modal synchronization also brings new difﬁculties on stable control (Olson, 2010; Liu et al., 2021).
In this work, we address the delay problem in learning-based vision-guided control with a simple yet effective randomization technique.
Delay Randomization in a Multi-Modal Control System
The Markov property is fundamental to most RL algorithms rely on (Puterman, 1994).
However, the latency in real robot hardware jeopardizes the Markov property.
This makes it difﬁcult to train policies deployed in the real world.
Worse still, in a multi-modal robot system that perceives information from multiple modalities, e.g. vision and proprioception, the latency of different sensors varies greatly.
Thus, multiple sensor data fusion will become a non-trivial task in the real world, compared to the synchronized settings that are commonly used in RL literature (Ibarz et al., 2021), which do not face these difﬁculties.
To tackle this challenge, we propose Multi-Modal Delay Randomization (MMDR), which explicitly models the latency from different sources in a real robot system, as shown in Figure 2.
In this following section, we will ﬁrst introduce the latency of a real robot in Section3.1, then discuss how we mitigate the latency issue via our MMDR.
3.1 Latency in Real Robot RL
[Figure]
In common RL settings, an agent perceives observations from the environment, computes the next step action based on the policy, then forwards this action to the environment.
Most physical simulators (Coumans, 2015; Todorov et al., 2012; Koenig & Howard, 2004; Makoviychuk et al., 2021) assume a synchronized observationaction loop.
In this synchronized setting, the environment will wait for the next action and stay unchanged before that.
However, the real world is asynchronous and never waits for the policy.
Any latency in the observation-action loop will inﬂuence the execution of the action, which in turn lowers the task performance.
Figure 2 illustrates delays in multiple stages: (i) Transmission delay to receive sensory readings; (ii) Processing delay to compute the action from an observation, commonly the time required by the neural network; (iii) Actuation delay to execute the action via the motors.
Among all three types of delays, the processing delay is the most signiﬁcant.
With a battery-based power supply and limited memory, the computational resources on-board often can not afford instant network inference, especially for high-dimensional visual input.
By the time the control signals are applied, the robot’s surroundings have already changed.
3.2 Multi-Modal Delay Randomization
[Figure]
To simulate the same control ﬂow as in the real world, MMDR provides randomized latency and asynchronous multi-modal observation in the simulation.
We randomize the sampling of the proprioceptive state and visual observation separately.
This allows us to utilize the domainspeciﬁc characteristics and simulate independent latency for different modalities.
We represent the proprioceptive state as an 84D vector, where each digit of the vector has speciﬁc physical meaning.
To maintain the ﬁdelity of the simulation for RL, the simulation frequency is generally several times higher than the control frequency of the robot.
Our method reads the proprioceptive state at every simulation step and uses a queue with a ﬁxed length to store historical observations from the near past.
We also assume the proprioceptive state is changing smoothly in the real world.
Therefore, we sample a proprioceptive delay for each episode.
Then we use linear interpolation to calculate the delayed observation from two adjacent states in the entire buffer, with the sampled delay.
We represent the visual observation as a four-stacked depth image, in order to maintain temporal information.
A straightforward way to simulate latency for visual observations is to use the same randomized delay as for the proprioceptive state during training.
However, in the real world, depth cameras can provide much fewer observations (around 30 Hz) compared to proprioceptive sensors (around 1K Hz).
In this case, the transition from one frame to the next is no longer smooth.
To simulate the perception latency for visual observation at a lower frequency, we obtain the simulated visual observation at every control step and store the near past frames into a queue.
As illustrated in
Figure 3, we store the most recent 4 k depth maps as our visual observation buffer, split the whole visual observation buffer into four sub-buffers, then sample one depth map from each sub-buffer to create the visual input with randomized latency.
Though there are multiple kinds of latency as we discussed in 3.1, what really inﬂuences the real-world deployment is the accumulated latency across the different stages.
Therefore, the separated randomized latency for proprioceptive state and visual observation is enough for the policy training.
Reinforcement Learning in Simulation
With our MMDR, we are able to simulate the real-world latency in simulators.
We train locomotion polices in the simulation using RL, under different settings for real-world deployment and comparison.
4.1 Simulation Environment
We perform our simulation using PyBullet (Coumans & Bai, 2016–2021).
We train agents to control a Unitree A1(Unitree, 2018) robot to maneuver in an environment with obstacles and a complex terrain.
The simulated obstacles are cuboid rigid bodies with random positions and shapes, which remain static throughout the episode.
To force the robot to learn how to walk on natural, uneven grounds, we create complex terrain with random height ﬁelds, as shown in Figure 4a.
In all of the experiments, we use the same observation space, action space and reward function across all environments.
Observation Space.
The observation of the robot is made up of proprioceptive state and visual observation.
The proprioceptive state consists of (i) 12D robot joint rotation, (ii) 4D IMU sensor reading (angle and angular velocity of roll and pitch), (iii) 12D last action executed by the policy.
The proprioceptive input is an 84D vector containing three proprioceptive states.
The visual input consists of four stacked depth images of shape 64 × 64 .
All the depth images come from the depth camera mounted on the head of the robot.
To constrain the scale of the visual observation, depth values in visual observation are clipped to [0 . 3 , 10] m.
Action Space.
The action space for the policy is the target joint angle for each joint of the robot.
Target angles are converted to motor torques using a default PD controller.
Rewards.
In general, our reward function encourages the robot to safely move forward (along the x -axis in simulation) with a target speed (0.35m/s) while minimizing the energy cost.
Speciﬁcally, the reward function is given by:
R =
α moving R moving + α safe R safe + α energy R energy where α moving = 1 , α safe = 0 .
005 , α energy = 0 .
The moving reward contains two terms.
One encourages the robot to move forward at target speed and the other penalizes the speed along the z-axis to keep the robot walking smoothly.
The safe reward is 1.0 for each step until the robot falls, in which case the episode terminates.
The energy reward penalizes the energy consumption represented by the square of motor torques.
4.2 Network Architecture and Training Details.
We train all methods with 10M samples.
We use PPOp Schulman et al.
(2017) for policy training and utilize the generalized advantage estimator (GAE) (Schulman et al., 2016) to stabilize the training process (Peng et al., 2018a, 2020).
We use a batch-size of 16384 and split it into 16 mini-batches.
We use
Adam optimizer (Kingma & Ba, 2015) with learning rate of 1e-4 for both policy and value networks.
[Figure]
In our experiment, the policy network and value network share the same architecture as Figure 5.
We use a 2-Layer MLP encoder for proprioceptive input and a 3-Layer CNN for visual input to get encoded features from both modalities.
The encoded features from both modalities are 256D vectors.
We then concatenate the encoded features to get a uniﬁed feature and feed it into the additional 2-Layer MLP to get the action distribution or value prediction.
For better sample efﬁciency and stable training, the CNN for visual input is shared by the policy and the value function to capture a consistent visual representation.
4.3 Domain Randomization
[Table]
Besides the multi-modal latency, we leverage domain randomization on various physical parameters to narrow the sim2real gap in the observation space.
We sample a group of values within speciﬁc ranges at the beginning of each episode.
The randomized parameters and the corresponding ranges are listed in Table 1.
Ambient light in the real world and the stereo nature of a depth camera result in occlusion in the depth image.
Therefore, the original depth estimation from the camera is not available in certain visual regions of the depth map.
We ﬁll in all the missing values with the maximum depth (10m).
As shown in Figure 4b, the shape of obstacles breaks because of these missing values, which makes the deployment of the visual-guided locomotion policy even more challenging.
To simulate this phenomenon, we randomly sample 3-30 pixels in each depth map, and set the value of these pixels to the maximum depth.
The modiﬁed depth map are visualized in Figure 4a.
4.4 Baselines
We compare MMDR with two baselines both in simulation and the real world.
The No-Delay baseline is trained without delay randomization, and stacks the most recent 4 depth maps as visual input.
The Frame-Extract baseline stores the most recent 4 k frames and stacks the ﬁrst frames from every consecutive k frames as visual input to provide input with more temporal information.
[Figure] As shown in Figure 6, our method uses the same amount of temporal information as the FrameExtract baseline, but models the real-world latency by performing random selection.
Experimental Results and Analysis
In our experiments, we evaluate the performance of policies using two metrics: (i) Moving Distance : the distance in the forward direction covered by the robot; (ii) Collision Steps : the number of robot steps during which the robot is in contact with an obstacle.
The moving distance represents stability of the gait while the collision steps reﬂects safety.
Please refer to our project page and supplementary video for visualizations.
5.1 Results in Simulation
[Figure]
All methods show similar sample efﬁciency in the environment that it’s trained on, even if MMDR learns in a noisy environment.
When evaluated in environment with random delay, our MMDR outperform all other baselines.
We evaluate all methods in an environment with random delays varying from [0.04, 0.12]s.
Aside from the static obstacles, we also compare the policies in environment with moving obstacles to evaluate the generalization ability in simulation.
Though MMDR introduces more uncertainty into the observation, the robot maintains comparable learning efﬁciency as shown in Figure 7.
When tested in the same scenario with random delays, the policy learned with
MMDR moves further with less collisions.
This demonstrates the potential to better adapt to the real world.
To test the generalization ability further, we include a modiﬁed scenario in which the obstacles will continuously move at a random speed and direction.
Such a dynamic environment requires more accurate perception, so a slight delay may bring a very large impact like crashing.
Table 2 shows our superiority on both traversing skill and safety, tested against an unseen dynamic environment.
MMDR signiﬁcantly increases the forward distance by nearly 100% , and reduces the Collision Steps by 475% to No-Delay and 340% to Frame-Extract.
We conjecture it is because asynchronously randomizing multi-modal delays provides data-augmentation for relative position and speed changes of surroundings.
This allows the policy to be more adaptable to a dynamic environment during testing.
Ablation.
We add two ablation studies to show the substance of vision and random selection.
First, we train the blind policy in the same environment without visual input.
We use a 4-Layer MLP for training, corresponding to State-Only in Figure 7.
The State-Only agent learns almost nothing for such a complex task.
Second, like how we model the proprioceptive state delay, we sample a delay t d for each episode and use linear interpolation to calculate visual observation from two adjacent observations around t − t d at each step t (Interpolation in Figure 7).
We compare MMDR with this smooth interpolation to show discrete sampling is more appropriate for vision.
MMDR consistently outperforms them during training stages.
We also ﬁnd that though Interpolation does not perform well compared to both baselines during training, when evaluated on environment with moving obstacles, it generalizes better.
It shows that when temporal information is crucial for decision-making, it’s essential to take latency into consideration.
We stop the two methods in simulator rather than real-world deployment due to the disadvantage in training phase. [Figure] [Table]
5.2 Results in Real World
Given the performance of different methods in simulation, we conduct real-world experiments with these three methods: Ours, No-Delay, and Frame-Extract.
Robot Setup.
In the real world, we perform evaluations using the Unitree A1 Robot (Unitree, 2018), a low-cost quadruped robot with 18 links and 12 degree of freedoms (3 for each leg).
Our policy computes the target joint angles at 25 Hz, while the underlying PD controller computes the motor torques at 400 Hz.
Kp and Kd are 40 and 0.6 respectively.
We ﬁrst evaluate all the methods in scenarios similar to the training environments.
We launchtrials on 3 seeds for each setting.
The evaluation metric used in deployment is Collision Count , i.e. number of times robot hits the obstacle.
This differs from Collision Steps used in simulation because in outdoors environments it is dangerous to let the robot continuously collide with objects.
We place differently-sized boxes on a sloped lawn with different density.
We refer to these environments as
Sparse box placement (Box.)
(as in Figure 8(a)) and Dense box placement (Dense Box.)
(as in
Figure 8(b)).
In these two environments, as demonstrated in Table 3, MMDR excels baselines by a large margin.
MMDR moves twice as far as the other baselines do without colliding into any boxes in Sparse box placement , and maintains a similar performance when the obstacles become much denser.
In contrast, Frame-Extract performs poorly in Sparse box placement (Box.)
and the performance becomes worse as the density of boxes increases.
We speculate that using visual observation covering longer time span makes the policy become more sensitive to the latency in the real world.
This phenomenon suggests that modeling the real world latency is essential.
Beyond static boxes on a sloped lawn, we deploy policies in two more challenging scenarios: (i) Moving box (Moving Box.) :
the box is moving slowly in the same sloped lawn as previous environments as Figure 8(c) shows; (ii) Forest covered with branches and fallen leaves (Forest.)
as Figure 8(d) shows.
These two tasks are more challenging than the static box environments, for the diverse lighting condition, complex terrains, and dynamic obstacles.
We ﬁnd that in the Moving Box environment, the improvement our method obtained over the baselines becomes larger.
We conjecture that this happens because when the environment is dynamical, the latency in the multi-modal control system becomes more inﬂuential.
So it gets harder to predict the movement of the obstacles and act accordingly with latency.
This result also indicate that it’s necessary to simulate latency for vision-guided locomotion policy.
For the Forest environment, the challenges mostly come from the unseen obstacles (trees), the branches, and leaves on the ground.
In Forest environment, the policy trained with MMDR still obtains large improvement over the baselines in the moving distance, while having similar collision results.
Stepping on branches or fallen leafs can cause unexpected state transition that don’t happen Box.
Dense
[Table]
[Figure]
to analyze the detailed locomotion. Lines denote stepping to the ground. During the left leg continuously steps to the twisting adjusting direction, robot turns to run faster in bouncing in the simulation. Thanks to our asynchronous randomization for multi-modal input, our method can learn policies that are more robust to unseen transitions.
To study the learned locomotion skills in more details, we visualize the positive (down) torque pattern during testing in Forest environment as shown in Figure 9.
We selected two essential frames when the robot is turning around a tree.
In Figure 9(a), the front left leg pushes the ground longer for power and the two right legs take small steps to change the body orientation.
Once the robot faces to correct direction, it uses the bouncing gait for acceleration to pass the tree (Figure 9(b)).
Such a complex locomotion sequence demonstrates that robot combines multi-modal information well to adapt to the complex environments.
Conclusion
Latency, as a crucial reality gap, exists in many parts of the robot control pipeline.
We propose the
Multi-modal Delay Randomization technique to address the latency issues in real-world deployment for vision-guided quadruped locomotion control.
Our approach shows great advantages on two realistic metrics in both the simulator and real world.
It also improves generalization and adaptation ability in unseen scenarios so as to help the robot pass through arbitrary barriers in real world.
These results suggest MMDR can be universally applied not only to legged locomotion, but potentially also to many other visual robotic control tasks.