Towards Real Robot Learning in the Wild: A Case Study in Bipedal Locomotion
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Algorithms for self-learning systems have made considerable progressin recent years, yet safety concerns and the need for additional instrumentationhave so far largely limited learning experiments with real robots to well controlled lab settings.
In this paper, we demonstrate how a small bipedal robot canautonomously learn to walk with minimal human intervention and with minimalinstrumentation of the environment.
We employ data-efﬁcient off-policy deep reinforcement learning to learn to walk end-to-end, from scratch, using rewards thatare computed exclusively from proprioceptive sensing.
To allow the robot to autonomously adapt its behaviour to its environment, we additionally provide theagent with raw RGB camera images as input.
By deploying two robots in different geographic locations while sharing data in a distributed learning setup, weachieve higher throughput and greater diversity of the training data.
Our learning experiments constitute a step towards the long-term vision of learning “in thewild” for legged robots, and, to our knowledge, represent the ﬁrst demonstrationof learning a deep neural network controller for bipedal locomotion directly onhardware.
Introduction
Enabling robots to learn and adapt completely autonomously only through interaction with theirenvironment is one of the long-standing goals of AI and robotics.
The idea that robots might one daylearn and cohabit outside controlled laboratory environments, taking full advantage of the richnessof the real world, is appealing especially for mobile robots.
However, even though algorithms forself-learning systems have made considerable progress in recent years in terms of data efﬁciency androbustness (e g. [1, 2]), concerns around safety and the need for instrumentation (for instance, forreward computation or resets) have so far largely limited experiments with real robots to controlledlab settings [3, 4] or to sim-to-real transfer [5, 6, 7].
While transfer of behaviours from simulation to real can sidestep the challenges around data efﬁ- ciency and safety, it requires signiﬁcant engineering effort in the design of high ﬁdelity simulationsthat represent not only the correct dynamics of the robot but also offer increasingly complex andrich scenarios.
More importantly, learning only in simulation exposes the agent to the sim-to-realgap, leaving questions around adaptation to unfamiliar situations after deployment.
Despite ﬁrst successful applications (e.g. [8]), learning from scratch and “in the wild” is oftenthought of as an infeasible method for real legged robots.
In this paper, we show that using stateof-the-art model-free deep reinforcement learning (RL) and intrinsically deﬁned rewards, we canlearn to walk and avoid walls autonomously in uncontrolled environments.
In particular, we showhow small humanoid robots can learn to walk and interact with their environment in a learning sce- nario outside of the laboratory, relying only on on-board sensors and limited prior knowledge of the hardware.
The robot uses proprioceptive sensing consisting of an inertial measurement unit (IMU) and jointposition encoders together with egocentric RGB vision to perceive its own body and the environ-
[Figure]
t
[Figure]
t
[Figure]
ment.
Rewards and termination signals are derived from the proprioceptive sensors only.
We employa data-efﬁcient off-policy algorithm [1] that allows for distributed training with a variable numberof actors.
The learning setup does not rely on additional external hardware for state estimation orresets.
This allows us to train multiple robots simultaneously, outside of the controlled laboratoryenvironment, in different geographic locations, exposing the robot to a range of environments withvarying shape, appearance, and lighting conditions; see Figure 1 (right).
Our results show that such an intrinsic learning signal designed to promote forward movement (combined with the sensory capabilities of the robot and the partly structured environment) is sufﬁcientto learn bipedal gaits and simple wall avoiding behavior requiring only a minimal amount of priorknowledge, see Figure 1 (left) and supplementary video.
The distributed training with multiplerobots improves data throughput and thus the complexity of the behavior that can be learned.
Overall, the results provide a glimpse of the possibility of real-world learning for legged robots.
Our key contributions are: 1.
First demonstration of learning a deep non-parametric policy forunconstrained bipedal locomotion on hardware; 2.
A data-efﬁcient autonomous learning setup forlegged robots that can be easily deployed in different environments and run on multiple robots simultaneously; 3.
Learned vision guided legged locomotion with simple obstacle avoidance capabilities.
Related Work
Conventional approaches to legged locomotion typically rely on models and optimization [9, 10] andare concerned with concepts of gait stability, such as zero moment point [11] and limit cycles [12].
While optimization based methods can yield highly effective solutions, they are laborious to implement, require rigorous domain understanding, and can struggle to generalize to diverse real-worldenvironments and high-dimensional input modalities, such as vision.
Learning and data-driven approaches hold the promise to circumvent these limitation by avoidingthe need for accurate modelling.
Deep reinforcement learning for locomotion has been studiedextensively in simulation [13, 14], and recent papers have shown progress in sim-to-real transferfor both quadrupeds [15, 7, 16] and bipeds [17, 6, 5].
Compared to these works, we largely lift theassumption of the availability of a simulation model.
Closest to our approach is the line of work where learning is directly applied on hardware.
Unlikeours, these methods consider either a constrained, hand-designed gait parameterization [18, 19, 20,21] or apply unconstrained deep reinforcement learning to an inherently stable [22] or a quadrupedalrobot [3, 4].
None of these works consider an unconstrained humanoid robot or vision observations,and, unlike our approach, they require external instrumentation such as motion capture or othermeans to quantify the obtained gait.
Methods
3.1 Generalizable Instrumentation and Learning Autonomy
We aim to create a learning setup for a real robot to learn to walk in different uncontrolled environments (e.g. households).
This means that we cannot rely on external sensing such as motion capturesystems and we cannot use reset mechanisms such as a gantry.
Moreover, the setup has to be able todeal with natural, possibly cluttered, and irregularly shaped environments, and with variable naturaland artiﬁcial lighting.
In this paper, we focus on a simpler instantiation of this setting in which the robot is tasked tomaintain an upright position, and to move forward as fast as possible in an area surrounded by walls.
This forces the robot to use its legs for locomotion while avoiding the walls.
Access to cameraobservations enables the robot to look ahead and adapt its behavior to its surroundings and therebyto deal with a variety of environments.
For generality and robustness reasons, we do not employ any(possibly brittle) image pre-processing steps, e.g. for wall detection.
Instead, we learn the mappingfrom the camera pixels to the control signal end-to-end.
The design of our rewards and episode termination conditions is based on [8].
The reward is generated when the feet exhibit a positive forward velocity in the robot’s body coordinate frame.
To assessthis, the velocity of the robot body is computed by assuming that the lowest foot is in contact withthe ground and the contact point is stationary (no-slip condition).
While these assumptions may notalways hold, Hafner et al. have shown experimentally that this reward may still be sufﬁcient to learnforward motion when combined with a reward for keeping the torso upright.
This uprightness is inturn estimated from a gravity vector obtained via the IMU’s internal orientation estimate (based ona complementary ﬁlter).
Given that the velocity reward encourages forward movement in the robocentric coordinate frame, there is a risk of promoting walking in small circles.
To avoid this, wepenalize large changes in the heading direction, again based on the IMU’s orientation estimate.
Thethree reward terms (velocity, uprightness, and no turning) are linearly combined to form the ﬁnalreward.
Episodes are terminated whenever the robot’s body inclination exceeds a certain threshold(see the supplementary material for more details about the reward).
Finally, we make use of a simple reset mechanism to allow autonomous learning: At the end ofeach episode, if the robot has fallen (as inferred from the IMU orientation estimate), we use a robustpre-programmed feed-forward controller to stand up.
On rare occasions multiple attempts may berequired.
The robot does not need to be moved and simply continues from the location where itended the previous episode.
3.2 Robot Platform
We focus on the off-the-shelf OP3 bipedal robot platform [23], or its simulated analog modelled in
MuJoCo [24].
This platform is built of 20 Dynamixel servo motors (XM430-W350-R) that we usein position control mode.
All controllers use 20Hz control frequency.
In real robot experiments, allcontrollers run directly on the onboard Intel NUC i3 computer using ROS [25] to send commands tothe actuators.
While the low weight and low cost of the robot are advantageous for real robot learningexperiments, the low actuator and sensory speciﬁcations increase the difﬁculty of the learning task.
3.3 Distributed Data Efﬁcient Reinforcement Learning
The input to our policy consists of proprioceptive observations listed in Table 1, and the two mostrecent RGB camera observations down-sampled to 64x64 resolution.
The output of the policy is adistribution over 20 dimensional actions which represent position commands for the robot’s
[Table] Our policies are trained using an asynchronous distributed version of the actor-critic MPO algorithm [1, 26] which uses a distributional critic [27] (DMPO), and which has shown to be effective inaddressing high-dimensional tasks with continuous action space.
The algorithm is implemented in
JAX [28] using the code 1 released as part of [29] as a reference. All hyperparameters are listed inthe supplementary material.
The robots’ onboard computers run actor processes which synchronize the policy’s neural networkweights with a remote learner process at the beginning of each episode.
After an episode of executing the policy on the robot, the actor process sends all data experienced in that episode to a remotereplay buffer implemented using the Reverb library [30].
The learner process continuously readsdata from the replay buffer, and updates the network weights via gradient descent on the DMPOactor and critic losses.
In order to prevent overﬁtting, the learner process is blocked when the totalnumber of gradient steps exceeds some threshold times the number of control steps executed by allrobots.
In our experiments, this threshold is 1 / 4 .
The remote learner process runs in the cloud.
Communication with the robots is implemented using the Launchpad API [31].
One advantage of this setup is that we can scale from one robot to multiple robots in a practicaland application-relevant way.
But it also shifts the computational load away from the robot and thelocation of the experiment.
This in principle allows to learn with robots having weaker onboardcomputing power, basically requiring only an internet connection to allow learning experiments.
fed to separate policy and (action-conditioned) critic networks; see text for more details on tensor shapes and losses.
3.4 Network Architecture
The network architecture is depicted in Figure 2 and consists of an image encoder shared betweenthe policy and critic networks.
It processes the latest two RGB images received from the robot’scamera.
The outputs of the encoder are concatenated with proprioceptive observations (see Table 1)and processed by separate policy and critic networks.
The shared encoder is trained using gradientsfrom critic losses only.
The output of the critic is 51-dimensional corresponding to the logits pa- rameterizing the distribution of future returns [27].
The policy outputs the mean and the standarddeviation of a diagonal Gaussian distribution from which we sample the actions for the actuators.
Experiments
For safety reasons and to ensure autonomous learning, the workspace in our experiments consistsof a rectangular court delimited by foam walls.
This prevents the robot from walking away, and, incombination with stand-up resets, represents a very effective way to minimize human intervention.
In some sense, this is similar to the play pens used for babies when they start learning to walk.
Exceptfor this, all other environmental conditions are uncontrolled, e.g. we train in different daylightconditions, with various furniture items present around the workspaces, and different court sizes(see Figure 1).
To avoid entanglement with the power cable, real robots are powered by a batterypack which needs to be changed roughly every 30 min.
As we train from scratch, the policy starts purely random, causing the robots to fall very often,especially at the beginning of training.
To reduce the risk of damaging the hardware, we equip therobots with a 3D-printed body armour covering the front and back of the torso.
In addition, welimit the range of the joints to prevent self-collisions that might damage individual actuators or otherhardware parts.
We also apply an exponential action ﬁlter on the actions sampled from the policybefore sending the reference position commands to the actuators.
History of these ﬁltered actions isincluded in the policy’s inputs (see Table 1) in order to mitigate partial observability due to delays.
4.1 Simulation Experiments
A simulation environment is used to make preliminary tests ahead of the real world experiments.
Aside from using it to tune network architecture and hyperparameters, we want to know if thesample complexity is within a reasonable range and whether vision can be used as viable alternativeto instrumented state estimation.
We roughly replicate the real world setup – a 4 × 4 m court surrounded by walls – in simulation (see Figure 3).
We randomize the textures for the ﬂoor and walls in every episode to roughly mimic thediversity of visual inputs in the real world setup where the learner receives trajectories from actorsrunning in different locations and under various lighting conditions.
The position and orientationof the robot are randomised at the beginning of every episode.
We do not make additional effortsto make this simulated environment more realistic as we do not aim for sim-to-real transfer (see Section 1).
Given the lack of rich textures, realistic lighting conditions, and motion blur it would beunlikely for a policy to transfer (see Fig.
6 in supplementary). Figure 4 depicts the training curves of three different agents: 1) a vision agent using raw images,2) a pose agent having access to ground-truth robot pose (position and orientation), and 3) a blindagent without additional non-proprioceptive measurements.
All are trained in an open square courtbut tested either with or without an additional disturbing wall (see Figure 3).
Starting with the blindagent, we can observe that the attained performance is only marginally affected by the presence ofan additional wall.
Looking at the learned policy shows how the robot uses its arm to detect thepresence of a wall which is a behavior that is fairly robust to changes of wall conﬁguration (seesupplementary material).
When having access to ground-truth pose ( pose agent), the robot learns very quickly to walk withinthe court.
But when adding a wall at test time the performance strongly degrades as the robotcollides with the wall and is unable to recover.
This is improved by the proposed vision agent.
While performing almost on par with the pose agent in the simple square court, it is also ableto naturally generalise when the wall conﬁguration is altered.
This shows that there is a signiﬁcantbeneﬁt when including vision in real world settings were space is usually limited and more cluttered.
The difference in behavior can also be appreciated when visualising the policy (see Figure 3): Thepolicy with vision avoids the wall (which was not present during training) and turns into a differentdirection .
Overall, the vision agent achieves good performance after about 1M environment steps.
On a real robot, and taking into account repair and maintenance, this could be collected within 5days.
Given the limited space of our environments all experiments have the tendency to develop circularmotion patterns (see Figure 3) despite the “no turning” component of our reward.
This is expected– only use the “no turning” reward to discourage walking in very small circles which would preventthe robot from exploring the whole workspace
4.2 Real robot experiments
Our robot experiments focus on sample complexity relative to simulation experiments, on assessingwhether our trained policies make use of vision, and on evaluating the degree to which they cangeneralize to different robots and environments.
Figure 5 (left) shows the learning curve for the distributed two-robot experiment.
We plot the episoderesults experienced by different robots in different colors to demonstrate that the robots had to beoccasionally stopped for repairs (while the other robot would keep collecting data).
Interesting behaviors start to emerge after about 25k environment steps of training which corresponds to 2-3 hours of experiment time.
This includes battery changes, time spent on resets andlogging data in between episodes.
It takes much longer to collect 25k environment steps at the beginning of the experiment than at later stages because the robot falls a lot and the experiment timeis dominated by the “stand up” reset behaviors (see supplementary material).
At this point the robothas learned to balance and to avoid falling which causes the episode lengths and the returns from the steadily
Figure 5 (left) also shows a learning curve from an analogous simulation experiment.
Once therobot learns to avoid falling, the rate of learning progress is about half that of the simulation.
Webelieve this gap is caused by several factors which make the real environment more complex thanthe simulation: inaccurate model of delays in simulation; changing robot dynamics due to wear andtear of the joints causing them to become loose and eventually completely broken; changing robotdynamics due to battery voltage ﬂuctuations; and non-stationary lighting conditions.
We also ran a shorter, single robot experiment and observed a similar sample complexity as thatof the two-robot experiment (see learning curve in supplementary material).
This is not obvious apriori because larger diversity of inputs (as is the case in the two-robot experiment) typically rendersreinforcement learning more difﬁcult.
The single robot experiment also serves as an additional“seed” and provides evidence for the reproducibility of our setup.
To demonstrate the increasedthroughput of the two-robot experiment, Figure 5 (right) shows the dependence of individual robots’performances on their uptime, and compares it to the same curve for the robot in the single robotexperiment.
We see that individual robots in the two-robot experiment, where data is shared, improvefaster, pointing to a promising direction of scaling up real experiments.
Visual wall avoidance To demonstrate that the learned policy uses camera observations to avoidwalls, Figure 6 depicts the reaction of the robot when encountering the wall.
Because of the policy’sright turning bias, we initialize the robot so that a wall is on its right (red square marks the robot’sinitial position).
If no wall is present, then the robot turns in a large circle (top row).
In comparison,when a wall is present, the robot takes a sharper turn as to only just brush the surface of the wall(bottom row).
Further robustness tests We evaluated the ﬁnal policy obtained in the two-robot experiment intwo additional settings: 1.
on a third robot which was not used during training, and 2.
on the samerobot which was used in the experiment but using a black ﬂoor instead of a blue one.
Table 2 showsthe results of this evaluation.
The policy works well on Robot 3 achieving only a slightly lowerper step reward than the robot used during training ( Robot 1 ).
However, it is less stable on Robot 3causing the robot to fall more frequently.
While this difference can be due to transfer, it is likely thatthe state of Robot 3 was unfavorable – as mentioned before, the dynamics of the robots vary acrosstime
and things like loose joints or screws can impact what a robot can do.
The policy performs considerably worse on a black ﬂoor demonstrating that the visual module isadapted to the blue ﬂoors encountered during training (also see supplementary video).
However, asdemonstrated in the simulation experiments, this could be improved by exposing the agent to blackﬂoors during training as well.
[Table]
Conclusions
In this paper we showed that we can learn a walking gait for a bipedal, off-the-shelf robot, fromscratch, outside a well-controlled lab infrastructure.
Using a state-of-the-art RL method (DMPO)renders learning with minimal prior knowledge feasible in terms of data efﬁciency.
Scalable reinforcement learning in the real world is still in its infancy, and to the best of our knowledge, this is one of the largest scale experiments for RL learning from scratch for legged robots.
Limiting ourselves to on-board sensors only and collecting data from multiple robots at various geographic locations allows us to scale up the diversity of environments which the robots encounter.
While in this paper we used only two locations with different lighting conditions, arena sizes, andbackground visuals, in future this can include many more locations with varying terrains and obstacles.
This should be contrasted to lab settings with immobile equipment for external state estimation.
We further also intend to remove the requirement for a reset controller by learning to stand-up.
Speciﬁcally, we plan to explore learning more dynamic behaviours, as well as tasks speciﬁed byrewards derived from camera images such as searching for an object in a cluttered environment.
Such tasks require memory, complex visual processing, and longer horizon decision making, henceit will be necessary to train for much longer than 1M steps.
We believe that the presented approachis well suited for learning at such scale.
We also plan to follow a “curriculum” and progressivelymove, or completely remove, the protective walls.
Social Impact Statement Embodied agents using perception and locomotion have the potential toenable applications with signiﬁcant positive social impact.
E.g. they can be deployed in search-andrescue operations (required in force majeure events like earthquakes, landslides, nuclear disasters),infrastructure inspections that might pose risks for humans (e.g. assessing if a bridge or tunnel is ina healthy condition), and many others.
However, there is also the risk that AI-driven technology ingeneral can be used for harmful purposes, e.g. in military applications.
We strongly oppose such usecases.
We believe that our particular application presents a low risk of being used in such situationsas it has limited potential for causing large-scale damage, but we actively support discussions aroundsafety and fairness in Robotics.