Rapid Exploration for Open-World Navigation with Latent Goal Models
Author s)
Abstract: We describe a robotic learning system for autonomous exploration andnavigation in diverse, open-world environments.
At the core of our method is alearned latent variable model of distances and actions, along with a non-parametrictopological memory of images.
We use an information bottleneck to regularize thelearned policy, giving us (i) a compact visual representation of goals, (ii) improvedgeneralization capabilities, and (iii) a mechanism for sampling feasible goals forexploration.
Trained on a large ofﬂine dataset of prior experience, the model acquires a representation of visual goals that is robust to task-irrelevant distractors.
We demonstrate our method on a mobile ground robot in open-world explorationscenarios.
Given an image of a goal that is up to 80 meters away, our methodleverages its representation to explore and discover the goal in under 20 minutes,even amidst previously-unseen obstacles and weather conditions
Introduction
Robustness is a key challenge in learning to navigate diverse, real-world environments.
A roboticlearning system must be robust to difference between an ofﬂine training dataset and the real world(i.e.
, it must generalize), and must be robust to non-stationary changes in the real world itself (i.e.,it must ignore visual distractors), and it must be equipped with mechanisms to actively explore togather information about connectivity and traversability.
Different environments typically exhibitsimilar physical structures, and these similarities can be used to accelerate exploration of new environments.
Learning-based methods provide an appealing approach for learning a representation ofthis shared structure using prior experience.
In this work, we consider the problem of navigating to a user-speciﬁed goal in a previously unseenenvironment.
The robot has access to a large and diverse dataset of experience from other environments, which it can use to learn general navigational affordances.
Our approach to this problemuses a an information bottleneck architecture to learn a compact representation of goals.
Learnedfrom prior data, this latent goal model encodes prior knowledge about perception, navigational affordances, and short-horizon control.
We use a non-parametric memory to incorporate experiencefrom the new environment.
Combined, these components enable our system to learn to navigate togoals in a new environment after only a few minutes of exploration.
The primary contribution of this work is a method for exploringnovel environments to discover user-speciﬁed goals.
Our methodoperates directly on a stream of image observations, without relying on structured sensors or geometric maps.
An important partof our method is a compressed representation of goal images thatsimultaneously affords robustness while providing a simple mechanism for exploration.
Such a representation allows us, for example,to specify a goal image at one time of day, and then navigate to thatsame place at a different time of day: despite variation in appearance, the latent goal representations must be sufﬁciently close thatthe model can produce the correct actions.
Robustness of this kindis critical in real-world settings, where the appearance of landmarkscan change signiﬁcantly with times of day and seasons of the year.
We demonstrate our method, R apid E xploration C ontrollers for O utcome-driven N avigation (RECON), on a mobile ground robot (Fig. 1) and rigorously evaluate against 6 competitive baselines
[Figure]
spanning over 100 hours of real-world experiments in 8 distinct open-world environments (Fig. 2).
Our method can discover user-speciﬁed goals up to 80m away after just 20 minutes of interaction ina new environment.
We also demonstrate robustness in the presence of visual distractors and novelobstacles.
Related Work
Exploring a new environment is often framed as the problem of efﬁcient mapping, posed in termsof information maximization to guide the robot to uncertain regions of the environment.
Some priorexploration methods use local strategies for generating control actions for the robots [1–4], whileothers use use global strategies based on the frontier method [5–7].
However, building high-ﬁdelitygeometric maps can be hard without reliable depth information.
Further, such maps do not encodesemantic aspects of traversability, e.g., tall grass is traversable but a wire fence is not.
Inspired by prior work [8–12], we construct a topological map by learning a distance function anda low-level policy.
We estimate distances via supervised regression and learn a local control policyvia goal-conditioned behavior cloning [13, 14].
However, these prior methods do not describe howto learn to navigate in new , unseen environments.
We equip RECON with an explicit mechanismfor exploring new environments and transferring knowledge across environments.
Well-studied methods for exploration in reinforcement learning (RL) utilize a novelty-based bonus,computed from a predictive model [15–21], information gain [22, 23], or methods based on counts,densities, or distance from previously-visited states [24–26].
However, these methods learn to reasonabout the novelty of a state only after visiting it.
Recent works [27, 28] improve upon this bypredicting explorable areas for interesting parts of the environment to accelerate visual exploration.
While these methods can yield state-of-the-art results in simulated domains [29, 30], they come atthe cost of high sample complexity (over 1M samples) and are infeasible to train in open-worldenvironments without a simulated counterpart.
Instead, our method enables the robot to explore anenvironment from scratch in just 20 minutes, using prior experience from other environments.
The problem of reusing experience across tasks is studied in the context of meta-learning [31–33]and transfer learning [34–38].
Our method uses an information bottleneck [39], which serves adual purpose: ﬁrst, it provides a representation that can aid the generalization capabilities of RLalgorithms [40, 41], and second, it serves as a measure of task-relevant uncertainty [42], allowingus to incorporate prior information for proposing goals that are functionally-relevant for learningcontrol policies in the new environment.
The problem of learning goal-directed behavior has been studied extensively using RL [43–46] andimitation learning (IL) [13, 14, 47–50].
Our method builds upon prior goal-conditioned IL methodsto solve a slightly different problem: how to reach goals in a new environment.
Once placed in anew environment, our method explores by carefully choosing which goals to visit, inspired by priorwork [51–55].
Unlike these prior methods, however, our method makes use of previous experiencein different environments to accelerate learning in the current environment.
Problem Statement and System Overview
We consider the problem of goal-directed exploration for visual navigation in novel environments:a robot is tasked with navigating to a goal location G , given an image observation o g taken at G .
Broadly, this consists of three separate stages: (1) learning from ofﬂine data, (2) building a mapin a new environment, and (3) navigating to goals in the new environment.
We model the task ofnavigation as a Markov decision process with time-indexed states s t ∈ S and actions a t ∈ A .
Wedo not assume the robot has access to spatial localization or a map of the environment, or accessto the system dynamics.
We use videos of robot trajectories in a variety of environments to learngeneral navigational skills and build a compressed representation of the perceptual inputs, whichcan be used to guide the exploration of novel environments.
We make no assumption on the natureof the trajectories: they may be obtained by human teleoperation, self-exploration, or as a result of apreset policy.
These trajectories need not exhibit intelligent behavior.
Since the robot only observesthe world from a single on-board camera and does not run any state estimation, our system operatesin a partially observed setting.
Our system commands continuous linear and angular velocities.
3.1 Mobile Robot Platform
We implement RECON on a Clearpath Jackal UGV platform (see Fig. 1).
The default sensor suiteconsists of a 6-DoF IMU, a GPS unit for approximate global position estimates, and wheel encodersto estimate local odometry.
In addition, we added a forward-facing 170 ◦ ﬁeld-of-view RGB cameraand an RPLIDAR 2D laser scanner.
Inside the Jackal is an NVIDIA Jetson TX2 computer.
The GPSand laser scanner can become unreliable in some environments [56], so we use them solely as safetycontrollers during data collection.
Our method operates only using images taken from the onboard
RGB camera, without other sensors or ground-truth localization.
3.2 Self-Supervised Data Collection & Labeling
Our aim is to leverage data collected in a wide range of different environments to enable the robotto discover and learn to navigate to novel goals in novel environments.
We curate a dataset ofself-supervised trajectories collected by a time-correlated random walk in diverse real-world environments (see Fig.
2 (d,e)).
This data was collected over a span of 18 months and exhibits signiﬁcantvariation in appearance due to seasonal and lighting changes.
We make this dataset publicly available and provide further details in Appendix A.
RECON : A Method for Goal-Directed Exploration
Our objective is to design a robotic system that uses visual observations to efﬁciently discover and reliably reach a target image in a previously unseen environment.
RECON consists of two componentsthat enable it to explore new environments.
The ﬁrst is an uncertainty-aware, context-conditionedrepresentation of goals that can quickly adapt to novel scenes.
The second component is a topological map, where nodes represent egocentric observations and edges are the predicted distancebetween them, constructed incrementally from frontier-based exploration, maintaining a compactmemory of the target environment.
4.1 Learning to Represent Goals
Our method learns a compact representation of goal images that is robust to task-irrelevant factorsof variation.
We learn this representation using a variant of the information bottleneck architecture [42, 57].
We use a context-conditioned representation of goals to learn a control policy in thetarget environment (Fig. 3 describes the graphical model).
Letting I ( · ; · ) denote mutual information,the objective in Eq.
1 encourages the model to compress the incoming goal image o g into a representation z gt conditioned on the current observation o t that is predictive of the best action a gt and thetemporal distance d gt to the goal (upper-case denotes random variables):
[Equation]
Following [42], we approximate the intractable objective in Eq.
1 with a variational posterior anddecoder (an upper bound), resulting in the maximization objective:
[Equation]
Encoder
[Equation]
Decoder q θ ( a gt ,d gt | z tg ,o t )
O g
Z gt
A gt
O t
D gt where we deﬁne the prior r ( z gt ) (cid:44) N (0 , I ) and D is a dataset of trajectories characterized by( o t , o g , a gt , d gt ) quadruples.
The ﬁrst term measuresthe model’s ability to predict actions and distancesfrom the encoded representation, and the secondterm measures the model’s compression of incoming goal images.
As the encoder p φ and decoder q θ are conditioned ono t , the representation z gt only encodes informationabout relative location of the goal from the context– this allows the model to represent feasible goals.
If, instead, we had a typical VAE (in which the inputimages are autoencoded), the samples from the priorover these representations would not necessarily represent goals that are reachable from the currentstate.
This distinction is crucial when exploring new environments, where most states from thetraining environments are not valid goals.
4.2 Goal-Directed Exploration with Topological Memory
The second component of our system is a topological memory constructed incrementally as therobot explores a new environment.
It provides an estimate of the exploration frontier as well asa map that the robot can use to later navigate to arbitrary goals.
To build this memory, the robotuses the model from the previous section to propose subgoals for data collection.
Note that thisis done in the exploration phase and have a latent goal model pre-trained on the ofﬂine dataset.
Given a subgoal, our algorithm (Alg. 1) proceeds by executing actions towards the subgoal for aﬁxed number of timesteps (Alg. 1 L12).
The data collected during subgoal navigation expands thetopological memory (Alg. 1 L14) and is used to ﬁne-tune the model (Alg. 1 L15).
Thus, the task ofefﬁcient exploration is reduced to the task of choosing subgoals.
Subgoals are represented by latent variables in our model, which may either come from the posteriorp φ ( z | o t , o g ) , or from the prior r ( z ) .
Given a subgoal z and observation o t , the model decodes it into Algorithm 1 RECON for Exploration: RECON takes as input an encoder p φ , a decoder q θ , prior r , the current observation o t and goal observation o g .
δ 1 , δ 2 , (cid:15), β ∈ R + ; H, γ ∈ N are hyperparameters. RECON
G (cid:46) Initialize
] do (cid:46)
4: o n ← LeastExploredNeighbor
( G , o t 2 ) 5: z gt ∼ p φ ( z | o t , o g ) (cid:46)
Encode goal 6: if goal is feasible [ r ( z gt ) > (cid:15) ] then 7: z wt ← z gt (cid:46) Will go to the goal 8: else if robot at frontier [ ¯ d nt <
δ 1 ] then 9: z wt ∼ r ( z ) (cid:46) Will explore from frontier 10: else 11: z wt ∼ p φ ( z | o t , o n ) (cid:46) Will go to frontier 12:
D w , o t ← SubgoalNavigate( z wt ; H ) 13:
D ← D ∪ D w 14:
ExpandGraph( G , o t ) 15:
Step L ( φ, θ ; D , β ) for γ epochs (cid:46) Eq.16: return networks p φ , q θ and graph G an action and distance pair q ( a gt , d gt | z, o t ) ; the action is used to control the robot towards the goal,and the distance is used to construct edges in the topological graph.
The choice of intermediatesubgoal to navigate toward at any step is based on the robot’s estimate of the goal reachability andits proximity to the frontier.
To determine the frontier of the graph, we track the number of timeseach node in the graph was selected as the navigation goal; nodes with low counts are considered tobe on the frontier.
In the following, we use ¯ z gt to denote the mean of the encoder p φ ( z | o t , o g ) , and¯ d gt to denote the distance component of the mean of the decoder q θ ( a t , d t | ¯ z gt , o t ) (i.e., the predictednumber of time steps from o t to ¯ z gt ).
The choice of subgoal at each step is made as follows:
Algorithm exploration, uses the topological graph
G to quickly navigate towards the goal o g . 1: procedure GoalNavigate ( G , o t , o g ; H ) 2: v t ← AssociateToVertex( G , o t ) 3: v g ← AssociateToVertex( G , o g ) 4: ( v t , . .
. , v g ) ← ShortestPath( G , v t , v g ) 5: for v ∈ ( v t , . . .
, v g ) do 6:
z ← p φ ( z | o t , o g = v. o) 7:
D w , o t ← SubgoalNavigate( z ; H ) (i) Feasible Goal: The robot believes it canreach the goal and adopts the representation ofthe goal image as the subgoal (Alg. 1 L7).
Therobot’s conﬁdence in reaching the goal is basedon the probability of the current goal embedding z gt under the prior r ( z ) .
Large r ( z gt ) implies the relationship between the observationo t and the goal o g is in -distribution , suggestingthat the model’s estimates of the distances is reliable – intuitively, this means that the model is
conﬁdent about the distance to o g and can reach
it.
(ii) Explore at Frontier:
The robot is at
the “least-explored node” (frontier) o n and ex-
plores by sampling a random conditional subgoal latent z wt from the prior (Alg. 1 L9).
The robotdetermines whether it is at the frontier based on the distance (estimated by querying the model) to its“least explored neighbor” ¯ d nt – the node in the graph within a distance threshold ( δ 2 ) of the currentobservation that has the lowest visitation count.
If the distance to this node ¯ d nt is low (threshold δ 1 ),then the robot is at the frontier.
(iii) Go to Frontier:
The robot adopts its “least-explored neighbor” o n as a subgoal (Alg. 1 L11).
The SubgoalNavigate function rolls out the learned
policy for a ﬁxed time horizon H to navigate to the
desired
subgoal
latent z wt , by querying the decoder q θ ( a t , d t | z wt , o τ ) with a ﬁxed subgoal latent.
The endpoint of such a rollout is used to update the visitation counts in the graph G .
At the end ofeach trajectory, the ExpandGraph subroutine is used to update the edge and node sets {E , V} ofthe graph G to update the representation of the environment.
We provide the pseudocode for thesesubroutines in Appendix B.1.
We also share broader implementation details including choice ofhyperparameters, model architectures and training details in Appendix B.2.
4.3 System Summary
RECON uses the latent goal model and topological graph to quickly explore new environments anddiscovers user-speciﬁed goals.
Our complete system consists of three stages:
A) Prior Experience: The goal-conditioned distance and action model (Sec. 4. is trained using experience from previously visited environments. Supervision
for
training
our model is obtained by using time steps as a proxy for distances and a relabeling scheme (Appendix A). B) Exploring a Novel Environment: When placed in a new environment, RECON uses a combination of frontier-based exploration and latent goal-sampling with the learned model.
The learnedmodel is also ﬁne-tuned to this environment.
These steps are summarized in Alg. 1 and Sec.
4.2.C)
Navigating an Explored Environment: Given an explored environment (represented by a topological graph G ) and the model, RECON uses G to navigate to a goal image by planning a pathof subgoals through the graph.
This process is summarized in Alg.
Experimental Evaluation
We designed our experiments to answer four questions:
Q1.
How does RECON compare to prior work for visual goal discovery in novel environments? [Table]
Q2.
After exploration, can RECON leverage its experience to navigate to the goal efﬁciently?
Q3.
What is the range of perturbations and non-stationary elements to which RECON is robust?
Q4.
How important are the various components of RECON, such as sampling from an informationbottleneck and non-parametric memory, to its performance?
5.1 Goal-Directed Exploration in Novel Environments
We perform our evaluation in a diverse variety of outdoor environments (examples in Fig. 2), including parking lots, suburban housing, sidewalks, and cafeterias.
We train our self-supervisednavigation model using an ofﬂine navigation dataset (Sec.3.2) collected in a distinct set of trainingenvironments, and evaluate our system’s ability to discover user-speciﬁed goals in previously unseenenvironments.
We compare RECON to six baselines, each trained on the same 20 hours of ofﬂinedata as our method, and ﬁnetuned in the target environment with online interaction.
PPO + RND: Random Network Distillation (RND) is a widely used prediction bonus-based
exploration strategy in RL [18], which we use with PPO [59, 60].
This comparison is represen-
tative of a frequently used approach for exploration in RL using a novelty-based bonus.
InfoBot:
An ofﬂine variant of InfoBot [41], which uses goal-conditioned information bottle-
neck, analogous to our method, but does not use the non-parametric memory.
Active Neural SLAM (ANS):
A popular indoor navigation approach based on metric spatialmaps proposed for coverage-maximizing exploration [21].
We adapt it to the goal-directed task by using the distance function from RECON to detect when the goal is nearby.
Neural Topological SLAM (NTS): A recent approach that uses image-space predictions of
semantically rich areas to explore, with topological maps serving as an episodic memory [27].
This baseline represents the state-of-the-art in visual navigation for structured indoor environ-
ments that use high-ﬁdelity simulators, such as Habitat or AI2-THOR [29, 30].
Visual Navigation with Goals (ViNG): A method that uses random action sequences to exploreand incrementally build a topological graph without reasoning about visitation counts [11].6.
Episodic Curiosity (ECR): A method that executes random action sequences at the frontier ofa topological graph for exploration [20].
We implement this as an ablation of our method thatsamples random action sequences, rather than rollouts to sampled goals (Alg.
1 Line 7).
We evaluate the ability of RECON to discover visually-indicated goals in 8 unseen environmentsand navigate to them repeatedly.
For each trial, we provide an RGB image of the desired target (oneper environment) to the robot and report the time taken by each method to (i) discover the desiredgoal ( Q1 ), and (ii) reliably navigate to the discovered goal a second time using prior exploration( Q2 ).
Additionally, we quantify navigation performance using Success weighed by Completion
Time (SCT), a success metric that takes into account the agent’s dynamics [58].
We show quantitative results in Table 1, and visualize sample trajectories of RECON and the baselines in Fig. 4.
RECON outperforms all the baselines, discoveringgoals that are up to 80m away in under 20 minutes, including instances where no other baselinecan reach the goal successfully.
RECON+ECR
and
ViNG discover the goal in only the easier environments, and take up to 80% more time to discoverthe goal in those environments.
RND, InfoBot and
ANS are able to discover goals that are up to 25 maway but fails to discover more distant goals, likelybecause using reinforcement learning for ﬁne-tuningis data-inefﬁcient.
We exclude reporting metrics on
NTS, which fails to successfully explore any environment, likely due to overﬁtting to the ofﬂine trajectories.
Indeed, the simulation experiments reported in each of these online algorithms requireupwards of 1 M timesteps to adapt to new environments [21, 27, 41].
We attribute RECON’s successto the context-conditioned sampling strategy (described in Sec. 4.1), which proposes goals that canaccelerate the exploration of new environments.
We then study RECON’s ability to quickly reachgoals after initial discovery.
Table 1 shows that RECON variants are able to quickly recall a feasiblepath to the goal.
These methods create a compact topological map from experience in the targetenvironment, allowing them to quickly reach previously-seen states.
The other baselines are unsuccessful at recalling previously seen goals for all but the simplest environments.
Fig.
4 shows an aerialview of the paths recalled by various methods in one of the environments.
Only the RECON variantsare successfully able to navigate to the checkerboard goal; all other baselines result in collisions inthe environment.
Further, RECON discovers a shorter path to the goal and takes 30% less time tonavigate to it than ECR ablation.
5.2 Exploring Non-Stationary Environments
Outdoor environments exhibit non-stationarity due to dynamic obstacles, such as automobiles andpeople, as well as changes in appearance due to seasons and time of day.
Successful exploration andnavigation in such environments requires learning a representation that is invariant to such distractors.
This capability is of central interest when using a non-parametric memory: for the topologicalmap to remain valid when such distractors are presented, we must ensure the invariance of thelearned representation to such factors ( Q3 ).
To test the robustness of RECON, we ﬁrst had RECON explore in a new “junkyard” to learn toreach a goal image containing a blue dumpster (Fig.
5-a).
Then, without any more exploration,we evaluated the learned goal-reaching policy when presented with previously unseen obstacles(trash cans, trafﬁc cones, and a car) and and weather conditions (sunny, overcast, and twilight).
Fig.
5 shows trajectories taken by the robot as it successfully navigates to the goal in scenarios withvarying obstacles and lighting conditions.
These results suggest that the learned representations are
[Table]
invariant to visual distractors that do not affect robot’s decisions to reach a goal (e.g., changes inlighting conditions do not affect the trajectory to goal, and hence, are discarded by the bottleneck).
5.3 Dissecting RECON
RECON explores by sampling goals from the prior distribution over state-goal representations.
Toquantify the importance of this exploration strategy ( Q4 ), we deploy RECON to perform undirectedexploration in a novel target environment without building a graph of the environment.
We comparethe coverage of trajectories of the robot over 5 minutes of exploration when: (a) it executes randomaction sequences [20], and (b) it performs rollouts towards sampled goals.
We see that performingrollouts to sampled goals results in 5 x faster exploration in novel environments (see Fig. 6).
We also evaluate several variants of RECON that ablate its goal sampling and non-parametric memory on the end-to-end task of visual goal discovery in novel environments:
- Reactive: our method deployed without the topological graph for memory.
- Random Actions: a variant of our method that executes random action sequences at the frontierrather than rollouts to sampled goals.
This is identical to the ECR baseline described in Sec. 5.1.
- Vanilla Sampling: a variant of our method which learns a goal-conditioned policy and distances
without an information bottleneck to obtain compressed representations.
We deploy these variants in a subset of the unseen test environments and summarize their performance in Table 2.
These results corroborate the observations in Fig.
6: learning a compressed goalrepresentation is key to the performance of RECON.
“Vanilla Sampling”, despite sampling froma joint prior, performs poorly and is unable to discover distant goals.
We hypothesize that ourmethod is more robust because the information bottleneck helps learn a representation that ignorestask-irrelevant information.
We also observe that “Reactive” experiences a smaller degradation inexploration performance, suggesting that goal-sampling can help with the exploration problem evenwithout the graph.
However, we ﬁnd a massive degradation in its ability to recall previously discovered goals, suggesting that the memory is key to the navigation performance of RECON.
Discussion
We proposed a system for efﬁciently learning goal-directed policies in new open-world environments.
The key idea behind our method is to use a learned goal-conditioned distance model witha latent variable model representing visual goals for rapid goal-directed exploration.
The problemsetup studied in this paper, using past experience to accelerate learning in a new environment, isreﬂective of real-world robotics scenarios: collecting new experience at deployment time is costly,but experience from prior environments can provide useful guidance to solve new tasks.
In future work, we aim to provide theoretical guarantees for when and where we can expect stochastic policies and the information bottleneck to provide efﬁcient exploration.
One limitation of thecurrent method is that it does not explicitly account for the value of information.
While this makesthe resulting method very simple, accounting for such states can generate a better goal-reaching policy.
Another exciting possibility is using additional modalities for task speciﬁcation, including GPS or language, which can extend the range and efﬁciency of the current system to enable long-rangenavigation at the scale of miles. Adding reliable long-range cues like GPS can greatly increase thescope of practical applications of our method. A Dataset
In this work, we emphasize that data collected from prior experience in unrelated environments canbe a rich source of supervision, even if the interactions in the dataset are suboptimal.
To demonstratethis, we curate a dataset of over 5000 self-supervised trajectories collected over 9 distinct realworld environments.
These trajectories capture the interaction of the robot in diverse environments,including phenomena like collisions with obstacles and walls, getting stuck in the mud or pits, orﬂipping due to bumpy terrain.
The dataset contains measurements from a wide range of sensorsincluding a pair of stereo RGB cameras, thermal camera, 2D LiDAR, GPS and IMU to supportofﬂine evaluation using an alternative suite of sensors. While a lot of these sensor measurementscan be noisy and unreliable, we believe that learning-based techniques coupled with multimodalsensor fusion can provide a lot of beneﬁts in the real-world. This dataset was collected over a spanof 18 months and exhibits signiﬁcant variation in appearance due to seasonal and lighting changes 1 . [Figure]
A.1 Self-Supervised Data Collection and Labeling
We design the data collection methodology to enable gathering large amounts of diverse data withminimal human intervention.
Due to the high cost of gathering data with real-world robotic systems,we choose to use an off-policy learning algorithm in order to be able to gather data using any controlpolicy and train on all of the gathered data.
To ensure that the control policy achieves sufﬁcient coverage of the environment while also ensuring that the action sequences executed by the robotare realistic, we use a time-correlated random walk to gather data.
A na¨ıve uniform random controlpolicy is inadequate because the robot will primarily drive straight due to the linear and angularvelocity action interface of the robot, which will result in both insufﬁcient exploration and unrealistictest time action sequences.
During data collection using the random control policy, the robot requires a mechanism to detect ifit is in collision or stuck, and an automated controller to reset itself in order to continue gatheringdata.
We detect collisions in one of two ways, either using the LIDAR to detect when an obstacle isnear or the IMU to detect when the robot is stuck due to an obstacle or uneven terrain.
We programan automated backup maneuver that drives the robot out of collision (whenever possible)
so it caninitiate a new trajectory.
We also use these collision detectors as a weak source of supervision by generating event labels for the collected trajectories, giving us a self-supervised relabeling pipeline as proposed in
BADGR [56].
We consider three different events: collision, bumpiness, and position.
A collisionevent is detected the LIDAR measures an obstacle to be close or, in off-road environments,
when the
IMU detects a sudden drop in linear acceleration (jerk) and angular velocity magnitudes.
A bumpiness event is calculated as occurring when the angular velocity magnitudes measured by the IMUare above a certain threshold.
The position is determined by an onboard state estimator that fuseswheel odometry and the IMU to form a local position estimate.
Note that all experiments reportedin this paper only use the collision labels; these labels are used to dissect the random walks intosmooth trajectories that end in collision.
A.2 Environments
To learn general navigational affordances across a wide range of environments, we curate over 40hours of trajectories in 9 diverse open-world environments of varying complexity (see Figure 7).
Figure 8 shows the exploration and navigation performance of RECON and the baselines (see Sec.
5for details) on the individual environments.
As the environment complexity increases, most methodsare not able to explore the environment efﬁciently to discover the goal.
For videos of our systemexploring these environments, please check out the supplemental video submission.
B Reproducibility
[Table]
B.1 Algorithmic Components
The SubgoalNavigate function rolls out the learned policy for a ﬁxed time horizon H to navigate to the desired subgoal latent z wt , by querying the decoder q θ ( a t , d t | z wt , o τ ) in an open loopmanner.
The endpoint of such a rollout is used to update the visitation counts v in the graph Gusing the AssociateToVertex subroutine.
To nudge the robot to the frontier, we use a heuristic
LeastExploredNeighbor routine that uses the visitation counts of the neighbors to identify unexplored areas in the local neighborhood.
At the end of each trajectory, the ExpandGraph subroutineis used to update the edge and node sets {E , V} of the graph G to update the non-parametric representation of the environment.
Pseudocode for these subroutines are given in Alg.
B.2 Implementation Details
[Table]
Inputs to the encoder p φ are pairsof observations of the environment– current and goal – represented bya stack of two RGB images obtained from the onboard camera ata resolution of 160 × 120 pixels.p φ is implemented by a MobileNetencoder [61] followed by a fullyconnected layer projecting the 1024dimensional latents to a stochastic, context-conditioned representa- tion z gt of the goal that uses 64 -dimensions each to represent the mean and diagonal covarianceof a Gaussian distribution.
Inputs to the decoder q θ are the context (current observation) – processedwith another MobileNet – and z gt .
We use the reparametrization trick [62] to sample from the latentand use the concatenated encodings to learn the optimal actions a gt and distances d gt .
Details of ournetwork architecture are provided in Table 4.
During pretraining, we maximize Eq.
{+2 with a batchsize of 
+}
and perform gradient updates using the Adam optimizer with learning rate λ
= 10 − 4until convergence.
We provide the hyperparameters associated with our algorithms in Table 3.
C Supplemental Video
For more results and videos of our system deployed in unstructured, outdoor environments in thereal-world, please check out the supplemental video submission.
1: function SubgoalNavigate ( z wt ; H ) 2: trajectory ← () 3: for t ∈ [1 , . . .
, H ] do 4: trajectory .
append(( o t , a t , t )) 5: a t , d gt ∼ q θ ( a t , d t | z wt , o τ ) (cid:46)
Sample action 6:
o t ← Env .
step( a t ) (cid:46) Execute action 7: v
H ← AssociateToVertex( G , o H ) 8: v H .
count ← v H .
count +9:
D w ← (( o t , o H , a t , H − t ) for ( o t , a t , t ) ∈ trajectory) 10: return D w , o H 1: function AssociateToVertex ( G =
( V , E ) , o t ) 2:
d ← sort(( ¯ d vt , v ) for v ∈ V ) (cid:46) Predict distances 3: v, d ← d [0] (cid:46) Associate o t with nearest vertex 4: return v 1: function LeastExploredNeighbor ( G =
( V , E ) , o t , δ 2 ) 2: v ← AssociateToVertex( G , o t ) 3:
V n ← { v (cid:48) : E ( v, v (cid:48) ) < δ 2 , v (cid:48) ∈ V} (cid:46)
Retrieve neighbors 4: c ← sort(( v (cid:48) .
count , v (cid:48) .
o) for v (cid:48) ∈ V n ) 5: v c , o c ← c [0] (cid:46) Retrieve neighbor with smallest count 6: return o c 1: procedure ExpandGraph ( G =
( V , E ) , o t ) 2: v t ← Node(count = 1 , o = o t ) (cid:46)
Create node for o t 3:
E ← E ∪ { ( v t
[Table]
v g )
[Table]
¯ d gt
[Table]
∈ V}
[Table]
V ← V ∪ { v t }
[Table]