E NVIRONMENT
P
REDICTIVE C ODING FOR V ISUAL
N AVIGATION
Anonymous authors
Paper under double-blind review
A BSTRACT
We introduce environment predictive coding , a self-supervised approach to learn environment-level representations for embodied agents.
In contrast to prior work on self-supervised learning for individual images, we aim to encode a 3D environment using a series of images observed by an agent moving in it.
We learn these representations via a masked-zone prediction task , which segments an agent’s trajectory into zones and then predicts features of randomly masked zones, conditioned on the agent’s camera poses.
This explicit spatial conditioning encourages learning representations that capture the geometric and semantic regularities of 3D environments.
We learn the representations on a collection of video walkthroughs and demonstrate successful transfer to multiple downstream navigation tasks.
Our experiments on the real-world scanned 3D environments of Gibson and Matterport3D show that our method obtains 2 - 6 × higher sample-efﬁciency and up to 57% higher performance over standard image-representation learning.
I NTRODUCTION
In visual navigation, an intelligent embodied agent must move around a 3D environment using its stream of egocentric observations to sense objects and obstacles, typically without the beneﬁt of a pre-computed map.
Signiﬁcant recent progress on this problem can be attributed to the availability of large-scale visually rich 3D datasets (Chang et al., 2017; Xia et al., 2018; Straub et al., 2019), and high-quality 3D simulators (Kolve et al., 2017; Savva et al., 2019a; Xia et al., 2020).
End-to-end reinforcement learning (RL) has been shown to achieve state-of-the-art navigation performance (Savva et al., 2019a;
Wijmans et al., 2020).
However, these approaches suffer from sample inefﬁciency and incur signiﬁcant computational cost.
Recent approaches try to mitigate these limitations by pre-training image representations ofﬂine and transferring them for navigation (Mirowski et al., 2016; Sax et al., 2020), or by performing auxiliary tasks and data augmentation in an online fashion during RL policy learning (Gordon et al., 2019;
Kostrikov et al., 2021; Ye et al., 2020; 2021).
Current ofﬂine representation learning methods are ﬂexible — once learned, the representations can be transferred to improve multiple embodied tasks.
However, they are limited to learning image feature extractors (Gupta et al., 2017; Sax et al., 2020), or image-level proximity functions (Savinov et al., 2018; Chaplot et al., 2020c; Chang et al., 2020).
Since embodied agents typically operate with limited ﬁeld-of-view sensors, image representations only encode small parts of the scene in the nearby locality of the agent, and do not consider the broader context from the rest of the environment.
We contend that embodied agents must learn higher-level semantic and geometric representations of the larger 3D environment around them, conditioned on their entire history of observations.
To that end, we introduce environment predictive coding (EPC), a self-supervised approach to learn environment-level representations that are transferrable to a variety of navigation-oriented tasks.
The key idea is to learn an encoding of a 3D environment from a series of egocentric observations so as to be predictive of visual content that the agent has not yet observed.
Consider the example in
Fig.
An agent has observed the living room, lounge, and the bedroom in an unfamiliar house.
The agent’s encoding of the observed spaces (i.e., the red trajectory) should be predictive of the visual features at an unseen location ( x, y, θ ), e.g., the green viewpoint, and enable inferences like “there is a small kitchen”, and “it contains a sink and an oven”.
Learning such predictive representations can equip an agent with the ability to reason about 3D environments as it starts performing various
[Equation]
[Equation] [Equation]
navigation-oriented tasks.
The proposed EPC model aims to learn such representations that capture the natural statistics of real-world environments in a self-supervised fashion, by simply watching video walkthroughs recorded by other agents.
To achieve this, we devise a self-supervised masked-zone prediction task in which the model learns environment embeddings in an ofﬂine fashion, by watching pre-collected video walkthroughs recorded by other agents navigating in 3D environments (See Fig. 1).
The videos contain RGB-D and odometry.
Speciﬁcally, we segment each video, into zones of temporally contiguous frames which capture local regions of the 3D environment.
Then, we randomly mask out zones, and predict features for the masked zones conditioned on both the unmasked zones’ views and the masked zones’ camera poses.
Since the overlap in scene content across zones is typically limited, the model needs to reason about the geometry and semantics of the environment to ﬁgure out what is missing.
Our general strategy can be viewed as a context prediction task in sequential data (Devlin et al., 2018;
Sun et al., 2019b; Han et al.
, 2019)—but, very differently, aimed at learning high-level semantic and geometric representations of 3D environments to aid embodied agents acting in them.
Unlike any prior video-based feature learning, our approach learns features conditioned on camera poses, explicitly grounding them in a 3D space; we demonstrate the impact of this important distinction.
Through simulated experiments in photorealistic scenes from Matterport3D (Chang et al., 2017) and
Gibson (Xia et al., 2018), we show that transferring the EPC environment-level representations leads to 2 - 6 × higher sample-efﬁciency and up to 57% better performance compared to only image-level transfer on 4 navigational tasks: room goal navigation, object visitation, ﬂee, and area coverage.
Our contributions are: (1) we propose environment-predictive coding (EPC), a self-supervised approach to represent the underlying 3D environment given the observation sequences of an embodied agent, (2) we propose the proxy task of masked-zone prediction to learn environment-level representations from video walkthroughs captured by other agents, (3) we perform extensive experiments on Gibson and Matterport3D to demonstrate that EPC leads to good improvements on multiple navigation-oriented tasks, and study EPC’s design choices and noise robustness.
R ELATED WORK
Self-supervised visual representation learning : Prior work leverages self-supervision to learn image and video representations from large unlabelled datasets.
Image representation learning attempt proxy tasks such as inpainting (Pathak et al., 2016) and instance discrimination (Oord et al., 2018;
Chen et al., 2020a; He et al., 2020), while video representation learning leverages signals such as temporal consistency (Jayaraman & Grauman, 2015; Wei et al., 2018; Kim et al., 2019) and contrastive predictions (Han et al., 2019; Sun et al., 2019a).
VideoBERT (Sun et al., 2019a;b) jointly learns video and text representations from videos by ﬁlling in masked out information.
Dense Predictive Coding (Han et al., 2019; 2020) learns video representations that capture the slow-moving semantics in videos.
Whereas these methods tackle human activity recognition in videos, we aim to learn geometric and semantic cues in 3D spaces for embodied agents.
Unlike the existing video models (Sun et al., 2019a;b; Han et al., 2019), which simply infer missing frame features conditioned on time, our approach explicitly grounds its predictions in the spatial structure of 3D environments.
Representation learning via auxiliary tasks for RL : Reinforcement learning approaches often suffer from high sample complexity, sparse rewards, and unstable training.
Prior work tackles these using auxiliary tasks for learning image-level representations during online RL training (Mirowski 𝑍 !" 𝑍 #" 𝑍 $% 𝑍 $" 𝑍 #% 𝑍 ! % 𝑍 &" 𝑍 ’%
Zone decoder 𝑝 !"
𝑓 !"
𝑓 # . . . 𝑓 $% . . . Contrastive loss 𝑍 &’ . . . 𝑍 &" 𝑍 (’ . . .
𝑍 )"
Environment encoder .
𝑜,𝑝 ∈ [𝑍 $" ,...,𝑍 &" ] 𝜀 . . .
𝑒𝑒 𝑡 . . .
[Figure]
et al., 2016; Gordon et al., 2019; Shen et al., 2019; Ye et al., 2020; 2021; Stooke et al., 2021).
In contrast, we learn environment-level representations in an ofﬂine fashion from previously captured videos.
Recent work also learns state representations via future prediction and implicit models (Ha & Schmidhuber, 2018; Gregor et al., 2019; Hafner et al., 2019; Guo et al., 2020).
In particular, neural rendering approaches achieve impressive reconstructions for arbitrary viewpoints (Eslami et al., 2018; Kumar et al., 2018).
However, unlike our idea, they focus on pixelwise reconstruction, and their success has been limited to synthetically generated environments like DeepMind Lab (Beattie et al., 2016).
In contrast to any of the above, we use egocentric videos to learn predictive feature encodings that capture the naturally occurring regularities of photorealistic 3D environments.
Scene completion : Prior scene completion work performs pixelwise (Jayaraman & Grauman, 2018;
Ramakrishnan et al., 2019) or voxelwise (Song et al., 2017) reconstruction of 360 panoramas, image inpainting (Pathak et al., 2016), and image-level extrapolation of depth and semantics (Song et al., 2018; Yang et al., 2019).
Recent work on visual navigation extrapolates maps of roomtypes (Wu et al., 2019; Narasimhan et al., 2020) and occupancy (Ramakrishnan et al., 2020).
While our approach is also motivated by anticipating unseen elements, we learn to extrapolate in a highdimensional feature space (rather than pixels, voxels, or semantic categories) and in a self-supervised manner without relying on human annotations.
Further, the proposed model learns from egocentric video sequences captured by other agents without assuming access to detailed 3D reconstructions.
Learning representations for visual navigation : Researchers are making steady advances in visual navigation architectures (Gupta et al., 2017;
Chen et al., 2019; Fang et al., 2019; Wijmans et al., 2020; Chaplot et al., 2020b; Lenton et al., 2020).
Prior work further improves navigation by pretraining representations using supervised image annotations (Gupta et al., 2017; Anderson et al., 2018b;
Chen et al., 2019; Sax et al., 2020; Chaplot et al., 2020c), mined object relations (Yang et al., 2018), unannotated videos (Chang et al., 2020), and active exploration (Du et al., 2021).
However, these approaches are limited to learning image-level functions.
In contrast, we learn environmentlevel representations of image sequences conditioned on the camera poses.
We show that the two are complementary by augmenting a SoTA navigation architecture from Fang et al.
A PPROACH
We propose environment predictive coding (EPC) to learn environment-level representations via selfsupervision on video walkthroughs (Sec. 3.1).
To demonstrate the utility of these representations, we integrate them into a transformer-based architecture and reﬁne them for individual navigation tasks (Sec. 3.2).
Finally, we describe our procedure for generating video walkthroughs (Sec. 3.3).
3. E NVIRONMENT PREDICTIVE CODING
Our hypothesis is that it is valuable for an embodied agent to learn a predictive coding of the environment.
The agent must not just encode the individual views it observes, but also learn to leverage the encoded information to anticipate the unseen parts of the environment.
We propose to train an encoder-decoder model that observes a subset of views from a video walkthrough in a 3D environment, and then infers the features of unobserved views conditioned on their camera poses .
To successfully infer features from unobserved views, the encoder must build a predictive representation of the underlying physical environment using the observed views.
By transferring this encoder to a navigation agent, we equip the agent with the structural and semantic priors of 3D environments to quickly perform new tasks in new spaces, like mapping the house or room goal navigation.
We propose the self-supervised task of masked-zone prediction to achieve this goal (see Fig. 2).
For this task, we use a dataset of egocentric video walkthroughs containing (possibly noisy) RGBD and odometer sensor readings collected by other agents deployed in various unseen simulated environments (Fig. 2, top).
These environments are inaccessible for interactive RL training, and the agent policies are ignorant of our eventual downstream tasks (see Sec. 3.3)
Our method works as follows.
First, we automatically segment each video into “zones” which contain temporally contiguous sets of frames.
We then learn an environment encoder via the self-supervised maskedzone prediction task on the segmented videos.
Finally, we transfer the learned environment encoder to an array of downstream navigation-oriented tasks.
We explain each step in detail next.
Zone generation At a glance, one might ﬁrst consider masking arbitrary individual frames in the training videos.
However, doing so can result in poor representation learning since shared content from nearby unmasked frames can make the prediction task trivial.
Instead, our approach masks zones of frames at once.
We deﬁne a zone to be a set of temporally contiguous frames in the video.
By choosing a large-enough temporal window, we can reduce the amount of shared content with temporally adjacent zones.
Given a video walkthrough of size L , we divide it into zones { Z 0 , Z 1 , · · · } of length N (selected through validation):
[Equation]
where t s = i × N, t e = min
( ( i + 1) × N, L ) , o t is the RGB-D sensor reading, and p t is the camera pose obtained by accumulating odometer readings from time 0 to t (see Fig.
2, bottom left).
While two zones may share visual content, we ﬁnd that this simple approach works better than strictly limiting the overlap between zones (see Appendix. A8).
Thus, the learning is guided by predicting parts of the environment that were never seen as well as those seen from different viewpoints.
Masked-zone prediction Having segmented the video into zones, we next present our EPC masked-zone prediction task to learn environment embeddings (see Fig. 2)
The main idea is to infer unseen zones in a video by previewing the context spanning multiple seen zones.
We randomly divide the zones into seen zones { Z si } ni =1 and unseen zones { Z ui } mi =1 .
Given the seen zones and the mean camera pose from an unseen zone p ui , we need to infer a feature encoding of the unseen zone Z ui .
To perform this task efﬁciently, we ﬁrst extract visual features x t from each RGB-D frame o t in the video using pretrained CNNs (described in Sec. 3.2).
These features are concatenated with the corresponding pose p t and projected using an MLP M to obtain the image-level embedding.
The target features for the unseen zone Z ui are obtained by averaging 1 all the MLP projected features:
[Equation]
where we mask out the pose (i.e., p = −→ 0 ) in the target to avoid trivial solutions.
We use a transformer encoder-decoder model (Vaswani et al., 2017) to infer the zone features (see Fig.
2, bottom).
An environment encoder uses self-attention over the image-level embeddings from all the seen zones, i.e., {M ([ x,p ])
| ∀ ( x,p ) ∈
Z si , ∀ i ∈ [1 ,n ] } , to generate the environment embeddings E .
A zone decoder then attends to E conditioned on the camera pose p ui from the unseen zone and predicts the zone features:
[Equation]
Following Fang et al.
(2019), we transform all poses in the input zones relative to p u i before encoding, which provides the model an egocentric view of the world.
As we will show in experiments, conditioning on pose is critical to learn useful representations.
The environment encoder, zone decoder, and projection function M are trained end-to-end using noise-contrastive estimation (Gutmann & Hyv¨arinen, 2010).
We use ˆ f ui as the anchor and f ui from Eqn. 2 as the positive.
We sample
[Figure]
# ∈ [' !"
[Figure]
#$ !
[Figure] [Figure]
$" !
[Figure]
[Figure]
negatives from other unseen zones in the same video 2 and from all zones in other videos.
The former discourages the model from simply learning video-speciﬁc textures and patterns; the latter expands the pool of negatives which was found to be beneﬁcial in prior work (Chen et al., 2020b;a).
The loss for the
i th unseen zone in the video is:
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation] (4) where f (cid:48) k are zone features from other videos, and sim ( q, k ) =
exp (cid:0) q · k | q || k | 1 τ (cid:1) is a similarity measure with temperature τ = 0 .
The idea is to predict zone representations that are closer to the ground truth, while being sufﬁciently different from the negative zones.
Since the seen and unseen zones may only have limited overlap, the model needs to effectively reason about the geometric and semantic context in the seen zones to perform this task.
We qualitatively analyse the masked-zone prediction results from the learned EPC model in Fig. 4.
3. E NVIRONMENT EMBEDDINGS FOR EMBODIED AGENTS
Once a self-supervised environment encoder is trained, we transfer the encoder to various agents for performing navigation-oriented tasks.
To this end, we integrate our pretrained environment encoder into the Scene Memory Transformer (SMT) from Fang et al.
While our idea is potentially applicable to other memory models, our choice of SMT is motivated by the recent successes of transformers in NLP (Devlin et al., 2018) and vision (Sun et al., 2019b; Fang et al., 2019).
We brieﬂy overview the SMT architecture (Fig. 3, center).
We extract visual features from each
RGB-D input (concatenated along the channel dimension) using a ResNet-18 image encoder (He et al., 2016).
We store the visual features and agent poses { ( x i , p i ) } ti =0 observed during the episode in a scene memory.
We then use an environment encoder to perform self-attention over the scene memory and generate a rich set of environment embeddings E .
We use a policy decoder to attend to
E conditioned on the inputs o t =
[ x t , p t ] at time t, and use the decoder outputs to sample an action a t and estimate the value v t .
We detail each component in the Appendix A2.
To incorporate our EPC environment embeddings into SMT, we ﬁrst initialize the image encoder using CNNs pre-trained using a state-of-the-art MoCo-v2 method (Chen et al., 2020b) (see Fig.
3, right).
The image encoder is pre-trained for 7,000 epochs on RGB-D images sampled from the video walkthroughs generated for EPC pre-training.
Note that this is the CNN used to pre-extract RGB-D image features during EPC masked-zone prediction.
Next, and most importantly, we initialize the environment encoder with our EPC pre-training on masked-zone prediction (see Fig. 3, left).
We then ﬁnetune the SMT model end-to-end on the downstream task—whether that is room navigation, object visitation, ﬂee, or area coverage (cf.
Sec. 4).
As we will demonstrate in the experiments, initializing the environment encoder using EPC leads to 2-6x higher sample efﬁciency and better performance when compared to initializing only the image encoder.
[Figure]
3. V IDEO WALKTHROUGH GENERATION
As discussed previously
EPC relies on a dataset of video walkthroughs containing RGB-D and odometer readings for self-supervision.
Since EPC aims to learn the natural statistics of real-world environments (and not agent behavior), it sufﬁces to use walkthroughs recorded in diverse environments by agents performing their day-to-day tasks.
In particular we do not require that the scenes used to generate walkthroughs are available for interactive policy learning, nor that the agent behaviors used for generating walkthroughs are tied to the downstream navigation tasks of interest.
This means that we can, in parallel, collect videos from different agents operating in a large number of environments across geographical locations.
We now realize this process in photorealistic Gibson scenes; we leave leveraging in-the-wild consumer videos as a challenge for future work.
We generate 2 sets of egocentric video walkthroughs by deploying simulated agents in photorealistic scanned indoor environments from Gibson 3 (Xia et al., 2018).
Strong Exploration: an SMT agent that was trained for area coverage on MP3D, and 2) Weak Exploration: a heuristic forwardbiased navigation agent that moves forward until colliding, then turns randomly, and repeats.
The weak exploration agent generates less-informative walkthroughs as it tends to repeatedly explore the same regions.
Experimenting with EPC on both types of walkthrough allows us to test its dependence on walkthrough quality.
Methods robust to this quality are desirable since it may be easier to collect lower quality walkthroughs on a large-scale.
In both cases, the agents explore each Gibson environment starting from multiple locations for 500 steps per location and record the RGB-D and odometer readings.
This results in 5,047 videos per agent, which we divide into an 80-20 train/val split (i.e., ∼ 2M training frames).
As we will show in Sec. 4, the EPC encoders learned on these walkthroughs are applicable to navigation tasks not tied to the exploration agents.
E XPERIMENTS
First, we review the experimental setup (Sec. 4.1).
We then evaluate the pre-trained EPC embeddings on multiple downstream navigation-oriented tasks in unmapped environments (Sec. 4.2).
Finally, we present an ablation study of the data and proxy task used for EPC (Sec. 4.3).
4. E XPERIMENTAL SETUP FOR DOWNSTREAM TASKS
We perform experiments on the Habitat simulator (Savva et al., 2019b) with Matterport3D (MP3D) (Chang et al., 2017) and Gibson (Xia et al., 2018), two challenging and photorealistic 3D datasets with 90 and 572 scanned real-world indoor environments, respectively.
Our observation space consists of 171 × 128 RGB-D images and agent pose p
= ( x, y, θ ) relative to the starting pose at t = 0 (obtained by accumulating odometer readings).
Our action space consists of: MOVEFORWARD by 25cm , TURN - LEFT by 30 ◦ , TURN - RIGHT by 30 ◦ , and STOP (only for RoomNav).
For all methods, we assume noise-free sensing during training, then evaluate with both noise-free and noisy sensing (pose, depth).
We perform interactive RL training on 61 MP3D train scenes.
We evaluate the learned policies on 11 val and 18 test scenes in MP3D, and 14 val scenes in Gibson.
These are disjoint from the
Gibson train scenes used for walkthroughs.
We use episode lengths of T = 1000 for MP3D, and
T = 500 for Gibson since the scenes are smaller.
We further divide Gibson results into small and large environments (Ramakrishnan et al., 2020; Chaplot et al., 2020b).
We evaluate our approach on four standard navigation tasks from the literature:
Area coverage (Chen et al., 2019; Chaplot et al., 2020b; Ramakrishnan et al., 2021):
Agent is trained to maximize the area covered (in m 2 ) within a ﬁxed time budget.
Flee (Gordon et al., 2019): Agent is trained to maximize the ‘ﬂee distance’, i e., the geodesic distance (in m ) between its start and end positions, within a ﬁxed time budget.
Object visitation (Fang et al., 2019; Ramakrishnan et al., 2021):
Agent is trained to maximize the # object categories visited within a ﬁxed time budget.
An object is ‘visited’ if it is visible to the agent within 1 .
5m of the agent’s position.
We report both the # categories and # instances visited.
RoomNav (Savva et al., 2017; Wu et al., 2019; Narasimhan et al., 2020):
Agent is trained to ﬁnd the nearest room instance of a provided room category.
We evaluate using the two standard metrics: SPL and Success (Anderson et al., 2018a).
Please see Appendix A4 for the task details.
We choose these tasks since they capture different forms of geometric (tasks 1 & 2) and semantic (tasks 3 & 4) inference in 3D environments.
All tasks are different from the agent behavior in the weak exploration trajectories.
While task 1 is well-aligned with the strong exploration trajectories, all other tasks are distinct.
Object visitation is a challenging task since the agent must navigate within 1 .
5m of each object and directly observe it.
RoomNav is the most challenging task since the goal is to visit a speciﬁc room (not arbitrary exploration).
We compare our approach to several baselines.
Scratch baselines : We randomly initialize the visual encoders and policy, and train them end-to-end for each task.
Images are encoded using ResNet-18.
Agent pose and past actions are encoded using
FC layers.
Reactive (scratch) has no memory.
RNN (scratch) uses a 2-layer LSTM as the temporal memory.
SMT (scratch) uses a Scene Memory Transformer (SMT) for aggregating observations.
SMT (MidLevel) :
This SMT-based model uses image encoders pre-trained for various supervised visual-tasks.
The image encoders are kept frozen during RL (Sax et al., 2020; Zamir et al., 2020).
SMT MoCo
This SMT-based model uses a ResNet-18 image encoder pre-trained using MoCov2 (Chen et al., 2020b) and ﬁnetunes it end-to-end during RL.
This is an ablation of our model from
Sec. 3.2 where only the image encoder is pre-trained on the video walkthroughs.
This SoTA imagelevel pre-training is critical to isolate the impact of our proposed environment-level pre-training.
SMT (Video) :
This SMT-based model is inspired by Dense Predictive Coding (Han et al., 2019).
The image encoder is initialized using weights from SMT (MoCo).
It pre-trains the environment encoder as a ‘video-level’ model on the video walkthroughs.
It takes 25 consecutive frames as input and predicts the features from the next 15 frames (following timespans used in Han et al.
During SSL, we mask the input camera poses and query based on time, unlike EPC which uses input poses and queries based on pose during SSL.
ANS :
This is our implementation of the SoTA hierarchical policy from Active Neural SLAM (Chaplot et al., 2020b), upgraded to perform depth-based occupancy mapping (instead of using only RGB).
All RL agents receive RGB-D images and pose inputs.
All models are trained for 13M-15M frames with 60 parallel processes.
We train the scratch and ANS baselines for 2M more frames to account for the 2M frames in video walkthroughs used for other methods.
Both SMT (MoCo) and SMT (Video) are given the advantage of using strong exploration walkthroughs for SSL (see Sec. 3.3).
However, we test EPC with either strong or weak exploration walkthroughs (variants denoted as S.E and W.E, respectively).
See Appendix A5 for optimization hyperparameters.
4. D OWNSTREAM TASK PERFORMANCE
We transfer the EPC features to downstream navigation tasks.
Tab. 1 shows the complete results.
For brevity in the text, we also report the change in a method’s performance relative to another [Table] [Figure] as 4 values: (∆ AreaCoverage % , ∆ Flee % , ∆ ObjectVisitation % , ∆ RoomNav %) , where each value is the mean relative change over all datasets and metrics for the corresponding task in Tab. 1.
SMT (scratch) outperforms RNN (scratch) and Reactive (scratch) on area coverage and object visitation.
This is in line with results reported by Fang et al.
and veriﬁes our SMT implementation.
Image-level pre
-training from SMT (MoCo) provides good gains over SMT (scratch), particularly on Flee and RoomNav.
It relatively improves over SMT (scratch) by (4% , 20% , [Table]
Environment Predictive Coding (EPC) effectively combines environment-level and image-level pretraining to substantially improve over only image-level pre-training in SMT (MoCo).
EPC is most beneﬁcial for the challenging semantic tasks of object visitation and RoomNav.
For example, EPC (W.E) improves by (9% , 24% , 24% , 56%) relative to SMT (MoCo).
Furthermore, SMT (Video)— the video-level pretraining strategy—signiﬁcantly underperforms EPC, as it ignores the underlying spatial information during self-supervised learning.
This highlights EPC’s value in reasoning about the underlying 3D spaces of the walkthrough videos instead of treating them simply as video frames.
We also don’t observe a statistically signiﬁcant difference between EPC (W.E) and EPC (S.E) on majority of the tasks.
Thus, EPC is robust to the walkthrough quality, and does not require the agent behavior in walkthroughs to be tied to downstream tasks.
This is advantageous since diverse and lower quality walkthroughs may be easier to obtain in practice.
Finally, EPC matches or outperforms the state-of-the-art ANS (Chaplot et al., 2020b) on both geometric and semantic tasks.
We relatively improve over ANS by (2% , 42% , 13% , N/A ) with EPC (W.E) and (2% , 40% , 15% , N/A ) with EPC (S.E).
On area coverage, while ANS performs better on Gibson,
EPC outperforms it on the larger MP3D environments.
On RoomNav, ANS fails to learn the STOP action resulting in 0% success.
Note that ANS was not originally designed with a STOP action.
Unlike the other agents with 4 actions, ANS has 24 × 24 location actions (see Appendix A3).
Adding a STOP action to this huge action space makes RL optimization difﬁcult.
In Fig.
5, we plot the validation metrics of SMT (MoCo) and EPC over the course of training.
We see that environment-level pretraining offers signiﬁcantly better sample efﬁciency: both EPC variants reach the best performance of SMT (MoCo) up to 6 × faster.
This conﬁrms our hypothesis: transferring environment-level encoders learned via spatial reasoning helps embodied agents learn faster compared to the current approach of transferring image-level encoders alone (Sax et al., 2020). [Table]
4. A BLATION STUDY
We now present an ablation study of EPC in Tab. 2 to answer three key questions.
Does EPC need spatial conditioning for SSL?
One of our key contributions is to infer the unseen zone features conditioned on pose p ui from the unseen zone (see Eqn. 2).
We now try conditioning only on the time t ui when the agent visited the unseen zone.
Note that the dataset, inputs to the environment encoder, and the loss function still remain the same.
Consider rows A and B in Tab. 2.
The performance for EPC w/o query-p dramatically declines by (12% , 25% , 30% , 61%) relative to
EPC.
Thus, it is critical to use spatial queries for SSL with EPC.
Does EPC need noise-free sensors to gather videos for SSL?
In Tab. 1, we assumed noise-free depth and odometer sensors for gathering video walkthroughs.
We now inject noise into the depth and odometer readings from videos to analyze EPC’s robustness.
We use popular existing noise models: the depth noise uses disparity-based quantization, high-frequency noise, and low-frequency distortion (Choi et al., 2015); the odometry noise is based on data collected from a LoCoBot, and accumulates over time leading to drift in pose (Ramakrishnan et al., 2020; Chaplot et al., 2020b). Additionally, we replace the GT pose with pose estimates from RGB-D based visual odometry (Zhao et al., 2021).
Consider rows A, C, D, and E in Tab. 2.
EPC is robust to sensory noise in the videos and its transfer performance is stable.
The performance declines on area coverage and ﬂee when we inject depth noise (noisy-d), and on object visitation and RoomNav when we further inject odometry noise (noisy-dp). The performance remains similar when using estimated pose from visual odometry (noisy-d+vo).
Overall, EPC retains its advantages over the best baselines from Tab. 1.
Does EPC need global input context during SSL?
We use the global input context from multiple seen zones for EPC masked-zone prediction (Fig. 2).
We now reduce this context to a more localized part of the video.
Speciﬁcally, we modify the task to use the local context from a randomly sampled chunk of 25 contiguous frames as input, and infer features from the next 15 frames conditioned on pose.
Consider rows A and E in Tab. 2.
When we remove the global context (EPC w/o global), the performance deteriorates by (3% , 20% , 4% , 2%) relative to EPC.
As shown in Appendix A7, we observe an even more drastic reduction of (7% , 17% , 8% , 21%) for EPC w/o global with the strong exploration videos.
This suggests that the global input context is beneﬁcial for SSL with EPC.
In Appendix A9, we further demonstrate the robustness of EPC’s agent policies to injection of sensory noise during evaluation.
C ONCLUSIONS
We introduced Environment Predictive Coding, a self-supervised approach to learn environmentlevel representations for embodied agents.
By training on video walkthroughs generated by other agents, our model learns to infer missing content through a masked-zone prediction task.
When transferred to multiple downstream embodied agent tasks, the resulting embeddings lead to 2 - 6 × higher sample efﬁciency and upto 57% better performance when compared to the current practice of transferring only image-level representations.
Our work highlights the advantages of self-supervised learning in agent videos by predicting features grounded in the spatial structure of 3D environments.
In future work, we aim to extend our idea to in-the-wild videos and apply it to multimodal tasks such as embodied question answering, instruction following, and audio-visual navigation.
R EPRODUCIBILITY STATEMENT
We describe our method in Sec. 3 with architecture diagrams and equations to support the textual description.
We describe our experimental setup in Sec. 4.1, and provide the necessary hyperparameters in Appendix A5.
In Appendix A12, we provide more details about the overall experimental pipeline for EPC and refer to appropriate sections in the main paper and appendix as needed.
We will release the source code for generating the walkthroughs, performing EPC pre-training and downstream task ﬁnetuning, and the episode datasets for RoomNav upon acceptance.
E THICS STATEMENT
Our idea for self-supervised learning relies on having a dataset of video walkthroughs captured in a diverse set of environments.
In our experiments, we demonstrate a proof-of-concept by generating these walkthroughs in photorealistic Gibson scenes using simulated agents.
However, obtaining such a large-scale dataset of walkthroughs in real-world environments can be challenging due to privacy concerns.
To create such a dataset, we would need to carefully consider the privacy of the people and personal belongings recorded as a part of this data.
We perform experiments on the publicly available Gibson and Matterport3D datasets.
Since these datasets primarily contain data from work places and economically well-off residences, our results may be inﬂuenced by the social and geographical distribution of the scanned buildings in these datasets.
In experiments, we demonstrate good generalization of embodied policies trained in MP3D train scenes to novel MP3D test scenes and novel Gibson val scenes.
However, it is possible that the generalization may be impacted when the policies are evaluated in environments that are significantly outside the distribution of these datasets.
Appendices
We provide additional information about the experimental settings as well as quantitative results to support the experiments from the main paper.
Below is a summary of the sections in the Appendix:
• (§A1)
Supplementary visualization • (§A2) Scene memory transformer • (§A3)
Model architectures • (§A4)
Additional task details • (§A5)
Hyperparameters • (§A6)
Sample efﬁciency on Gibson • (§A7)
Ablations of EPC with strong exploration videos • (§A8)
Complex zone creation schemes for EPC • (§A9)
Noise robustness of learned policies • (§A10) ANS performance on Flee • (§A11) Downstream task performance vs. time • (§A12) EPC pipeline for reproducibility • (§A13) Relation to visual place recognition
A S UPPLEMENTARY VIDEO
The supplementary visualization “epc visualization supp.pdf” provides a brief overview of environment predictive coding and the masked-zone prediction task.
We also show an example of intervideo retrievals (similar to Fig.
4 from main paper).
In the demonstrated example, we show that
our
EPC model is able to accurately retrieve the masked-zone from the current scene, and even retrieve similar zones from other scenes.
A S CENE MEMORY TRANSFORMER
We provide more details about individual components of the Scene Memory Transformer (Fang et al., 2019).
As discussed in the main paper, the SMT model consists of a ResNet-18 visual encoder that extracts features x t from RGB-D inputs, and a scene memory for storing the visual features { x i } ti =0 and agent poses { p i } ti =0 seen during an episode.
The environment encoder uses self-attention on the scene memory to generate a richer set of environment embeddings { e i } t i =1 .
The policy decoder attends to the environment embeddings using the inputs o t at time t, which consist of the visual feature x t , the agent pose p t , and optionally, the navigation goal g t .
The outputs of the policy decoder are used to sample an action a t and estimate the value v t .
Next, we discuss the details of the individual components.
V ISUAL ENCODER
At each time step t , the visual encoder extracts features x t from the RGB-D input.
[Equation]
where r t and d t are the RGB and depth inputs, respectively.
We consider two visual encoders for our work.
The ﬁrst variant is a modiﬁed ResNet-18 encoder from (Wijmans et al., 2020) where the number of output channels are halved and the BatchNorm layers are replaced by GroupNorm.
We used this encoder for the scratch models, SMT (MoCo), SMT (Video), and the EPC variants in the main paper.
Next, we consider MidLevel features derived from various pre-trained CNNs to solve midlevel perception tasks (Sax et al., 2020; Zamir et al., 2020).
For RGB inputs, we extract features from the pre-trained models in the max-coverage set proposed in Sax et al.
These include 4 encoders trained for predicting surface normals, keypoints, semantic segmentation, and 2.5D segmentation.
For depth inputs, we extract features from pre-trained models that predict surface normals and keypoints from depth (Zamir et al., 2020).
These encoders are kept frozen during reinforcement learning, following Sax et al.
MidLevel features were used for SMT (MidLevel) in the main paper.
We do not use the MidLevel encoder for EPC since we found the
ResNet-18 encoder to be more efﬁcient and achieved better performance.
S CENE MEMORY
It stores the visual features x t derived from the input images and the agent poses p t at each time-step t .
Following Fang et al.
(2019), the pose p t consists of the 2D coordinates ( x, y ) of the agent, heading θ , and the time t .
While we assumed that the ground-truth pose ( x t , y t , θ t ) was available for downstream tasks in the main experiments, we evaluate the impact of noisy odometry in Appendix A9.
A TTENTION MECHANISM
Following the notations from Vaswani et al.
(2017), we deﬁne the attention mechanism used in the environment encoder and policy decoder.
Given two inputs X ∈ R n 1 × d x and Y ∈ R n 2 × d y , the attention mechanism attends to Y using X as follows:
[Equation]
where Q X ∈ R n 1 × d k , K Y ∈ R n 2 × d k , V Y ∈ R n 2 ×
d v are the queries, keys, and values computed from X and Y as follows: Q X = XW q , K Y =
Y W k , and V Y
=
Y W v .
W q , W k , W v are learned weight matrices.
The multi-headed version of Attn generates multiple sets of queries, keys, and values to obtain the attended context C ∈ R n 1 × d v . [Equation]
[Equation]
We use the transformer implementation from PyTorch (Paszke et al., 2019).
Here, the multi-headed attention block builds on top of MHAttn by using residual connections, LayerNorm (LN) and fully connected (FC) layers to further encode the inputs.
[Equation]
where H = LN ( MHAttn ( X, Y ) + X ) , and MLP has 2 FC layers with ReLU activations.
We now describe the memory attention mechanisms used in SMT.
At time t , the agent recieves features x t , pose p t , and optionally the navigation goal g t .
First, we transform the pose vectors { p i } ni = from the scene memory relative to the current agent pose p t .
This allows the agent to maintain an egocentric view of the past inputs (Fang et al., 2019).
Next, the environment encoder performs selfattention between the features stored in the scene memory M to obtain the environment encoding:
[Equation]
Next, the policy decoder attends to the environment encodings E using a linear projection of the current observation f =
M ([ x t , p t , g t ]) .
[Equation]
P OLICY
It consists of two FC layers corresponding to the actor π and the critic V .
Given the output D of the PolicyDecoder, the policy samples an action a t ∼ π
( D ) and predicts the value function v t =
V ( D ) .
Please see the next section for more details on the exact architectures used.
A M ODEL ARCHITECTURES
We provide the architectures for the RNN (scratch), SMT (scratch), and ANS baselines in Fig.
A1.
For Reactive (scratch), we remove the “Transformer Encoder” and “Transformer Decoder” blocks from SMT (scratch) and directly feed the input features to the policy.
For SMT (MidLevel), we replace the ResNet encoder with the MidLevel encoders discussed in Sec. A2.
For SMT (MoCo),
SMT (Video), and EPC, we use the same architecture as SMT (scratch) and initialize pre-trained weights according to the method used.
For ANS, Fig.
A1 describes a global policy that samples as actions an (x, y) location or STOP.
The input occupancy map is a 300 × 300 × 2 egocentric map where the two channels track occupied [Figure] and explored regions.
It captures a 48m × 48m region around the agent.
A local policy, which is a low-level navigator, then navigates to the (x, y) location for 25 steps.
We use an analytical mapper+planner as the local policy as this was found to perform as well as a learned policy (Chaplot et al., 2020b;a).
For area coverage, ﬂee, and object visitation, we use a 48m × 48m action space for the global policy.
We try both discrete ( 24 × 24 actions) and continuous action spaces, and pick the best choice based on the validation performance.
For RoomNav, we use a discrete action space and add STOP as an additional action, giving us 24 × 24 + 1 = 577 actions.
A A DDITIONAL TASK DETAILS
We provide more details about the object visitation and RoomNav tasks below.
O BJECT VISITATION
To determine if an object is visited, we check if it is within 1 . 5m of the agent, present in the agent’s ﬁeld of view, and if it is not occluded Ramakrishnan et al.
We use a shaped reward function that rewards the agent for visiting a new object category and a cell-coverage reward to encourage exploration (similar to Fang et al.
[Equation]
(11) where O t , C t are the number of object categories and 2D grid-cells visited by time t .
For MP3D, we use all 21 object categories deﬁned for the ObjectNav task in Habitat (Batra et al., 2020; Savva et al., 2019a): chair, table, picture, cabinet, cushion, sofa, bed, chest of drawers, plant, sink, toilet, stool, towel, tv monitor, shower, bathtub, counter, fireplace, gym equipment, seating, clothes
For Gibson, we use the semantic annotations from the 3D Scene Graph dataset (Armeni et al., 2019).
We evaluate on the following categories:
We report both the number of categories and instances of the above objects visited.
For MP3D, we evaluate on 550/900 validation/test episodes.
For Gibson, we evaluate on 300 validation episodes.
We do not split Gibson semantic results into small/large since there are only 6 val semantic scenes.
R OOM N AV
For an agent to successfully reach a speciﬁed room category, it needs to be within 0 .
1m of a navigable location inside any room belonging to that category.
We use a shaped reward function (Savva et al., 2019a):
[Equation]
(12) where d t is the distance to the nearest target room location, S t indicates success at time t , and 0 .is
a slack penalty.
We evaluate on Gibson and MP3D rooms of the following categories: bathroom, bedroom, office,
kitchen
, living room, dining room
We measure both success rate and SPL, i.e., success weighted by path length (Anderson et al., 2018a).
We calculate SPL similar to (Batra et al., 2020), where the shortest path length is deﬁned based on the target room closest to the agent’s starting point.
We evaluate on 410/ val/test episodes on MP3D, and 598 val episodes on Gibson.
We use 500-step episodes following Narasimhan et al.
(2020) (unlike 1000-step episodes for other tasks on MP3D).
A H YPERPARAMETERS
All models are trained in PyTorch (Paszke et al., 2019) with DD-PPO (Wijmans et al., 2020) for 13M-15M frames with 60 parallel processes.
Since SMT (MoCo), SMT (Video) and EPC beneﬁt from 2M frames of off-policy experience in the video walkthroughs, we train the scratch and ANS baselines for 2M more frames to account for this.
Note that SMT (MidLevel) is already pre-trained on 4M frames of annotated data.
We detail the list of hyperparameter choices for different tasks and models in Tab.
A1.
For ANS, we use 4 PPO mini-batches, 4 PPO epochs (not 2), and entropy coefﬁcient of 0 . 003 .
For SMT (Video), we randomly sample 40 consecutive frames in the video and predict the ﬁnal 15 frames from the initial 25 frames (following time-spans from Han et al.
For EPC , we randomly mask out m zones of size N in the video and predict them from the remaining video.
We selected these values based on a grid-search and the validation performance on downstream tasks.
For area coverage, ﬂee, and object visitation, we found N = 5 , m = 6 to work best.
For RoomNav, we found N = 5 − 40 , m = 6 to work best on weak exploration videos, and N = 40 , m = 6 to work best on strong exploration videos. The hyperparameter search results are shown in Figs. A2, A3.
A S AMPLE EFFICIENCY ON G IBSON
We plot the Gibson validation performance as a function of training experience in Fig.
A4.
EPC achieves 2 - 6 × higher sample efﬁciency through environment-level pre-training when compared to the only image-level pre-training from SMT (MoCo).
A A BLATIONS OF EPC WITH STRONG EXPLORATION VIDEOS
In Sec.
4.3 from the main paper, we presented an ablation study of EPC (W.E).
We now provide the corresponding results for EPC (S.E) in Tab. A2.
Our conclusions remain the same as in Sec. 4.3.
(1) EPC needs spatial conditioning, i
e., to query using pose during SSL (instead of only time).
(2) EPC is robust to noise in depth and odometer sensors used for collecting the video walkthroughs.
(3) EPC beneﬁts from global input context during SSL (instead of a local input context of 25 frames).
Over the course of arriving at our ﬁnal EPC zone creation scheme, we tried an alternative scheme that attempted to strictly minimize the overlap between the seen and unseen zones. We measured [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
j e c t c a t ego r i e s
S u cc e ss r a t e
[Figure]
F l ee ( m ) # ob j e c t c a t ego r i e s
S u cc e ss r a t e
[Figure]
Object visitation
RoomNav
EPC zone size # c a t ego r i e s
v
i s i t ed      (5, 2) (5, 4) (5, 6) (20, 6) (40, 6) (5-40, 6)
EPC zone size
R oo m N a v
S u cc e ss    (5, 2) (5, 4) (5, 6) (20, 6) (40, 6) (5-40, 6)
Area coverage
Flee
Object visitation
RoomNav
A r ea ( m 2 )
F l ee ( m ) # ob j e c t c a t ego r i e s
S u cc e ss r a t e
A r ea ( m 2 )
F l ee ( m ) # ob j e c t c a t ego r i e s
S u cc e ss r a t e
RL Optimization
Optimizer
Adam
Learning rate 0.
# parallel actors 60/
PPO mini
-batches 2*
PPO epochs 2*
PPO clip param 0.
Value loss coefﬁcient 0.
Entropy coefﬁcient 0.
Advantage estimation
GAE
Normalized advantage?
Yes*
Training episode length 1000*
GRU history length
# training steps (in millions) 13SMT hyperparameters Hidden size Scene memory length Scene memory stride 2/ # attention heads # encoder layers
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
the 3D point-cloud overlap between every pair of frames in a video walkthrough (projected using depth, camera pose, and camera intrinsics), and clustered the frames that have signiﬁcant overlap together using hierarchical agglomerative clustering (Lukasov ´ a, 1979).
We deﬁned each cluster to be a zone, and used the same SSL training pipeline from Sec. 3.
Tab. A3 shows the results on the
MP3D validation split on 3 tasks.
The spatial overlap variant works reasonably well.
However, it is signiﬁcantly more complex than our original proposal (requiring camera intrinsics, 3D point-cloud overlap computations and clustering), and performs worse. Since the videos are unconstrained, the agent might revisit the same areas multiple times, causing too many frames to be grouped together into a single zone.
This results in very irregular zone sizes for masked-zone prediction across videos.
This affects the quality of EPC SSL training. By not strictly limiting the overlap between zones, the agent learns by both inferring seen features (but from different viewpoints), as well as unseen features. Our current method is simpler, and works better.
A N OISE ROBUSTNESS OF LEARNED POLICIES
In the main paper, we assumed the availability of ground-truth depth and pose sensors for downstream tasks (In Tab. 2, we added pose and depth noise to the walkthrough videos only).
Now, we relax these assumptions and re-evaluate all methods by injecting noise in the depth and odometer sensors for downstream tasks (same noise models from prior work that we applied in Sec. 4.3), without any noise-correction.
This is a common evaluation protocol for assessing noise robustness (Chen et al., 2019; Ramakrishnan et al., 2021; Chen et al., 2021).
Note that the noise in agent’s pose estimate increases over time since the per-step noise in odometer readings are accumulated.
We compare the noise-robustness of all methods in Tab. A4.
As expected, the performance declines as we add noise to more sensors (depth, then pose).
We observe larger deterioration in performance when noise is added to depth.
Note that this is likely due to the domain shift in the depth inputs between noise-free training and noisy evaluation.
Nevertheless, most approaches are reasonably stable.
EPC outperforms all methods when all noise sources are added.
ANS declines rapidly in the absence of noise-correction due to accumulated map errors. [Table]
A ANS PERFORMANCE ON F LEE
ANS relies on a global policy that samples a spatial goal location for navigation.
A local navigation policy then executes a series of low-level actions to reach that goal.
Our qualitative analyses indicate that the global policy overﬁt to large MP3D environments.
It often samples far away exploration targets, relying on the local navigator to explore the spaces along the sampled direction.
However, this strategy fails in the small Gibson-S environments (typically a single room).
Selecting far away targets results in the local navigator oscillating in place trying to exit a single-room environment.
This does not affect area coverage much because it sufﬁces to stand in the middle of a small room and look at all sides.
A D OWNSTREAM TASK PERFORMANCE VS . TIME
We show the downstream task performance as a function of time in Fig.
A5.
For each model, we train with 2 random seeds and evaluate with 3 random seeds.
We report the mean and the 95% conﬁdence interval in the plots.
EPC converges faster than using only image-level pretraining in
SMT (MoCo), and outperforms a state-of-the-art Active Neural SLAM (ANS) on most tasks.
A EPC PIPELINE FOR REPRODUCIBILITY
We provide more details regarding the EPC pipeline for pre-training and downstream task evaluation in order to aid reproducibility.
We will release the source code upon acceptance.
The EPC pipeline involves the following key steps.
Generating video walkthroughs: We deploy weak and strong exploration agents in Gibson 2+ training environments (Wijmans et al., 2020) to gather 2 sets of video walkthroughs.
Pre-training image-level representations: We pre-train a ResNet-18 encoder using MoCo v2 (Chen et al., 2020b) on images sampled from walkthroughs.
Extracting features for video walkthroughs: Using the pre-trained ResNet-18 encoder, we extract visual features for each image all video walkthroughs.
Pre-training environment-level features via EPC Given the dataset of video walkthrough features + agent camera poses, we pre-train the environment encoder and projection MLP M . 5.
Transferring to downstream tasks: We transfer both the image-level and environment-level encoders to an SMT agent and ﬁnetune it end-to-end a navigation task.
We will now detail each of the above individual steps.
Generating video walkthroughs We use the 332 Gibson 2+ training scenes deﬁned in Wijmans et al.
(2020) for generating video walkthroughs.
These walkthroughs consist of RGB-D and odometer readings for each step.
We use two types of agents to generate video walkthroughs (see Sec. 3.3).
The strong exploration agent is a SMT (scratch) model trained to perform area coverage on 61 MP3D training scenes for 10M frames.
The weak exploration agent is a simple forward-biased heuristic that keeps moving forward till colliding.
Upon collision, it rotates left or right by a randomly sampled angle between [0 ◦ , 150 ◦ ] , and then continues to move forward.
Given an environment, each agent is spawned on a set of different locations and executed for 500 steps from each start location.
We select the start locations by clustering the navigable locations within the environment into K clusters using K-means clustering.
The number of clusters K is decided based on the navigable area in each environment. .
Across all the Gibson 2+ train environments, we generate 2 sets of 5, video walkthroughs (one set per exploration agent).
We randomly divide each set of videos into an 80-20 train/val split.
Pre-training image-level representations To learn a strong image-level encoder from the video walkthroughs, we randomly sample RGB-D images from the walkthroughs and train a ResNetencoder using MoCo-v2 for 7,000 epochs.
Speciﬁcally, we sample a batch of N = 128 videos, and sample M = 4 frames per video.
The frames are uniformly spread across the video.
This gives us a batch of 512 images for each MoCo-v2 update.
We use the publicly available MoCo-v implementation from https://github.com/facebookresearch/moco .
Since this image-level pre-training corresponds to the important SMT (MoCo) baseline, we provide it the beneﬁt of using strong exploration walkthroughs.
Extracting features for video walkthroughs For lower memory consumption and faster training, we ﬁrst pre-extract the visual features for each image in the video walkthroughs dataset.
We extract features using the ResNet-18 encoder pre-trained using MoCo-v2 on the strong exploration walkthroughs.
We use the same image encoder for extracting features from both the weak and strong exploration walkthroughs.
We observed in experiments that the quality of the image-encoder did not vary signiﬁcantly based on the walkthroughs used.
By using a single pre-trained image encoder for the EPC variants, we can consistently evaluate the impact of walkthrough quality on EPC pre-training.
Note that the image encoders are kept frozen during EPC pre-training.
Pre-training environment-level features via EPC Having extracted the visual features for the walkthrough dataset (either strong or weak exploration), we perform the environment-level pretraining of the environment-encoder E and projection MLP M (see Fig.
2 from the main paper).
We use our proposed masked-zone prediction task for self-supervised pre-training.
Given a video walkthrough of length L , we divide it into zones { Z 0 , Z 1 , · · · } of length N as indicated in Eqn.from the main paper.
We randomly mask m zones (i.e., the unseen zones) from the video, and learn representations by predicting the features for an each unseen zone Z ui .
Given the list of visual features and camera poses ( x, p ) sampled from the seen zones, we transform the poses p relative to the average camera pose p ui from zone Z ui .
We then transform each ( x, p ) using the projection
MLP M .
The outputs of the projection MLP M are used as inputs for the environment encoder which generates rich environment-embeddings E using self-attention.
We then use the environment embeddings E and the average camera pose p ui as inputs to the zone decoder, which predicts the zone feature ˆ f ui .
The target feature for the zone is the average of MLP projected features from the zone (see Eqn. 2).
The model is trained using the contrastive loss in Eqn. 4.
The hyperparameters are provided in Appendix A5.
Transferring to downstream tasks For transferring the EPC pre-trained encoders, we need to transfer both the image-level ResNet-18 encoder pre-trained on MoCo-v2, and most importantly, the environment encoder pre-trained on EPC.
The SMT model is initialized using the image and environment encoders and ﬁne-tuned end-to-end of the downstream navigation task of interest.
Please refer to Sec.
3.2 from the main paper for details regarding the transfer.
For area coverage, ﬂee, and object coverage, we use the starting positions from the PointNav-v1 dataset available on https://github.com/facebookresearch/habitat-lab .
For RoomNav, we create our own habitat-compatible datasets for Gibson and MP3D which we will release along with the source code.
A R ELATION TO VISUAL PLACE RECOGNITION
Prior work on visual place reognition build maps and recognize whether the current visual information is from a place recorded in the map (Lowry et al., 2015).
A place could be an exact location (Kuipers, 2000), rooms (Kuipers, 2000), or locally distinctive regions (Kuipers & Byun, 1991;
Bailey et al., 1999). Zones in EPC can be viewed as a speciﬁc ‘place’ in 3D environments. However, instead of learning place-level features for recognizing previously seen places (Chen et al., 2017), we learn environment-level features by inferring features for unseen places. [Table]
A r ea c ov e r e d ( m 2 )
F l ee
[Figure]
i
[Figure]
a
[Figure]
t e go r i e s # i n s t a n ce s
[Figure]
r e d ( m 2 )
F l ee d i s t a n ce ( m )
[Figure]
l ee d i s t a n
[Figure]
( m
[Figure]
( x100 )
[Figure]