Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020)

Abstract: We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and validators. We establish baseline scores for monolingual and multilingual settings and multitask learning when including Room-to-Room annotations (Anderson et al., 2018). We also provide results for a model that learns from synchronized pose traces by focusing only on portions of the panorama attended to in human demonstrations. The size, scope and detail of RxR dramatically expands the frontier for research on embodied language agents in photorealistic simulated environments.

## introduction vision-and-language navigation (vln) tasks require computational agents to mediate the relationship between language, visual scenes and movement. datasets have been collected for both indoor @xcite @xcite and outdoor @xcite environments; success in these is based on clearly-defined, objective task completion rather than language or vision specific annotations. these vln tasks fall in the goldilocks zone: they can be tackled -but not solved -with current methods, and progress on them makes headway on real world grounded language understanding. we introduce room-across-room (rxr), a vln dataset that addresses gaps in existing ones by (1) * first two authors contributed equally. including more paths that (2) counter known biases in existing datasets, and (3) collecting an order of magnitude more instructions for (4) three languages (english, hindi and telugu) while (5) capturing annotators' 3d pose sequences. as such, rxr includes dense spatiotemporal grounding for every instruction, as illustrated in figure 1 . we provide monolingual and multilingual baseline experiments using a variant of the reinforced cross-modal matching agent @xcite . performance generally improves by using monolingual learning, and by using rxr's follower paths as well as its guide paths. we also concatenate r2r and rxr annotations as a simple multitask strategy @xcite : the agent trained on both datasets obtains across the board improvements. rxr contains 126k instructions covering 16.5k sampled guide paths and 126k human follower demonstration paths. the dataset is available. 1 we plan to release a test evaluation server, our annotation tool, and code for all experiments. ## motivation a number of vln datasets situated in photorealistic 3d reconstructions of real locations contain human instructions or dialogue: r2r @xcite , touchdown @xcite , cvdn @xcite and reverie @xcite . rxr addresses shortcomings of these datasetsin particular, multilinguality, scale, fine-grained word grounding, and human follower demonstrations ( multilinguality. the dominance of high resource languages is a pervasive problem as it is unclear that research findings generalize to other languages @xcite . the issue is particularly severe for vln. @xcite translated(∼1k) english navigation instructions into chinese for a game-like simulated 3d environment. otherwise, all publicly available vln datasets we are aware of have english instructions. to enable multilingual progress on vln, rxr includes instructions for three typologically diverse languages: english (en), hindi (hi), and telugu (te). the english portion includes instructions by speakers in the usa (en-us) and india (en-in). unlike @xcite and like the tydi-qa multilingual question answering dataset @xcite ), rxr's instructions are not translations: all instructions are created from scratch by native speakers. this especially matters for vln, as different languages encode spatial and temporal information in idiosyncratic ways-e.g., how contact/support relationships are expressed @xcite , frame of reference @xcite , and how temporal accounts are expressed @xcite . scale. embodied language tasks suffer from a relative paucity of training data; for vln, this has led to a focus on data augmentation @xcite , pre-training @xcite @xcite , multi-task learning @xcite and better generalization through piece-wise curriculum design @xcite . to address this shortage, for each language rxr contains 14k paths with 3 instructions per path, for a total of 126k instructions and 10m words (based on whitespace tokenization). as illustrated in fine-grained grounding. like r2r, rxr's instructions are collected by immersing guide annotators in a simulated first-person environment backed by the matterport3d dataset @xcite and asking them to describe predefined paths. rxr also enhances each instruction with dense spatiotemporal groundings. guides speak as they move and later transcribe their audio; our annotation tool records their 3d poses and time-aligns the entire pose trace with words in the transcription. instructions and pose traces can thus be aligned with any matterport data including surface reconstructions (figure 1 ), rgb-d panoramas (figure 4 ), and 2d and 3d semantic segmentations. ## two-level path sampling we satisfy desiderata 1-3 using a two-level procedure. at a high-level, each path visits a sequence of rooms; these are simple paths with no repeated (room) vertices. such paths are not necessarily shortest paths. the low-level sequence is then the shortest panorama path, constrained by the room sequence. given the set of all such paths across all houses, the fourth desiderata is satisfied by iteratively selecting the path that most improves coverage while maintaining a bias against shortest paths. preliminaries movement in the simulator is based on a navigation graph. vertices correspond to 360-degree panoramic images, captured at approximately 2.2m intervals throughout 90 indoor environments. edges are navigable links between panoramas. @xcite also partition panoramas via human-defined room annotations. let p be an undirected graph of interconnected panoramas, with vertices p @xmath0, p j )∈e(p ). let a r be a set of disjoint room annotations; each room r @xmath1, as shown in figure 2a . we abbreviate (p 1 , • • • , p m ) as p 1:m . path generation we generate the set of all simple paths in r that traverse at most 5 rooms and two building levels. let r p @xmath2. as shown in figure 2b , for each room path r 1:n , we construct a directed graph p [r 1:n ] in which an edge (p i , p j ) exists if r p @xmath3, r p j ) is an edge in the room path. given p [r 1:n ], we sample the start p 1 and goal p m uniformly from r 1 and r n , respectively. the full panorama path p 1:m is then the shortest path between p 1 and p m in p [r 1:n ]. room size varies greatly, so this approach produces high path length variance. it also satisfies naturalness because people tend to ground instructions at the room level (e.g., exit through the carved wooden door on the other side of the room). we find such paths easy to describe even with as many as 20 edges. finally, these paths can approach their goal indirectly, as exemplified in figure 2b . ## data collection and metrics we immerse annotators in our own web-based version of the matterport3d simulator using the panoramic images and the navigation graph. compared to @xcite , our annotation tool has additional capabilities including speech collection, virtual pose tracking, and timealignment between transcript and pose. figure 4 gives an example instruction with accompanying guide and follower pose traces. here, we describe our collection process, analysis of the data, path evaluation metrics and simple baselines. guide task like r2r, our simulator has camera controls allowing continuous heading and elevation changes and movement between panoramas. guides look around and move to explore a provided path and attempt to create an instruction others can follow. r2r's guides create written instructions. now you are standing in-front of a closed door, turn to your left, you can see two wooden steps, climb the steps and walk forward by crossing a... now you are standing in-front of a closed door, turn to your left, you can see two wooden steps, climb the steps and walk forward by... ...crossing a wall painting which is to your right side, you can see open door enter... ...by crossing a wall painting which is to your right.. ...enter into it. this is a gym room, move forward, walk... ...right side, you can see open door enter into it. this is a gym room, move forward, walk... ...walk till the end of the room, you can see a grey... ...walk till the end of the room, you can... ...grey colored ball to the corner of the room, stand there, that's... ...can see a grey colored ball to the corner... ...that's your end point. ...corner of the room, stand there... ...that's your end point. in contrast, rxr's guides speak and the tool logs their entire virtual camera pose sequence. we use a 640 × 480 pixel viewing canvas and a camera vertical field of view of 75 degrees. this process is inspired by localized narratives @xcite , an image captioning dataset for which annotators move mouse pointers around images while talking about them. as with localized narratives, rxr guides transcribe their own recordings; this produces high quality text versions of the instructions. to align text and pose traces, we generate a time-stamped transcription using automatic speech recognition. 2 the transcription and asr output are aligned using dynamic time warping. the output of the guide task is an audio file, a tokenized, timestamped, manually-transcribed instruction, and a pose trace (a series of timestamped 6-dof camera poses). on average, guide task annotations (including both steps, performed back-to-back) take 458 seconds. for each language (english, hindi and telugu) we annotate 14k paths with three instructions each. in the english dataset, each path gets one us english instruction and two indian english instructions. of the 14k paths per language, 12.8k paths are common across all three languages, and 1.2k paths in each language are unique (equaling 16.5k paths in total). the fact that most paths are annotated 9 times (3 per language) creates interesting opportunities to study aligned instructions across languages. unique paths add variety and coverage. follower task as followers, annotators begin at the start of an unknown path and try to follow the guide's instruction. they observe the environment and navigate in the simulator as the guide's audio plays. they can pause, rewind and skip forward in the instruction. if they believe they have reached the the end of the path, or give up, they indicate they are done and rate the instruction's clarity and their confidence in their own navigation. on average, follower tasks take 132 seconds. the follower tasks objectively validate the quality of guide instructions based on whether the follower can succeed (i.e., reaching within 3m of the last panorama in the path). if the follower doesn't succeed, the guide instruction is paired with a second follower. if the second follower succeeds, the first follower annotation is discarded and replaced. if the second follower also fails, then the path is reenqueued to generate another guide and follower annotation. the most successful of the three resulting guide-follower pairs is selected for inclusion in rxr and the others are discarded. in addition to validating data quality, the follower task also trains annotators to be better guides-following bad instructions often helps one see how to produce better instructions. most importantly, we collect the pose trace of the follower as they execute the instruction. this provides an alternative path with dense grounding that we can compare to the guide's pose trace and use as an additional training signal. ## conclusion rxr represents a significant evolution in the scale, scope and possibilities for research on embodied language agents in simulated, photo-realistic 3d environments. rxr's paths better ensure that language itself will play a fundamental role in better agents. evaluating on three typologically diverse languages will help the community avoid overfitting to a particular language and dataset. we have only begun to explore the possibilities opened up by pose traces. whereas others have retro-actively refined r2r's annotations to get alignments between sub-instructions and panorama sequences @xcite , rxr provides wordlevel alignments to specific pixels in panoramas. this is obtained as a by-product of significant work on the annotation tooling itself and designing the process to be more natural for guides. finally, every instruction is accompanied by a follower demonstration, including a perspective camera pose trace that shows a play-by-play account of how a human interpreted the instructions given their position and progress through the path. we have shown that these can help with agent training, but they also open up new possibilities for studying grounded language pragmatics in the vln setting, and for training vln agents with perspective cameras -either in the graph-based simulator or by lifting rxr into a continuous simulator @xcite . a supplementary material annotators in total, 247 annotators contributed to rxr, with 97 based in the usa and the remainder based in india and contributing to the indian english, hindi and telugu annotations. the annotators were paid hourly wages that are competitive for their locale. they have standard rights as contractors. they were fluent in the language they were tasked with. we ensure that a guide does not annotate the same path twice. as followers, annotators do not follow their own guide instructions. furthermore, we have provided annotators multiple forms of feedback as they complete tasks. after a round of pilot instructions were collected, we provided detailed analysis of common patterns that produced poor instructions and clear guidelines for producing better instructions. annotators provided ui suggestions and interesting corner cases to us that allowed us to refine the simulator and annotation process before kicking off the full annotation process. throughout the process, annotators have had access to a dashboard that shows them their success rate as both guide and follower. we indicated that their success as guide and a follower should be above 80%. any annotator whose success is lower is either given further training or is taken off the task. unfortunately, we cannot release the audio instructions yet due to the impact of covid-19: our annotators had to complete the tasks from home, so we need to review all recordings for safety and privacy. we hope to include the audio in a future release.