P an GEA : The Panoramic Graph Environment Annotation Toolkit (2021)

Abstract: PanGEA, the Panoramic Graph Environment Annotation toolkit, is a lightweight toolkit for collecting speech and text annotations in photo-realistic 3D environments. PanGEA immerses annotators in a web-based simulation and allows them to move around easily as they speak and/or listen. It includes database and cloud storage integration, plus utilities for automatically aligning recorded speech with manual transcriptions and the virtual pose of the annotators. Out of the box, PanGEA supports two tasks – collecting navigation instructions and navigation instruction following – and it could be easily adapted for annotating walking tours, finding and labeling landmarks or objects, and similar tasks. We share best practices learned from using PanGEA in a 20,000 hour annotation effort to collect the Room-Across-Room dataset. We hope that our open-source annotation toolkit and insights will both expedite future data collection efforts and spur innovation on the kinds of grounded language tasks such environments can support.

## introduction the release of high-quality 3d building and street captures @xcite @xcite @xcite has galvanized interest in developing embodied navigation agents that can operate in complex human environments. based on these environments, annotations have been collected for a variety of tasks including navigating to a particular class of object (objectnav) @xcite , navigating from language instructions aka visionand-language navigation (vln) @xcite @xcite , and vision-and-dialog navigation @xcite . to date, most of these data collection efforts have required the development of custom annotation tools. to expedite future data collection efforts, in this paper we introduce pangea, an open-sourced annotation toolkit designed for these settings. 1 specifically, pangea assumes an environment represented by discrete navigation graphs connecting high-resolution 360°panoramas, where each node represents a unique viewpoint in the environment and actions involve moving between these viewpoints. examples of suitable environments include the indoor buildings from matterport3d @xcite (using the navigation graphs from @xcite ) and the street-level environments from streetlearn @xcite . out of the box, pangea supports two annotation modes: the guide task and the follower task. in the guide task, guides look around and move through an environment to follow a pre-defined path and attempt to create a navigation instruction for others to follow. in the follower task, annotators listen to a guide's instructions and attempt to follow the path. these annotation modes are based on the visionand-language navigation (vln) setting proposed by @xcite . however, compared to similar annotation tools, pangea includes substantial additional capabilities, notably: we next describe pangea's capabitilities in more detail. in the final section we share some best practices learned from using pangea to collect rxr, which required more than 20,000 annotation hours. during the guide task, in parallel to the annotator's voice recording, pangea captures a timestamped record of the annotator's virtual camera movements, which we call a pose trace. by default, pangea is configured to use firebase foot_1 , saving the guide's audio recording to a cloud storage bucket, and the transcript, pose trace and other metadata to a cloud database for post processing. inspired by localized narratives (pont-tuset et al., 2020), pangea includes a utility to automatically align each guide's pose trace with the manual transcript of their audio recording. this is achieved by using a speech to text service foot_2 to first generate a noisybut-timestamped automatic transcription. pangea then using dynamic time warping to align tokens in the automatic transcript to the manual transcript before propagating timestamps from the automatic to the manual transcription (figure 2 ). the result is fine-grained synchronization between the transcribed text, the pixels seen, and the actions taken by the guide. follower task in the follower task (figure 1 , right), followers begin at a specified starting point in an environment and are asked to follow a guide's 2020) instructions. they observe the environment and navigate as the guide's audio plays. followers can skip forward or backward in the audio recording by clicking on an audio waveform representation of the guide's recording. this allows them to skip over periods of silence or to listen to part of the audio again. once the follower believes they have reached the the end of the path, or they give up, they indicate they are done and the task ends. note that although the follower task supports audio instructions, it can be easily adapted to replace the audio instruction with a textual instruction. this was the approach taken by @xcite . as with the guide task, the follower's pose trace is recorded and saved to a cloud database, along with the timestamp of the guide's audio that the follower listened to at each moment. this allows the follower's visual percepts and actions to be accurately aligned with text tokens in the guide's instructions. similarity between the annotated (guide) path and the follower path is also a natural measure of the joint quality of both the guide and the follower annotations. in the experiments for rxr, the path extracted from the follower's pose trace was also used as additional supervision when training follower agents, since it represents a step-by-step account of how a human solved the task and the visual inputs they focused on in order to do so @xcite . deployment pangea comes with several demos using a very simplistic environment. to deploy pangea for a new large-scale collection effort requires completing 3 main steps: ## observations and best practices pangea was developed for the collection of the rxr dataset, a 20,000+ hour annotation effort based on matterport3d indoor scenes. many of the lessons learned during this collection effort are codified in the pangea toolkit. for example, we found that uploading recorded audio at the end of the guide task was time consuming, and so in the final version of pangea the wav file is uploaded in the background while the annotator is busy transcribing their audio. we also found that audio annotations could include long periods of silence, so we provided follower annotators with an audio waveform visualization and an interface to skip over silence. some other observations and best practices for reducing annotation times and improving annotation quality are shared in this section. ## future applications there are many potential future applications of pangea and tools that could be built based on the design decisions discussed above. we are particularly excited about multi-agent problems that collect pose traces from multiple participants as they coordinate via language, such as hide and seek games or tasks where items must be moved from one location to another to satisfy goals or solve puzzles, similar to cerealbar @xcite .