Transformer-based Localization from Embodied Dialog with Large-scale Pre-training (2022)

Abstract: We address the challenging task of Localization via Embodied Dialog (LED). Given a dialog from two agents, an Observer navigating through an unknown environment and a Locator who is attempting to identify the Observer’s location, the goal is to predict the Observer’s final location in a map. We develop a novel LED-Bert architecture and present an effective pretraining strategy. We show that a graph-based scene representation is more effective than the top-down 2D maps used in prior works. Our approach outperforms previous baselines.

## introduction a key goal in ai is to develop embodied agents that can accurately perceive and navigate an environment as well as communicate about their surroundings in natural language. the recently-introduced where are you? (way) dataset @xcite provides a setting for developing such a multimodal and multi-agent paradigm. this dataset (collected via amt) contains episodes of a localization scenario in which two agents communicate via turntaking natural language dialog: an observer agent moves through an unknown environment, while a locator agent attempts to identify the observer's location in a map. the observer produces descriptions such as 'i'm in a living room with a gray couch and blue armchairs. behind me there is a door.' and can respond to instructions and questions provided by the locator: 'if you walk straight past the seating area, do you see a bathroom on your right?' via this dialog (and without access to the observer's view of the scene), the locator attempts to identify the observer's location on a map (which is not available to the observer). this is a complex task for which a successful localization requires accurate situational grounding and the production of relevant questions and instructions. one of the benchmark tasks supported by way is 'localization via embodied dialog (led)'. in this task a model takes the dialog and a representation of the map as inputs, and must output a prediction of the final location of the observer agent. the model's performance is based on error distance between the predicted location of the observer and its true location. led is a first step towards developing a locator agent. one challenge of the task is to identify an effective map representation. the led baseline from @xcite uses 2d images of top down (birds-eye view) floor maps to represent the environment and an (x,y) location for the observer. this paper provides a new solution to the led task with two key components. first, we propose to model the environment using the first person view (fpv) panoramic navigation graph from matterport @xcite , as an alternative to top-down maps. second, we introduce a novel visiolinguistic transformer model, led-bert, which scores the alignment between navigation graph nodes and dialogs. led-bert is an adaption of vilbert @xcite for the led task, and we show that it outperforms all prior baselines. a key challenge is the small size of the way dataset (approximately 6k episodes), which makes it challenging to use transformer-based models given their reliance on large-scale training data. we address this challenge by developing a pretraining approach -based on @xcite ) -that yields an effective visiolinguistic representation. contributions: to summarize: in the masked language modeling schema, 15% of the input tokens are replaced with a [mask] token. the model is then trained to predict the true value of the input tokens which are masked using the other tokens as context. in the next sentence prediction schema, the model is trained to predicted if the two input sentences follow each other or not. bert is specifically trained on wikipedia and bookscorpus @xcite . vilbert vilbert @xcite ) is a multimodal transformer that extends the bert architecture @xcite to learn joint visiolinguistic representations. similar multi-modal transformer models exist @xcite @xcite @xcite . vilbert is constructed of two transformer encoding streams, one for visual inputs and one for text inputs. both of these streams use the standard bert-base @xcite backbone. the input tokens for the text stream are text tokens, identical to bert. the input tokens for the visual stream are a sequence of image regions which are generated by an object detector pretrained on visual genome @xcite . the input to vilbert is then a sequence of visual and textual tokens which are not concatenated and only enter their respective streams. the two streams then interact using co-attention layers which are implemented by swapping the key and value matrices between the visual and textual encoder streams for certain layers. co-attention layers are used to attend to one modality via a conditioning on the other modality, allowing for attention over image regions given the corresponding text input and vise versa.