VISITRON : Visual Semantics-Aligned Interactively Trained Object-Navigator (2022)

Abstract: Interactive robots navigating photo-realistic environments need to be trained to effectively leverage and handle the dynamic nature of dialogue in addition to the challenges underlying vision-and-language navigation (VLN). In this paper, we present VISITRON, a multi-modal Transformer-based navigator better suited to the interactive regime inherent to Cooperative Vision-and-Dialog Navigation (CVDN). VISITRON is trained to: i) identify and associate object-level concepts and semantics between the environment and dialogue history, ii) identify when to interact vs. navigate via imitation learning of a binary classification head. We perform extensive pre-training and fine-tuning ablations with VISITRON to gain empirical insights and improve performance on CVDN. VISITRON’s ability to identify when to interact leads to a natural generalization of the game-play mode introduced by Roman et al. (2020) for enabling the use of such models in different environments. VISITRON is competitive with models on the static CVDN leaderboard and attains state-of-the-art performance on the Success weighted by Path Length (SPL) metric.

## introduction large pre-trained transformer-based language models @xcite are ubiquitous in natural language processing (nlp) and have performed very well in interactive settings such as open-domain @xcite and task-oriented dialogue @xcite . the success of transformers and the pretrain/fine-tune paradigm in nlp has also inspired their adoption in vision-and-language research, with cross-modal representations being learned @xcite and utilized towards tasks like image and object captioning, visual question answering, visual commonsense reasoning and visual dialogue. ## approach the policy for ndh (and vln) can be decomposed into an encoder-decoder setup, π θ = f θ e • f θ d : we model π θ by visitron, a visio-linguistic transformer-based model. visitron's encoder is structurally similar to oscar's transformer @xcite . this is by design to enable easy transfer of visual semantics-aligned representations learned from disembodied image-text data. we make navigation-specific modifications to oscar, but they are all structured as augmentations of modules instead of removal of network components, thus enabling us to use the pre-trained weights of oscar's transformer to initialize large portions of our encoder. the augmentations are described in section 3.1. as with oscar, the input to vis-itron's encoder is represented as word-tag-image (w, q, v), where w and q are the sequence of word embeddings of the text and object tags respectively, and v is the sequence of region features of the image. we represent the panorama in 36 views, extract faster r-cnn @xcite region features r ′ from each view and add positional vector p, @xmath0, p). to incorporate 3d direction, we add direction embedding d to the region features, @xmath1. d is a 128-dimensional orientation vector represented by repeating [sin ϕ; cos ϕ; sin ω; cos ω] cls] [tar] plant [nav] go left from here? [gui] no, pass the bed, [mask] the table take right [sep] bed table [mask] [sep] 32 times where ϕ and ω are heading and elevation poses. in addition to the standard [cls] and [sep], we also use [tar], [nav], [gui] as delimiter tokens for the initial target hint, navigator's questions and the guide's answers respectively. while this input structure is dialogue-specific, it is amenable to instructions-based datasets for multi-tasking. ## experiments in this section, we first describe the evaluation metrics we adopt. we then describe and discuss our experimental observations from performing ablations during visitron pre-training and finetuning respectively. we present our observations for question-asking classification for cvdn, establishing a strong baseline for future models. we finally present and discuss our observations from submitting our model checkpoints to the static evalai leaderboard for cvdn. ## conclusion and future work we presented visitron, a transformer-based navigator designed to identify and acquire visiolinguistic concepts and semantics and make decisions, all key traits for interactive navigation inherent to cvdn. we demonstrated the efficacy of our approach via experiments and ablations. we proposed generalizing the game-play regime introduced with rmm @xcite to enable interactive fine-tuning and evaluation of visitronlike models with pre-trained guides. the trade-off between gp and spl in dialogue-based navigation, sim-to-real transfer @xcite and robustness in dialogue-based navigation in presence of speech recognition errors @xcite are all important problems that merit detailed investigation in future work.