RMM : A Recursive Mental Model for Dialogue Navigation (2020)

Abstract: Language-guided robots must be able to both ask humans questions and understand answers. Much existing work focuses only on the latter. In this paper, we go beyond instruction following and introduce a two-agent task where one agent navigates and asks questions that a second, guiding agent answers. Inspired by theory of mind, we propose the Recursive Mental Model (RMM). The navigating agent models the guiding agent to simulate answers given candidate generated questions. The guiding agent in turn models the navigating agent to simulate navigation steps it would take to generate answers. We use the progress agents make towards the goal as a reinforcement learning reward signal to directly inform not only navigation actions, but also both question and answer generation. We demonstrate that RMM enables better generalization to novel environments. Interlocutor modelling may be a way forward for human-agent RMM where robots need to both ask and answer questions.

## introduction a key challenge for embodied language is moving beyond instruction following to instruction generation, which can require understanding the listener. the turn-based dialogue paradigm raises a myriad of new research questions, from grounded versions of traditional problems like co-reference resolution @xcite to explicitly modeling theory of mind in order to consider the listener's ability to understand generated instructions @xcite . in this paper, we develop end-to-end dialogue agents to navigate photorealistic, indoor scenes to reach goal rooms. we train agents using the human-human collaborative vision-and-dialogue navigation (cvdn) @xcite dataset. cvdn dialogues are turn-based, with a navigator following guide instructions and asking questions when needed. modeling turn-based dialogues involves four core challenges: c1 a navigator deciding when to ask a question. c2 generating navigator questions. c3 generating guide question answers. c4 generating navigator actions. prior work has addressed individual components of turn-based dialogue modeling. this work is the first to train navigator and guide agents to perform end-to-end, collaborative dialogues with question generation (c2), question answering (c3), and navigation (c4) conditioned on dialogue history. theory of mind @xcite ) posits that efficient questions and answers build on a shared world of experiences and referents. to communicate efficiently, people model both a listener's mental state and the effects of their actions on the world. modeling future worlds in navigation @xcite and control @xcite are open research questions, and we approximate solutions through a recursive mental model (rmm) of a conversational partner. our agent spawns instances of itself to simulate the ef-fects of dialogue acts before asking a question or generating an answer to estimate their effects on navigation. viewed as a single system, the agents cooperatively search through the space of dialogues to efficiently perform embodied navigation. ## related work and background we build on research in multimodal navigation and the wider literature involving goal oriented dialogue. instruction following tasks an embodied agent with interpreting natural language instructions and visual observations to reach a goal @xcite @xcite @xcite . these instructions describe step-by-step actions the agent needs to take, and can involve the creation of speaker models for data augmentation that provide additional instructions @xcite . this paradigm has been extended to longer trajectories and outdoor environments @xcite , as well as to agents in the real world @xcite . in this work, we focus on the the simulated, photorealistic indoor environments of the matterport dataset @xcite , and go beyond instruction following alone to a twoagent dialogue setting. navigation dialogues task a navigator and a guide to cooperate to find a destination. previous work includes substantial information asymmetry between the navigator and guide @xcite . information asymmetry can take the form of the navigator seeing a bird's eye, abstract semantic map while the guide sees egocentric simulation frames @xcite , affecting the kind of dialog possible when low-level visual cues cannot be grounded by the navigator. other work only investigates the navigation portion of the dialogue without considering text question generation and answering @xcite . going beyond models that perform navigation from dialogue history alone @xcite @xcite , or decide when to ask navigator questions but do so as a simple "help" flag with oracle responses @xcite , in this work we train two agents: a navigator agent that asks questions, and a guide agent that answers those questions. anderson et al. (2018) fried et al. (2018) narayan-chen et al. (2019) nguyen and daumé iii (2019) chi et al. (2020) thomason et al. (2019) multimodal dialogue takes several forms. in visual dialogue @xcite , an agent answers a series of questions about an image that may require dialogue context. reinforcement learning gives strong performance on this task @xcite , and such paradigms have been extended to producing multi-domain visual dialogue agents @xcite . @xcite presents a similar paradigm, where agents use visual properties of objects to reason about which referent meets various constraints. identifying visual attributes can also lead to emergent communication between pairs of learning agents @xcite . goal oriented dialogue systems can help a user achieve a predefined goal, from booking flights to learning kitchen tasks @xcite @xcite . modeling goal-oriented dialogue requires skills that go beyond language modeling, such as asking questions to clearly define a user request, querying knowledge bases, and interpreting results from queries as options to complete a transaction. many recent task oriented systems are data-driven and trained end-to-end using semisupervised or transfer learning methods @xcite . however, these datadriven approaches may lack grounding between the text and the environment state. reinforcement learning-based dialogue modeling @xcite @xcite can improve completion rate and user experience by helping ground conversational data to environments. our work creates a two-agent dialogue task, building on the cvdn dataset @xcite of human-human dialogues. in that dataset, a human n avigator and guide collaborate to find a goal room containing a target object. the n avigator moves through the environment, and the guide views this navigation until the n avigator asks a question in natural language (c1, c2). then, the guide can see the next few steps a shortest path planner would take towards the goal, and produces a natural language response (c3). dialogue continues until the n avigator arrives at the goal (c4). we model this dialogue between two agents: 1. questioner (q) & navigator (n ) 2. guide (g) we split the first agent into its two roles: question asking (c2) and navigation (c4). as input, the agents receive the same data as their human counterparts in cvdn. specifically, both agents (and all three roles) have access to the entire dialogue and visual navigation histories, in addition to a textual description of the target object (e.g., a plant). the n avigator uses this information to execute on a sequence of actions composed of: forward, left, right, look up, look down, and stop. the questioner asks for specific guidance from the guide. the guide is presented with the navigation and dialogue histories as well as the next five shortest path steps to the goal, given as a sequence of image observations those steps produce. agents are trained on human-human dialogues of natural language questions and answers from cvdn. individual question-answer exchanges in that dataset are underspecified and rarely provide simple step-by-step instructions like "straight, straight, right, ...". instead, exchanges rely on assumptions of world knowledge and shared context @xcite , which manifest as instructions rich with visuallinguistic co-references such as should i go back to the room i just passed or continue on? the cvdn release does not provide any baselines or evaluations for the interactive dialogue setting we present, and instead focuses solely on navigation (c4). we use the same metric as that work, "goal progress" in meters-the distance reduction between the n avigator's starting position and ending position with respect to the goal location. dialogue navigation proceeds by iterating through the three roles until either the n avigator algorithm 1: dialogue navigation @xmath0; @xmath1; a ∼ n (hist); loc, @xmath2, loc, hist); while @xmath3, loc) ; // question @xmath4, goal, @xmath5; o ∼ o(hist, loc, q, s) ; // answer hist ← hist + (q, o); for @xmath6; // move hist ← hist + a; end end ## models we introduce the recursive mental model (rmm) as an initial approach to our new full dialogue cvdn task formulation. key to this approach is allowing component models (n avigator, questioner, and guide) to learn from each other and roll out possible dialogues and trajectories. we compare our model to a traditional sequence-to-sequence baseline, and we explore data augmentation using the speaker-follower method @xcite . ## results in ## analysis we analyze the lexical diversity and effectiveness of generated questions by the rmm. ## conclusions and future work we present a two-agent task paradigm for cooperative vision-and-dialogue navigation (cvdn). existing work in vision-and-language navigation is largely limited to navigation only (c4), sometimes with limited additional instructions (c4,c3). by contrast, this work requires navigation (c4), question asking (c2), and question answering (c3) components for learned, end-to-end dialogue. we find that simple speaker models are insufficient for the dialogue setting, and demonstrate promising results from a recursive rl formulation with turn taking informed by theory of mind.