Goal-oriented Vision-and-Dialog Navigation via Reinforcement Learning (2022)

Abstract: Vision-and-dialog navigation is a recent benchmark for evaluating the AI capabilities of perception, interaction, and decision making. While existing methods developed for this benchmark have demonstrated great successes, they mostly rely on large datasets, where data collection can be a challenge, and the learned policies are not adaptive to domain changes. In this paper, we focus on a new problem, referred to as goal-oriented vision-and-dialog navigation (GVDN), where an agent uses reinforcement learning techniques to compute dialog-navigation policies from trial and error. A robot conducts visual navigation to locate target objects, and can talk to a remote human operator as needed. Our remote human is able to provide guidance on navigation only if the robot correctly conveys its location through dialog. Experiments have been conducted using photo-realistic simulation environments. Results suggest that, our agent outperforms competitive baselines in success rate.

## introduction embodied mobile robots in human spaces need navigation capabilities to move from one place to another, while at the same time interacting with people. while many sensory modalities have been applied to robot navigation, vision is particularly attractive due to the significant achievements @xcite @xcite @xcite . as a result, researchers have studied the problem of vision-based robot navigation to demonstrate and evaluate a robot's simultaneous capabilities of perception and decision making @xcite . when we further incorporate people into the loop, dialog systems @xcite @xcite @xcite @xcite @xcite become important for human-robot communication @xcite @xcite . given the importance of robot interaction with both people and the environment, researchers have developed the "vision-and-dialog navigation" (vdn) benchmark, where a robot visually perceives an environment, talks to people using natural language, and makes decisions for navigation @xcite @xcite . existing vdn research focuses on training robots to follow language instructions, which describe the unambiguous or ambiguous goal and how to reach the goal @xcite @xcite @xcite . current vdn tasks have been modeled as sequence-to-sequence translation problems, and tackled using different supervised learning methods in the literature, e.g., @xcite ; @xcite . those methods require massive amounts of labeled data of robots performing dialog and navigation tasks collected from crowd-sourcing platforms. first, data collection is very expensive. a second issue is that solutions computed using those datasets are usually domaindependent, and tend to soon become less applicable or outdated due to unforeseen domain changes. as a result, there is a need of computing vdn policies from the experience of interacting with both people and the environments @xcite @xcite . to this end, we propose the goal-oriented vision-and-dialog navigation (gvdn) task, as a new benchmark, where a robot learns policies for vision-based perception, control for navigation, and dialog-based interaction. this new benchmark (gvdn) is our first contribution. a gvdn agent must clearly communicate its current location in dialog, so as to get useful guidance for navigation from a human operator. some areas are more confusing than the others for naviwell, i also see a tv. is that good enough? figure 1 : our gvdn framework includes two agents for overall coordination and dialog respectively. a central agent learns navigation policies, and decides when to initiate a conversation. a dialog agent, as its name indicates, helps the robot making decisions on what dialog actions to take, and which object to tell human. after a dialog is concluded, the remote human provides the robot with a sequence of navigation actions, whose usefulness depends on whether the dialog is successfully performed or not. those factors together make it necessary for the robot to figure out when, where, and how a dialog is performed. our second contribution is a novel gvdn approach that includes two reinforcement learning (rl) agents for overall coordination and dialog respectively, as shown in figure 1 . we have implemented and evaluated our gvdn approach using matterport3d @xcite . we evaluated our robot system based on its performance on the success rate and accumulative reward in gvdn tasks. compared with visual navigation baselines @xcite with and without a dialog system, we demonstrated the superiority of our rl-based approach on gvdn tasks. ## methodology in gvdn tasks, a robot needs to compute a policy to navigate to a predefined target object within an indoor environment. a remote operator can help the robot by suggesting navigation actions via natural language. the remote operator is helpful only if the robot can correctly communicate its current location; otherwise, the suggested navigation actions will be misleading due to the operator's wrong belief about the robot's location. at the beginning of each task, the robot is given as input a single target word token w t . given the robot's initial pose which includes the spatial position along with heading and elevation angles, the robot observes an initial rgb image, and the utterance of the operator. at each turn, the robot is expected to choose either a locomotive action to execute in the home environment or a dialog action to the operator for help to get closer to the target poses. the locomotive action set a loc consists of five actions corresponding to left, right, up, down, and forward. the forward action is defined to always move to the reachable viewpoint that is closest to the centre of the robot's visual field. the left, right, up and down actions are defined to move the camera by 30 degrees. the dialog actions a dial include inform, request, ok, unknown, and greeting. the inf orm action is to tell the operator objects which the robot has seen. the challenge of gvdn tasks comes from the combination of two action spaces, referring to a loc ∪ a dial . the large action space brings difficulties in naively learning from trial and error. to address this challenge, we exploit the main idea of hierarchical reinforcement learning for action space decomposition. algorithm overview: our gvdn framework includes two interdependent agents, namely a central agent, and a dialog agent. the central agent gives the commands to an action executor and the dialog agent. when the action executor receives locomotive actions in a loc , it will execute it in the environment. when the dialog agent receives a dialog activation action, it collects environment information and begins a dialog with the operator. the dialog agent aims to get commands (a sequence of navigation actions) from the operator which leads the robot to the target. to complete this sub-task, the dialog agent needs to learn to select a dialog action in a dial in each turn of the dialog. thus, the original action space is decomposed. the action space of the central agent is denoted as a cent shown in equation 1 . the dialog activation action is dispatched to the dialog agent to execute. the action space of the dialog agent is inherited from a dial . the central agent chooses locomotive actions to enable the robot to navigate towards the target algorithm 1 our gvdn algorithm 1: initialize qcent(s, a; θqc) of the central agent agent cent and q dial (s, a; θ qd ) of the dialog agent agent dial 2: initialize experience replay buffers b c and b d for the interaction of the central agent and the dialog agent 3: while a new target wt arrives do 4: initialize action sequence acts h from the operator, and collect initial state, s 5: while s / ∈ term do // start an interaction with the 3d environment 6: select a ← centralf unc(qcent, s, wt , acts h ) of the central agent 7: collect next state s , and reward r, then add (s, a, r, s ) to b c 13: randomly sample a batch from b c , and update agent cent via dqn 16: randomly sample a batch from b d , and update agent dial via dqn 17: end while poses. otherwise, the central agent queries the dialog agent and expects next suggested commands that help lead the robot from the current pose to the target. before beginning an episode of dialog, the dialog agent collects necessary information of the environment from the central agent. the information contains the current objects around the robot, which may be requested by the operator. at the end of each dialog, the central agent will get the next commands from the operator if the dialog was successful. otherwise, nothing is transferred from the dialog agent to the central agent. once the central agent has received the suggested commands from the dialog agent, it dispatches them to the action executor to execute. the two agents collaborate as above, until the robot arrives at the target poses. our gvdn algorithm: our gvdn robot is mainly composed of the central agent and the dialog agent that are respectively learned from the interactions with the 3d environment and the operator. algorithm 1 shows the learning process of our gvdn algorithm. algorithm 1 starts with an initialization of the two agents' experience replay buffers (b c and b d respectively), the value function q cent (s, a; θ qc ) of the central agent, and the value function q dial (s, a; θ qd ) of the dialog agent. before the start of each episode, action sequence acts h is initialized as empty for storing the next actions from dialogs. in the while loop (starting algorithm 2 dialoginteract input: scent, state from the central agent; q dial (•) for the dialog agent; b d replay buffer to store the dialog tuples output: acts, locomotive action sequence to be executed 1: initialize the next action sequences acts ← ∅ 2: collect initial dialog state, s 3: while s / ∈ term do 4: select a ← argmax a q dialog (s, a ; θ qd ), and execute a 5: collect next state s , and reward r 6: add dialog turn @xmath0, a, r, s ) to b d 7: the dialog agent directly interacts with the operator, where the interaction (invoked in algorithm 1) is presented in algorithm 2. in addition to the value function q dial (s, a; θ qd ) and the replay buffer b d for the dialog agent, there is parameter s cent for navigation state. algorithm 2 starts with an initialization of the next action sequence acts. the dialog agent interacts with the operator while collecting and saving experience in b d . at the end of each dialog episode, the dialog agent extracts the next action from the dialog history. the output of algorithm 2, acts, is returned to algorithm 1 to guide the central agent's navigation actions. ## experiments experiments have been conducted in a realistic 3d indoor simulation platform, called matter-port3d @xcite , which allows an embodied agent to virtually move throughout a scene by adopting poses coinciding with panoramic viewpoints. each panoramic viewpoint is comprised of 18 images captured from a single 3d position. we manually add action noise into the simulated robot's navigation behaviors, i.e., there is 0.1 probability that the robot fails to execute selected actions and keeps still. we conduct experiments in two house environments, referred to as house-i and house-ii. house-i includes 20 viewpoints, 8 rooms, and 200 objects; house-ii is bigger and includes 50 viewpoints, 15 rooms, and 274 objects. 1 task specification: we on average select 4 objects as target sets and annotate the target locations and poses in each house environment. the start pose of the robot is randomly selected, which is no more than 15 steps from the target locations. if acting optimally, the average number of actions to reach the target from the starting position is 11. when the robot arrives at the target pose @xmath1, it will receive reward 10 (otherwise -5). ## conclusion in this work, we focus on goal-oriented visionand-dialog navigation (gvdn), where an agent is not provided with any data, and needs to learn navigation-dialog policies from trial and error toward achieving long-term goals. our gvdn algorithm enables robots to learn from the simultaneous experiences of interacting with the environment via navigation, and interacting with the human via dialog. our gvdn algorithm outperforms competitive baselines that are incapable of performing dialog actions or use rule-based dialog systems in success rate and task completion efficiency.