Spatially-Aware Speaker for Vision-and-Language Navigation Instruction Generation (2024)

Abstract: Embodied AI aims to develop robots that can understand and execute human language instructions, as well as communicate in natural languages. On this front, we study the task of generating highly detailed navigational instructions for the embodied robots to follow. Although recent studies have demonstrated significant leaps in the generation of step-by-step instructions from sequences of images, the generated instructions lack variety in terms of their referral to objects and landmarks. Existing speaker models learn strategies to evade the evaluation metrics and obtain higher scores even for low-quality sentences. In this work, we propose SAS (Spatially-Aware Speaker), an instruction generator or Speaker model that utilises both structural and semantic knowledge of the environment to produce richer instructions. For training, we employ a reward learning method in an adversarial setting to avoid systematic bias introduced by language evaluation metrics. Empirically, our method outperforms existing instruction generation models, evaluated using standard metrics. Our code is available at https://github.com/gmuraleekrishna/SAS.

## introduction incorporating language understanding in robots has been a long-standing goal of the nlp and robotic research community. specifically, the vision-language navigation (vln) task requires robots to follow natural language instructions grounded on vision to navigate in human living spaces. although humans generally follow navigational instructions well, training robots to follow natural language instructions remains a challenging problem. detailed navigation instructions may include landmarks, actions, and destinations. recent work has succeeded in improving instruction understanding of robots by augmenting instruction and trajectory training data @xcite 2023). they showed that using machine-generated instructions from a large number of navigational paths sampled from real houses helps robots navigate successfully even in previously unseen environments. however, there is still room for improvement, as the quality of machine-generated instructions is clearly lower compared to human annotations @xcite . in this work, we present a novel instruction generation model that can produce a variety of humanlike instructions using semantic and structural cues from the environment. our method uses rooms, interesting landmarks, objects, inter-object relations, object locations, and spatial features to produce richer instructions that can be used by robots and humans alike (fig. 1 ). our spatially-aware speaker (sas) model generates information-rich instructions by leveraging expert demonstrations that map trajectories to verbal directions. incorporating spatial references within these instructions is critical, as they convey the environmental layout, highlight key landmarks relevant to the actions taken, and gauge the progression of navigation. at its core, sas employs a sequence learning framework that is fine-tuned through a combination of adversarial learning rewards and multiple objec-tives aimed at enhancing its linguistic generation capabilities. the architecture of sas is based on an encoder-decoder model, which processes a sequence of viewpoints and corresponding actions that define a navigational path, subsequently generating a coherent set of instructions. during the encoding phase, the model extracts vital information from visual inputs, such as object categories (e.g., cupboard, bed), spatial relationships between objects (e.g., on top of, near, under), object placements, and significant landmarks within the viewpoint (e.g., bedroom, kitchen). these elements are combined with navigational actions to form a comprehensive vision-action representation that computes the temporal order. the decoding phase acquires linguistic capabilities by linking this latent representation to the instructions encountered during training. an adversarial learning objective is introduced to encourage the generation of varied sentences, mitigating the potential biases that automatic evaluation metrics introduce. through this novel approach, sas outperforms existing instruction generation models on vln datasets evaluated using standard language evaluation metrics. our contribution is as follows. ## problem definition here we present the task aimed at generating linguistic instructions from navigational demonstrations. the generation of instructions is posited as the converse operation to the standard vln task, where an agent executes a navigation instruction in a house. for this, we develop a speaker agent that synthesises a coherent set of natural language instructions from a sequence of navigational actions along a trajectory. at each discrete time step t, the speaker agent is presented with a panoramic visual observation o t and a directional action a t that signifies the transition to the next viewpoint within the trajectory. upon completion of a navigational episode, the agent outputs the complete instruction of the traversed path, that is, @xmath0, . . . , w l }. formally, the objective of the speaker is to minimise the negative log-likelihood for ground-truth instruction x conditioned on trajectory @xmath1, . . . , o n } with parameter θ: prior work in vln have shown that more instruction examples can improve an agent's performance in previously unseen environments @xcite . hence, to supplement training data, we mix parts of trajectories from the fgr2r dataset @xcite (which is derived from the r2r @xcite dataset) to obtain additional instruction-trajectory pairs. r2r dataset contains paths and humanannotated instructions for navigating inside 3d scanned house environments from the matter-port3d (mp3d) dataset @xcite . further fine-grained (turn-by-turn) or microinstructions are available in the fgr2r dataset. we adopt fgr2r to enhance our training data by algorithmically combining parts of its trajectories, creating new instruction-trajectory pairs. unlike rem @xcite , which randomly mixes data from different houses, we only mix trajectories from the same house to ensure visual and object referral consistency. first, we identify key edges in the graph to mix trajectories, focusing on start edges ε start and end edges ε end of navigation paths. these edges are crucial as they relate to start and end of instructions (e.g. "walk away from the desk, turn right"). edges with micro-instructions 1 that do not contain a noun, verb are ignored to avoid partial or inconsequential (wait there) actions. then, we mix the remaining transition edges ε trans to form a trajectory from ε start to ε end . nodes that are too close to each other, leading to a lack of visual diversity or repetitive micro-instructions are also avoided. we connect edges based on the following criteria: the final instruction combines these microinstructions, and the trajectory is a sequence of edges (as shown in fig. 2 ). using this method, we generated 162k instruction-trajectory pairs with path lengths ranging from 5m to 20m. the dataset foot_1 averages 7.27 views per trajectory, a mean path length of 14.4m, and about 82 words per instruction. more dataset statistics are added to the supplementary material. ## spatially-aware speaker (sas) model in this section, we present our spatially-aware speaker model. sas is an encoder-decoder model which generates an instruction for a trajectory when the sequence of viewpoints and actions are provided. the trajectory encoder produces visualaction context from the trajectory viewpoints and the corresponding actions. this context is used by instruction decoder to generate instruction. to incorporate spatial and semantic awareness, we provide three crucial pieces of information to the model, namely: action encoding, structural encoding and semantic encoding. these are explained in the following sections. action encoding the action taken between viewpoints in a trajectory is represented by action encoding. the visual action encoding is the current heading and elevation of the agent with respect to the next view direction. the relative elevation (θ) and heading (ϕ) angles are encoded as e @xmath2, sin θ, cos ϕ, sin ϕ]. structural encoding structural encoding provides knowledge of the egocentric locations of objects with respect to the speaker. in the panoramic view of each viewpoint, we extract an object's location in the image frame, as well as its size and distance to the agent in order to represent a complete pose of the object relative to the agent. the image frame location is obtained from the location of the object's bounding box detected by a faster r-cnn @xcite detector trained on the visual genome @xcite dataset. the size and distance of the object are obtained by projecting (inverse pinhole camera projection) the bounding box to the point cloud and measuring the centroid and volume of the contained point cloud. this provides an estimate of the size and distance to form the object descriptions. effectively, the structural encoding is a combination of object features f o , object location (c x , cy), size s o and distance d o , respectively, i.e. semantic encoding to provide inter-object relationships we use 1600 object classes faster-rcnn and relationships extracted from concept-net @xcite . the object-to-objectroom semantic features are a combination of glove embedding g of the respective token, furthermore, we encode the relationship from room to object using the in the relation as for each viewpoint, we encode one e room and one e obj per view direction with the highest detection confidence. ## results we evaluate two variations of sas to measure the effectiveness of the proposed method (1) sas with teacher-forced sas t f using pm augmentation and temporal alignment (tal) and (2) sas arl+t f a mixture of tf and arl using all augmentation and supervised learning objectives. from the results ( ## conclusion this work proposes a novel navigation instruction generation model that can produce diverse instructions by attending to several structural and semantic cues from the environment. by providing objects, their locations, and their relationships from the scene, spatially-aware speaker can refer to important aspects of the scene in the instruction. an adversarial reward learning method encourages the model to generate diverse instructions. the results show that our method improves on the standard evaluation metrics and performs better than the baselines. future work in future work, integrating the sas model with multimodal transformer architectures will be crucial for enhancing the generation of openvocabulary embodied instructions. this direction promises to overcome the limitations posed by current dataset constraints and improve performance in instruction generation tasks.