A rra M on: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments (2020)

Abstract: For embodied agents, navigation is an important ability but not an isolated goal. Agents are also expected to perform specific tasks after reaching the target location, such as picking up objects and assembling them into a particular arrangement. We combine Vision-andLanguage Navigation, assembling of collected objects, and object referring expression comprehension, to create a novel joint navigation-and-assembly task, named ARRAMON. During this task, the agent (similar to a PokeMON GO player) is asked to find and collect different target objects one-by-one by navigating based on natural language (English) instructions in a complex, realistic outdoor environment, but then also ARRAnge the collected objects part-by-part in an egocentric grid-layout environment. To support this task, we implement a 3D dynamic environment simulator and collect a dataset with human-written navigation and assembling instructions, and the corresponding ground truth trajectories. We also filter the collected instructions via a verification stage, leading to a total of 7.7K task instances (30.8K instructions and paths). We present results for several baseline models (integrated and biased) and metrics (nDTW, CTC, rPOD, and PTC), and the large model-human performance gap demonstrates that our task is challenging and presents a wide scope for future work.

## introduction navigation guided via flexible natural language (nl) instructions is a crucial capability for robotic and embodied agents. such systems should be capable of interpreting human instructions to correctly navigate realistic complex environments and reach destinations by understanding the environment, and associating referring expressions in the navigation phase: turn around to face the speed limit sign. go to the sign and then turn right around the corner. go to the booth and a little past it and to the right there is a brown hourglass. pick it up. figure 1 : navigation and assembly phases (2 turns), via nl (english) instructions in a dynamic 3d environment. in the navigation phase, agents are asked to find and collect a target object. in the assembly phase, agents have to egocentrically place the collected object at a relative location (navigation turn 2 starts where turn 1 ends; we only show 3 snapshots here for space reasons, but the full simulator and its image set will be made available). ## task the arramon task consists of two phases: navigation and assembly. we define one turn as one navigation phase plus one assembly phase (see figure 1 ). both phases are repeated twice (i.e., 2 turns), starting with the navigation phase. during the navigation phase, an agent is asked to navigate a rich outdoor city environment by following nl instructions, and then collect the target object identified in the instructions via diverse referring expressions. during the assembly phase, the agent is asked to place the collected object (from the previous navigation phase) at a target location on a grid layout, using a different nl instruction via relative spatial referring expressions. target objects and distracter objects are selected from one of seven objects shown in figure 2 and then are given one of two different patterns and one of seven different colors (see figure 11 in appendices). in both phases, the agent can take 4 actions: forward, left, right, and an end pickup/place action. forward moves the agent 1 step ahead and left/right makes agents rotate 30°in the respective direction. 6 3.1 environment navigation phase. in this phase, agents are placed at a random spot in one of the seven disjoint subsections of the city environment (see figure 3 ), provided with an nl instruction, and asked to find the target object. the city environment is filled with background objects: buildings and various objects found on streets (see figure 4 ). there are also a few distracter objects in the city that are similar to target objects (in object type, pattern, and color). during this phase, the agent's end action is 'pickup'. the pick-up action allows agents to pick up any collectible object within range (a rectangular area: 0.5 unit distance from an agent toward both their left and right hand side and 3 unit distance forward). assembly phase. once the agent picks up the collectible object in the navigation phase, they enter the assembly phase. in this phase, agents are again provided with an nl instruction, but they are now asked to place the target object they collected in the 6 in our task environment, holistically, the configuration of the set of objects dynamically changes as agents pick-up and place or stack them relative to the other objects, which is one challenging interaction between the objects. dumpster 2 traffic light 1 stop sign speed limit sign traffic barrier 1 barrels traffic light 2 streetlamp tire pile traffic barrier 2 mailbox trash can 2 traffic barrier 4 metal bench wooden bench traffic cones traffic barrier 5 dumpster 1 banners garbage bags trash can 1 hydrant phone booth traffic barrier 3 previous phase at the target location identified in the instruction. when the assembly phase begins, 8 decoy basic-type objects (figure 2 ) with random pattern and color, are placed for use as distractions. in this phase, agents can only move on a 4-by-5 grid layout. the grid is bordered by 4 walls, each with a different texture/pattern (wood, brick, spotted, striped) to allow for more diverse expressions in the assembly phase. their end action is 'place', which puts the collected object onto the grid one step ahead. agents cannot place diagonally and, unlike in the navigation phase, cannot move forward diagonally. hence, to accomplish the overall joint navigation-assembly task, it is required for agents to have integrated abilities. during navigation they must take actions based on understanding the egocentric view and aligning the nl instructions with the dynamic visual environment to successfully find the target objects (relevant metrics: ndtw and ctc-k, see section 3.2). during assembly, from an egocentric view, they must understand 3d spatial relations among objects identified by referring expressions in order to place the target objects at the right relative location. (relevant metrics: ptc and rpod, see section 3.2). foot_5 ## arramon dataset our arramon navigation-assembly dataset is a collection of rich human-written nl (english) instructions. the navigation instructions explain how to navigate the large outdoor environments and describe which target objects to collect. the assembly instructions provide the desired target locations for placement relative to objects. each instruction set in the dataset is accompanied by ground truth (gt) trajectories and placement locations. data was col-lected from the online crowd-sourcing platform amazon mechanical turk (amt). ## data analysis a total of 8,546 instruction sets were collected. each set included two pairs of navigation and assembly instructions (thus, 34,184 instructions in total). after filtering from stage 2 results, there remained 7,692 instruction sets (30,768 instructions in total). our dataset size is comparable to other similar tasks, e.g., touchdown @xcite 3d depth to guide the agent; implying that the combined navigation and assembly task requires that agents posses a full understanding of object relations in a 3d environment. our analysis showed other linguistic properties, such as frequent directional references, ego and allocentric spatial relations, temporal conditions, and sequencing (see appendix c.1 for the details and examples). dataset statistics. figure 5 shows that the most frequently occurring words in our dataset. these words are primarily directional or spatial relations. this implies that agents should be able to understand the concept of direction and the spatial relations between objects, especially as they change with movement. ## models we train an integrated vision-and-language model as a good starting point baseline for our task. to verify that our dataset is not biased towards some specific factors, we trained ablated and random walk models and evaluated them on the dataset. vision-and-language baseline. this model uses vision and nl instruction features together to predict the next actions (figure 7 ). we implement each module for navigation/assembly phases as: we train the model with the teacher-forcing approach @xcite and cross entropy loss: p t (a t ) = softmax(logit at ); @xmath0, where a * t is ground truth action at time step t. vision/language only baseline. to check the unimodality bias, we evaluate vision and language only baselines on our dataset. these exploit only single modality (visual or language) to predict the appropriate next action. to be specific, they use the same architecture as the vision-and-language baseline except the cross-attn module. random walk. agents take a random action at each time step without considering instruction and environment information. shortest path. this baseline simulates an agent that follows the shortest path provided by a* algorithm @xcite to show that the gt paths are optimal in terms of trajectory distances. ## experiments we split the dataset into train/val-seen/valunseen/test-unseen. we assign the city sub-sections 1 to 5 to train and val-seen, sub-section 6 to valunseen, and section 7 to test-unseen splits. we randomly split data from sub-sections 1 to 5 into 80/20 ratio to get train and val-seen splits, respectively. thus, the final number of task samples for each split is 4,267/1,065/1,155/1,205 (total: 17,068/4,260/4,620/4,820). the stage 1 workers are equally distributed across the city sub-sections, so the dataset splits are not biased toward specific workers. we also keep the separate 2 sections (i.e., section 6 and 7) for the unseen dataset following @xcite , which allows the evaluation of the models' ability to generalize in new environments. note that for agents to proceed to the next phase, we allow them to pick up the closest target object (in the navigation phase) or place collected object at the closest location (in the assembly phase) when they do not perform the required actions. training details: we use 128 as hidden size. for word and action embedding sizes, we use 300 and 64, respectively. we use adam (kingma and ba, 2015) as the optimizer and set the learning rate to 0.001 (see appendix e.2 for details). ## results and analysis as shown in if perfect performance is assumed in the navigation phase, rpod and ptc are higher than if there were low ctc-k scores in navigation (e.g., 0.382 vs. 0.044 for ptc of the vision-and-language model on val-seen: see appendix f for the comparison). this scoring behavior demonstrates that phases in our arramon task are interweaved. also, comparing scores from turn 1 and 2, all turn 2 scores are lower than their turn 1 counterparts (e.g., 0.222 vs. 0.049 ndtw of the vision-and-language model on val-seen split; see appendix f for the detailed turn-wise results). this shows that the performance of the previous turn strongly affects the next turn's result. note that to relax the difficultly of the task, we consider ctc-3 (instead of ctc-0; see section 3.2) as successfully picking up the target object and then we calculate the assembly metrics under this assumption. if this was not done, then almost all the metrics across assembly would be nearly zero. ## conclusion we introduced arramon, a new joint navi-gation+assembling instruction following task in which agents collect target objects in a large realistic outdoor city environment and arrange them in a dynamic grid space from an egocentric view. we collected a challenging dataset via a 3d synthetic simulator with diverse object referring expressions, environments, and visuospatial relationships. we also provided several baseline models which have a large performance gap compared to humans, implying substantial room for improvements by future work.