DOROTHIE : Spoken Dialogue for Handling Unexpected Situations in Interactive Autonomous Driving Agents (2022)

Abstract: In the real world, autonomous driving agents navigate in highly dynamic environments full of unexpected situations where pre-trained models are unreliable. In these situations, what is immediately available to vehicles is often only human operators. Empowering autonomous driving agents with the ability to navigate in a continuous and dynamic environment and to communicate with humans through sensorimotor-grounded dialogue becomes critical. To this end, we introduce Dialogue On the ROad To Handle Irregular Events (DOROTHIE), a novel interactive simulation platform that enables the creation of unexpected situations on the fly to support empirical studies on situated communication with autonomous driving agents. Based on this platform, we created the Situated Dialogue Navigation (SDN), a navigation benchmark of 183 trials with a total of 8415 utterances, around 18.7 hours of control streams, and 2.9 hours of trimmed audio. SDN is developed to evaluate the agent’s ability to predict dialogue moves from humans as well as generate its own dialogue moves and physical navigation actions. We further developed a transformer-based baseline model for these SDN tasks. Our empirical results indicate that language guided-navigation in a highly dynamic environment is an extremely difficult task for end-to-end models. These results will provide insight towards future work on robust autonomous driving agents

## introduction in embodied agents such as autonomous vehicles (avs), highly dynamic environments often lead to unexpected situations, such as challenging environment conditions (e.g., caused by weather, light, obstacles, etc.), influence of other agents, and change of the original goals. in these situations, the agent's pre-trained models or existing knowledge may not be adequate or reliable to make a corresponding decision. what is immediately available to help the agent is often only human partners @xcite . as they are not programmers who can readily change the code in the field, approaches that enable natural communication and collaboration between humans and autonomy become critical @xcite . although recent years have seen an increasing amount of work in natural language communication with robots, and especially the many benchmarks that have been developed for navigation by instruction following @xcite @xcite , little work has been done to study language communication under unexpected situations, particularly in the context of avs. to address this limitation, we have developed dialogue on the road to handle irregular events (dorothie), an interactive simulation platform built upon the carla simulator @xcite to specifically target unexpected situations. the dorothie simulator supports wizard-of-oz (woz) studies through a novel duo-wizard setup: a collaborative wizard (co-wizard) that collaborates with the human to accomplish the tasks, and an adversarial wizard (ad-wizard) that generates unexpected situations (e.g., creating road obstacles, changing weather conditions, adding/changing goals, etc.) on the fly. using dorothie, we collected the situated dialogue navigation (sdn) dataset of 183 trials between a co-wizard and human subjects to collaboratively resolve unexpected situations and complete navigation tasks through spoken dialogue. the sdn dataset contains multi-faceted and timesynchronized information (e.g., first-person view of the environment, speech input from the human, discrete actions, continuous trajectory and control sig- environment communication granularity data collection instruction type action space name domain fidelity continuity turn form language control lang. demo. modal. replan. adp. nav. man. continuity sdn (ours) sim c m freeform h & l h & l h h lvms -d & c cdnli (roh et al., 2020) sim c m multi inst l h & l h+t p lvm --d & c lcsd (sriram et al., 2019) sim c s multi inst l h h p lvm ---d ttw (de vries et al., 2018) pano d m freeform h & l h h h lvm ---d talk2nav (vasudevan et al., 2021) pano d s multi inst l h h p lvm ---d touchdown (chen et al., 2019) pano d s multi inst l h h p lvm ---d street nav (hermann et al., 2020) pano d m multi inst l h t p lvm ---d map2seq (schumann and riezler, 2021) pano d s multi inst l h h p lm ---d run (paz-argaman and tsarfaty, 2019) outdoors pano d s multi inst l h h h lm ---d teach (padmakumar et al., 2022) sim c m freeform h & l h h h lv -d dialfred (gao et al., 2022) sim c m restricted h & l h h+t p lv -d alfred (shridhar et al., 2020) sim c s multi inst h & l h h p lv -d hanna (nguyen and daumé iii, 2019) pano d m multi inst h & l h h p lv --d robotslang (banerjee et al., 2020) phy c m freeform h & l h h p lv ---d ttt and wtw (ilyevsky et al., 2021) phy c s restricted h & l h h p lm ---d robo-vln (irshad et al., 2021) pano c s multi inst l h & l h p lv ---c vln-ce @xcite pano c s multi inst l h h p lv ---d cvdn @xcite pano d m restricted l h h h lv ---d r2r @xcite al., 2018 nals) as well as fine-grained annotation of dialogue phenomena at multiple levels. sdn challenges autonomous driving agents to navigate in continuous and dynamic environments, engage in situated communication with humans, and handle unexpected events on the fly. as an initial step, we developed the temporally-ordered task oriented transformer (toto), a transformer-based baseline model for three tasks: (1) predicting dialogue moves from human utterances; (2) generating dialogue moves in response to humans; and (3) generating navigation actions towards the goal. we present our empirical results and discuss key challenges and opportunities. to the best of our knowledge, this is the first effort on language communication under unexpected situations in autonomous vehicles. our contributions are the following: (1) a novel, high-fidelity simulation platform, dorothie, that can be used to create unexpected situations on the fly during human-agent communication, (2) a fine-grained benchmark, sdn, for continuous, dynamic, interactive navigation with sensorimotor-grounded dialogue, and (3) a transformer-based model for action prediction and decision-making which serves as a baseline for future development. ## related work our work is mostly related to language-conditioned navigation tasks @xcite @xcite and particularly recent work on embodied agents that learn to navigate by following language instructions @xcite . replanning in unexpected situations. most simulated environments assume that only the tasked agent can change the state of the world through navigation and/or manipulation. in outdoor settings, the agent operates in a highly dynamic environment where unexpected changes to the world can often occur due to, e.g., walking pedestrians, moving vehicles, lighting, and weather conditions. while previous studies have explored misleading @xcite or perturbed @xcite instructions, no prior work has looked into how language instructions can help agents adapt in these unexpected situations. to our knowledge, sdn is the first dataset where language is used to assist agents to replan their goals, paths, and trajectories. free-form communication. most prior work adopts either simple instruction-following @xcite @xcite , or restricted qa dialogue @xcite @xcite that only allows the agent to ask for help. except for some recent work in human-robot dialogue @xcite @xcite , few efforts have supported fully free-form communication where agents can ask, propose, explain, and negotiate under ambiguity or confusion. to the best of our knowledge, sdn is the first benchmark to enable navigation in autonomous driving agents conditioned on free-form spoken dialogue. continuous navigation. in discrete navigation, agents take discrete actions, e.g., tele-transport in a pre-defined grid world @xcite or a navigation graph with sparsely sampled panoramas at each node @xcite . more recently, researchers proposed a continuous navigation setting @xcite by converting discrete paths on navigation graphs into trajectories. unfortunately, these agents are still limited with a discrete action space such as forward 0.25m. this becomes unnatural in outdoor settings because the default behaviour of outdoor driving agents (e.g., autonomous vehicles) is lane-following instead of staying still. we instead follow the settings of mobile robot navigation @xcite , where the agents are controlled by a continuous action space with physics like throttle and steering, leading to continuous control signals with long-range trajectories. ## dialogue on the road to handle irregular events (dorothie) simulator motivated by the wide availability of software simulations for autonomous vehicles @xcite , we set up our experiment in carla @xcite , a driving simulator for autonomous vehicles. we developed a novel framework, dialogue on the road to handle irregular events (dorothie) (as shown in figure 1 ), to study situated communication under unexpected situations based on the wizard-of-oz (woz) paradigm @xcite @xcite . in woz, a human participant is typically instructed to interact with an autonomous agent to complete a set of tasks. the agent's behaviors, however, are controlled by a human "wizard" (i.e., a researcher). one important novelty of our framework is that it extends the traditional woz approach by introducing a pair of wizards. in our duo-wizard setup, a co-wizard controls the agent's behaviors and carries out language communication with the human participant to jointly achieve a goal, and an ad-wizard creates unexpected situations on the fly. the co-wizard and the participant need to resolve these unexpected situations as they arise. ## situated dialogue navigation (sdn) our data collection effort has led to the situated dialogue navigation (sdn), a fine-grained outdoor navigation benchmark. each session was replayed at 10 fps following prior work @xcite ) to obtain multi-faceted and time-synchronized information, e.g., a first-person view of the environment, speech input from the participant, discrete actions, dialogue moves initiate respond instruct explain ask inform communicate initiation? response? command? statement? question? domain information? communication status? confirm communication status align confirm an inferable domain information check simple yes/no query queryyn complex wh query queryw query for domain information query contain just the information requested reply contain amplified information clarify simple no reply replyn complex wh reply replyw confuse indicate confusion acknowledge indicate successful communication simple yes reply replyy simple unsure reply replyu contain repeated information ## task definition while many challenging tasks can be tackled using sdn, within the scope of this paper, we formulate three tasks that are critical for enabling situated dialogue for navigation. we first introduce some notations then describe our task for-mulation. the agent is provided with domain knowledge k, including a list of street names {str i } and (possibly incomplete) landmarks {lm i } on the map topology m . at time t, the interaction history (possibly empty) is represented as h @xmath0, u h t-1,hum , u h t-1,bot } which includes visual observations (o h t-1 ) and dialogue utterances from the human (u h t-1,hum ) and the agent (u h t-1,bot ). the action history, represented as a t , captures the sequence of navigation actions and the dialogue moves from both the human and the agent. given these representations, we define three tasks based on the sdn benchmark. ## experiments and results we summarize the experiment results in town 2) is significantly smaller in size and simpler in map topology. overall, our experiment has shown that the tasks in sdn are challenging. comparatively, the episodic transformer (e.t.) (pashevich et al., 2021) baseline particularly underperforms in the inference task, i.e., ufn move prediction. the fine-tuned language model baseline can handle dialogue move predictions very well, but significantly fails on other tasks. we further provide a set of ablation studies, and discuss potential reasons why sdn is a challenge for end-to-end models. additional results are available in appendix c.4. ## conclusion we introduced dorothie, a high-fidelity simulation platform to support woz studies for situated communication with autonomous driving agents that can adapt to unexpected events. we defined and collected sdn, a fine-grained benchmark for continuous, dynamic, interactive naviga-tion with sensorimotor-grounded dialogue. our dorothie platform, together with our sdn benchmark, contribute a valuable resource for several lines of work in robotics navigation and human-robot communication. we presented temporally-order task oriented transformer (toto), a fully transformer-based baseline model for the sdn task. our empirical results have shown that such longhorizon navigation tasks with rich dialogue phenomena and unexpected situations can be very challenging for end-to-end approaches. this work has shown that language-guided navigation in a highly dynamic environment (e.g., in the context of avs) is an extremely difficult task. our dorothie simulation environment, the sdn benchmark, and baseline models provide a stepping stone towards future efforts in this challenging space.