Audio-Visual Understanding of Passenger Intents for In-Cabin Conversational Agents (2020)

Abstract: Building multimodal dialogue understanding capabilities situated in the in-cabin context is crucial to enhance passenger comfort in autonomous vehicle (AV) interaction systems. To this end, understanding passenger intents from spoken interactions and vehicle vision systems is an important building block for developing contextual and visually grounded conversational agents for AV. Towards this goal, we explore AMIE (Automated-vehicle Multimodal In-cabin Experience), the in-cabin agent responsible for handling multimodal passenger-vehicle interactions. In this work, we discuss the benefits of multimodal understanding of in-cabin utterances by incorporating verbal/language input together with the non-verbal/acoustic and visual input from inside and outside the vehicle. Our experimental results outperformed text-only baselines as we achieved improved performances for intent detection with multimodal approach.

## introduction understanding passenger intents from spoken interactions and visual cues (both from inside and outside the vehicle) is an important building block towards developing contextual and scene-aware dialogue systems for autonomous vehicles. when the passengers give instructions to the in-cabin agent amie, the agent should parse commands properly considering three modalities (i.e., verbal/language/text, vocal/audio, visual/video) and trigger the appropriate functionality of the av system. for in-cabin dialogue between car assistants and driver/passengers, recent studies explore creating a public dataset using a woz approach @xcite and improving asr for passenger speech recognition @xcite . another recent work @xcite attempts to classify sentences as navigation-related or not using the cu-move in-vehicle speech corpus @xcite , a relatively old and large corpus focusing on route navigation. we collected a multimodal in-cabin dataset with multi-turn dialogues between the passengers and amie using a wizard-of-oz (woz) scheme via realistic scavenger hunt game. in previous work @xcite , we experimented with various rnn-based models to detect the utterancelevel intents (i.e., set-destination, change-route, gofaster, go-slower, stop, park, pull-over, drop-off, open-door, other) along with the intent keywords and relevant slots (i.e., location, position/direction, object, gesture/gaze, time-guidance, person) associated with these intents. in this work, we discuss the benefits of a multimodal understanding of in-cabin utterances by incorporating verbal/language input together with the non-verbal/acoustic and visual cues, both from inside and outside the vehicle (e.g., passenger gestures and gaze from in-cabin video stream, referred objects outside of the vehicle from the road view camera stream). ## data our amie in-cabin dataset includes 30 hours of multimodal data collected from 30 passengers (15 female, 15 male) in a total of 20 sessions. in 10 sessions, a single passenger was present, whereas the remaining 10 sessions include two passengers interacting with the vehicle. participants sit in the back of the vehicle, separated from the driver and the human acting as an agent at the front. the vehicle is modified to hide the operator and the woz amie agent from the passengers, using a variation of the woz approach @xcite . in each ride/session, which lasted about 1 hour or more, the participants were playing a realistic scavenger hunt game on the streets of richmond, bc, canada. passengers treat the vehicle as av and communicate with the woz amie agent mainly via speech commands. game objectives require passengers to interact naturally with the agent to go to certain destinations, update routes, give specific directions regarding where to pull over or park (sometimes with gestures), find landmarks (refer to outside objects), stop the vehicle, change speed, get in and out of the vehicle, etc. further details of the data collection protocol and dataset statistics can be found in @xcite . see fig. 1 for the vehicle instrumentation to enable multimodal data collection setup. ## methodology we explored leveraging multimodality for the natural language understanding (nlu) module in the spoken dialogue system (sds) pipeline. as our amie in-cabin dataset has audio and video recordings, we investigated three modalities for the nlu: text, audio, and visual. for text (verbal/language) modality, we employed the hierarchical & joint bi-lstm model @xcite @xcite , namely h-joint-2. this is a 2-level hierarchical joint learning model that detects/extracts intent keywords & slots using sequence-to-sequence bi-lstms first (level-1), then only the words that are predicted as intent keywords & valid slots are fed into the joint-2 model (level-2), which is another sequence-to-sequence bi-lstm network for utterance-level intent detection, jointly trained with slots & intent keywords. this architecture was chosen based on the bestperforming uni-modal results presented in previous work @xcite for utterance-level intent recognition and slot filling on our amie dataset. these initial uni-modal results were obtained on the transcribed text with pre-trained glove word embeddings @xcite . in this study, we explore the following multimodal features to better assess passenger intents for conversational agents in self-driving cars: word embeddings for text, speech embeddings and acoustic features for audio, and visual features for the video modality. ## experimental results performance results of the utterance-level intent recognition models with varying modality and feature concatenations can be found in ## conclusion and future work in this work, we briefly present our initial explorations towards the multimodal understanding of passenger utterances in autonomous vehicles. we show that our experimental results outperformed the uni-modal text-only baseline results, and with multimodality, we achieved improved performances for passenger intent detection in av. this ongoing research has the potential impact of exploring real-world challenges with humanvehicle-scene interactions for autonomous driving support via spoken utterances. there exist various exciting recent work on improved multimodal fusion techniques @xcite @xcite . in addition to the simplified feature and modality concatenations, we plan to explore some of these promising tensor-based multimodal fusion networks @xcite @xcite for more robust intent classification on amie dataset as future work.