Achieving Common Ground in Multi-modal Dialogue (2020)

Abstract: All communication aims at achieving common ground (grounding): interlocutors can work together effectively only with mutual beliefs about what the state of the world is, about what their goals are, and about how they plan to make their goals a reality. Computational dialogue research offers some classic results on grouding, which unfortunately offer scant guidance to the design of grounding modules and behaviors in cutting-edge systems. In this tutorial, we focus on three main topic areas: 1) grounding in human-human communication; 2) grounding in dialogue systems; and 3) grounding in multi-modal interactive systems, including image-oriented conversations and human-robot interactions. We highlight a number of achievements of recent computational research in coordinating complex content, show how these results lead to rich and challenging opportunities for doing grounding in more flexible and powerful ways, and canvass relevant insights from the literature on human–human conversation. We expect that the tutorial will be of interest to researchers in dialogue systems, computational semantics and cognitive modeling, and hope that it will catalyze research and system building that more directly explores the creative, strategic ways conversational agents might be able to seek and offer evidence about their understanding of their interlocutors.

## description all communication aims at achieving common ground (grounding): interlocutors can work together effectively only with mutual beliefs about what the state of the world is, about what their goals are, and about how they plan to make their goals a reality @xcite . computational dialogue research, in particular, has a long history of influential work on how implemented systems can achieve common ground with human users, from formal results on grounding actions in conversation @xcite to machine learning results on how best to fold confirmation actions into dialogue flow @xcite . such classic results, however, offer scant guidance to the design of grounding modules and behaviors in cutting-edge systems, which increasingly combine multiple communication modalities, address complex tasks, and include the possibility for lightweight practical action interleaved with communication. this tutorial is premised on the idea that it's time to revisit work on grounding in human-human conversation, particularly brennan's general and important characterization of grounding as seeking and providing evidence of mutual understanding @xcite , in light of the opportunities and challenges of multi-modal settings such as human-robot interaction. in this tutorial, we focus on three main topic areas: 1) grounding in human-human communication; 2) grounding in dialogue systems; and 3) grounding in multi-modal interactive systems, including image-oriented conversations and humanrobot interactions. we highlight a number of achievements of recent computational research in coordinating complex content, show how these results lead to rich and challenging opportunities for doing grounding in more flexible and powerful ways, and canvass relevant insights from the literature on human-human conversation. we expect that the tutorial will be of interest to researchers in dialogue systems, computational semantics and cognitive modeling, and hope that it will catalyze research and system building that more directly explores the creative, strategic ways conversational agents might be able to seek and offer evidence about their understanding of their interlocutors. grounding in human-human communication. @xcite argued that communication is accomplished in two phases. in the presentation phase, the speaker presents signals intended to specify the content of the contributions. in the second phase, the participants work together to establish mutual beliefs that serve the purposes of the conversation. the two phases together constitute a unit of communication-contributions. @xcite show how this model applies to coordinated action, while @xcite applies the model to text-and-video presentations. coherence is key. grounding in dialogue systems. computer systems achieve grounding mechanistically by ensuring they get attention and feedback from their users, tracking user state, and planning actions with reinforcement learning to resolve problematic situations. we will review techniques for maintaining engagement @xcite @xcite and problems that arises in joint attention @xcite and turn taking such as incremental interpretation (devault @xcite , ambiguity resolution (de-vault and @xcite and learning flexible dialogue management policies @xcite . similar questions have been studied in the context of instruction games @xcite @xcite , and interactive tutoring systems @xcite . grounding in multi-modal systems. multimodal systems offer the ability to use signals such as nodding, certain hand gestures and gazing at a speaker to communicate meaning and contribute to establishing common ground @xcite . however, multi-modal grounding is more than just using pointing to clarify. multi-modal systems have diverse opportunities to demonstrate understanding. for example, recent work has aimed to bridge vision, interactive learning, and natural language understanding through language learning tasks based on natural images @xcite @xcite . the work on visual dialogue games @xcite brings new resources and models for generating referring expression for referents in images @xcite , visually grounded spoken language communication @xcite , and captioning ( levinboim et al., 2019; alikhani and stone, 2019 ), which can be used creatively to demonstrate how a system understand a user. figure 1 shows two examples of models that understand and generate referring expressions in multimodal settings. similarly, robots can demostrate how they understand a task by carring it out-in research on interactive task learning in human-robot interaction @xcite as well as embodied agents perform-show me a restaurant by the river, serving pasta/italian food, highly rated and expensive, not child-friendly, located near cafe adriatic. @xcite crystal island, an interactive narrativecentered virtual learning environment @xcite figure 2 : content and medium affect grounding. this figure shows two examples of interactive multimodal dialogue systems. we pay special attention to non-verbal grounding in languages beyond english, including german @xcite , swedish (kontogiorgos, 2017), japanese @xcite , french @xcite , italian @xcite , spanish @xcite , russian @xcite , and american sign language @xcite . these investigations often describe important language-dependent characteristics and cultural differences in studying non-verbal grounding. ## outline we begin by discussing grounding in humanhuman communication (∼20 min). after that, we discuss the role of grounding in spoken dialogue systems (∼30 min) and visually grounded interactions including grounding visual explanations in images and multimodal language grounding for human-robot collaboration (∼90 min). we then survey methods for developing and testing multimodal systems to study non-verbal grounding (∼20 min). we follow this by describing common solution concepts and barrier problems that cross application domains and interaction types (∼20 min). ## prerequisites and reading list the tutorial will be self-contained. for further readings, we recommend the following publications that are central to the non-verbal grounding framework as of late 2019: 1. grounding in communication, herb clark and susan brennan. (clark et al., 1991) 2. meaning and demonstration by una stojnic and matthew stone (stone and stojnić, 2015) 3. using reinforcement learning to model incrementality in a fast-paced dialogue game, ramesh manuvinakurike, david devault and kallirroi georgila. (manuvinakurike et al., 2017) 4. language to action: towards interactive task learning with physical agents, joyce y. chai by joyce y. chai et al.(chai et al., 2018) 5. it's not what you do, it's how you do it: grounding uncertainty for a simple robot, julian hough and david schlangen. (hough and schlangen, 2017) 6. learning effective multimodal dialogue strategies from wizard-of-oz data: bootstrapping and evaluation rieser-lemon by verena rieser and oliver lemon. (rieser and lemon, 2008) 7. a survey of nonverbal signaling methods for non-humanoid robots by elizabeth cha et al. (cha et al., 2018) 8. the devil is in the details: a magnifying glass for the guesswhich visual dialogue game by alberto testoni et al. (testoni et al., 2019) 4 authors malihe alikhani is a 5th year ph.d. student in the department of computer ssience at rutgers university ma1195@cs.rutgers.edu, advised by prof. matthew stone. she is pursuing a certificate in cognitive science through the rutgers center for cognitive science and holds a ba and ma in mathematics. her research aims at teaching machines to understand and generate multimodal communication. she is the recipient of the fellowship award for excellence in computation and data sciences from rutgers discovery informatics institute in 2018 and the anita berg student fellowship in 2019. before joining rutgers, she was a lecturer and an adjunct professor of mathematics and statistics for a year at san diego state university and san diego mesa college. she has served as the program committee of acl, naacl, emnlp, aaai, icrl, icmi, and inlg and is currently the associate editor of the mental note journal. email: ma1195@cs.rutgers.edu, webpage: www.malihealikhani.com matthew stone is professor and chair in the department of computer science at rutgers university; he holds a joint appointment in the rutgers center for cognitive science. his research focuses on discourse, dialogue and natural language generation; he is particularly interested in leveraging semantics to make interactive systems easier to build and more human-like in their behavior. he was program co-chair for naacl 2007, general co-chair for sigdial 2014. he has also served as program co-chair for inlg and iwcs, as an information officer for sigsem, and on the editorial board for computational linguistics. email: mdstone@cs.rutgers.edu, website: www.cs.rutgers.edu/ ˜mdstone/ laura a janda. 1988. the mapping of elements of cognitive space onto grammatical relations: an example from russian verbal prefixation. topics in cognitive linguistics, 50:327-343. sahar kazemzadeh, vicente ordonez, mark matten, and tamara berg. 2014. referitgame: referring to objects in photographs of natural scenes. in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pages 787-798. caroline kery, francis ferraro, and cynthia matuszek. 2019. ¿ es un plátano? exploring the application of a physically grounded language acquisition system to spanish. in proceedings of the combined workshop on spatial language understanding (splu) and grounded communication for robotics (robonlp), pages 7-17. hyounghun kim, hao tan, and mohit bansal. 2020. modality-balanced models for visual dialogue. dimosthenis kontogiorgos. 2017. multimodal language grounding for improved human-robot collaboration: exploring spatial semantic representations in the shared space of attention. in proceedings of the 19th acm international conference on multimodal interaction, pages 660-664. acm. dimosthenis kontogiorgos, vanya avramova, simon alexanderson, patrik jonell, catharine oertel, jonas beskow, gabriel skantze, and joakim gustafson. 2018. a multimodal corpus for mutual gaze and joint attention in multiparty situated interaction. in international conference on language resources and evaluation (lrec 2018). staffan larsson. 2018. grounding as a side-effect of grounding. topics in cognitive science, 10(2):389-408. séverin lemaignan and rachid alami. 2013. talking to my robot: from knowledge grounding to dialogue processing. in 2013 8th acm/ieee international conference on human-robot interaction (hri), pages 409-409. ieee. e. levin, r. pieraccini, and w. eckert. 1998. using markov decision process for learning dialogue strategies. in proceedings international conference on acoustics, speech and signal processing (icassp). tomer levinboim, ashish thapliyal, piyush sharma, and radu soricut. 2019. quality estimation for image captions based on large-scale human evaluations. arxiv preprint arxiv:1909.03396. changsong liu and joyce yue chai. 2015. learning to mediate perceptual differences in situated humanrobot dialogue. in twenty-ninth aaai conference on artificial intelligence.