A Survey of Challenges and Methods in the Computational Modeling of Multi-Party Dialog (2023)

Abstract: Advances in conversational AI systems, powered in particular by large language models, have facilitated rapid progress in understanding and generating dialog. Typically, task-oriented or open-domain dialog systems have been designed to work with two-party dialog, i.e., the exchange of utterances between a single user and a dialog system. However, modern dialog systems may be deployed in scenarios such as classrooms or meetings where conversational analysis of multiple speakers is required. This survey will present research around computational modeling of “multi-party dialog”, outlining differences from two-party dialog, challenges and issues in working with multi-party dialog, and methods for representing multi-party dialog. We also provide an overview of dialog datasets created for the study of multi-party dialog, as well as tasks that are of interest in this domain.

## introduction dialog systems are increasingly a part of our personal and professional lives, and have made their way into domains such as healthcare @xcite , business @xcite , and education @xcite . predominantly, research on dialog systems investigates how to develop task-oriented or open-domain systems that individual users can interact with, to accomplish routine tasks or engage in chit-chat. conversations in such settings tend to be two-party or dyadic conversations, that is, involve only two participants, the system and the user, who may typically alternate turns while speaking. however, for applications such as classroom tutoring assistants or meeting summarization, dialog systems need to be able to understand and participate in multi-party dialog -interactions between multiple humans. however, multi-party dialog is structurally different from dyadic dialog, requiring systems to be designed with their characteristics in mind. for figure 1 : an example of a multi-party interaction, with speakers and threads marked. figure from @xcite instance, looking at the chat conversation in figure 1 , we see that the conversations are non-linear and interleaved, and utterances can be implicitly addressed to a specific participant(s). conversational analysis of this interaction would require understanding each sub-dialog, and require resolving the speaker and addressees of each utterance. responses by the dialog agent would also require determining which participant the response should be directed to. if multiple dialog agents are present, response management also requires determining which agent takes the turn. for the purposes of this study, we only consider scenarios with multiple human participants, and one dialog agent. in this paper, we survey research that investigates the computational modeling of multi-party dialog 1 . we first introduce the characteristics of multi-party dialog based on early work in conversational analysis, focusing on ways in which they differ from two-party dialog. based on these differences, we outline some of the challenges that face systems operating in this setting, and their solutions that have been investigated by the field. in section 5, we present a comprehensive overview of representation learning methods for multi-party dialog, focusing on the merits of modeling information flow through graph structures, and discuss deep learning methods for obtaining and encoding these structures. finally, we conclude with a discussion of opportunities for future work in multi-party dialog modeling. ## characteristics of multi-party dialog participant roles: the defining characteristic of multi-party dialog is the presence of multiple participants or interlocutors in a conversation. while in a two-party interaction, one participant takes on the role of the speaker in a turn and the other participant takes on the role of listener or "addressee", an utterance in a multi-party conversation not only has multiple candidate addressees, but could also be directed at multiple listeners at the same time. @xcite further defines participant roles based on their degree of participation at various stages in the conversation: in-context listeners have heard all the previous utterances and may interpret the current utterance differently from a listener with no prior context; active participants are engaged in the conversation and play the roles of speakers and addressees, whereas overhearers may receive utterances but do not participate in the conversation. initiative and turn-taking @xcite observe that while many two-party dialog systems are mixed-initiative or user-initiative driven, multiparty dialog tends to be asymmetric in displaying initiative, with some participants dominating. multi-party dialog may also include simultaneous conversations about multiple distinct topics @xcite . @xcite analyze spontaneous social conversations in small groups, focusing on the nature of turn-taking in simultaneous conversations. of particular interest are conversational floors @xcite , which are structures that can be composed of one turn at a time such as in a therapy session, or can contain multiple alternating turns -for example, when a speaker has the floor and another speaker takes a turn to ask a question, but does not take the floor @xcite . they find that multi-party conversations tend to have multiple simultaneously active floors, with a single session (of up to an hour) having an average of 1.79 active floors, and a maximum of 4 active floors. they further find that floors are dynamic, particularly when the participants are young (ages 14-24) -in sessions with youth there are upto 70 distinct floors over the course of the conversation, each lasting about 44 seconds. dialog structure: research has also studied how structures such as dialog acts or discourse relations can shed light on the nature of multi-party dialog. @xcite examine how dialog act structures differ between two-party and multi-party dialog (specifically, three-party dialog in their study). they first find that dialog act sequences most frequently involve only two speakers, particularly in sequences of length three to five. looking at distances between utterances and their antecedents, @xcite find that long range dependencies are more prevalent in multi-party dialog than in two-party dialog. discourse relations prevalent in multi-party dialog also tend to be distinctive: @xcite find feedback elicitation to be more prevalent than in two-party dialog, whereas @xcite find that the most frequent relations are questionanswer pairs or follow-up questions. ## challenges and sub-tasks the unique characteristics of multi-party dialog imply the existence of challenges that cannot be handled by traditional two-party dialog systems. these challenges are occasionally treated as part of the larger system design @xcite , but for the most part have been isolated as separate sub-tasks. we list a few major problems, and discuss solutions proposed in the literature. ## datasets corpora for studying multi-party conversations span a variety of modalities -spoken @xcite , written @xcite , or accompanied by video @xcite ; they also span multiple genres, including chat forums for software discussions, movies and tv dialog, formal discourse in meetings and interviews, and informal discourse during gameplay. in this survey, we do not focus on comprehensively describing all available datasets, but provide an overview of three datasets which serve as benchmarks for modeling multi-party dialog, and have been extensively used in the models described below. for a detailed survey of datasets specifically, we refer the reader to @xcite . ubuntu irc corpora internet relay chat (irc), a text-based chat interface, contains channels for discussion about specialized topics. typically, discussions consist of users posting questions, and other users replying with solutions, and all messages (or utterances), contain the identity of the sender (speaker). corpora built from this interface have been used for the tasks of conversation disentanglement, speaker and addressee recognition, and response generation. @xcite were the first to use conversations from the ##linux channel, which they manually annotate for threads, for the task of disentanglement. this yields 80 conversations, with a total of about 1500 utterances. uthus and aha (2013) scrape six years of chats from the ##ubuntu channel (which contains messages in english), as well as seven non-english channels including the languages chinese, russian, spanish, portuguese, italian, polish and swedish. this corpus contains over 26 million messages, but without any annotations. @xcite present the ubuntu dialog corpus, which contains 1 million english conversations totalling 7 million utterances. each utterance contains speaker id, and they also heuristically extract addressee ids and disentangle conversations, as shown in figure 2 . @xcite present the largest manually annotated corpus from this domain, for the task of conversation disentanglement, with 70k utterances. finally, @xcite introduce the molweni challenge corpus by annotating the ubuntu corpus with reading comprehension style questions and answers, resulting in 33k question-answer pairs. meeting corpora the ami project @xcite provides a corpus for multimodal conversational analysis of formal discourse -specifically, in multi-party meetings. the ami corpus consists of 100 hours (175 sessions) of scenario-oriented meetings between four participants, where video and audio are recorded, along with artifacts such as digital pen movements and whiteboard content. they providing access to videos, manually transcribed speech, abstractive and extractive summaries of the conversations, and annotations for dialog acts, topic segments, gaze and positional information, and gestures. other corpora under the umbrella of the ami project includes the icsi corpus @xcite , which contains 72 hours of naturally-occuring meetings (not elicited by a scenario). meld corpus another multi-modal multiparty dataset that is widely used in the models below is the meld corpus @xcite , designed for emotion recognition from conversations. it consists of 1433 conversations from the tv show friends, providing access to video, audio, and transcripts. they include annotations at the utterance level indicating one out of seven emotions (such as anger, surprise, etc.) expressed by the utterance. ## representation learning for mpd in this section, we will describe how machine learning models represent and encode multi-party dialog in order to leverage its inherent structural properties for tasks such as response generation. early work such as @xcite represent the entire conversational context sequentially, where all prior utterances to the current one that fall in a window are concatenated. improvements such as @xcite model relationships between the current utterance and the context through a hierarchical rnn. however, given that multi-party dialog can have multiple addressees, multiple replies, as well as simultaneous conversations, such sequential structures cannot represent all relationships between utterances in the dialog. as a solution, recent successful models experiment with graph structures to represent the flow of information in multi-party dialog. typically, this approach treats the utterances as nodes, and the relations between them (such as reply-to) as edges. the graphs thus obtained are encoded through a suitable neural network architecture @xcite , and the resulting embeddings are used for the downstream task, in combination with decoders or classification layers. below, we look at specific sub-components and strategies for this workflow. ## discussion one of the salient findings from our survey is that most recent work on multi-party dialog modeling, particularly using the graph-based methods, are centered around corpora from a limited set of domains; in fact, almost all of the models in section 5 are evaluated on the ubuntu chat corpus or on tv show transcript corpora. a possible reason for this is the availability of annotated structure in these datasets, including speaker and addressee information, as well as threads. however, we argue that the time is ripe for researchers to investigate how to extend modeling innovations to other available corpora and domains. this is an important next step for two reasons, namely real-world applicability, and robustness. natural dialog, such as spontaneous interactions between humans, is typically not well-represented in datasets such as typed chat, or scripted tv dialog. with the growing influence of dialog systems in daily lives, if our goal is to build better technology for the real world, like classrooms or businesses, we need to demonstrate that these stateof-the-art models perform equally well on probable, real-world conversations. moreover, as seen in @xcite , numerous datasets satisfying these properties are actually available, although they do not necessarily contain explicit annotations for structure. however, as this survey shows, we have a large body of work that tells us how to go from natural conversations to more structured representations through tasks such as speaker and addressee recognition, turn prediction, and conversation disentanglement. using these tasks as scaffolds for downstream tasks like response generation would enable us to leverage the expressivity of graph-based modeling on new and realistic domains. in terms of other important next steps for this field of research, one interesting direction is exploring strategies for obtaining silver-standard graph structures through unsupervised methods -we so far only find one paper constructing a reply-to relation graph unsupervisedly. additionally, to answer the robustness question, a systematic assessment of the advantages and shortcomings of graph-structured methods on rarer domains such as meetings @xcite could be highly valuable, particularly for practitioners interested in studying the phenomena exhibited in such conversations. more broadly in this direction, given how the methods we have seen are predominantly focused on english multi-party dialog, the applicability of these methods to languages other than english @xcite , as well as conversations with code-switching @xcite , also needs to be evaluated. finally, with the growing adoption and effectiveness of large language models (llms) in nlp research, a natural next question is to determine how these models can be used in understanding multi-party dialog, and what their limitations are. current directions with promising results include using llms for conversation synthesis @xcite , where high-quality multi-party conversations are synthesized through prompting, and the conversations can be grounded in specific characters or personas. such synthesized conversations may also help adapt methods for conversation analysis and response generation to rarer domains that may not be well-represented in natural corpora. ## conclusion our survey provides an overview of research in computationally modeling multi-party dialog. we identify major challenges based on differences from two-party dialog, and discuss how sub-tasks have been designed for solving them. we comprehensively describe recent advances in representation learning for multi-party dialog, focusing in particular on graph-based structures. finally, we discuss some key directions that future work in this area can explore.