In Search of the Lost Arch in Dialogue: A Dependency Dialogue Acts Corpus for Multi-Party Dialogues (2025)

Abstract: Understanding the structure of multi-party conversation and the intentions and dialogue acts of each speaker remains a significant challenge in NLP. While a number of corpora annotated using theoretical frameworks of dialogue have been proposed, these typically focus on either utterance-level labeling of speaker intent, missing wider context, or the rhetorical structure of a dialogue, losing fine-grained intents captured in dialogue acts. Recently, the Dependency Dialogue Acts (DDA) framework has been proposed to for modeling both the fine-grained intents of each speaker and the structure of multi-party dialogues. However, there is not yet a corpus annotated with this framework available for the community to study. To address this gap, we introduce a new corpus of 33 dialogues and over 9,000 utterance units, densely annotated using the Dependency Dialogue Acts (DDA) framework.Our dataset spans four genres of multi-party conversations from different modalities: (1) physics classroom discussions, (2) engineering classroom discussions, (3) board game interactions, and (4) written online game chat logs. Each session is doubly annotated and adjudicated to ensure high-quality labeling. We present a description of the dataset and annotation process, an analysis of speaker dynamics enabled by our annotation, and a baseline evaluation of LLMs as DDA parsers. We discuss the implications of this dataset understanding dynamics between speakers and for developing more controllable dialogue agents.

## introduction understanding and representing speaker intention has long been a fundamental challenge in dialogue analysis, spanning multiple disciplines, including * jon.z.cai@colorado.edu † bking2@ucsc.edu @xcite bp: okay, w ho w ants to go first ? over the years, various annotation frameworks and corpora have been developed to analyze discourse structure and speaker intentions in dialogue. switchboard damsl (swbd; jurafsky 1997) pioneered utterance-level annotation by categorizing dialogue acts such as questions, statements, and back-channels, in two-person phone conversations. meanwhile, rhetorical structure theory (rst) @xcite and penn dis-course treebank (pdtb) @xcite focused on structural text coherence by identifying discourse relations such as elaboration, contrast, and causality, laying the groundwork for understanding how speakers build meaning beyond isolated utterances. segmented discourse representation theory (sdrt) @xcite extended this approach to dialogue, modeling multiturn dependencies and coherence relations crucial for capturing conversational flow. while these frameworks and corpora have significantly advanced computational dialogue understanding, they face challenges in multi-party conversations, where interactions are more fragmented, overlapping, and dynamic. where two speakers on the phone might share uninterrupted joint attention, real-world multi-party dialogues are situated in complex tasks and contain deviations whose structure could be missed in frameworks like swbd, which focus on utterance-level labels. alternatively, frameworks focused on rhetorical structure might miss subtle distinctions in a speaker's communicative intent, such as whether a speaker is 'proposing' someone take an action, which is open to rejection, or 'directing' they do, which is not. to better capture the complexities encountered in multi-party dialogue, @xcite propose the dependency dialogue act (dda) framework, a general purpose framework for capturing multiparty speaker intentions and structure. a visualization of the dda structure is shown in figure 1 . to our knowledge, no publicly available corpus has previously been annotated with the dda framework, limiting its adoption and evaluation. in this work, we present the first publicly released corpus of multi-party dialogues annotated with the dda framework, capturing deep, structured representations of speaker intentions. our corpus spans four domains, including both spoken and written (chat) modalities across 33 english language multi-party dialogues. our primary goal is to develop benchmark data that enables the evaluation of ai systems in collaborative multi-party conversations. unlike dyadic exchanges, multi-party dialogues present additional challenges, such as overlapping speech, conflicting intentions, implicit coordination strategies, and emotional dynamics. human communicators are not always perfectly rational, meaning interactions are often shaped by personal biases, misunderstandings, and differing levels of assertiveness. as such, an ai system capable of accurately identify-ing speaker intentions and behavioral patterns can help reduce unnecessary tension, promote effective collaboration, and support more structured group discussions. despite the importance of this goal, existing frameworks and benchmarks for analyzing ai systems' understanding of multi-party dialogue remain limited. our corpus provides a deep, structured approach for evaluating an ai system's understanding of multi-party dialogue. structured representations like dda also provide an opportunity for controllable generation. while the rise of large language model-based (llm) ai chat agents has reshaped the dialogue systems landscape, current llms still require elaborate prompt engineering to produce desirable behaviors consistently, even after fine-tuning @xcite . their dialogue actions are not inherently controllable, which means that modifying an agent's behavior often requires trial-and-error adjustments to the prompts rather than direct intervention at the intent and strategy level. by equipping conversational ai systems with the ability to reason about complex speaker intentions within the dda framework, we move closer to developing ai agents that can navigate collaborative scenarios, facilitate productive discussions, and promote synergy between human participants. this structured framework will help ai agents understand, anticipate, and appropriately respond to multi-party dialogue dynamics, ultimately improving explainability, controllability, and safety in conversational ai applications. the contributions of this paper: ## related work the study of dialogue acts and speaker intention modeling has driven the development of various annotation schemes and corpora, serving as benchmarks for dialogue systems. in this section, we review key annotation frameworks, multi-party dialogue corpora, and recent evaluations of llms' capabilities in multi-party dialogue settings. ## coding scheme we closely follow the dda framework definition and provide our domain-aware customizations of the coding scheme of dda. slash unit (su) as atomic unit of analysis: in the dda framework, a slash unit (su) is the fundamental unit of analysis and annotation. it approximates the minimal functional segment of an utterance, ensuring that each unit corresponds to a meaningful communicative act. dialogue act (da) categories: each su is assigned one or more dialogue act (da) labels to represent its communicative function. ## dataset formation and quality control we manually annotated multi-party conversational data across various domains. in this section, we show the annotation process, an overview of the core characteristics of the dataset, distributions of the dda labels and how our annotation process ensures the quality of the annotation. as previously mentioned, all annotations were performed on manually transcribed conversations. however, where available, multimodal cues such as speaker expressions, gestures, gaze patterns, and vocal tones were taken into account to ensure a more comprehensive representation of speaker intentions and actions. we construct dda through a web-based dda annotation application. 4 each session is doubly annotated by trained annotators with linguistics backgrounds. we employ a systematic adjudication process as our primary strategy for quality control and to address the initially moderate inter-annotator agreement. this process not only resolves minor discrepancies but also enhances overall annotation quality, even though in-depth interpretation and annotation of dialogue data remain inherently challenging. the adjudication process has evolved from a group discussion and voting approach to a more asynchronous collaboration method, improving efficiency while maintaining annotation quality. initially, group discussions required at least two annotators to deliberate and reach consensus. as annotators became more familiar with the dda coding scheme, the process transitioned to an asynchronous workflow. in the asynchronous adjudication process, once independent annotations are completed, the two annotators attempt to justify their disagreements on dda edges per su. this is done through: first, densely paraphrasing -rewriting the original utterance to make dialogue acts and discourse structure more explicit while preserving the intended meaning. second, documenting decision rationale on edges with disagrement. a third annotator then reviews these justifications and casts a deciding vote in the event of a tie. among all dialogue sessions, pulley k12 conversations and 3267 out of 4765 sus in the sensor immersion k12 datatset were adjudicated via real-time group discussion, while the reminder of the released dataset was adjudicated asynchronously. these annotations and visualizations serve as a foundation for further analysis of dialogue flow, discourse coherence, and conversational engagement patterns within multi-party interactions. ## inter-annotator agreement since dda is not a straightforward classification task but still relies on a finite set of labels, we adapt krippendorff's alpha @xcite to evaluate inter-annotator agreement for our annotations. each dialogue consists of a set of sus, denoted as @xmath0, . . . , u n }. each annotator @xmath1,a for each u i . here x i,a is a set of edges or labels, depending on the aspect we consider. we define a distance function δ(x, y) ≥ 0 on the annotation domain ω. krippendorff's alpha can be defined as: to have a better understanding of the intricate annotation task, we incorporate a label hierarchy within the dda label set. ## downstream use cases the development of the dda corpus is driven not only by theoretical interest but also by its practical applications. one key use case of the dda framework is in collaborative learning analysis, where it can be used to assess students' interaction dy-namics in group discussions. by aggregating dda edges across a dialogue session, we can construct a holistic representation of speaker interaction behaviors. figure 4 illustrates this approach, where each node represents a speaker in a given session, and incoming edges denote dda response edges directed toward the speaker's utterance, while outgoing edges indicate the opposite. this speaker dynamics graph captures interaction patterns throughout an entire session, both when considering all response dependencies or particular subsets. for example in figure 4 (a),(b) and (c), considering all response dependencies shows more frequent communication between particular speakers in a group for different domains. in (d1) and (d2), we can isolate subsets of relations to consider, such as those with forward communicative function (asking a 'question', making a 'proposal', etc.) or backward communicative function (providing an 'answer', 'accept', or 'acknowledgment'). where (c) shows significant communication between the pilot and messenger or pilot and engineer, (d1-2) highlight that the pilot is driving most of the discussion in each case. 5 with an automated dda parser, such analyses could be generated dynamically, enabling real-time analysis of group discussions. another important application of the dda structure is in dialogue policy development, particularly for improving ai-driven conversational agents. by leveraging dda annotations, ai systems can infer speaker intentions more effectively, even when users do not explicitly verbalize their thought processes. the structured representation of speaker relations provides a richer context for response generation, facilitating a more transparent, controllable, and interpretable dialogue system. ## dda parsing with llms we evaluate whether llms can analyze conversational structure and intents using the dda coding scheme. to do this, we prompt a variety of proprietary and open-source models with the task of incrementally constructing a dda parse. ## conclusion and future work while we produce dda resources for structured speaker intention modeling and their characteristics and potential impact, several directions remain open for our future exploration. one of our concurrent efforts is to further evaluate the impact of dda as an intention representation in facilitating dialogue policy learning and response generation. we aim to assess whether structured intention representation improves dialogue coherence and response controllability and will compare dda-driven models against dialogue act tagging based approaches. additionally, we plan to expand dda annotations to additional datasets, both within our existing domains and in new conversational settings. this expansion will enable cross-domain analysis, helping to identify commonalities and variations in speaker intention patterns across educational, collaborative, and social dialogue contexts.