A Multi-Layered Annotation Protocol for Polyadic Conversation: Structuring Interactional Data in the G a MMA Corpus (2025)

Abstract: Computational models of dialogue often struggle to capture the nuanced structures of spontaneous conversation - specifically in polyadic, real-world settings. We introduce a multilayered annotation protocol designed for the GaMMA corpus, a Danish dataset of four-person conversations recorded in both quiet and noisy environments. The protocol targets key interactional phenomena: Turn Construction Units, backchannels, floor transfer attempts, and repair sequences. Each annotation layer is grounded in Conversation Analysis while remaining machine-actionable, enabling alignment with multimodal data such as gaze and motion. We report inter-annotator agreement metrics across annotation tiers and discuss how the protocol supports both fine-grained interaction analysis and the training of context-aware dialogue models.

## introduction multimodal conversation corpora are essential to dialogue systems, yet few resources provide systematic annotation schemes tailored specifically for spontaneous, polyadic interactions in real-world noisy environments @xcite . this paper introduces an annotation taxonomy developed for the gamma corpus @xcite conversations among four danish speakers recorded in both quiet and noisy conditions. inspired by foundational conversation analysis (ca) literature @xcite and multimodal annotation standards @xcite , our protocol captures complex interactional phenomena relevant to both ca and speech technology research. current computational methods, such as speech diarization, voice activity detection (vad), and automatic speech recognition (asr), lack robustness in detecting socially meaningful conversational structures like turn-construction units (tcus), backchannels, or failed floor transfers, especially in polyadic conversations @xcite . polyadic interactions pose unique challenges, including managing overlapping speech and negotiating speaking turns dynamically @xcite . we posit that structured human-annotated conversational behavior can provide valuable training and evaluation layers for context-sensitive computational models. ## related work turn-taking detection and facilitation in speech models often relies on silence thresholds, which work in some cases but fail during mid-sentence pauses or hesitant speech. for instance, openai's advanced voice mode interrupts on the user's backchannel speech, limiting the potential for fluid, conversational interaction. ideally, systems should maintain low latency after genuine utterance completions without mistaking pauses for turn ends @xcite @xcite . machine learning can help address this by learning to anticipate turn-taking using cues such as syntax, semantics, and prosody @xcite @xcite . recent evaluations of asr segmentation highlight further limitations. @xcite found minimal correspondence between automatic and human-annotated utterance boundaries, with openai's asr model, whisper, frequently over-segmenting and google's asr merging distinct turns or truncating mid-word. to address this, @xcite propose an end-of-turn detector (etd), speculativeetd, that serves as a classifier to distinguish between pauses and end-of-turns, alongside the etd dataset. while it is a valuable dataset of synthetic and real dyadic conversations, labeled with speech-state transitions (speaking unit, pause, gap), their annotations focus solely on low-level audio segmentation, and does not address interactional structures such as tcus, backchannels, or floor transfer attempts (fta). our protocol complements this work by offering ca-informed, multilayered annotations necessary to model conversational timing and turn negotiation more robustly. another relevant observation is that whisper's performance degrades on danish, and even further on dialectal danish, but the røst model, fine-tuned on the dialect and gender-balanced coral dataset, improves accuracy to levels comparable with english asr @xcite . these findings may generalize to the ca domain, with performance on tasks such as etd potentially affected by language-and dialect-specific acoustic cues. supporting this intuition, @xcite show that modern asr systems perform significantly worse on conversational speech than on read speech. these limitations reflect a broader trend: asr research remains focused on solo, voice-assistant-style interactions, with limited attention to polyadic, spontaneous conversation. this research gap limits the potential of speech dialogue systems in realistic group interaction settings, such as collaborative projects, where multiple speakers dynamically share the floor. ## corpus context and goals the gamma corpus @xcite consists of 44 conversations, each lasting between 13 and 19 minutes, among four normal-hearing danish speakers, recorded under controlled quiet and noisy conditions. audio and motion data were captured using head-mounted sensors @xcite . our objective was to develop a robust annotation protocol that could be reliably used by annotators with a basic background in conversation analysis, clearly labeling verbal interaction events critical to understanding conversational dynamics. ## annotation taxonomy our annotation protocol integrates insights from two primary traditions: (1) conversation analysis, particularly @xcite 's concept of tcus, which emphasizes the emergent and negotiable nature of conversational turns, and (2) multimodal annotation frameworks such as @xcite . additional influence stems from iso 24617-2 and mumin standards, although these often lack the flexibility or granularity needed for detailed polyadic interaction analyses @xcite . see while nuanced categories are needed to capture the complexity of natural conversation, an overly fine-grained taxonomy can overwhelm annotators and reduce consistency @xcite . to mitigate this, several subcategoriessuch as highly specific repair types -are deliberately excluded. ## annotation workflow and reliability annotation guidelines were developed with input from three annotators, of which two remained through the full training process. annotators independently labeled a subset of the corpus to assess inter-rater agreement (iaa). following protocol finalization, the full dataset was annotated by a single trained annotator. agreement statistics are reported for the two primary annotators. annotations were conducted using elan @xcite and custom matlab scripts for pre-alignment and label consistency. to accommodate the free segmentation style of conversational annotation (e.g., tcus), we compute agreement using an overlap-weighted metric inspired by @xcite . this score reflects the proportion of overlapping segment duration where labels match: agreement to account for structural differences between annotation tiers, we apply different agreement metrics: for segment-based tiers (e.g., tcus), we report the overlap agreement along with intersectionover-union (iou), using @xmath0. for binary classification tiers (e.g., backchannels), we compute accuracy, precision, recall, f1, and cohen's κ. repair labels are evaluated via overlapsensitive label matching and macro-f1. ## applications and outlook this structured taxonomy supports the study of conversational phenomena such as overlap negotiation, backchannel timing, and dynamic floortaking, offering actionable insights for both theoretical analysis and computational modeling. the gamma annotations provide rich training data for machine learning models focused on detecting tcus, backchannels, and trps, thereby enhancing the context sensitivity of real-time dialogue systems @xcite . ## discussion throughout the annotation process, several interactional patterns and challenges emerged. it became evident that interlocutors often exhibit distinct conversational habits -some frequently offer verbal backchannels, while others remain largely silent listeners. much of their speech is easily categorized using the available tiers, but certain outliers require more nuance -as with utterances like "so. . . ", which can variably continue a prior message, serve as a stalling device, or signal turn completion, and are not always disambiguated by prosody alone. see in appendix ?? for elan example. this variability presents difficulties for transcription-based methods for speech dialogue systems, which may struggle to accurately identify backchannels or delineate tcus. in the same elan example, another issue arises when two interlocutors speak simultaneously, resulting in an ambiguous floor transfer. this ambiguity can often be resolved using the surrounding conversational context or, even more reliably, by consulting video data. however, in some cases, it still requires subjective interpretation when the nuances are too subtle. additionally, speech overlap tended to decrease during periods of parallel dyadic interaction within the polyadic setting, suggesting that this structure may be useful for training models in managing multi-party turn-taking. another consideration is that interlocutors in varying degrees produce utterances that are syntactically incomplete yet pragmatically sufficient, which should pose challenges to the types of asr solutions that rely on transcription-based methods, and this underlines the importance of speech systems trained specifically for etd tasks. to capture more nuances, we propose several avenues for future continuation of annotation work on the gamma (or similar) dataset, based on the experience of the main annotator. these include marking short floor changes where an interjection does not fully claim the floor but is acknowledged through micro-pauses or gaze; annotating overlapping speech in which speakers simultaneously produce similar or identical content, effectively dividing the floor; identifying explicit floor control devices (e.g., "can i just say-"); and labeling collaboratively constructed tcus, where utterances are jointly completed by interlocutors. such additions would enrich the representation of conversational structure, particularly in complex multi-speaker environments. appendix a3: aggregated descriptive statistics for each conversation. metrics include total counts and durations of turn-construction units (tcu) and backchannels (bc) in seconds, bc and front-channel (fc) ratios, tcu completion rate, repair types (@xmath1, @xmath2, and inter-participant talking-time (talkshare) variability.