An LLM Benchmark for Addressee Recognition in Multi-modal Multi-party Dialogue (2025)

Abstract: Handling multi-party dialogues represents a significant step for advancing spoken dialogue systems, necessitating the development of tasks specific to multi-party interactions. To address this challenge, we are constructing a multi-modal multi-party dialogue corpus of triadic (three-participant) discussions. This paper focuses on the task of addressee recognition, identifying who is being addressed to take the next turn, a critical component unique to multi-party dialogue systems. A subset of the corpus was annotated with addressee information, revealing that explicit addressees are indicated in approximately 20% of conversational turns. To evaluate the task’s complexity, we benchmarked the performance of a large language model (GPT-4o) on addressee recognition. The results showed that GPT-4o achieved an accuracy only marginally above chance, underscoring the challenges of addressee recognition in multi-party dialogue. These findings highlight the need for further research to enhance the capabilities of large language models in understanding and navigating the intricacies of multi-party conversational dynamics.

## introduction the rapid advancements in dialogue systems, fueled by the emergence of large language models (llms) capable of generating human-like text and engaging in natural conversations, have been largely confined to the realm of dyadic interactions. while these systems have demonstrated remarkable progress, they fail to capture the complexities inherent in multi-party dialogues, involving three or more participants. these dialogues are characterized by intricate information flow, dynamic participant roles, and nuanced social cues, posing significant challenges for system development. previous research has explored specific aspects of multi-party dialogues, including turntaking @xcite ; skantze et al., 2015), addressee recognition @xcite @xcite , and dialog act recognition @xcite . however, existing benchmarks are limited by their reliance on textbased or acted dialogue data, failing to reflect the spontaneity and multi-modality inherent in natural human interactions. to address this crucial gap, this paper introduces a novel, spontaneous, and multi-modal multi-party dialogue corpus specifically designed to facilitate research on triadic (three-participant) dialogue systems. this research further focuses on the critical, yet under-explored, task of addressee recognitionthe identification of the intended recipient of a turn -which is foundational for enabling dialogue systems to navigate and participate effectively in multiparty settings. unlike dyadic interactions where the addressee is implicitly defined, turn-taking in multi-party settings is far more complex. the intended recipient might be a specific participant or the group as a whole, and behavioral signals are often subtle and inconsistent @xcite . this work makes two key contributions: ultimately, this research aims to establish a strong foundation for the development of advanced multiparty dialogue systems capable of understanding and responding effectively in complex, real-world conversational settings. ## teidan corpus we begin by briefly describing the teidan multiparty corpus. unlike other datasets that involve specific contexts, such as meetings @xcite , task-oriented interactions @xcite , or game-based scenarios @xcite @xcite , the teidan corpus captures goal-free discussions. additional multi-party corpora also exist for online discussions @xcite . the discussions involved triads (groups of three participants). each participant was seated in a circle around a table placed at the center, as shown in figure 1 . cameras recorded each participant's face, and individual pin microphones were used to capture their speech separately. participants discussed three general topics: (1) which city would be best suited as an alternative capital of japan, (2) which items would be essential to bring to a desert island, and (3) where they would like to travel on the weekend. each triad completed three discussion sessions, one for each topic. each triad conversed for approximately 5 to 10 minutes per session, with no requirement to reach a conclusion. data were collected from 10 triads, resulting in a total of 30 discussion sessions. note that this corpus is in the japanese language. ## annotation of addressee we annotated a subset of the teidan corpus for addressee information. the annotation process consisted of the following steps: this labeling process considered both textual and visual cues, such as gaze behavior. initially, a single session was annotated and discussed to ensure inter-rater agreement by the authors. subsequently, the remaining four sessions were annotated by one of the authors. we have so far annotated five sessions from the teidan corpus. ## benchmark for addressee recognition to evaluate the task's complexity, we tested the performance of a multimodal large language model (gpt-4o) on addressee recognition. the model was given a prompt as follows: in the following conversation among a, b, and c, please infer who is addressed as the next speaker in the last utterance. answer with one of the following: "a, b, c, or o". "a, b, and c" represent the participants, and "o" represents the case where no one is addressed, and anyone can take the turn next. the output should only contain the label "a, b, c, or o" and should not include any other characters. this was followed by five context turn utterances with the current utterance, and also it contained the name of the discussion topic and designated identifier of the participants. note that the utterances were manually transcribed. the gpt-4o achieved an accuracy of 80.9%, which is only marginally above the chance level (80.6%). this indicates that the model struggles to identify the addressee in multi-party dialogues. the output by the llm is summarized in we then analyzed samples to examine how gpt-4o deals with addressee recognition, as illustrated below: ## adding gaze features to see the effect of gaze information in the current task, we processed the video of each participant (shown at the bottom of figure 1 ) and automatically annotated their gaze throughout the discussion. openface 2.0 was used to estimate the eye gaze vector @xcite . we could then generate a gaze vector 30 times a second. for every gaze timestamp, we then estimated whether the gaze of the participant who had the turn (speaker) was directed at either one of the other participants or at nobody in particular. as each participant was seated in an approximately equilateral triangle, we used a simple heuristic to test if the speaker was looking at another participant. the y (up-down) portion of the gaze vector must be within a certain range (0.2), and the x (leftright) gaze vector had to be out of a certain range (-0.2 to 0.2). if this heuristic was met, then the gaze timepoint was labeled as the speaker looking at the relevant participant, else the gaze timestamp was labeled as o (no participant). we also labeled the turn of a speaker as opposed to continuous timestamps. we based our approach on previous research which found that end-of-turn gaze was important @xcite and labeled the majority gaze in the turn's final second, specifically the gaze label which was present in over 50% of the timestamps, or o if this was not reached. this information was added to the previous prompt to assess if adding gaze information in this way could improve the result. however, as shown in ## benchmark for next speaker prediction we are also interested in predicting the actual next speaker in the multi-party scenarios @xcite . this is distinguished from addressee annotation, where subsequent information on who took the turn is unknown, and there is no 'o' label as somebody must take the turn. it is possible that the addressee and the actual next speaker differ because of interruptions during the turn. we then evaluated gpt-4o's performance on this next speaker prediction task. using a prompt similar to that used for addressee recognition, the model was tasked with predicting the actual next speaker. the output label was limited to a, b, or c, with a chance-level accuracy of 50%, as either of the other two participants could take the turn. as a result, gpt-4o attained an accuracy of 46.0% on this task, performing below chance level. this outcome further suggests that the model struggles to effectively capture the dynamics of turn-taking in spontaneous multi-party dialogues. ## conclusion this study investigated the challenges of addressee and next speaker prediction in multi-party dialogues. we introduced a new multi-party dialogue corpus and analyzed the performance of an llm (gpt-4o) on these tasks. the findings revealed that llms struggle with the complexities of multi-party interactions. they perform only marginally above chance level in addressee recognition and below chance level in the next speaker prediction task. although the llm was given the simple gaze feature, it did not improve the performance. these results underscore the need for further research to improve llms' understanding of multiparty conversational dynamics. future work should explore more sophisticated methods for incorporating contextual information, including gaze and other non-verbal cues, and develop new models that can better capture the intricate interplay between participants in multi-party conversations.