Analyzing Dialogue System Behavior in a Specific Situation Requiring Interpersonal Consideration (2025)

Abstract: In human-human conversation, interpersonal consideration for the interlocutor is essential, and similar expectations are increasingly placed on dialogue systems. This study examines the behavior of dialogue systems in a specific interpersonal scenario where a user vents frustrations and seeks emotional support from a long-time friend represented by a dialogue system. We conducted a human evaluation and qualitative analysis of 15 dialogue systems under this setting. These systems implemented diverse strategies, such as structuring dialogue into distinct phases, modeling interpersonal relationships, and incorporating cognitive behavioral therapy techniques. Our analysis reveals that these approaches contributed to improved perceived empathy, coherence, and appropriateness, highlighting the importance of design choices in socially sensitive dialogue.

## introduction interpersonal consideration plays a crucial role in human-human conversation, as extensively discussed in prior research @xcite @xcite . while these studies primarily focus on humanhuman interactions, interpersonal consideration should also be incorporated into human-machine dialogue, given the growing demand for automated systems that take such factors into account @xcite . although implementing interpersonal consideration in dialogue systems has long been a challenge, recent advances in large language models (llms) have significantly mitigated these limitations, making it increasingly feasible to realize forms of interpersonal sensitivity that were previously unattainable. this study aims to examine the behavior of dialogue systems in scenarios where such consideration is required and to analyze the subjective assessment of human evaluators to identify effective approaches and remaining challenges. although consideration is important in a variety of contexts, such as attending to user needs in task-oriented dialogue, this study focuses on scenarios that involve conversations between friends, where interpersonal consideration is particularly crucial. to conduct this analysis, we focus on the 7th dialogue system live competition (dslc7) that we organized @xcite . using evaluation results and system logs from the participating systems in dslc7, we conduct a cross-system analysis to investigate how interpersonal consideration is handled in dialogue systems, identify effective approaches, and highlight remaining challenges. ## results of dslc7 situation track in the situation track of dslc7, 14 teams participated, and along with the baseline system developed by the organizers, a total of 15 systems were subject to evaluation in the preliminary round. a wide range of methods were explored by participating teams. notable design choices adopted by participating teams included the following: • prompt control based on real conversation examples: msu, anonymous2 the results of the preliminary round are shown in ## quantitative analysis using system logs the system logs output by each dialogue system allowed us to examine the relationship between system behavior and evaluation scores. the results are presented in all evaluation metrics were negatively correlated with the interval, which is the average time between a user utterance and the system's response. this indicates that slower response times had a detrimental impact on perceived quality, revealing response latency as a key challenge for future system design. ## qualitative analysis of evaluation comments in addition to ucs and mes, evaluators in the preliminary round were also asked to provide freeform qualitative comments on each system. we conducted a qualitative analysis of these comments and identified three frequently mentioned aspects: dialogue flow (flow), empathy expressed by the system (empathy), and multimodal behavior (multimodal). each comment was manually annotated by the authors according to these three categories, and the proportion of comments that mentioned each category positively was calculated for each system. the results are reported in the "# positive comment" columns of the following subsections examine how the system design influenced user impressions along these three aspects. ## conclusion in this study, we conducted a cross-system comparative analysis of the characteristics and evaluation results of dslc7. through this analysis, we examined the strategies and techniques used by each system and confirmed their effectiveness. we found that explicitly defining detailed phases in the dialogue was particularly effective, and that incorporating a psychotherapeutic approach contributed to enhancing perceived empathy. furthermore, we observed that the use of gestures positively influenced the evaluation. in the multimodal dimension, certain systems received high evaluations specifically for facial expressions or gestures, and these features were shown to positively influence overall system ratings. having confirmed the effectiveness of individual multimodal strategies, future research can explore integrated approaches that combine multiple modalities in a coordinated manner. given the wide range of strategies and design choices available for dialogue systems, competitions that enable cross-system comparative analyses are considered to hold a significant value @xcite . this study highlights the importance of such competitions in advancing dialogue system research. further analyses of the collected data are expected to provide deeper insights into the factors that influence system performance.