How Stylistic Similarity Shapes Preferences in Dialogue Dataset with User and Third Party Evaluations (2025)

Abstract: Recent advancements in dialogue generation have broadened the scope of human–bot interactions, enabling not only contextually appropriate responses but also the analysis of human affect and sensitivity. While prior work has suggested that stylistic similarity between user and system may enhance user impressions, the distinction between subjective and objective similarity is often overlooked. To investigate this issue, we introduce a novel dataset that includes users’ preferences, subjective stylistic similarity based on users’ own perceptions, and objective stylistic similarity annotated by third party evaluators in open-domain dialogue settings. Analysis using the constructed dataset reveals a strong positive correlation between subjective stylistic similarity and user preference. Furthermore, our analysis suggests an important finding: users’ subjective stylistic similarity differs from third party objective similarity. This underscores the importance of distinguishing between subjective and objective evaluations and understanding the distinct aspects each captures when analyzing the relationship between stylistic similarity and user preferences. The dataset presented in this paper is available online.

## introduction recent advances in dialogue generation have markedly improved the ability of systems to produce appropriate and natural responses @xcite @xcite . building on this progress, personalization has become a central focus in dialogue system research, wherein system outputs are tailored to individual user preferences @xcite @xcite . unlike standardized evaluations such as appropriateness, individual user preferences are inherently difficult 1 https://github.com/ikuminumaya/duo-dataset to generalize, highlighting the need for system design that accounts for user-specific characteristics. prior studies suggest that, beyond semantic content, stylistic similarity-the extent to which a system's utterances resemble a user's speaking stylemay contribute to more positive impressions toward dialogue systems @xcite . however, a critical question is often overlooked : whose perception of stylistic similarity matters, and how it affects users' impressions. while prior studies often rely on third party annotators to judge stylistic similarity and user engagement, it remains unclear whether users' own perceptions of stylistic similarity or third party evaluations of it accurately capture users' preferences. to investigate this question, we constructed a novel dataset incorporating users' preferences, subjective stylistic similarity based on users' perception, and objective ratings provided by third party annotators. an overview of the dataset construction is shown in figure 1 . the dataset was based on two widely studied open-domain settings: empatheticdialogues @xcite , focusing on empathetic communication, and wizard of wikipedia @xcite , involving knowledge-grounded conversation. in opendomain dialogues, the primary goal is not explicit task completion but the cultivation of positive user impressions through conversational engagement @xcite @xcite . focusing on this setting, we use the newly constructed dataset to analyze the factors that influence users' subjective preferences. correlations analysis of the subjective evaluations revealed a strong positive relationship between subjective stylistic similarity and user dialogue preference, indicating that stylistic alignment plays a key role in shaping positive user impressions. in contrast, the correlation between objective stylistic similarity and user dialogue preference was weak, and no clear relationship emerged between subjective and objective stylistic similarity. these findings underscore the importance of distinguishing between subjective and objective stylistic similarity in analyzing dialogue quality and user preferences. the key contributions of this study are: open-domain dialogue. advancements in response generation have greatly enhanced the capacity of dialogue systems to engage in coherent, contextually appropriate multi-turn interactions (ope-nai, 2024; shuster et al., 2022; zhang et al., 2020). in open-domain dialogue research, the primary objective is not task completion but the facilitation of user-centered, engaging communication. the blended skill talk (bst) framework (smith et al., 2020) was introduced to evaluate dialogue systems that can blend multiple conversational skills in open-domain settings. the framework includes personalized dialogue based on profiles @xcite @xcite , emotion-aware responses that reflect users' affective states @xcite @xcite , and knowledge-grounded dialogue designed to support in-depth topic exploration @xcite @xcite . these settings are widely used in multi-turn response generation studies. in such interactions, systems are implicitly expected to maintain appropriate communication and foster user preferences. effects of entrainment on speakers. entrainment, which is also referred to as accommodation or convergence, describes the gradual alignment of linguistic and prosodic features (e.g., lexical choices, response timing, and vocal pitch) between interlocutors during dialogue. within the framework of communication accommodation theory (cat), @xcite presented concrete examples of entrainment in human-human interaction, emphasizing its positive social effects. prior research has shown that entrainment can increase perceived attractiveness @xcite , foster interpersonal involvement (lafrance, 1979), and strengthen relational bonds @xcite . beyond these social outcomes, @xcite introduced a method for quantifying entrainment in dialogue by analyzing word frequency patterns, demonstrating its role in enhancing perceived naturalness and task success in human dialogue. building on these findings, recent research has investigated the incorporation of entrainment into dialogue systems to enhance user experience @xcite . aligned with this line of research, our study explores how entrainment between users and systems influences users' subjective evaluations. ## dataset construction this study introduces an open-domain dialogue dataset designed to support subjective and objective evaluations. to facilitate the analysis of the relationship between dialogue preference and stylistic similarity, data were collected under two opendomain settings, followed by annotation from both users and third party evaluators. we named this dataset duo (dialogue dataset with user subjective and objective evaluations). figure 2 summarizes the overall dataset construction process described in the remainder of this section. setting empatheticdialogues emotion excited episode i'm moving out west, i'm excited about it. dialogue user: i'm going to be making a big move in 2 weeks. system: that's exciting! how are you feeling about it? ... ratings pref. sb : 1, cons. sb : 5, styl. sb : 1, emp. sb : 4 pref. ob : 3.67, cons. ob : 4.33, styl. ob : 3.67, emp. ob : 4.00 ## dataset analysis this section presents analyses based on evaluations collected during the dataset construction. hereafter, subjective (sb) and objective (ob) evaluation labels are abbreviated as follows: preference (pref. sb / pref. ob ), consistency (cons. sb / cons. ob )stylistic similarity (styl. sb / styl. ob ), empathy for ed (emp. sb / emp. ob ), and engagingness for wow (engag. sb / engag. ob ). ## conclusion this study focused on stylistic similarity as a key factor influencing individual users' dialogue preferences. while prior work has pointed to the potential relevance of stylistic similarity, it remained unclear whether users' own perception of stylistic similarity or third party assessments play a more critical role. to explore this question, we collected human-bot dialogues in open-domain settings and obtained both users' subjective preferences and stylistic similarity ratings, as well as third party annotations. our findings revealed a strong correlation between subjective stylistic similarity and user preference, whereas objective stylistic similarity showed only a weak correlation. moreover, there was no significant association between subjective and objective stylistic similarity. they suggest that subjective and objective evaluations reflect different aspects of style, underscoring the need to analyze them separately. the observed discrepancies suggest that whether the evaluation is made by a participant or a third party may matter not only for stylistic similarity but also for other aspects of dialogue assessment. future work could consider the impact of such perspective differences on system evaluation.