A Similarity Measure for Comparing Conversational Dynamics (2025)

Abstract: The quality of a conversation goes beyond the individual quality of each reply, and instead emerges from how these combine into interactional dynamics that give the conversation its distinctive overall “shape”. However, there is no robust automated method for comparing conversations in terms of their overall dynamics. Such methods could enhance the analysis of conversational data and help evaluate conversational agents more holistically.In this work, we introduce a similarity measure for comparing conversations with respect to their dynamics. We design a validation procedure for testing the robustness of the metric in capturing differences in conversation dynamics and for assessing its sensitivity to the topic of the conversations. To illustrate the measure’s utility, we use it to analyze conversational dynamics in a large online community, bringing new insights into the role of situational power in conversations.

## introduction in a conversation, individual utterances combine to form interactional patterns, such as changes in tone (e.g., from passive-aggressive to defusing), conversational strategies (e.g., analogies, concessions, or challenges), and interaction sequences (e.g., extended back-and-forth vs. one-sided rants). each of these patterns contributes to shaping the conversation's overall dynamics, but none of them alone is sufficient to characterize it @xcite . these emerging conversational dynamics are closely tied to the perceived quality of the conversation and its outcome @xcite @xcite . as such, a measure comparing conversations with respect to their overall dynamics can enhance our ability to analyze human-human and human-ai conversational data. for example, it can be used to group conversations according to their dynamics and distinguish those that are likely to lead to positive outcomes. this type of analysis could enable a * equal contribution. more holistic evaluation of conversational agents, one that goes beyond optimizing for the quality of each response to encourage overall dynamics that are desirable. however, developing a method for comparing conversations with respect to their overall dynamics presents several challenges. the first challenge is finding an appropriate way of representing the dynamics of a conversation: it is not sufficient to detect individual patterns separately (e.g., speech acts, empathy, politeness, sarcasm), as done by prior work @xcite @xcite . instead, a representation of the overall dynamics must capture how relevant patterns of different types connect to each other. for example, a passive-aggressive tone changing into a defusing tone leads to a very different dynamic than when a defusing tone is followed by a passive-aggressive tone. the second challenge arises when comparing dynamics. dynamics take place at multiple scales, with some patterns spanning single exchanges (e.g., a sarcastic response) and others spanning the entire conversation (an increasingly escalating tone). furthermore, a single utterance can contribute to multiple patterns (e.g., an utterance can be a sarcastic response and simultaneously be part of an increasingly escalating tone). this inherent overlap makes it hard to align the dynamics of two conversations in order to quantify how similar they are. in this work, we address these challenges to introduce a similarity measure for conversational dynamics: condyns (read as "condense"). we address the first challenge by representing dynamics as a sequence of relevant interactional patterns in a conversation (a sequence of patterns, henceforth the sop), extracted from a summary of conversational dynamics @xcite . this representation captures not only which interaction patterns are present in a conversation, but also the order in which they follow each other. we address the second challenge by designing an asymmetric procedure for aligning conversational dynamics (figure 1 ). the main intuition behind this procedure is to combine the advantage of the sop representation-which allows checking the order in which interaction patterns appear-with the advantage of a simple transcript representationin which we can find patterns with high-recall, even when they are overlapping. to validate the effectiveness of condyns and compare it with baseline measures using other representations or alignment methods, we introduce a human-in-the-loop procedure for generating labeled data. condyns recovers these labels with over 90% accuracy, substantially outperforming the baselines, while being robust against topical confounds. we further demonstrate how a similarity measure for conversational dynamics can enable new types of analysis by applying condyns to conversations from a large online debate community. first, it allows us to adapt standard similarity-based techniques-clustering, inter-group similarity, and intra-group diversity-to study conversational dynamics. second, we use our measure to investigate which participants are more likely to influence the dynamics of a conversation, providing new insights into the role of situational power in conversations. in summary, in this work we: we additionally explore the versatility of our measure by applying it to two other conversational domains, including scripted casual conversations between friends and non-english discussions held in a collaborative setting. to encourage further use and development, we release the code for condyns publicly as part of convokit, including demos on multiple datasets. 1 2 background and related work conversational dynamics. we use the term "conversational dynamics" to denote how different interactional patterns combine and unfold over the course of a conversation to shape its overall temporal trajectory. this is arguably a broad and under-defined concept, as it depends on what type of interactional patterns are considered most salient in a given context, and how these patterns are identified and tracked over the course of the interaction. prior literature has studied it by focusing on the temporal sequence of specific utterance-level patterns in isolation, such as argumentation strategies @xcite , sentiment @xcite , communicative acts @xcite , or dialog acts @xcite . other work has focused on tracking patterns that only emerge at a higher multi-turn or conversational-level, such as turn-taking @xcite , time-sharing @xcite , coordination @xcite , or changes in topical entropy @xcite . in this work, we follow @xcite to take a holistic approach that does not limit the types of patterns that can contribute to the overall dynamics. by adopting this flexible framework, our measure can compare dynamics involving patterns that are potentially novel or domain-specific, rather than imposing strict theoretical constraints in advance. ## measure measuring the similarity between the dynamics of two conversations involves (1) representing these dynamics and (2) comparing them. below, we discuss several options for these steps, which combine to form condyns and several baseline measures. here we describe the general approach, and defer to section 4 for details about the operationalization in the specific domains we analyze in this work. ## data and operationalization online debate discussions. to validate and demonstrate applications of condyns, we use a dataset of conversations from the changemyview subreddit (cmv), retrieved from convokit @xcite . the objective of this platform is for participants (challengers) to persuade the original poster (op) to change their viewpoint on an opinion they hold. 3 the dataset includes conversations from the subreddit's inception in 2015, up to 2018, and is thus not polluted by content generated by large language models. in this paper we use a total of 9,138 cmv conversations, selected as described in sections 5 and 6. this setting has several properties that make it particularly suitable for developing a similarity metric for conversational dynamics. first, it has been a resource for many studies analyzing how conversational features connect to different outcomes-such as successful persuasion @xcite @xcite or conversation derailment @xcite @xcite )-documenting its richness in conversational dynamics. second, a key feature of the dataset is the "delta" (∆) mechanism through which the op can award a ∆ to a challenger that successfully changed their view. this mechanism provides explicit persuasion labels for each conversation, which we will use to interpret our results. finally, @xcite developed the scds procedure on this dataset. as such, they distribute humanwritten scds and provide a validated procedural prompt for automatically generating scds, which grounds our method and validation procedure in an established framework. other conversation settings. as discussed in section 2, we adopt a flexible framework for conversational dynamics in order to allow adapting the measure to other domains where different patterns might be at play. to explore the versatility of condyns, we apply it to two additional settings (section 7). the first setting involves 50 fictional conversations from the friends tv show @xcite which are scripted to resemble everyday face-to-face interactions and reflect entertainment-driven dialogue. the second is a collaborative setting in a non-english language, specifically 100 conversations from the german wikipedia talk-pages @xcite . operationalization. we release a modular implementation of condyns in convokit, making it easy to swap specific components to facilitate adaptation to different settings. we use google's gemini 2.0 flash model's api @xcite for generating scds, extracting sop, and quantifying the alignment of dynamics. 4 to generate scds, we use the procedural prompt validated by @xcite for our main cmv setting, and modified versions that include domain-specific examples for the additional settings. all generated scds are distributed together with the respective datasets in convokit. to measure the alignment scores s i , we use a few-shot in-context learning prompt with human-constructed examples (showing scoring and reasoning) to quantify alignment. all prompts are included in appendix a. ## validation no data with labels for similarity of conversational dynamics is available, and the vast space of possible dynamics and their complexity makes humanannotation highly subjective and prohibitively timeconsuming @xcite . therefore, to validate our measure and compare it with baseline measures, we design a human-inthe-loop procedure for obtaining synthetic data in which the relative similarity of conversational dynamics is known (figure 2 ). specifically, we construct triplets of conversations with (1) an anchor conversation that serves as the reference for comparison, (2) a positive conversation with a dynamic that is known to be similar to that of the anchor, and (3) a negative conversation with a dynamic that is known to be different from that of the anchor. 5 given a collection of such triplets, we calculate the accuracy of a similarity measure as the proportion of triplets where the anchor-positive pair receives a higher similarity score than the anchor-negative pair. anchor-positive pairs. recent work has demonstrated that llms can be used to reliably simulate conversations with specific properties @xcite . we use a similar idea and prompt an llm to simulate a conversation that closely follows the dynamics of a given anchor conversation. a manual check of the resulting pairs, however, reveals that directly providing the anchor's transcript in the prompt often leads the model to directly replicate its surface-level features, such as topic, word choice, or speaker turn order, rather than creating an entirely new conversation. to rectify this, we rely on the scd abstraction to remove such surface-level features while maintaining the desired dynamics. we prompt the llm to generate a conversation following the dynamics summarized in the anchor's scd. we use human-written (rather than machine-generated) scds since they are guaranteed to accurately represent the dynamics as perceived by humans @xcite , while also avoiding circularity with the measures using machine-generated scds. this procedure results in a conversation that, while completely new, follows similar dynamics to the anchor conversation, forming the anchor-positive pair. ## applications having verified its effectiveness, we now demonstrate possible applications of condyns in analyzing conversational datasets. we start with showcasing three types of standard data analysis techniques for which a similarity metric is neededclustering, comparing inter-group similarity, and comparing intra-group diversity-and show that condyns leads to intuitive results in our online discussions setting (outlined in figure 3 ). we then use our measure to answer new questions about a speaker's tendency to engage in similar dynamics across different conversations and about how a speaker's role in a conversation mediates their influence over its dynamics (figure 4 ). to ensure that results are not driven by basic structural differences like participant count or conversation length, we focus on conversations that involve only the op and one challenger (who always initiates) and that are at least 4 utterances long (and thus are long enough to allow dynamics to develop). we also consider a stricter length control, in which all conversations are between 4 and 6 utterances. both conditions lead to similar qualitative and numerical results, with small changes in significance levels. in what follows we report the results with the strict length control, and report those without strict length control in appendix f. ## application to other settings in developing condyns, we purposely adopted a flexible framework that can accommodate a broad range of conversational dynamics (section 4). this design choice facilitates the extension of our measure to new domains with diverse dynamics. casual dialogues between friends. to explore this versatility, we apply our method to 50 dialogues from the friends tv show @xcite which are scripted to resemble casual face-toface dialogues. the only component of condyns that needs to be adapted is the prompt for generating the scds. to tailor these summaries to the dynamics of this particular setting, we adjust the prompt to indicate its casual nature, incorporating a few hand-written examples of summaries (figure 7 in the appendix). we apply the same procedure from section 6 to provide a qualitative analysis of the two top-level clusters obtained using con-dyns in this setting. the clusters ( overall, this exploration suggests condyns is versatile enough to be adapted to a variety of conversational settings and to account for different types of conversational dynamics. the ability of condyns to take into account specific aspects of conversational dynamics ultimately depends on the extent to which the scd-creation procedure can be guided towards surfacing those aspects. ## conclusion in this work, we introduce a similarity measure for conversational dynamics and develop a validation procedure to compare different representations and alignment methods. we showcase the measure's utility in the context of an online debate community, adding to the literature on the relation between situational power and influence in conversations. our measure joins a growing toolkit of computational methods for conversational analysis. in future work, our measure could be used towards a more holistic evaluation of llms' conversational ability, going beyond the quality of each individual reply to compare systems with respect to the dynamics they engender. for instance, our metric could be applied to compare the dynamics of mental health therapy conversations held by human therapists versus ai therapists, a timely problem with important societal ramifications. similarity measures like condyns can also be an important step towards providing conversationallevel feedback to ai agents to encourage dynamics that are similar to those preferred by humans. for example, post-training reinforcement-learning methods could be extended to consider rewards based on similarity with human-preferred dynamics, in addition to human-preferred replies.