How did we get here? Summarizing conversation dynamics (2024)

Abstract: Throughout a conversation, the way participants interact with each other is in constant flux: their tones may change, they may resort to different strategies to convey their points, or they might alter their interaction patterns. An understanding of these dynamics can complement that of the actual facts and opinions discussed, offering a more holistic view of the trajectory of the conversation: how it arrived at its current state and where it is likely heading.In this work, we introduce the task of summarizing the dynamics of conversations, by constructing a dataset of human-written summaries, and exploring several automated baselines. We evaluate whether such summaries can capture the trajectory of conversations via an established downstream task: forecasting whether an ongoing conversation will eventually derail into toxic behavior. We show that they help both humans and automated systems with this forecasting task. Humans make predictions three times faster, and with greater confidence, when reading the summaries than when reading the transcripts. Furthermore, automated forecasting systems are more accurate when constructing, and then predicting based on, summaries of conversation dynamics, compared to directly predicting on the transcripts.

## introduction conversations take place on at least two different levels @xcite . on one level, participants directly communicate ideas, facts, and opinions, providing the topical context of the discussion. on the other level, the dynamics of their interactions reveal how they feel about each other, through changes in their tone-e.g., polite @xcite , condescending @xcite , or sarcastic @xcite -, conversational strategies they employ-e.g., rhetorical questions @xcite -and even the patterns of their exchanges @xcite . ## human-written summaries to start, we introduce a procedure for writing scds and a collection of such summaries for an existing dataset of online conversations. procedure for writing summaries. to construct the first collection of scds, we iteratively designed a writing procedure that addresses the selection and synthesis challenges described in the introduction. in early iterations in which we asked a single annotator to both read the transcript and write its scd, we observed that they consistently omitted key information that they take for granted, perhaps because some aspects of the dynamics are often processed non-consciously @xcite . to address this issue, we devise a procedure that uses interaction between two annotators to surface key elements of the conversation dynamics that readers who cannot see the transcript would consider relevant. thus, we settle on a procedure that has two parts-one in which an annotator works individually and one in which they interact with another annotator-which we briefly outline here (and detail in appendix a). for the individual work, annotator a will draft several summaries for a transcript in 4 steps: in the interactive part, annotator b will write the scd, by interacting with annotator a with a goal of understanding the conversation trajectory. in this process, annotator b may read the summaries written in the previous steps by annotator a, but not the transcript, and may make inquiries on details they deem important to understand the trajectory, such as 'was this said neutrally, or is there something about the tone that i should note?' or 'is the comment overtly rude, or is it just passive-aggressive or blunt?', surfacing key aspects that were not explicitly mentioned in annotator a's summaries. conversation transcripts data. we apply this procedure to summarize conversations from the conversations gone awry -changemyview (cga-cmv) dataset (chang and danescu-niculescu-mizil, 2019), foot_1 a conversation corpus collected from the 'change my view' subreddit, where people actively seek to have others challenge their views on controversial topics. this community has been studied extensively in part because of their explicit norms against toxic behavior, and corresponding labels inferred from the moderators' interventions. in the cga-cmv corpus, conversations are paired such that every conversation that derailedi.e., ended in a toxic comment removed by moderators-is matched with another conversation on the same topic that did not. for us, these labels provide an opportunity to test the extent to which scds provide an intuition about the future trajectory of the conversation (i.e., will it derail or not). to focus on the future trajectory, we use truncated transcripts obtained by removing the last 3 utterances from every conversation (in addition to the toxic comment, if there was one). since our interest is in summarization, we only keep pairs where both conversations are longer than 10 utterances. collection of human-written summaries. we produce human-written summaries for 50 conversations from the train split of cga-cmv. the summary writing process took roughly 140 annotatorhours. 3 summaries are on average 71 words long (annotators are instructed to keep them under 80); for comparison, the transcripts are on average 940 words long. an example summary is shown in figure 1 , and a qualitative analysis is provided in section 5. ## machine-generated summaries we now turn to explore several simple baselines for generating scds, setting the stage for developing more specialized methods in future work. models in the gpt family have achieved remarkable results in various summarization benchmarks @xcite . among them, chatgpt is particularly suitable for adapting to new tasks like ours without demanding a sizable train set. thus, for the first group of baselines, we query openai's chatgpt (gpt-3.5turbo-0613) api with default parameters using different prompts, from the most common prompt for 4 an equivalent check could be implemented by providing one summary segment and three transcripts to pick from. this method corresponds to the existing literature in communication-based evaluations for natural language generation @xcite , and implements the idea that an informative summary should capture the salient information that makes the source text stand out with respect to other related texts @xcite . however, this equivalent method would require substantially longer time due to the lengths of the transcripts. we also experimented with few-shot in-context learning on a small subset of the training set, but manual inspection did not reveal an increase in quality. thus, due to significantly higher api costs, we did not pursue this path. appendix d includes more discussion of our prompt engineering. finetuning. finally, we experimented with finetuning gpt-3.5-turbo and smaller dialogue summarization systems (bart-large and dialogled) @xcite using the 50 human-written summaries and the corresponding transcripts. we provide details in appendix d. ## downstream evaluation: forecasting derailment popular metrics for summarization-e.g., rouge @xcite , bertscore @xcite , and qa-based metrics-are notoriously unreliable when evaluating llm-generated summaries or summaries of long documents @xcite . we thus follow recommendations of @xcite and perform a downstream evaluation, in which we quantify the extent to which scds provide an understanding of the conversation trajectory. specifically, we choose the task of forecasting whether a conversation will eventually derail into toxic behavior @xcite . unlike previous work in which the prediction was made based on a truncated transcript of the conversation (for a comprehensive discussion of prior models see section 6), here we aim to make the prediction directly on the scd of that truncated transcript. in addition to providing means to evaluate and compare current and future models for generating scds, this derailment forecasting task is also important in itself, as it was shown to enable important practical applications: automated forecasts can be used to inform users during ongoing discussions @xcite while human forecasts are made by moderators in their everyday workflow @xcite (see section 8 for practical and ethical considerations of real-world deployment). besides the potential practical importance of this task, it is also worth noting its difficulty. because the forecaster never actually sees the toxic comment, it must rely on subtle signals and overall trajectory of the conversation dynamics. this inherent difficulty and its implications on the design of forecasting models were extensively discussed in @xcite . we first compare the usefulness of scds for automated forecasting systems. then we devise an experiment to estimate their usefulness for human forecasts. throughout, the forecasts are done on a balanced dataset of derailing and non-derailing conversations paired by topic, following the setup of the cga-cmv dataset (section 2); thus the overall topic of the discussion plays a minimal role and the random baseline is 50%. to leave room for future work we leave the original cga-cmv test set untouched. using truncated transcripts from the original train split, we construct a new train set (234 conversations), a new development set (100), and a new test set (100); the new test set includes the 50 conversations for which we also have human-written summaries (section 2). ## qualitative analysis to complement our quantitative evaluation and understand what might drive the differences between human and machine-generated summaries, we now turn to the actual content of the scds. through a close reading of the 20 human-written and 20 machine-generated summaries used in the experiment described above, we identify, annotate, and compare several aspects that were shown to provide clues about the conversation trajectories. 8 this difference continues to hold when only considering correct guesses. also, reassuringly, confidence in correct guesses is higher than in incorrect ones throughout. 9 we also experimented with directly asking participants to report their understanding of the trajectory of the conversation, on a scale from 1 to 5. there was no significant difference between human and machine-written summaries (4.0 and 3.9 respectively), perhaps due to the difficulty of briefly explaining what a trajectory is and how it differs from the derailment prediction, a confusion that surfaced during debriefing. tone. whether 'polite,' 'rude,' 'aggressive,' 'condescending,' or 'sarcastic' @xcite , the tone employed by the participants is a prominent feature of the scds. tone can be explicitly stated, as in 'speaker1 disagrees [...] in a somewhat passive-aggressive tone.' other times, especially in human-written summaries, it is expressed as modifying a speech act, as in 'contradicts sarcastically', 'disagrees rudely,' and 'adamantly defends.' overall, tone is indicated less frequently in the machine-generated summaries (75% of them mention tone at least once) than in the human-written summaries of the same conversations (all mention tone at least once), suggesting a potential path for improvement. ## further related work our work falls at the intersection of three broad areas of nlp: studies of conversation dynamics, summarization, and conversation forecasting. conversation dynamics. we are primarily inspired by extensive computational work on modeling various aspects of conversation dynamics. some studies have focused on identifying specific aspects, such as such as politeness @xcite @xcite , formality @xcite , passiveaggressiveness @xcite , condescension @xcite or sarcasm @xcite . others have studied changes along these dimensions during the discussion @xcite @xcite . a separate but related thrust focused on persuasive strategies in-terlocutors employ in a conversation, mostly in the context of debates (see @xcite for a survey). unlike these studies, the goal of scds is not to exhaustively identify occurrences of either one of these phenomena, but to convey how such key aspects combine towards an understanding of the conversation's trajectory. nevertheless, the dataset of scds that we release (with annotated aspects of conversation dynamics) can constitute an additional resource for studying these phenomena and the context in which they occur. dialogue summarization. the vast majority of dialogue summarization systems focus on the content of the utterances, rather than on the more subtle non-topical dynamics. early approaches to dialogue summarization focused on using external tools to explicitly model dialogue structures, such as topic segmentation and conversation stages @xcite , dialogue acts (goo and chen, 2018), discourse dependency and speaker-action relations @xcite , which are processed into features that can help language models. later, pretraining on dialogue corpora also attracted increasing research interest and achieved state-of-the-art results on many datasets @xcite . most recently, extensively pretrained, instruction-tuned llms, such as the gpt family models, have achieved superior results on various summarization leaderboards @xcite @xcite . in dialogue summarization, these instruction-tuned llms possess strong in-context-learning capabilities @xcite , making them strong candidates for solving new summarization tasks that have limited training data. conversation forecasting. we motivate and evaluate dynamics summaries with applications requiring an understanding of a conversation's trajectory. beyond forecasting derailment @xcite @xcite , other tasks include forecasting thread growth @xcite , prosocial outcomes @xcite , editorial decisions @xcite , controversy @xcite , the outcomes of negotiations @xcite or team resilience @xcite . it would be interesting to consider the extent to which scds can aid with these other forecasting tasks, and how to obtain summaries specifically dedicated to these tasks. to the best of our knowledge, all conversational forecasting systems operate directly on conversa-tion transcripts. the early work by @xcite adopts a recurrent network and applies unsupervised training to learn a representation of conversation dynamics. more recently, @xcite explores pretraining and various training paradigms for this task, @xcite applies a graph convolutional network, and yuan and singh (2023) uses a hierarchical transformer-based framework to combine utterance-level and conversationlevel information. however, since it aims to guess the future, this task remains challenging. unlike detecting the toxic language after the fact @xcite , the signs of future derailment are subtle and require a more thorough understanding of the conversation trajectory. our results suggest that scds can provide this information concisely and effectively, suggesting a new summarize-then-forecast approach to conversational forecasting tasks. this inspires future work to integrate scds in real-time forecasting systems, which would require tackling shorter conversations where summaries might not be appropriate, as well as the 'unknown horizon' problem: not knowing when to trigger the prediction @xcite . ## conclusions in this work, we introduce the task of summarizing the dynamics of interaction between participants in a text-based conversation. by introducing human and automated procedures for writing such summaries, we show that they can capture information that is mostly missing from traditional summaries, such as the tone in which the participants write and how it changes throughout a conversation. summaries of these dynamics are useful to both humans and automated systems for understanding the overall trajectory of the conversation, as shown through the downstream evaluation task of forecasting whether a conversation will eventually derail or not. humans can make similarly accurate forecasts three to four times faster by starting from scds than by reading the transcripts. when compared to directly predicting on the transcripts, automated systems make better forecasts when generating scds as an intermediate step. going beyond the gains in accuracy and efficiency, scds add interpretability to the forecasting task. interpretability is key to enabling applications where forecasts are used by humans for decision making. for example, when warning interlocutors (or moderators) that a conversation is at risk of derailing @xcite , scds could provide them with insights into why that might be the case, helping them decide whether to heed the warning and how to deescalate the situation. this pertains to settings beyond online forums, such as customer service conversations or mental health counseling conversations, in which supervisors need to decide fast whether (and how) to intervene in conversations that seem to enter a non-desirable path. in addition to forecasting, capturing and succinctly describing the dynamics of conversations can be useful for a series of applications, including training conversationalists (e.g., by providing them with a way to review their prior conversations), providing context to people (re)joining a conversation (e.g., to aid a therapist prepare for their next session with their client), or identifying common trajectories in human-ai conversations. scds could even aid conversational analysis researchers more efficiently explore individual conversations and reveal patterns in the intricate ways in which they develop @xcite .