What’s Mine becomes Yours: Defining, Annotating and Detecting Context-Dependent Paraphrases in News Interview Dialogs (2024)

Abstract: Best practices for high conflict conversations like counseling or customer support almost always include recommendations to paraphrase the previous speaker. Although paraphrase classification has received widespread attention in NLP, paraphrases are usually considered independent from context, and common models and datasets are not applicable to dialog settings. In this work, we investigate paraphrases across turns in dialog (e.g., Speaker 1: “That book is mine.” becomes Speaker 2: “That book is yours.”). We provide an operationalization of context-dependent paraphrases, and develop a training for crowd-workers to classify paraphrases in dialog. We introduce ContextDeP, a dataset with utterance pairs from NPR and CNN news interviews annotated for context-dependent paraphrases. To enable analyses on label variation, the dataset contains 5,581 annotations on 600 utterance pairs. We present promising results with in-context learning and with token classification models for automatic paraphrase detection in dialog.

## introduction repeating or paraphrasing what the previous speaker said has time and time again been found to be important in human-to-human or human-tocomputer dialogs: it encourages elaboration and introspection in counseling @xcite @xcite , can help deescalate conflicts in crisis negotiations @xcite @xcite , can have a positive impact on relationships @xcite , can increase the perceived response quality of dialog systems @xcite and generally provides tangible understanding-checks to ground what both speakers agree on @xcite . fortunately, in nlp, paraphrases have received wide-spread attention: researchers have created guest: and people always prefer, of course, to see the pope as the principal celebrant of the mass. so that's good. that'll be tonight. and it will be his 26th mass and it will be the 40th or, rather, the 30th time that this is offered in round the world transmission. and it will be my 20th time in doing it as a television commentator from rome so. host: yes, you've been doing this for a while now. figure 1 : context-dependent paraphrase in a news interview. the interview host paraphrases part of the guest's utterance. it is only a paraphrase in the current context (e.g., doing something 20 times and doing something for a while are not generally synonymous). our annotators provide word-level highlighting. the color's intensity shows the share of annotators that selected the word. here, most annotators selected the same text spans, some included "from rome" as part of what is paraphrased by the host. we underline the paraphrase identified by our fine-tuned deberta token classifier. agreement single example with high variation dataset acc. α shortened example vote balanced 0.71 0.32 guest: [...] maybe the money will help. host: it can't hurt, let's put it that way. 9/20 random 0.72 0.23 g: so both parties agree that we need to stop horrific acts of violence against animals. but everyone is standing behind this. it is time to stop horrific acts of brutality on animals. h: britain's queen elizabeth's senior dresser writes "if her majesty is due to attend an engagement in particularly cold weather from 2019 onwards fake fur will be used to make sure she stays warm." it's a very stark example of a monarch following public opinion in the u.k. which is moving away from fur and it very much embraces prevention of cruelty to the animals. ## related work paraphrases have most successfully been classified by encoder architectures with fine-tuned classification heads @xcite and more recently using in-context learning with generative models like gpt-3.5 and llama 2 @xcite @xcite . to the best of our knowledge, only wang et al. (2022a) go beyond classifying paraphrases at the complete sentence level. they use a deberta token classifier to highlight text spans that are not part of a paraphrase, i.e., the reverse of our task. what? shortened example additional conclusions or facts ⊈ cp guest: if you're not in our country, there are no constitutional protections for you. host: so, you don't have a problem with facebook giving the government access to the private accounts of people applying to enter the u.s.? isolated equivalence ⊈ cp g: there are militant groups out there firing against the military. h: why did the army decide today to move in and clear out the camp? paraphrase taxonomies commonly go beyond binary classifications to make more fine-grained distinctions between paraphrase types, often including considerations w.r.t. the context of the text pairs. @xcite and @xcite describe substitutions and other lexical operations that result in paraphrases in a given sentential context. @xcite show that context information can reverse semantic relations between phrases. @xcite discuss text pairs that are equivalent when one presupposes encyclopedic or situational knowledge (e.g., referents or intentions foot_0 ), but exclude them as non-paraphrases. further, to the best of our knowledge, most previous work annotate sentence pairs without considering the document context, with @xcite being the only exception, and no previous work looking at detecting paraphrases in dialog. dialog act taxonomies aim to classify the communicative function of an utterance in dialog and commonly include acts such as summarize/reformulate @xcite . however, generally, communicative function can be orthogonal to meaning equivalence. for example, the paraphrase from ## context-dependent paraphrases in dialog in nlp, paraphrases typically are pairs of text that are approximately equivalent in meaning @xcite , since full equivalence usually only applies for practically identical strings @xcite -with some scholars even claiming that different sentences can never be fully equivalent in meaning @xcite @xcite . the field of nlp has mostly focused on paraphrases that are context-independent, i.e., approximately equivalent without considering a given context @xcite @xcite . some studies have operationalized paraphrases using more fine-grained taxonomies, where context is sometimes considered @xcite @xcite . however, only a few datasets include such paraphrases @xcite and to the best of our knowledge none that focus on context-dependent paraphrases or dialog data. we define a context-dependent paraphrase as two text excerpts that are at least approximately equivalent in meaning in a given situation but not necessarily in all non-absurd situations. 6 for example, consider the first exchange in context-dependent paraphrases range from clear (first example in we specifically exclude common cases of disagreements between annotators foot_2 that we consider not to be context-dependent paraphrases in dialog, see ## dataset generally, people do not paraphrase each other in every conversation. we focus on the news interview setting, because paraphrasing, or more generally active listening, is a common practice for journalists @xcite @xcite . we therefore also only consider whether the journalist (the interview host) paraphrases the interview guest and not the other way around. we use @xcite al. (2021 )'s mediasum corpus which consists of over 450k news interview transcripts and their summaries from 1999-2019 npr and 2000-2020 cnn interviews. 8 dataset size # paraphrases # anns/item balanced 100 54 20.1 random 100 13 5.7 para 400 254 7.5 total 600 321 9.3 ## annotation we first describe the annotation task ( §5.1). then, we discuss why the annotation task is difficult and a clear ground truth classification might not exist in many cases ( §5.2). therefore, we dynamically collect many judgments for text pairs with for pairs that at least two annotators classified a paraphrase, we display the average lexical overlap between the highlights (jaccard index displayed as a∩b a∪b ) and krippendorff's unitizing α over all words for guest and host highlights, see @xcite . ## modeling token classifier. similar to wang et al. (2022a), we fine-tune a large deberta model 15 @xcite on token classification to highlight the paraphrase positions (for hyperparameters, see app. d.2). we train two models: using all 3,896 training annotations ("all" in the 420 unique (guest, host) training pairs ("ag-gregated" in in-context learning. we further prompt the following generative models (see urls in app. d.1) to both classify and highlight the position of paraphrases: llama 2 7b and 70b @xcite , vicuna 7b @xcite , mistral 7b instruct v0.2 @xcite , openchat 3.5 @xcite , gemma 7b @xcite , mixtral 8x7b instruct v0.1 @xcite and gpt-4 foot_7 @xcite . we design the prompt to be as close as possible to the annotator training using a few-shot setup @xcite with all 8 examples shown during annotator training. we also provide explanations in the prompt @xcite and use selfconsistency by prompting the models 10 (gpt-4 and llama 70b: 3) times @xcite . for the prompt and further hyperparameter settings see app. d.1. results. for evaluation, we consider a pair to contain a paraphrase if it has been classified by a majority of crowd-workers and a word to be part of the paraphrase if it has been highlighted by a majority of crowd-workers. we leave softevaluation approaches to future work @xcite , among others because of challenges in extracting label distributions for in-context learning in a straight-forward way @xcite . see overall, gpt-4 and mixtral 8x7b achieve the best results in paraphrase classification. in highlighting, our deberta token classifiers and gpt-4 achieve the best overlap with human annotations. however, due to problems with extracting highlights from model responses (e.g., hallucinations, see app. d.3), our fine-tuned deberta token classifiers are probably the best choice to extract the position of paraphrases. while the deberta aggregated model achieves higher f1 scores, the deberta all model has the highest precision out of all models. we provide our best-performing deberta aggregated model (model with seed 202 and f1 score of 0.76) on the hugging face hub foot_9 and use it in the following error analysis. ## conclusion a majority of work on paraphrases in nlp has looked at the semantic equivalence of sentence pairs in context-independent settings. however, the human dialog setting is highly contextual and typical methods fall short. we provide an operationalization of context-dependent paraphrases and an up-scalable hands-on training for annotators. we demonstrate the annotation approach by providing 5,581 annotations on a set of 600 turn pairs from news interviews. next to paraphrase classifications, we also provide annotations for paraphrase positions in utterances. in-context learning and token classification both show promising results on our dataset. with this work, we contribute to the automatic detection of paraphrases in dialog. we hope that this will benefit both nlp researchers in the creation of llms and social science researchers in analyzing paraphrasing in human-to-human or human-to-computer dialogues on a larger scale.