TR o TR : A Framework for Evaluating the Re-contextualization of Text Reuse (2024)

Abstract: Current approaches for detecting text reuse do not focus on recontextualization, i.e., how the new context(s) of a reused text differs from its original context(s). In this paper, we propose a novel framework called TRoTR that relies on the notion of topic relatedness for evaluating the diachronic change of context in which text is reused. TRoTR includes two NLP tasks: TRiC and TRaC. TRiC is designed to evaluate the topic relatedness between a pair of recontextualizations. TRaC is designed to evaluate the overall topic variation within a set of recontextualizations. We also provide a curated TRoTR benchmark of biblical text reuse, human-annotated with topic relatedness. The benchmark exhibits an inter-annotator agreement of .811. We evaluate multiple, established SBERT models on the TRoTR tasks and find that they exhibit greater sensitivity to textual similarity than topic relatedness. Our experiments show that fine-tuning these models can mitigate such a kind of sensitivity.

## introduction as individuals, we often reuse someone else's words for diverse reasons and in various ways. this linguistic choice transcends cultural and temporal boundaries, representing an interesting phenomenon to study in linguistics @xcite . for instance, linguistic scholars have investigated theories of reception @xcite and @xcite to understand how individuals and communities interpret and reuse historical texts many years after they were written. with the advent of digitization, recent years have seen a growing interest in computational methods for studying text reuse, i.e., "the reuse of existing written sources in the creation of a new text" @xcite . existing methods focus on the main task of text reuse detection (trd). in trd, text reuses are all assumed as "topically related to the source" @xcite , the boundaries of reused text are unknown, and the goal is to detect text reuse across a diachronic corpus @xcite . whether and how the topic(s) or context(s) of a reused text differs from the source is generally overlooked. thus, new methods are needed for modeling recontextualization, i.e., "the dynamic transfer-andtransformation of a text from one discourse/text-incontext to another" @xcite . in this paper, we propose a framework, called topic relatedness of text reuse (trotr), to evaluate computational methods for capturing the different recontextualizations of text reuse. in trotr, the boundaries of reused text are known and the goal is to distinguish reuses of the same text according to their different, latent (i.e., unlabeled) topics. as an example, consider three recontextualizations of the biblical passage john 15:13 (in bold): greater love has no one than this: to lay down one's life for one's friends" in this example, the biblical passage is incorporated within three texts with different topic recontextualizations. in particular, the text (1) has a different topic with respect to text (2) and (3), while the texts (2) and (3) are topic related. in trotr, we support the recognition of such a kind of recontextualizations by leveraging the notion of topic relatedness. trotr represents a new opportunity in natural language processing (nlp) and can be used to distinguish recontextualizations of any kind of text reuse (e.g., proverbs, @xcite , to investigate phenomena such as the use of misquotations @xcite and dogwhistles @xcite , as well as to provide in-context interpretation to vague utterances, with special focus on enhancing the large language models (llms)' capabilities to this end @xcite . ## related work works related to trotr are about text reuse and recontextualization, semantic textual similarity and relatedness, and topic modeling and annotation. text reuse and recontextualization. although multiple facets of text reuse have been investigated, such as historical @xcite , cross-lingual @xcite , allusive @xcite , explicit @xcite , non-literal @xcite , and local @xcite , computational approaches primarily focuses on detecting instances of text reuse. to the best of our knowledge, studies extending beyond mere trd often leverage text metadata to analyze reuse within temporal and spatial graphs @xcite @xcite . however, these studies do not specifically focus on capturing how the reused text is recontextualized, thereby leaving a gap in the current literature. among recent advancements in nlp, some works are related to the recontextualization of text. @xcite focus on narrative analysis by investigating how the recontextualization of events across whole stories impacts word embeddings. @xcite introduce a benchmark for evaluating the llms' capability of generating proverbs in-context of narratives. over the past few years, there has been growing interest in quotations, i.e. "well known phrases or sentences that we use for various purposes such as emphasis, elaboration, and humor" @xcite . this interest extends to various forms of quotations spanning from epigraphs @xcite to biblical references @xcite . in particular, there has been a surge of attention in recommendation systems that offers off-the-shelf quotations based on provided context @xcite @xcite . ## the trotr framework the trotr framework consists of two tasks, called text reuse in-context (tric) and topic variation ranking across corpus (trac), along with a corresponding annotation process (see figure 1 ). tric and trac are grounded on human judgments of a specific facet of semantic relatedness that considers the extent to which two texts share a common topic. we call this facet topic relatedness (see when dealing with complex problems, such as recontextualization, a general approach involves starting with a smaller sub-problem to establish a focused foundation before further expanding. thus, we first present tric as a context-pair level task. then, we present trac as a more complex corpuslevel task that must be addressed to identify potential varying targets for real, in-depth analysis. ## evaluation setup we use the trotr tasks and benchmarks to evaluate the ability of sequence-level models to capture topic relatedness and variation in different text recontextualizations to set baselines for the tasks. because sentence-bert (sbert) models are recognized to be the state-of-the-art architecture for addressing sequence-level tasks @xcite , we choose a range of different sbert models tailored for sequence-level embeddings and textual similarity. ## evaluation results first, we evaluated an extensive set of pre-trained sbert models on the tric task (see as a general remark on our initial evaluation, we note that bi-encoder models consistently exhibit superior performance compared to cross-encoder models in both tric subtask 1 and subtask 2. this finding aligns with the recent comparisons by ishihara and shirai (2022) and @xcite for news article similarity and lsc, challenging the idea that the use of cross-attention benefits cross-encoder architectures in sequence-level tasks @xcite . in the following, we first present the results of our evaluation by comparing the use of pre-trained and fine-tuned models (+ft); then, we discuss the results in the masking settings (+mask, +ft+mask). we report in ## concluding remarks and future work to the best of our knowledge, this work represents a first pioneering effort in the computational modeling of recontextualization. we relied on the notion of topic relatedness to introduce a novel framework named topic relatedness of text reuse (trotr) with two tasks: text reuse in-context (tric) and topic variation ranking across corpus (trac). the tasks are inherently difficult as topic relatedness is under-defined, and under-researched, therefore this paper presents important steps forward. first, we presented a human annotated benchmark of text reuse instances extracted from twitter. this benchmark can be used to support linguistic recycling and reception studies, ranging from misuse and dog whistles to the study of author influence. using the framework, the benchmark can easily be extended in future work to cover more diverse sets of text reuse from other sources, e.g., literature and political text. next, we comprehensively evaluate sbert models on the tric and trac tasks. we find that the bi-encoder model outperform the cross-encoder models. additionally, we evaluate the considered models by masking the occurrences of text reuse and find that the models exhibit a greater sensitivity to semantic similarity rather than topic relatedness. these results now constitute a baseline for continued research and can be used as comparison for improved models and architectures. future work: text reuse is inherently diachronic and can take place both over short and long time spans. the trotr framework is applicable to address the recontextualization problem across time, space, or domain. in our ongoing work, we will extend the trotr benchmark by annotating historical text and explicitly modeling change in topical variation over time. this will allow us to track the evolution of a quote like to be or not to be where hamlet originally reflected on the struggles of existence and the fear of the unknown. over the centuries, the phrase has become deeply embedded in various languages and cultures, often improperly referenced, quoted, and parodied in diverse literary works, contexts, and topics @xcite .