Recovering Lexically and Semantically Reused Texts (2021)

Abstract: Writers often repurpose material from existing texts when composing new documents. Because most documents have more than one source, we cannot trace these connections using only models of document-level similarity. Instead, this paper considers methods for local text reuse detection (LTRD), detecting localized regions of lexically or semantically similar text embedded in otherwise unrelated material. In extensive experiments, we study the relative performance of four classes of neural and bag-of-words models on three LTRD tasks – detecting plagiarism, modeling journalists’ use of press releases, and identifying scientists’ citation of earlier papers. We conduct evaluations on three existing datasets and a new, publicly-available citation localization dataset. Our findings shed light on a number of previously-unexplored questions in the study of LTRD, including the importance of incorporating document-level context for predictions, the applicability of of-the-shelf neural models pretrained on “general” semantic textual similarity tasks such as paraphrase detection, and the trade-offs between more efficient bag-of-words and feature-based neural models and slower pairwise neural models.

## introduction when composing documents in many genresfrom news reports, to scientific papers, to political speeches-authors obtain ideas and inspiration from source documents and present them in the form of direct copies, quotations, summaries, or paraphrases. in the simplest case, e.g. in congressional bills, writers include text from earlier versions of the same document along with new material @xcite . in news media, journalists often paraphrase or quote speeches, press releases, and interviews @xcite . in academia, citations of papers usually appear along with summaries of their contributions @xcite . these are instances of lexical and semantic local text reuse, where both source and target documents contain lexically or semantically similar passages, surrounded by text that is unrelated or dissimilar. often, reused text is presented without explicit links or citations, making it hard to track information flow. while many state-of-the-art (sota) nlp architectures have been trained on the closely-related tasks of document-and sentence-pair similarity detection @xcite and ad-hoc retrieval @xcite , prior methods for local text-reuse detection (ltrd) are mostly limited to lexical matching @xcite @xcite @xcite with some dictionary expansion @xcite . to our knowledge, only @xcite has applied neural models to this problem, proposing hierarchical neural models that use a cross-document attention mechanism to model local similarities between two candidate documents. in this paper, we conduct a large-scale evaluation of several lexical overlap and sota neural models for ltrd. among the neural models, we benchmark not only the hierarchical neural models proposed by @xcite , but also study the effectiveness of three classes of models not yet applied to ltrd: 1) bert-based @xcite passage encoders trained on generic paraphrase, semantic textual similarity, and ir data @xcite ; 2) feature-based bert models with direct sentence-level supervision; and 3) finetuned bert-based models for sequence-pair tasks. we conduct evaluations on four datasets, including 1) pan and s2orc @xcite , benchmark ltrd datasets for plagiarism detection and citation localization; 2) pr2news @xcite , a dataset of text reuse in news articles labeled with a mix of expert, non-expert, and heuristic annotation; 3) arc-sim, a new, publicly available 1 citation localization dataset created using citation links in the acl arc @xcite . our experiments address a number of previouslyunexplored questions in the study of ltrd, including 1) the impact of training on weakly-supervised data on model accuracy; 2) the effectiveness of sota neural models trained on "general" semantic similarity data for ltrd tasks; 3) the importance of incorporating document-level context; 4) the effects of domain-adaptive pretraining @xcite on the accuracy of fine-tuned bert models; and 5) the trade-offs between more efficient lexical overlap and feature-based neural models and slower pairwise neural models. our work builds on the recent efforts of @xcite , who demonstrate the efficacy of hierarchical neural models in detecting instances of non-literal reuse where authors paraphrase, summarize, and heavily edit source content. however, as discussed in §1, we conduct a much larger set of experiments beyond those of @xcite . in addition to the hierarchical neural models with document-level supervision proposed by @xcite , we evaluate four sets of models: lexical overlap models, sota neural models trained for general paraphrase detection, hierarchical neural models with sentence-level supervision, and finetuned sequence-pair bert models. further, in 1 https://github.com/maclaughlin/arc-sim addition to evaluating models on the benchmark ltrd datasets introduced by @xcite , we conduct experiments on two more challenging datasets: arc-sim, a new citation localization dataset with hard negative examples, and pr2news @xcite , a dataset of text reuse in science news articles with heuristically-labeled training data. also related to our work is research studying sentence-pair problems, e.g. paraphrase detection (pd) @xcite , semantic textual similarity (sts) @xcite and textual entailment, @xcite , and documentranking problems, e.g. ad-hoc retrieval @xcite . in fact, it is trivial to adapt existing approaches to sentence-pair and document ranking problems to ltrd. as discussed in §3, we cast ltrd as sentence classification and ranking, identifying which sentences in a target text are lexically or semantically reused from some portion of the source. thus, in order to adapt sentence-pair models to this task, we simply compute scores for all pairs of (source sentence, target sentence), and use some function (e.g. max) to aggregate the scores for each target sentence. similarly, one can adapt existing ad-hoc retrieval approaches by treating each target sentence as a query and computing a score with the corresponding source. these approaches, however, may suffer from a lack of contextualization and/or efficiency issues. sentencepair models that encode each source and target sentence separately, while efficient, might miss important contextualizing information in surrounding sentences. similarly, neural ir models that process each target sentence as a separate query do not contextualize target sentences and also require a computationally-expensive forward pass for each query. we study the importance and impact of these limitations in our work, testing the effectiveness of multiple sota bert-based architectures for sequence-pair similarity and ranking. ## problem definition following @xcite , we define ltrd as two tasks: document-to-document (d2d) alignment and sentence-to-document (s2d) alignment. in d2d, for a given pair of documents (source document s, target document t), we aim to predict whether t reuses content from s. thus, each pair has a corresponding binary label of 1 if t reuses content, else 0. note, this is different than evaluating the similarity of the two documents as a whole, since, in this setting, only a small portion of t is adapted from s, and most of it is possibly unrelated. in s2d, given an (s, t) pair, we aim to predict which specific sentences t @xmath0. thus, each pair has n corresponding labels, one label for each sentence t @xmath1. foot_0 ## models we benchmark four classes of models on this task: ## datasets we benchmark the proposed models on four different datasets ( ## results & discussion as seen in tables 2 & 3 , bert-based models finetuned on ltrd data perform the best in general, outperforming lexical overlap, sbert, and hnm. overall, models achieve their best performances on pan. we suspect that this is because many positive (s, t) pairs are easy, containing many plagiarized passages with high lexical overlap, and since many negative ( s, t) pairs are topically unrelated and share little lexical or semantic overlap. on the other end of the spectrum is arc-sim, where models score relatively poorly. we hypothesize that this is because most t only contain one citing sentence and since, as discussed in §5.4, we focus on selecting hard negative target texts, t, sampled from the same document as the original t. ## conclusion we study methods for local text reuse detection, identifying passages in a target document that lexically or semantically reuse content from a source. through evaluations on four datasets, including a new citation localization dataset, we confirm the strong performance of bert models fine-tuned on our task. however, we also find that lexical-overlap methods, e.g. tfidf, provide strong baselines, frequently outperforming off-the-shelf neural passage encoders and hierarchical neural models. based on these findings, we suggest practitioners take one of two approaches: 1) in instances with little labeled training data or where most reuse is exact (i.e. copying), use traditional lexical overlap models; 2) in instances with large-scale labeled training data and where much of the reuse is nonliteral (e.g. summarization, paraphrasing), use a lexical overlap method to filter possible (s, t) pairs, then run a more expensive fine-tuned bert on that subset. we suggest users opt for fine-tuned bert models over pretrained passage encoders (sbert) or hnms for this second step since they achieve substantially higher performance. suggestion #2 follows current approaches to neural ir, where neural models only rerank smaller lists of documents retrieved by a cheaper lexical overlap method, e.g. tf-idf. performance may be further boosted by fine-tuning bert-based models that incorporate document-level context (i.e. longformer) or ones that are adapted to the target domain of interest (i.e. dapt), but often the standard roberta base achieves highly competitive results. 197 36 , 227 5 , 269 3 , 852 5 , 665 4 , 171 ment, next 45 sentences of target document), and so on. predictions for split examples are merged back together at test time.