Multilevel Text Alignment with Cross-Document Attention (2020)

Abstract: Text alignment finds application in tasks such as citation recommendation and plagiarism detection. Existing alignment methods operate at a single, predefined level and cannot learn to align texts at, for example, sentence and document levels. We propose a new learning approach that equips previously established hierarchical attention encoders for representing documents with a cross-document attention component, enabling structural comparisons across different levels (document-to-document and sentence-to-document). Our component is weakly supervised from document pairs and can align at multiple levels. Our evaluation on predicting document-to-document relationships and sentence-to-document relationships on the tasks of citation recommendation and plagiarism detection shows that our approach outperforms previously established hierarchical, attention encoders based on recurrent and transformer contextualization that are unaware of structural correspondence between documents.

## introduction aligning texts and understanding their relationships is a common problem for nlp tasks such as citation recommendation @xcite , comparable document mining @xcite @xcite , parallel sentence mining @xcite @xcite , plagiarism detection @xcite @xcite , paraphrase identification @xcite @xcite , and textual entailment @xcite @xcite . longer texts make the problem more challenging due to the potential complexity of the underlying correspondence. here, we develop a model to address this problem and demonstrate its applicability on three different tasks which require such understanding, namely on citation recommendation, citation localization, and plagiarism detection for general web documents. one key component of an nlp system for aligning documents is the encoding process. present approaches for comparing documents rely on hierarchically structured document encoders such as hierarchical attention networks (hans; @xcite , which independently represent the two documents as fixed-length vectors. the vectors are fed to a classifier which makes a decision on the relation between them @xcite . however, such methods do not provide insights about or leverage the underlying relationships across documents and are applicable only to a single, predefined level @xcite . importantly, when comparing documents, those methods ignore the structural cor-respondence between parts. figure 1 shows an example of predicting and localizing citations in scientific documents. to solve this problem in a cost-effective way, models need to be able to make joint predictions about these different tasks without relying on fine-grained annotations which are typically more expensive to obtain. in this paper, we propose a new approach for encoding documents that aligns document parts during the encoding process and is able to make predictions about their relationships across different levels (specifically, document-to-document and sentence-to-document). in particular, we equip a powerful class of models, namely hierarchical attention encoders @xcite @xcite with a cross-document attention component that "attends" to the structure of documents, enabling inferences about alignment of their parts (section 3). we introduce new benchmarks for joint document-to-document prediction and sentence-todocument localization of document relationships for citation recommendation and plagiarism detection (section 4). our experiments with variations find that cross-document attention is beneficial to strong baseline hierarchical encoders (section 5) on these challenging tasks. ## comparing documents many potential applications of natural language processing involve a comparative analysis of two (or more) documents. examples include: our experiments in section 5 will consider tasks inspired by the first two applications. note that, in each of these examples, the most useful analysis of the document-to-document relationship will include a more fine-grained analysis: which parts of the source document correspond to which parts of the target document? figure 1 illustrates an example for citation recommendation, in which the main relationship (does/should document a cite document b?) is actually composed of a number of more localized relationships between sentences in document a that contain citations and document b (or, perhaps, parts of document b). in general, whenever we seek to model relationships between documents, we believe that these relationships can be localized in one or both documents. we believe that automatic identification of these local correspondences is useful, both directly (e.g., mining parallel sentences for use in training a machine translation system), and for providing explanations (e.g., in plagiarism detection). of course, these fine-grained, localized correspondences are not typically directly observable in realistic datasets. here, we consider scenarios where positive and negative examples of documentlevel relationships are available for supervision, but fine-grained correspondences between their parts are not. we exploit simple decompositions of documents (into sentences and words) but follow earlier work @xcite in offering a general hierarchical model that could be extended to allow for additional levels in future work. the problem we aim to solve is: (i) given two documents (each decomposed, e.g., into sentences and words), automatically categorize whether a particular relationship holds between them, and (ii) which parts between them should be "aligned" in support of the relationship in (i). we will refer to these tasks respectively as document-to-document alignment (d2d) and sentence-to-document alignment (s2d), and will conduct experiments on tasks of both kinds in section 5. ## approach we next describe our solution to this problem, starting with a high-level overview (section 3.1). we build on a family of widely used models for document representation, known as hierarchical attention networks (hans; section 3.2), which is sensitive to predefined notions of hierarchy (here, sentences; @xcite . we augment the han with cross-document attention (section 3.3). ## a benchmark for document relation prediction and localization while many tasks and datasets focus on understanding the relationships between sentences or documents separately, to the best of our knowledge, there are no joint publicly available english benchmark for both d2d and s2d tasks. annotating document correspondences is expensive and time- pairs docs words sentences dataset count count avg std avg std aan 132k 13k 122.7 11.2 4.9 2.7 oc 300k 567k 190.4 16.3 7.0 3.5 s2orc 190k 270k 263.7 19.2 9.3 5.9 pan 34k 23k 1569.7 90.4 47.4 66.1 aan. contains computational linguistics papers published on acl anthology from 2001 to 2014, along with their metadata. for each paper, we extract its abstract and the abstracts of its citations and treat them as positive pairs without including full texts. for each positive pair's source paper, an (uncited, presumed irrelevant) negative paper is sampled at random to create a negative instance. since the dataset is not complete, we filter out pairs where either document lacks an abstract, but otherwise include all positive citation pairs. oc. contains about 7.1m papers in computer science and neuroscience. we follow a similar procedure to that for aan. here we only select one citation per source paper, for wider coverage. ## experiments our experiments are performed on the above benchmark to test the benefit of cross-document attention. we first evaluate our model on scientific document citation recommendation (section 5.2) followed by citation localization (section 5.3). then, we evaluate our model on web document plagiarism detection and localization (section 5.4). ## conclusion we augment hierarchical attention networks with cross-document attention, allowing their use in document-to-document and sentence-to-document alignment tasks. we introduce benchmarks, based on existing datasets, to evaluate model performance on such tasks. in controlled experiments, we observe a benefit from cross-document attention on three out of the four document-to-document tasks and two out of two sentence-to-document tasks. we provide more details about the datasets in the new benchmark, experimental setup, choices of hyperparameters, model design choices, and validation performances corresponding to the ones reported in the main paper. moreover, we provide qualitative examples that visualize the crossdocument attention component.