CDLM : Cross-Document Language Modeling (2021)

Abstract: We introduce a new pretraining approach geared for multi-document language modeling, incorporating two key ideas into the masked language modeling self-supervised objective. First, instead of considering documents in isolation, we pretrain over sets of multiple related documents, encouraging the model to learn cross-document relationships. Second, we improve over recent long-range transformers by introducing dynamic global attention that has access to the entire input to predict masked tokens. We release CDLM (Cross-Document Language Model), a new general language model for multi-document setting that can be easily applied to downstream tasks. Our extensive analysis shows that both ideas are essential for the success of CDLM, and work in synergy to set new state-of-the-art results for several multi-text tasks.

## introduction the majority of nlp research addresses a single text, typically at the sentence or document level. yet, there are important applications which are concerned with aggregated information spread across multiple texts, such as cross-document coreference resolution @xcite , classifying relations between document pairs @xcite and multi-hop question answering @xcite . existing language models (lms) @xcite @xcite , which are pretrained with variants of the masked language modeling (mlm) self-supervised objective, are known to provide powerful representations for internal text structure @xcite , which were shown to be beneficial figure 1 : an example from multi-news @xcite . circled words represent matching events and the same color represents mention alignments. in this paper, we point out that beyond modeling internal text structure, multi-document tasks require also modeling cross-text relationships, particularly aligning or linking matching information elements across documents. for example, in fig. 1 , one would expect a competent model to correctly capture that the two event mentions suing and alleges, from documents 1 and 2, should be matched. accordingly, capturing such cross-text relationships, in addition to representing internal text structure, can prove useful for downstream multi-text tasks, as we demonstrate empirically later. following this intuition, we propose a new simple cross-document pretraining procedure, which is applied over sets of related documents, in which informative cross-text relationships are abundant (e.g. like those in fig. 1 ). under this setting, the model is encouraged to learn to consider and represent such relationships, since they provide useful signals when optimizing for the language modeling objective. for example, we may expect that it will be easier for a model to unmask the word alleges in document 2 if it would manage to effectively "peek" at document 2, by matching the masked position and its context with the corresponding information in the other document. naturally, considering cross-document context in pretraining, as well as in finetuning, requires a model that can process a fairly large amount of text. to that end, we leverage recent advances in developing efficient long-range transformers @xcite , which utilize a global attention mode to build representations based on the entire input. overcoming certain restrictions in prior utilization of global attention (see section 2.1), we introduce a dynamic attention pattern during pretraining, over all masked tokens, and later utilize it selectively in finetuning. combining pretraining over related documents along with our global attention pattern yields a novel pretraining approach, that is geared to learn and implicitly encode informative cross-document relationships. as our experiments demonstrate, the resulting model, termed cross-document language model (cdlm), can be generically applied to downstream multi-document tasks, eliminating the need for task-specific architectures. we show empirically that our model improves consistently over previous approaches in several tasks, including cross-document coreference resolution, multihop question answering, and document matching tasks. moreover, we provide controlled experiments to ablate the two contributions of pretraining over related documents as well as new dynamic global attention. finally, we provide additional analyses that shed light on the advantageous behavior of our cdlm. our contributions are summarized below: ## evaluations and results this section presents experiments conducted to evaluate our cdlm, as well as the the ablations and baselines we used. for the intrinsic evaluation we measured the perplexity of the models. for extrinsic evaluations we considered event and entity cross-document coreference resolution, paper citation recommendation, document plagiarism detection, and multihop question answering. we also conducted an attention analysis, showing that our cdlm indeed captured cross-document and longrange relations during pretraining. 4 baseline lms recall that cdlm employs multiple related documents during pretraining, and assigns global attention to masked tokens. to systematically study the importance of these two components, we consider the following lm baselines: -longformer: the underlying longformer model, without additional pretraining. the data and pretraining hyperparameters used for the ablations above are the same as the ones used for our cdlm pretraining, except for the underlying longformer, which is not further pretrained, and the rand cdlm, that is fed with different document clusters (drawn from the same corpus). during all the experiments, the global attention weights used by the underlying longformer and by local cdlm are initialized to the values of their pretrained local attention weights. all the models above further finetune their global attention weights, depending on the downstream task. when finetuning cdlm and the above models on downstream tasks involving multiple documents, we truncate the longer inputs to the longformer's 4,096 token limit. ## related work recently, long-context language models @xcite introduced the idea of processing multi-document tasks using a single long-context sequence encoder. however, pretraining objectives in these models consider only single documents. here, we showed that additional gains can be obtained by mlm pretraining using multiple related documents as well as a new dynamic global attention pattern. processing and aggregating information from multiple documents has been also explored in the context of document retieval, aiming to extract information from a large set of documents @xcite @xcite . these works focus on retrieving relevant information from often a large collection of documents, by utilizing short-context lms, and then generate information of interest. cdlm instead provides an approach for improving the encoding and contextualizing information across multiple documents. as opposed to the mentioned works, our model utilizes long-context lm and can include broader contexts of more than a single document. the use of cross-document attention has been recently explored by the cross-document attention (cda) @xcite . cda specifically encodes two documents, using hierarchical attention networks, with the addition of cross attention between documents, and makes similarity decision between them. similarly, the recent dcs model @xcite suggested a cross-document finetuning scheme for unsuper-vised document-pair matching method (processing only two documents at once). our cdlm, by contrast, is a general pretrained language model that can be applied to a variety of multi-document downstream tasks, without restrictions on the number of input documents, as long as they fit the input length of the longformer. finally, our pretraining scheme is conceptually related to cross-encoder models that leverage simultaneously multiple related information sources. for example, the translation language model (tlm) @xcite encodes together sentences and their translation, while certain crossmodality encoders pretrain over images and texts in tandem (e.g., vilbert @xcite ). ## conclusion we presented a novel pretraining strategy and technique for cross-document language modeling, providing better encoding for cross-document (cd) downstream tasks. our contributions include the idea of leveraging clusters of related documents for pretraining, via cross-document masking, along with a new long-range attention pattern, together driving the model to learn to encode cd relationships. this was achieved by extending the global attention mechanism of the longformer model to apply already in pretraining, creating encodings that attend to long-range information across and within documents. our experiments assess that our crossdocument language model yields new state-of-theart results over several cd benchmarks, while, in fact, employing substantially smaller models. our analysis showed that cdlm implicitly learns to recover long-distance cd relations via the attention mechanism. we propose future research to extend this framework to train larger models, and to develop cross-document sequence-to-sequence models, which would support cd tasks that involve a generation phase.