  We introduce a new pretraining approach for language models that are geared
to support multi-document NLP tasks. Our cross-document language model (CD-LM)
improves masked language modeling for these tasks with two key ideas. First, we
pretrain with multiple related documents in a single input, via cross-document
masking, which encourages the model to learn cross-document and long-range
relationships. Second, extending the recent Longformer model, we pretrain with
long contexts of several thousand tokens and introduce a new attention pattern
that uses sequence-level global attention to predict masked tokens, while
retaining the familiar local attention elsewhere. We show that our CD-LM sets
new state-of-the-art results for several multi-text tasks, including
cross-document event and entity coreference resolution, paper citation
recommendation, and documents plagiarism detection, while using a significantly
reduced number of training parameters relative to prior works.
