Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering (2023)

Abstract: The integration of multi-document pre-training objectives into language models has resulted in remarkable improvements in multi-document downstream tasks. In this work, we propose extending this idea by pre-training a generic multi-document model from a novel cross-document question answering pre-training objective. To that end, given a set (or cluster) of topically-related documents, we systematically generate semantically-oriented questions from a salient sentence in one document and challenge the model, during pre-training, to answer these questions while “peeking” into other topically-related documents. In a similar manner, the model is also challenged to recover the sentence from which the question was generated, again while leveraging cross-document information. This novel multi-document QA formulation directs the model to better recover cross-text informational relations, and introduces a natural augmentation that artificially increases the pre-training data. Further, unlike prior multi-document models that focus on either classification or summarization tasks, our pre-training objective formulation enables the model to perform tasks that involve both short text generation (e.g., QA) and long text generation (e.g., summarization).Following this scheme, we pre-train our model - termed QAmden - and evaluate its performance across several multi-document tasks, including multi-document QA, summarization, and query-focused summarization, yielding improvements of up to 7%, and significantly outperforms zero-shot GPT-3.5 and GPT-4.

## introduction among recent nlp research, multi-document processing is gaining increasing attention, due to the need to handle and process an increasing amount of textual data and available documents online. a early solutions for multi-text processing were task-specific and used complex architectures that were difficult to generalize across different multidocument tasks @xcite @xcite . efficient lms @xcite recently demonstrated that by simply concatenating multiple documents into a single sequence, the transformer can offload the goal of identifying and connecting relevant information between the documents. recently, it was suggested that these long-context lms can be equipped with new pre-training objectives to enable them to process multiple documents more effectively @xcite @xcite . these pre-trained models demonstrated state-ofthe-art performance on a variety of multi-document downstream tasks, and outperformed underlying lms and task-specific architectures. such models are often pre-trained using a dataset where each instance is a set of related documents (e.g., news articles all discussing a specific event), which facilitates modeling of cross-text relationships. existing multi-document pre-training objectives involve unmasking tokens in a document @xcite , or generating a salient masked sentence @xcite , encouraging the model to recover missing information using other documents. while successful, these models are either limited to classification tasks @xcite or primarily designed for summarization @xcite . in this work, we propose a novel pre-training objective that supports both short and long text generation, resulting in a versatile and general multidocument language model. in particular, we hypothesize that using questions and answers involving multiple documents can encourage the model to better learn and incorporate both fine-grained information (by asking questions about core information units in a specific sentence) as well as coarsegrained cross-document relationships required to generate a long text such as a summary. we show that this approach holds not only for summarization, but for other multi-document downstream tasks as well. during the pre-training of existing multidocument language models, the goal is to unmask spans (for encoder-only models) or generate masked textual spans (for encoder-decoder models) under a multi-document context. to that end, multiple concatenated sequences of related documents are fed during pre-training, thus requiring a large number of sets of related documents for an effective pre-training phase @xcite . in a variety of existing multi-document benchmarks, such as multi-document summarization, only small to medium-scale document clusters are readily available. these are acquired either automatically with lexical similarity and retrieval @xcite or semi-automatically @xcite , but generally, this process requires a substantial amount of human effort for filtering instances and generating high quality corpora. by employing a novel multi-document question-answer generation procedure, we propose an effective method for expanding the multi-document pre-training corpora. our approach allows us to provide multiple views for every single cluster of documents, thereby artificially increasing the pretraining data size (in terms of number of instances) via augmentation. to expose the model to a variety of contexts and diversify the pre-training data, we propose to generate multiple pairs of questions and answers and condition them on a subset of the documents' cluster. we select a salient sentence in one held-out document and then employ a recent parser to generate a high-quality question-answer pair about one predicate in the selected sentence, using a systematic semantically-oriented approach @xcite . this new multi-document pre-training objective challenges the model to generate both the answer to the question as well as the salient sentence, while discarding the held-out document or parts of it (see figures 1 , 2 for illustration). this procedure exposes the model to a variety of contexts -a question and a different subset of the documents in the cluster per instance, in contrast to prior methods that provide only a single view of the cluster. our contributions are summarized below: ## related work long-context efficient text generation transformers @xcite extend earlier transformer models @xcite for processing long sequences, often using a sparse self-attention architecture. examples include the longformer encoder-decoder (led) @xcite , and longt5 @xcite . these models demonstrated that single-text approaches be can adapted to multi-document tasks by concatenat-ing multiple documents into a single sequence and processing them using their sparse attention patterns. they sparsify the full self-attention matrix of transformers by using a combination of a localized sliding window (called local attention), as well as a global attention pattern on a few specific input locations. led is build upon the bart model @xcite ) by using additional positional embeddings and global attention weights, and introduces the global attention mode that operates over pre-selected tokens. longt5 extends the t5 model @xcite ) by using a similar technique introduced in the etc and bigbird models @xcite , relieving the requirement to manually select global tokens by automatically globalizing the aggregated representations of groups of tokens. further strategies have been proposed for increasing these models' abilities in multi-document tasks. the cross-document language model (cdlm) @xcite suggested pretraining a longformer-encoder @xcite over sets of related documents, and showed superior performance results over several multidocument tasks. following this methodology, the authors of linkbert @xcite used a similar approach, but utilized wikipedia's hyperlinks in order to curate informative pairs of linked documents for lm pre-training. in order to adopt the multi-document pretraining approach for sequence-to-sequence tasks, primera @xcite , which is built on top of the longformer encoder-decoder model (led), selected salient sentences within clusters of related documents using a pyramid estimation approach, resembling the method presented for pre-training the single-document pegasus model @xcite . while this work is the closest to ours, it was pre-trained to generate masked salient sentences without any control, which makes the model potentially hallucinate while generating text, while our model uses a controlled qa-based objective. furthermore, unlike these works, our method generates significantly more data then used to pre-train primera, which is possible to obtain by the singledocument qa generation approach. our qa pretraining formulation allows us to generate multiple contexts per document cluster. another related line of work includes methods that incorporate large-scale qa-generated data for pre-training lms @xcite ; @xcite . these works hypothesize and show that pre-training by utilizing generated qa data can encourage contextual representations to encode useful semantic information for other non-qa downstream tasks. inspired by that, we conjecture that lms can strongly benefit from infusing qa during pre-training in the multi-document setup, for adding an additional signal for modelling cross-text relationships. ## augmenting the multi-document pre-training objective in this section, we provide the required steps for compiling the pre-training dataset for qamden. we next elaborate on the details of the data creation and provide analysis of the resulted corpus. recent works have shown that for text summarization, pre-training lms to generate a "summarylike" sequence, termed pseudo summary, inherently provides gains over general-purpose pre-trained lms (pegasus, primera; @xcite . the data in which the pegasus and primera models were pre-trained on was constructed using the gap sentence generation (gsg) method, which suggests masking highly-ranked salient sentences, where salience is pre-determined by a sentence-scoring method of interest. particularly, in pegasus, gsg has been adopted as its pre-training objective, where some sentences in a single document are masked in the input and the model is tasked to generate them. formally, for each sentence s i in a given input document d, pegasus computes its salience score based on its rouge score @xcite w.r.t the rest of the sentences within the document (d/{s i }), i.e. score(s i ) = rouge(s i , d/{s i }). intuitively, …pokemon sword and shield might have already been announced, but we now know there's another new pokemon game on the way from dena… qasem qa generation (klein et al., 2022) q1: what might been announced? a1: pokemon sword and shield. (answer length: 4) q3: who knows something? a: we. (answer length: 1) q2: where does someone know something? a: on the way from dena. (answer length: 5) contextualization (pyatkin et al., 2022) q: where did we know there's another new pokemon game? a: on the way from dena. ## experimental setup and results this section presents experiments conducted to evaluate qamden, as well as the the ablations and baselines we used. for the intrinsic evaluation we evaluated the models over multi-document qa tasks. for extrinsic evaluations we considered the multi-document abstractive summarization task. model implementation details following xiao et al. ( 2022 ), we use the large-sized longformer-encoder-decoder (led) @xcite for our model initialization. the length limits of input and output are 4096 and 1024, respectively. 3 following the huggingface implementation @xcite , we set the sliding window size to 1024 for local attention in the encoder part. similar to the primera model @xcite , when concatenating the documents and the question, we add a special document separator token (<doc-@xmath0. we also assign the global attention mode to these tokens which enables the model to share information across documents @xcite . for further hyperparameter and pre-training execution details, see app. b. ## conclusions in this work, we present a novel pre-training scheme for multi-document tasks. first, our approach suggests to augment the existing multidocument pre-training objectives into a crossdocument question answering task. second, we generate high-quality large-scale qa pre-training data using a controlled generation approach, in which each qa pair originates from a salient sentence in one of the documents in the set. during pre-training, we task the the longformer encoder-decoder (led) model to generate the answer and the salient sentence on the basis of the remaining context. this objective encourages the led model to elicit cross-document relationships, and stitch pieces of information across the input documents, which are relevant for performing multi-document tasks. the resulted model qamden shows significant performance improvements compared to prior models under extensive experimentation over multiple challenging multidocument summarization and qa datasets. future work can extend the ideas in this work for equipping decoder-only large lms with crossdocument modeling using our proposed method, also in the setup of in-context learning and prompt tuning. we foresee that our method should be significant specifically for retrieval-augmented language modeling setups @xcite , where there is a use of related documents as an outsourced external non-parametric knowledge source. finally, the use of a single document in order to trigger cross-document relationships, as firstly introduced in this work, might be further investigated.