Span Selection Pre-training for Question Answering (2020)

Abstract: BERT (Bidirectional Encoder Representations from Transformers) and related pre-trained Transformers have provided large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA). BERT is pretrained on two auxiliary tasks: Masked Language Model and Next Sentence Prediction. In this paper we introduce a new pre-training task inspired by reading comprehension to better align the pre-training from memorization to understanding. Span Selection PreTraining (SSPT) poses cloze-like training instances, but rather than draw the answer from the model’s parameters, it is selected from a relevant passage. We find significant and consistent improvements over both BERT-BASE and BERT-LARGE on multiple Machine Reading Comprehension (MRC) datasets. Specifically, our proposed model has strong empirical evidence as it obtains SOTA results on Natural Questions, a new benchmark MRC dataset, outperforming BERT-LARGE by 3 F1 points on short answer prediction. We also show significant impact in HotpotQA, improving answer prediction F1 by 4 points and supporting fact prediction F1 by 1 point and outperforming the previous best system. Moreover, we show that our pre-training approach is particularly effective when training data is limited, improving the learning curve by a large amount.

## introduction state-of-the-art approaches for nlp tasks are based on language models that are pre-trained on tasks which do not require labeled data @xcite @xcite @xcite . fine tuning language models to downstream tasks, such as question answering or other natural language understanding tasks, has been shown to be a general and effective strategy. bert is a recently introduced and highly successful model for language understanding. the general bert adaptation approach is to alter the model used for pre-training while retaining the transformer encoder layers. the model discards the layers used for the final prediction in the pretraining tasks and adds layers to predict the target task. all parameters are then fine tuned on the target task. bert is based on the transformer architecture @xcite , and trained on the following two unsupervised tasks: the masked lm or "cloze" task @xcite and next sentence prediction are auxiliary tasks @xcite requiring language understanding, and therefore train the model to acquire effective representations of language. however, the cloze pre-training task often poses instances that require only shallow prediction, or else require memorized knowledge. for many cloze instances the model simply requires syntactic or lexical understanding to answer. for example, in the cloze instances in other language models face the same challenge. in gpt-2 @xcite the entities present in a language generation prompt are expanded with type cloze syntactic in the 15th century, the blast furnace spread into what is now belgium where it was improved. lexical akebia quinata grows to 10 m (30 ft) or more in height and has compound leaves with five leaflets. collocation apollo 11 was launched by a saturn v rocket from kennedy space center on merritt island, florida memorized knowledge hadrian died the same year at baiae , and antoninus had him deified, despite opposition from the senate. as the preceding examples illustrate, in many cloze and conventional language model prediction instances, the correct prediction depends on a specific, narrowly relevant, bit of knowledge. further, pre-trained transformer models do indeed encode a substantial number of specific facts in their parameter matrices, enabling them to answer questions directly from the model itself @xcite . however, because the computational cost of transformers scales at least linearly with the number of parameters, it is expensive to encode all the facts that would enable the correct predictions. encoding a large amount of rarely useful information in parameters that are used for every instance is an inefficient use of model capacity if it is not needed for the downstream task. as the performance gains from gpt to gpt-2 and bert base to bert large show, increasing model capacity continues to provide gains. previous work also found seemingly limitless improvements from increasing model capacity @xcite , possible through sparse activation. our hypothesis is that making more efficient use of a fixed number of parameters can provide analogous gains. in mrc tasks, the model does not need to generate an answer it has encoded in its parameters. instead, the task is to use a retrieved passage, or passage set to extract an answer to the question. to better align the pre-training with the needs of the mrc task, we use span selection as an additional auxiliary task. this task is similar to the cloze task, but is designed to have a fewer simple instances requiring only syntactic or collocation understanding. for cloze instances that require specific knowledge, rather than training the model to encode this knowledge in its parameterization, we provide a relevant and answer-bearing passage paired with the cloze instance. we provide an extensive evaluation of the span selection pre-training method across four reading comprehension tasks: the stanford question answering dataset (squad) in both version 1.1 and 2.0; followed by the google natural questions dataset @xcite and a multihop question answering dataset, hotpotqa @xcite . we report consistent improvements over both bert base and bert large models in all reading comprehension benchmarks. the rest of the paper is structured as follows. in section 2 we describe earlier work on similar tasks and relate our extended pre-training to the broader research efforts on pre-training transformers. to provide context for our contribution, we review the most relevant parts of bert in section 3. next, we describe and formalize our pre-training task and the architectural adjustments to bert in section 4. finally we provide an extensive empirical evaluation in mrc tasks, describing benchmarks in section 5 and evaluating our approach in section 6. section 7 concludes the paper highlighting interesting research directiond for future work. ## related work since the development of bert there have been many efforts towards adding or modifying the pretraining tasks. @xcite introduced span-bert, a task that predicts the tokens in a span from the boundary token representations. note that, unlike span selection, there is no relevant passage used to select an answer span. ernie 2.0 @xcite trained a transformer language model with seven different pre-training tasks, including a variant of masked language model and a generalization of next-sentence prediction. xlnet @xcite introduced the permuted language model task, although it is not clear whether the success of the model is due to the innovative pre-training or larger quantity of pre-training. in this paper we focus on a pre-training task that has been specifically designed to support qa applications. previous related work has explored tasks similar to span selection pre-training. these are typically cast as approaches to augment the training data for question answering systems, rather than alleviating the pressure to encode specific facts in the pre-training of a language model. @xcite introduces a reading comprehension task constructed automatically from news articles with summaries. in this view the constructed dataset is used both for training and test. also, entities were replaced with anonymized markers to limit the influence of world knowledge. unlike our span selection pre-training task, this requires summaries paired with articles and focuses only on entities. a similar approach was taken in @xcite to augment training data for question answering. wikipedia articles were divided into introduction and body with sentences from the introduction used to construct queries for the body passage. phrases and entities are used as possible answer terms. @xcite constructed a question answering dataset where answers are always people. unlike other work, this did not use document structure but instead used a search index to retrieve a related passage for a given question. because the answers are always people, and there are only a few different people in each passage, the task is multiple choice rather than span selection. self training @xcite has also been used to jointly train to construct questions and generate self-supervised training data. bert was trained for one million batches, with 256 token sequences in each. although this is already a considerable amount of pre-training, recent research has shown continued improvement from additional pre-training data. xlnet @xcite used four times as much text, augmenting the wikipedia and bookscorpus @xcite with text from web crawls, the number of instances trained over was also increased by a factor of four. roberta @xcite enlarged the text corpus by a factor of ten and trained over fifteen times as many instances. this, along with careful tuning of the mlm task resulted in substantial gains. unfortunately, these very large-scale pre-training approaches require significant hardware resources. we restrict our experiments to extended pre-training with less than half the steps of bert (390k batches of 256). ## background in this section, we give the readers a brief overview of the bert @xcite pre-training strategy and some details which we modify for our novel span selection auxiliary task. ## span selection pre-training in the previous section we briefly discussed the bert framework along with its objective functions. in this section, we will propose a novel pre-training task for bi-directional language models called span selection. ## mrc tasks we follow previous work and evaluate our sspt architecture on several downstream tasks. our primary motivation is to improve question answering by improving the pre-trained language model. our qa benchmarks are the following: ## experiments tables 4, 5, and 6 show our results on the development set with extended span selection pre-training for bert relative to the pre-trained bert. we use the same hyperparameters on these tasks as the original bert. the best results for each dataset are in bold when significant relative to the bert baseline. the four question answering datasets are improved substantially with span selection pre-training. ## conclusion and future work span selection pre-training is effective in improving reading comprehension across four diverse datasets, including both generated and natural questions, and with provided contexts of passages, documents and even passage sets. this style of pretraining focuses the model on finding semantic connections between two sequences, and supports a style of cloze that can train deep semantic understanding without demanding memorization of specific knowledge in the model. the span selection task is suitable for pre-training on any domain, since it makes no assumptions about document structure or availability of summary/article pairs. this allows pre-training of language understanding models in a very generalizable way. in future work, we will address end-to-end question answering with pre-training for both the answer selection and retrieval components. we hope to progress to a model of general purpose language modeling that uses an indexed long term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.