Few-Shot Question Answering by Pretraining Span Selection (2021)

Abstract: In several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order of 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred training examples are available, and observe that standard models perform poorly, highlighting the discrepancy between current pretraining objectives and question answering. We propose a new pretraining scheme tailored for question answering: recurring span selection. Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans are replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select the answer span. The resulting model obtains surprisingly good results on multiple benchmarks (e.g., 72.7 F1 on SQuAD with only 128 training examples), while maintaining competitive performance in the high-resource setting.

## introduction the standard approach to question answering is to pretrain a masked language model on raw text, and then fine-tune it with a span selection layer on top @xcite @xcite . while this approach is effective, and sometimes exceeds human performance, its success is based on the assumption that large quantities of annotated question answering examples are available. for instance, both squad @xcite and natural questions @xcite contain an order of 100,000 question and answer pairs in their training data. this assumption quickly becomes unrealistic as we venture outside the lab conditions of english wikipedia, and attempt to crowdsource question-answer pairs in other languages or domains of expertise @xcite . how do question answering models fare in the more practical case, where an in-house annotation effort can only produce a couple hundred training examples? we investigate the task of few-shot question answering by sampling small training sets from existing question answering benchmarks. despite the use of pretrained models, the standard approach yields poor results when fine-tuning on few examples (figure 1 ). for example, roberta-base finetuned on 128 question-answer pairs from squad obtains around 40 f1. this is somewhat expected, since the pretraining objective is quite different from the fine-tuning task. while masked language modeling requires mainly local context around the masked token, question answering needs to align the question with the global context of the pas-figure 2 : an example paragraph before (a) and after (b) masking recurring spans. each color represents a different cluster of spans. after masking recurring spans (replacing each with a single @xcite token), only one span from each cluster remains unmasked, and is considered the correct answer to the masked spans in the cluster. the pretraining task is to predict the correct answer for each @xcite . we introduce splinter (span-level pointer), a pretrained model for few-shot question answering. the challenge in defining such a self-supervised task is how to create question-answer pairs from unlabeled data. our key observation is that one can leverage recurring spans: n-grams, such as named entities, which tend to occur multiple times in a given passage (e.g., "roosevelt" in figure 2 ). we emulate question answering by masking all but one instance of each recurring span with a special @xcite token, and asking the model to select the correct span for each such token. to select an answer span for each @xcite token in parallel, we introduce a question-aware span selection (qass) layer, which uses the [question] token's representation to select the answer span. the qass layer seamlessly integrates with fine-tuning on real question-answer pairs. we simply append the @xcite token to the input question, and use the qass layer to select the answer span (figure 3 ). this is unlike existing models for span selection, which do not include an explicit question representation. the compatibility between pretraining and fine-tuning makes splinter an effective few-shot learner. splinter exhibits surprisingly high performance given only a few training examples throughout a va-riety of benchmarks from the mrqa 2019 shared task @xcite . for example, splinter-base achieves 72.7 f1 on squad with only 128 examples, outperforming all baselines by a very wide margin. an ablation study shows that the pretraining method and the qass layer itself (even without pretraining) both contribute to improved performance. analysis indicates that splinter's representations change significantly less during fine-tuning compared to the baselines, suggesting that our pretraining is more adequate for question answering. overall, our results highlight the importance of designing objectives and architectures in the few-shot setting, where an appropriate inductive bias can lead to dramatic performance improvements. ## background extractive question answering is a common task in nlp, where the goal is to select a contiguous span a from a given text t that answers a question q. this format was popularized by squad @xcite , and has since been adopted by several datasets in various domains @xcite and languages @xcite , with some extensions allowing for unanswerable questions @xcite ) or multiple answer spans @xcite . in this work, we follow the assumptions in the recent mrqa 2019 shared task @xcite and focus on questions whose answer is a single span. the standard approach uses a pretrained encoder, such as bert @xcite , and adds two parameter vectors s, e to the pretrained model in order to detect the start position s and end position e of the answer span a, respectively. the input text t and question q are concatenated and fed into the encoder, producing a contextualized token representation x i for each token in the sequence. to predict the start position of the answer span, a probability distribution is induced over the entire sequence by computing the inner product of a learned vector s with every token representation (the end position is computed similarly using a vector e): the parameters s, e are trained during fine-tuning, using the cross-entropy loss with the start and end positions of the gold answer span. this approach assumes that each token representation x i is contextualized with respect to the question. however, the masked language modeling objective does not necessarily encourage this form of long-range contextualization in the pretrained model, since many of the masked tokens can be resolved from local cues. fine-tuning the attention patterns of pretrained masked language models may thus entail an extensive learning effort, difficult to achieve with only a handful of training examples. we overcome this issue by (1) pretraining directly for span selection, and (2) explicitly representing the question with a single vector, used to detect the answer in the input text. ## splinter we formulate a new task for pretraining question answering from unlabeled text: recurring span selection. we replace spans that appear multiple times in the given text with a special [question] token, except for one occurrence, which acts as the "answer" span for each (masked) cloze-style "question". the prediction layer is a modification of the standard span selection layer, which replaces the static start and end parameter vectors, s and e, with dynamically-computed boundary detectors based on the contextualized representation of each [question] token. we reuse this architecture when fine-tuning on questionanswer pairs by adding a [question] token at the end of the actual question, thus aligning the pretraining objective with the fine-tuning task. we refer to our pretrained model as splinter. ## a few-shot qa benchmark to evaluate how pretrained models work when only a small amount of labeled data is available for finetuning, we simulate various low-data scenarios by sampling subsets of training examples from larger datasets. we use a subset of the mrqa 2019 shared task @xcite , which contains extractive question answering datasets in a unified format, where the answer is a single span in the given text passage. split i of the mrqa shared task contains 6 large question answering datasets: squad @xcite , newsqa @xcite , triv-iaqa @xcite , searchqa @xcite , hotpotqa @xcite , and natural questions @xcite . for each dataset, we sample smaller training datasets from the original training set with sizes changing on a logarithmic scale, from 16 to 1,024 examples. to reduce variance, for each training set size, we sample 5 training sets using different random seeds and report average performance across training sets. we also experiment with fine-tuning the models on the full training sets. since split i of the mrqa shared task does not contain test sets, we evaluate using the official development sets as our test sets. we also select two datasets from split ii of the mrqa shared task that were annotated by domain experts: bioasq @xcite and textbookqa @xcite . each of these datasets only has a development set that is publicly available in mrqa, containing about 1,500 examples. for each dataset, we sample 400 examples for evaluation (test set), and follow the same protocol we used for large datasets to sample training sets of 16 to 1,024 examples from the remaining data. to maintain the few-shot setting, every dataset in our benchmark has well-defined training and test sets. to tune hyperparameters, one needs to extract validation data from each training set. for simplicity, we do not perform hyperparameter tuning or model selection (see section 5), and thus use all of the available few-shot data for training. ## experimental setup we describe our experimental setup in detail, including all models and baselines. ## results our experiments show that splinter dramatically improves performance in the challenging few-shot setting, unlocking the ability to train question answering models with only hundreds of examples. when trained on large datasets with an order of 100,000 examples, splinter is competitive with (and often better than) the baselines. ablation studies demonstrate the contributions of both recurring span selection pretraining and the qass layer. ## related work the remarkable results of @xcite have inspired a renewed interest in few-shot learning. while some work focuses on classification tasks @xcite , our work investigates few-shot learning in the context of extractive question answering. one approach to this problem is to create synthetic text-question-answer examples. both @xcite and @xcite use the traditional nlp pipeline to select noun phrases and named entities in wikipedia paragraphs as potential answers, which are then masked from the context to create pseudo-questions. @xcite use methods from unsupervised machine translation to translate the pseudo-questions into real ones, while @xcite keep the pseudo-questions but use information retrieval to find new text passages that can answer them. both works assume access to language-and domain-specific nlp tools such as part-of-speech taggers, syntactic parsers, and named-entity recognizers, which might not always be available. our work deviates from this approach by exploiting the natural phenomenon of recurring spans in order to generate multiple question-answer pairs per text passage, without assuming any language-or domain-specific models or resources are available beyond plain text. similar ideas to recurring span selection were used for creating synthetic coreference resolution examples @xcite , which mask single words that occur multiple times in the same context. @xcite combines this approach with a copy mechanism for predicting the masked word during pretraining, alongside the masked language modeling objective. unlike our approach, which was designed to align well with span selection, corefbert masks only single-word nouns (rather than arbitrary spans) and replaces each token in the word with a separate mask token (rather than a single mask for the entire multi-token word). therefore, it does not emulate extractive question answering. we did not add corefbert as a baseline since the performance of both corefbert-base and corefbert-large was lower than spanbertbase's performance on the full-data mrqa benchmark, and pretraining corefbert from scratch was beyond our available computational resources. ## conclusion we explore the few-shot setting of extractive question answering, and demonstrate that existing methods, based on fine-tuning large pretrained language models, fail in this setup. we propose a new pretraining scheme and architecture for span selection that lead to dramatic improvements, reaching surprisingly good results even when only an order of a hundred examples are available. our work shows that choices that are often deemed unimportant when enough data is available, again become crucial in the few-shot setting, opening the door to new methods that take advantage of prior knowledge on the downstream task during model development. for diverse, explainable multi-hop question answering. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369-2380, brussels, belgium. association for computational linguistics. deming ye, yankai lin, jiaju du, zhenghao liu, peng li, maosong sun, and zhiyuan liu. 2020. coreferential reasoning learning for language representation. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 7170-7186, online. association for computational linguistics. tianyi zhang, felix wu, arzoo katiyar, kilian q. weinberger, and yoav artzi. 2021. revisiting fewsample bert fine-tuning. in iclr. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in the ieee international conference on computer vision (iccv).