  In a number of question answering (QA) benchmarks, pretrained models have
reached human parity through fine-tuning on an order of 100,000 annotated
questions and answers. We explore the more realistic few-shot setting, where
only a few hundred training examples are available. We show that standard span
selection models perform poorly, highlighting the fact that current pretraining
objective are far removed from question answering. To address this, we propose
a new pretraining scheme that is more suitable for extractive question
answering. Given a passage with multiple sets of recurring spans, we mask in
each set all recurring spans but one, and ask the model to select the correct
span in the passage for each masked span. Masked spans are replaced with a
special token, viewed as a question representation, that is later used during
fine-tuning to select the answer span. The resulting model obtains surprisingly
good results on multiple benchmarks, e.g., 72.7 F1 with only 128 examples on
SQuAD, while maintaining competitive (and sometimes better) performance in the
high-resource setting. Our findings indicate that careful design of pretraining
schemes and model architecture can have a dramatic effect on performance in the
few-shot settings.
