In a number of question answering (QA) benchmarks, pretrained models have reached human parity through fine-tuning on an order of 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred training examples are available . We show that standard span selection models perform poorly, highlighting the fact that current pretraining objective are far removed from question answering. To address this, we propose a new pretraining scheme that is more suitable for extractive question answering . Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans are replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select the answer span. The resulting model obtains surprisingly good results on multiple benchmarks , e.g., 72.7 F1  with only 128 examples on SQuAD , while maintaining competitive (and sometimes better) performance in the high-resource setting . Our findings indicate that careful design of pretraining schemes and model architecture can have a dramatic effect on performance in the few-shot settings .
In several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order of 100,000 annotated questions and answers. We explore the more realistic few-shot setting, where only a few hundred training examples are available , and observe that standard models perform poorly, highlighting the discrepancy between current pretraining objectives and question answering. We propose a new pretraining scheme tailored for question answering: recurring span selection . Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span. Masked spans are replaced with a special token, viewed as a question representation, that is later used during fine-tuning to select the answer span. The resulting model obtains surprisingly good results on multiple benchmarks ( e.g., 72.7 F1 on SQuAD with only 128 training examples) , while maintaining competitive performance in the high-resource setting .
R
a number of question answering (QA)
several question answering
R
. We show that standard span selection
, and observe that standard
R
fact that current pretraining objective are far removed from
discrepancy between current pretraining objectives and
R
To address this, we
We
R
that is more suitable for extractive question answering
tailored for question answering: recurring span selection
R
,
(
A
on SQuAD
R
examples on SQuAD
training examples)
D
(and sometimes better)
D
. Our findings indicate that careful design of pretraining schemes and model architecture can have a dramatic effect on performance in the few-shot settings