F ewshot QA : A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models (2021)

Abstract: The task of learning from only a few examples (called a few-shot setting) is of key importance and relevance to a real-world setting. For question answering (QA), the current state-of-the-art pre-trained models typically need fine-tuning on tens of thousands of examples to obtain good results. Their performance degrades significantly in a few-shot setting (< 100 examples). To address this, we propose a simple fine-tuning framework that leverages pre-trained text-to-text models and is directly aligned with their pre-training framework. Specifically, we construct the input as a concatenation of the question, a mask token representing the answer span and a context. Given this input, the model is fine-tuned using the same objective as that of its pre-training objective. Through experimental studies on various few-shot configurations, we show that this formulation leads to significant gains on multiple QA benchmarks (an absolute gain of 34.2 F1 points on average when there are only 16 training examples). The gains extend further when used with larger models (Eg:- 72.3 F1 on SQuAD using BART-large with only 32 examples) and translate well to a multilingual setting . On the multilingual TydiQA benchmark, our model outperforms the XLM-Roberta-large by an absolute margin of upto 40 F1 points and an average of 33 F1 points in a few-shot setting (<= 64 training examples). We conduct detailed ablation studies to analyze factors contributing to these gains.

## introduction the task of question answering (qa) in natural language processing typically involves producing an answer for a given question using a context that contains evidence to support the answer. the latest advances in pre-trained language models resulted in performance close to (and sometimes exceeding) a human performance when fine-tuned on several qa benchmarks @xcite , @xcite , @xcite , @xcite . however, to achieve this result, these models need to be fine-tuned on tens of thousands of examples. in a more realistic and practical scenario, where only a handful of annotated training examples are available, their performance degrades significantly. for instance, @xcite show that, when only 16 training examples are available, the robertabase @xcite and spanbert-base @xcite obtain a f1 score of 7.7 and 18.2 respectively on squad @xcite . this is far lower than the f1 score of 90.3 and 92.0 when using the full training set @xmath0. through experimental analysis, we observe that this degradation is majorly attributed to the disparities between fine-tuning and pre-training frameworks (a combination of the input-output design and the training objective). and to address this, we propose a fine-tuning framework (referred to as fewshotqa hereby) that is directly aligned with the pre-training framework, in terms of both the input-output design and the training objective. specifically, we construct the input as a concatenation of the question, a mask token and context (in that order) and fine-tune a text-to-text pre-trained model using the same objective used during its pre-training to recover the answer. these text-totext pre-trained model(s) were originally trained to recover missing spans of text in a given input sequence. and since our proposed fine-tuning setup is very much identical to the pre-training setup, this enables the model to make the best use of the pre-training "knowledge" for the fine-tuning task of question answering. the effectiveness of our fewshotqa system is shown in its strong results (an absolute average gain of 34.2 f1 points) on multiple qa benchmarks in a few-shot setting. we show that the gains extend further when used with larger sized models. we also test fewshotqa on a multilingual benchmark by replacing the pre-trained model with its multi-bert* bart t5 ## few-shot fine-tuning framework design our proposed few-shot fine-tuning framework design involves a different choice of input-output design and the training objective than the current standard for qa fine-tuning frameworks. we provide a motivation for this design by comparison with the existing frameworks. figure 1 illustrates this in detail. the pre-training framework is also pictured for comparison. note that we focus on the bi-directional masked language models (mlms) instead of the auto-regressive language models (such as gpt-2 @xcite ) as the mlms typically are deemed superior for qa tasks @xcite , @xcite . ## conclusion we present an effective few-shot question answering (qa) system that combines the use of pre-trained text-to-text models and a fine-tuning framework aligned with their pre-training counterpart. through experimental studies on various qa benchmarks and few-shot configurations, we show that this system can produce significant gains including in scenarios where the training data is extremely scarce (an absolute gain of 34 f1 points on average in comparison to the current standard of the fine-tuning framework). we also present extensions to multilingual and larger model settings and show that the gains translate well to these settings (eg:-up to an absolute 40 f1 point gain in comparison to xlm-roberta + a span-selection objective). through ablation studies, we study the impact of model size, fine-tuning objectives, inputoutput design and illustrate the factors leading to such strong gains. for future, as our framework doesn't explicitly enforce the answer to be a span in the input text, it'd be interesting to consider its applications to generative qa tasks.