PPDAC : A Plug-and -Play Data Augmentation Component for Few-shot Extractive Question Answering (2024)

Abstract: “Extractive Question Answering (EQA) in the few-shot learning scenario is one of the most chal-lenging tasks of Machine Reading Comprehension (MRC). Some previous works employ exter-nal knowledge for data augmentation to improve the performance of few-shot extractive ques-tion answering. However, there are not always available external knowledge or language- anddomain-specific NLP tools to deal with external knowledge such as part-of-speech taggers, syn-tactic parsers, and named-entity recognizers. In this paper, we present a novel Plug-and-PlayData Augmentation Component (PPDAC) for the few-shot extractive question answering, whichincludes a paraphrase generator and a paraphrase selector. Specifically, we generate multipleparaphrases of the question in the (question, passage, answer) triples using the paraphrase gener-ator and then obtain highly similar statements via paraphrase selector to form more training datafor fine-tuning. Extensive experiments on multiple EQA datasets show that our proposed plug-and-play data augmentation component significantly improves question-answering performance,and consistently outperforms state-of-the-art approaches in few-shot settings by a large margin.”

## introduction extractive question answering (eqa) presents a significant challenge within the domain of machine reading comprehension, requiring a nuanced understanding of both questions and passages to accurately identify the text fragments that serve as answers. high-performance eqa algorithms are essential for enabling retrieval systems to swiftly locate relevant passages and text fragments based on user queries. however, the reality is that not all scenarios offer a wealth of labeled samples, leading to diminished effectiveness of current eqa algorithms. this gap underscores the importance of investigating eqa algorithms within few-shot learning contexts, which are not only more aligned with real-world situations but also more feasible for practical implementation. the primary issue confronting existing eqa algorithms in few-shot scenarios is performance degradation due to a disconnect between the pre-training objectives of masked language modeling (mlm) and the fine-tuning objectives of downstream machine reading comprehension (mrc) tasks. while @xcite @xcite have attempted to bridge this gap by integrating downstream tasks into pre-training or by aligning the fine-tuning framework with the pre-training framework (known as the prompt-based fine-tuning paradigm, or prompt-tuning), these approaches often require extensive computational resources and overlook the potential of external knowledge bases for data augmentation in few-shot eqa. in response to these challenges, another avenue of research has focused on leveraging external knowledge to either generate training data or enhance the representation of prompt-tuning paradigm methods. specifically, @xcite generate context, question, and answer triples via the traditional nlp pipeline from the external knowledge base, which selects noun phrases and named entities in wikipedia paragraphs as potential answers, and then masks from the context to create pseudo-questions. moreover, @xcite proposed a novel framework named knowledge enhanced contrastive prompt-tuning, which enhances the representation of the question by the support of the rich semantics from the external knowledge base and the passage context. despite the potential of these approaches, they face significant hurdles, including limited availability of external knowledge or language-and domain-specific nlp tools, the risk of error accumulation in the nlp pipeline, and a scarcity of data augmentation methods specifically designed for few-shot eqa tasks. to address these issues, this paper introduces a novel plug-and-play data augmentation component (ppdac), aimed at generating paraphrases of questions within @xcite triples to enrich the training data. the ppdac comprises a paraphrase generator, a paraphrasing filter, and a paraphrase selector, which work together to transform the original question statements into multiple new statements with similar semantics but varied syntactic structures. this approach substantially increases the volume of training data available for fine-tuning. our experimental evaluation across various eqa baselines in few-shot scenarios reveals significant performance improvements, showcasing the effectiveness of ppdac in addressing the challenges of few-shot eqa without relying on external knowledge bases. in summary, our contributions can be summarized as follows: ## related work extractive question answering (eqa) methods based on pre-trained language models resulted in performance close to (and sometimes exceeding) a human performance when fine-tuned on several qa benchmarks @xcite @xcite . these methods need to be fine-tuned on tens of thousands of examples to obtain this result. however, only a small amount of annotated training data is available in a more realistic and practical scenario, which results in their performance degrading significantly. to address this issue, researchers started from two aspects: 1) reducing the difference between the pre-training objective of masked language modeling (mlm) and the fine-tuning objective of downstream mrc tasks; 2) utilizing external knowledge to generate the training data or enhancing the representation. to reduce the difference between the pre-training objective and the fine-tuning objective, some researchers construct new pre-training tasks specifically tailored to the eqa task, and other researchers propose a new fine-tuning framework that is directly aligned with the pre-training framework. for instance, @xcite designed a new pretraining scheme tailored for few-shot question answering, i.e. using a recurring span selection as the objective of pre-training models. @xcite presents a decoding algorithm on the fine-tuning phase that efficiently finds the most probable answer span in the passage. and @xcite explore a simple fine-tuning framework that leverages pretrained text-to-text models and is directly aligned with the pre-training framework, in terms of both the input-output design and the training objective. however, these methods may cost a lot of computational resources during the pre-training phase and ignore the external knowledge base for data augmentation in few-shot question answering. therefore, some researchers explore another branch of the few-shot eqa that studies the data augmentation methods to generate the training data or enhance the representation from the external knowledge base. @xcite exploit the traditional nlp pipeline to select noun phrases and named entities in wikipedia paragraphs as potential answers, and then mask these potential answers from the context to create pseudo-questions. the difference is that @xcite c c l 2 0 2 4 utilizes unsupervised machine translation methods to translate these pseudo-questions into real ones, while @xcite exploits information retrieval to find new text passages that can answer the pseudo-questions. moreover, @xcite presents a novel framework named knowledge enhanced contrastive prompt-tuning, which enhances the representation of the question by the support of the rich semantics from the external knowledge base and the passage context. however, the above studies assume access to external knowledge or language-and domain-specific nlp tools such as part-of-speech taggers, syntactic parsers, and named-entity recognizers, which may not always be available. ppdac decoder @xmath0:a defect in the human homologue of the drosophila "period" gene was identified as a cause of the sleep …… paraphrase question :what is another name for the fruit fly gene? passage:a defect in the human homologue of the drosophila "period" gene was identified as a cause of the sleep …… paraphrase question :what is the drosophila gene also called? n ' q 1 ' q passage:a defect in the human homologue of the drosophila "period" gene was identified as a cause of the sleep …… paraphrase question :what is another name for the fruit fly gene? passage:a defect in the human homologue of the drosophila "period" gene was identified as a cause of the sleep …… paraphrase question :what is the drosophila gene also called? n ' q 1 ' q passage:a defect in the human homologue of the drosophila "period" gene was identified as a cause of the sleep …… paraphrase question :what is another name for the fruit fly gene? passage:a defect in the human homologue of the drosophila "period" gene was identified as a cause of the sleep …… paraphrase question :what is the drosophila gene also called? n ' q 1 ' q passage:a defect in the human homologue of the drosophila "period" gene was identified as a cause of the sleep …… question:what is the drosophila gene also known as? passage:a defect in the human homologue of the drosophila "period" gene was identified as a cause of the sleep …… question:what is the drosophila gene also known as? >φ linear encoder softmax bert probability figure 1: the overview of the plm-based framework for few-shot eqa with our proposed component ppdac. given a (question, passage, answer) triples, our proposed ppdac converts the statement of the question in (question, passage, answer) triples into multiple new statements by the paraphrase generator, and then obtains highly similar statements via paraphrase selector to form more training data for finetuning the pre-train language model. ## the proposed plug-and-play component in this section, we formally introduce the few-shot eqa task and our proposed plug-and-play component in detail. the overview of the plm-based framework for few-shot eqa with our proposed component is shown in figure 1 . ## experiments in this section, we conduct extensive experiments to verify the improvement of the proposed plug-andplay component ppdac for the few-shot eqa methods 2 . ## conclusion in this work, we introduce a novel plug-and-play data augmentation component (ppdac) designed to enhance the training of pre-trained language models for question-answering (qa) tasks. by generating paraphrases (q ′ ) of the original questions within (question, passage, answer) triples, ppdac significantly enriches the training dataset, creating additional (q ′ , passage, answer) combinations. this innovative approach is particularly beneficial in few-shot learning scenarios, where data scarcity often hampers model performance. our comprehensive experiments across various qa baselines demonstrate the efficacy of ppdac, with notable improvements observed even in extremely data-constrained environments. specifically, in a 16-shot learning setup on the squad dataset, ppdac facilitated performance enhancements of 3.6%, 1.9%, and 30.6% for the splinter, fewshotbart, and fewshott5 models, respectively. remarkably, the integration of ppdac with fewshott5 not only achieved significant gains over baseline models but also surpassed existing state-of-the-art methods in few-shot settings by a substantial margin. furthermore, our detailed ablation studies shed light on the critical roles of ppdac's components, particularly the paraphrase filter (pf) and paraphrase selector (ps). the studies reveal that removing any component leads to a universal decline in model performance, with the simultaneous exclusion of both pf and ps resulting in the most pronounced performance drops. however, even in the absence of c c l 2 0 2 4 these components, the models equipped with the remaining paraphrase generation capabilities of ppdac continued to outperform their original configurations. this underscores the transformative impact of ppdac on enhancing qa model performance in few-shot learning contexts, with the pf module, in particular, emerging as a key contributor by prioritizing quality over quantity in paraphrase generation. looking ahead, we acknowledge the effectiveness of large language models (llms) in data augmentation. however, the innovation of our approach lies in its novel framework and process. while our current implementation utilizes a specific paraphrase generation model, we recognize its adaptability. in future iterations, we plan to incorporate the latest llms for paraphrase generation, emphasizing our innovative approach tailored for question-answering tasks. the plug-and-play data augmentation component (ppdac) currently enhances question paraphrasing within the (question, passage, answer) framework. future research may expand this approach to include passage paraphrasing, thereby enriching the training dataset and establishing a more solid foundation for fine-tuning pre-trained language models. this expansion aims to advance few-shot learning in question answering, steering towards more sophisticated and effective natural language understanding systems.