Adapting Pre-trained Generative Models for Extractive Question Answering (2023)

Abstract: Pre-trained Generative models such as BART, T5, etc. have gained prominence as a preferred method for text generation in various natural language processing tasks, including abstractive long-form question answering (QA) and summarization. However, the potential of generative models in extractive QA tasks, where discriminative models are commonly employed, remains largely unexplored. Discriminative models often encounter challenges associated with label sparsity, particularly when only a small portion of the context contains the answer. The challenge is more pronounced for multi-span answers. In this work, we introduce a novel approach that uses the power of pre-trained generative models to address extractive QA tasks by generating indexes corresponding to context tokens or sentences that form part of the answer. Through comprehensive evaluations on multiple extractive QA datasets, including MultiSpanQA, BioASQ, MASHQA, and WikiQA, we demonstrate the superior performance of our proposed approach compared to existing state-of-the-art models.

## introduction an important subcategory of question-answering tasks is extractive question answering, where parts of a given context are selected as the answer to a question. in many settings, this is considered more reliable than abstractive question answering @xcite which is more powerful in general but less explainable. the extractive questionanswering task is primarily tackled using discriminative models. specifically, for datasets featuring single-span factoid answers, such as squad @xcite , models such as @xcite ; @xcite ; @xcite identify the start and end positions of the answer span. conversely, for datasets encompassing multi-span factoid answers, such as multispanqa @xcite and bioasq @xcite , researchers have proposed discriminative models based on "bio" tagging ('begin', 'inside', 'outside'), which works for both single and multi-span answers. in the case of long-form sentence-level qa datasets like mashqa @xcite and wikiqa @xcite , sentence classification models like multico @xcite have been employed. however, to date, the application of generative seq2seq models to address this extractive qa task remains unexplored. the main challenge that we may hope to overcome using a generative approach is that of sparsity. our observations indicate that extractive questionanswering tasks exhibit a high level of sparsity, where the answers comprise only a minuscule fraction of the tokens or sentences present in the given context (see the sparsity challenge encountered in extractive question answering is less daunting for generative approaches, as they explicitly model what is likely (via likelihood) rather than what is unlikely. ## adaptation of generative model for extractive question answering we now formalize the sentence-level answer extraction task and propose a novel approach for it. this can easily be extended for the token-level answer extraction tasks as well. we are given a context c and a question q. the context is a sequence of sentences {s 1 , s 2 , . . . , s n }, where n is the number of sentences in c. each sentence s i and similarly the question q is a sequence of tokens. each sentence is associated with a binary variable a i to indicate whether it is part of the extractive answer for q. the answer sentences, with a @xmath0. ## experimental results we include the previous sota performance and our best f1 score across the five datasets in ## conclusion in this work, we introduce a novel approach for extractive question-answering by leveraging a pretrained generative language model and fine-tuning it to generate indexes of answer tokens or sentences. discriminative models often necessitate dataset-specific customizations to achieve satisfactory performance due to the varying nature of label sparsity in such tasks. in contrast, we demonstrate that generative models can be readily adapted to address this challenge by generating indexes of tokens instead of directly generating the tokens themselves. through empirical evaluations, our proposed model surpasses specialized state-of-theart baselines across a range of diverse extractive qa benchmark datasets, showcasing its superior performance and effectiveness.