M in P rompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering (2024)

Abstract: Recent advances in few-shot question answering (QA) mostly rely on the power of pre-trained large language models (LLMs) and fine-tuning in specific settings. Although the pre-training stage has already equipped LLMs with powerful reasoning capabilities, LLMs still need to be fine-tuned to adapt to specific domains to achieve the best results. In this paper, we propose to select the most informative data for fine-tuning, thereby improving the efficiency of the fine-tuning process with comparative or even better accuracy on the open-domain QA task. We present MinPrompt, a minimal data augmentation framework for open-domain QA based on an approximate graph algorithm and unsupervised question generation. We transform the raw text into a graph structure to build connections between different factual sentences, then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text. We then generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model. Empirical results on several benchmark datasets and theoretical analysis show that MinPrompt is able to achieve comparable or better results than baselines with a high degree of efficiency, bringing consistent improvements in F-1 scores.

## introduction question answering (qa) provides accurate responses to a series of questions based on given narrative contexts. its diverse applications extend to areas such as chatbots @xcite , dialogue systems @xcite , and instant information retrieval @xcite , making it a key pursuit in the field of natural language processing (nlp). supervised learning has traditionally been the approach for developing efficient qa systems that deliver commendable results @xcite . however, this method is intrinsically restricted by its reliance on a large set of annotated qa training examples, which becomes problematic due to the substantial cost associated with acquiring expert-level annotations. our research focuses on the few-shot qa task, an effort to address the qa challenge with the presence of only a limited number of training examples. the prevalent approaches under the few-shot setting either introduce a new task and pre-train an extensive language model from scratch @xcite , or they fine-tune an already pre-trained model on the given training examples @xcite . the finetuning stage is crucial in the sense that it stimulates the power of the llms obtained during the pre-training stage and makes the model align with the input/output distribution of a certain domain or dataset. however, with an increasing data size for fine-tuning, the training duration increases accordingly, which is undesirable, especially when the model size is also large @xcite . as such, the importance of minimal data augmentation cannot be understated. the fine-tuning data, often a limited resource in our consideration (up to 128 shots), is directly used to adjust the parameters of a pre-trained model to enhance performance on the downstream task. the data is usually labeled by domain experts and thus could be time-consuming to obtain in large quantities. on the other hand, augmented data represents a broader dataset, generated in an unsupervised manner by converting statements into question-answer pairs. in qa tasks, it is vital for a model to be exposed to a diverse range of questions, answers, and contexts to develop a robust understanding of the language and the task at hand. however, not all parts of the training data hold equal relevance or significance for the model's learning process. some parts may contain more valuable information or more complex language structures that the model needs to under-stand to improve its performance. consequently, identifying and augmenting these critical portions of the training data could substantially enhance the model's capacity to answer questions accurately and comprehensively. to address the above challenges, we present minprompt, which consists of the following three modules: (1) a sentence graph construction module that leverages sentence graph representation to structurize the raw text. each node in the graph symbolizes a sentence, while edges illustrate the shared entities between sentences. this sentence graph effectively encapsulates the complex interconnections between various textual elements; (2) a data selection module that features an approximate minimal dominating set algorithm. the algorithm is applied to the sentence graph to identify the smallest set of sentences to cover all shared entities. this module ensures efficient use of computational resources, reduces the risk of overfitting, and enhances the model's generalization ability, resulting in an overall improvement in qa performance; and (3) a question generation module that transforms the selected plain factual sentences into qa pairs. the synthesized qa pairs are further turned into prompts, providing a condensed, yet comprehensive representation of the text. the generated prompts serve as high-quality, information-rich training instances for the qa model. this model trained on the compact and meaningful prompts is then capable of generating accurate answers to the posed questions, all without requiring any additional explicit supervision. in summary, our contributions are as follows: question generation. chen et al. (2019) presented an answer-aware question generation (qg) model that employs reinforcement learning for improved question quality. the model incorporates a coverage mechanism to alleviate the common issue of answer-related content being left out from the generated questions. ma et al. (2020) developed a more sophisticated approach to answer-aware question generation. their model uses sentence-level semantic matching and answer position inferring within a sequence-to-sequence framework, resulting in higher-quality questions. do et al. (2023) proposed a two-stage framework for conversational question generation (cqg). it selects sentences from a semantic graph to pick up coherent topics and then uses a classifier to determine the answer type of the question. their approach produces more natural dialogues, as real-life interlocutors often discuss relevant content that is non-sequential. mohammadshahi et al. (2022) introduces rquge, a novel metric for assessing the quality of automatically generated questions. traditional methods may unfairly penalize valid questions that don't mirror reference questions closely. rquge overcomes these issues by evaluating on the basis of the answerability of a question given the context. utilizing pre-trained models for its qa scorer modules, rquge does not require additional training. the paper presents evidence of rquge's high correlation with human judgment and robustness against adversarial corruption. few-shot qa. previous research in qa has mainly focused on either reusing pre-trained language models (plms) @xcite or training a model from scratch using synthetic qa data @xcite @xcite . however, both approaches require a large amount of annotated data from the downstream qa task to fine-tune the models, which can be impractical in real-world scenarios. to address this problem, several recent approaches have been developed that allow the model to adapt to the downstream task with only a small amount of annotated data @xcite . for example, @xcite proposed a pretraining scheme tailored for qa tasks by designing a recurring span selection objective that aligns with the common objective in extractive qa tasks. @xcite proposed a framework called fewshotqa, which leverages the capacity of ## minprompt: graph-based prompt data augmentation for few-shot qa as shown in figure 1 , our overall framework, min-prompt, is designed to extract the most semantically rich and factually dense sentences to serve as candidates for conversion into a prompt tuning qa dataset. this process is guided by the principal intuition that the most informative sentences are those that encompass facts or declarations concerning a greater number of entities. hence, these highimpact sentences should ideally cite more entities within their purview. to implement this, we start by extracting the co-reference of entities across sentences. essentially, it allows us to map the discourse in a way that allows us to understand which sentences are speaking about the same entities. next, we construct a graph to depict the higherorder coreference relationships. in this graph, the sentences serve as nodes, and sentences are connected if they mention the same entity. this representation allows us to establish and understand the intricate network of relationships between sen-tences and the entities they mention. employing graph-based algorithms, we are then able to identify and extract the most informative sentences. these are typically sentences that have a high degree of connectivity in the graph, indicating that they mention or discuss a larger number of entities. we then transform these selected sentences into a fine-tuning dataset. the transformation process entails restructuring the sentences to meet the format requirements of a qa dataset, which generally involves turning declarative sentences into question-and-answer pairs. this method thus combines insights from computational linguistics and graph theory to achieve its goal of creating a highquality fine-tuning dataset for qa tasks. the approach ensures that the dataset is not only rich in informative sentences, but also maps intricate entity relationships, thus providing a comprehensive context for each question and answer pair. this context helps in the training of more robust and nuanced qa systems. ## conclusion in this paper, we present minprompt, a robust data augmentation framework that leverages a graph-based algorithm and unsupervised question generation to extract minimally meaningful qa training samples from raw text. our contributions reside in the application of minimal data augmentation, enhancing computational efficiency and model performance while mitigating overfitting. through extensive experiments, our model consistently outperformed competitive methods in public benchmarks, demonstrating its effectiveness.