Can LLM s Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges (2024)

Abstract: Large Language Models (LLMs) have demonstrated impressive zero-shot performance on a wide range of NLP tasks, demonstrating the ability to reason and apply common sense. A relevant application is to use them for creating high-quality synthetic datasets for downstream tasks. In this work, we probe whether GPT-4 can be used to augment existing extractive reading comprehension datasets. Automating data annotation processes has the potential to save large amounts of time, money, and effort that goes into manually labeling datasets. In this paper, we evaluate the performance of GPT-4 as a replacement for human annotators for low-resource reading comprehension tasks, by comparing performance after fine-tuning, and the cost associated with annotation. This work serves to be the first analysis of LLMs as synthetic data augmenters for QA systems, highlighting the unique opportunities and challenges. Additionally, we release augmented versions of low-resource datasets, that will allow the research community to create further benchmarks for evaluation of generated datasets. Github available at https://github.com/vsamuel2003/qa-gpt4

## introduction machine reading comprehension (mrc) is a challenging nlp task where systems are designed to answer questions based on a given context. this task has significant practical value, as it answers user queries in diverse settings, from clinical contexts @xcite @xcite , to customer support @xcite and policy interpretation @xcite . bert-based models @xcite have achieved state-of-the-art performance when trained with extensive data from datasets like squad @xcite and natural questions @xcite . however, their † work does not relate to position at amazon. effectiveness diminishes in low-resource domains with limited data points @xcite . this limitation becomes particularly pronounced in newly emerging fields such as covid-19 @xcite , where substantial annotated instances are often lacking. data augmentation has been instrumental in enhancing performance across numerous lowresource nlp tasks @xcite @xcite . yet, much of the work on data augmentation for qa @xcite @xcite @xcite , hinges on the availability of unlabeled paragraphs from common sources, such as wikipedia, to produce new contextquestion-answer instances. this approach poses a challenge for specialized and mission-critical domains where such unlabeled contexts are scarcely available. bridging this gap, large language models (llms) exhibit a capability to generate texts that closely resemble human-authored content @xcite . this potential of llms can be harnessed to generate both novel contexts and their corresponding questionanswer pairs. addressing this gap, we introduce a gpt-4 (ope-nai, 2023) based data augmentation technique tailored for low-resource machine reading comprehension, specifically focusing on the extractive setting. in extractive qa, the system is provided with a context passage and a question, and the system must determine if the question is answerable using an extractive span from the passage. our approach begins by generating supplementary contexts, questions, and answers to augment training sets. to achieve this, we use in-context learning with passages, questions, and answers from the training set, ensuring minimal domain shift between the synthetically generated data and the original datasets. subsequently, we adopt cycle-consistent filtering to isolate high-quality training instances. em-pirical evaluations conducted on three pertinent real-world low-resource datasets covidqa @xcite , policyqa @xcite , and techqa @xcite reveal that our methodology improves the performance of bertbased mrc on covidqa by 23% and on policyqa by 5% in terms of exact match. notably, our approach attains state-of-the-art results on covidqa. ## related work language models have played a key role in the creation of synthetic datasets for various nlp tasks. models such as gpt-2 @xcite and ctrl @xcite have been applied to areas including general language understanding @xcite , classification @xcite , dialogue tasks @xcite , commonsense reasoning @xcite , and relation extraction @xcite , among others. recently, llms have significantly improved the quality and scope of synthetic dataset generation. they have been instrumental in augmenting datasets for tasks such as nli and sentiment analysis @xcite , classification @xcite , and even creating datasets for personalized dialogue generation @xcite , hate speech detection @xcite , and textual similarity (schick and schütze, 2021) to name a few. most prior work in synthetic data generation for qa @xcite @xcite has concentrated on generating questions from wikipedia passages to produce supplementary training examples. more recently, kalpakchi and boye introduced the use of gpt-3 for creating extra training data for swedish multiple-choice questions. our approach is the first to utilize in-context learning with llms for synthesizing contexts, questions, and answers for low-resource mrc. ## synthetic data generation we generate synthetic examples for each dataset using the in-context learning capabilities of the gpt-4 model. as part of our contribution, we release all synthetically augmented datasets to promote reproducibility and further research into refining the use of bootstrapping datasets with synthetically generated data. dataset statistics are included in the results section of this paper. furthermore, examples of original data instances and synthetically generated data instances are included in the appendix. the data generation process consists of two stages: ## results for policyqa, the largest dataset with over 12,000 examples, the best performance was achieved by augmenting with one-shot synthetic data without filtration, improving em by 1.6 points and f1 by 1.5 points over the baseline. this approach outperformed both two-shot and cyclefiltered variations. in the smallest dataset, techqa, with only 1,808 examples, synthetic data augmentation did not lead to clear improvements. the baseline model achieved the highest em score, with two-shot cycle filtered, one-shot filtered, and one-shot unfiltered configurations performing similarly. for f1, twoshot cycle filtered data obtained the second-highest score after the baseline. overall, synthetic data augmentation improved performance in covidqa and policyqa, with the best results from one-shot generation combined with round trip filtration for covidqa, and unfiltered one-shot generation for policyqa. in techqa, the small data size and high domain diversity limited the effectiveness of synthetic augmentation dataset statistics for the three datasets used are shown in ## opportunities and challenges our experiments demonstrate the significant potential of leveraging llms like gpt-4 for synthetic data generation. in domains like covidqa and pol-icyqa, augmenting with llm-generated synthetic examples consistently improved performance over the baseline, showcasing the few-shot generalization abilities of modern llms. one-shot synthetic data augmentation yielded the best results, surpassing other configurations. llms can significantly expand limited training sets for various nlp tasks, enhancing performance without the expense of human labeling. however, challenges remain, particularly in lowdata regimes like techqa, where llm-augmented models performed no better than the baseline. this highlights the difficulty llms face in synthesizing useful examples from scarce data. improving llms' few-shot learning, integrating external knowledge, and developing advanced filtering techniques are critical for maximizing the benefits of synthetic data generation. while llms hold promise for addressing limited training data, substantial challenges must be overcome to fully realize their potential in diverse nlp tasks.