Generative Data Augmentation using LLM s improves Distributional Robustness in Question Answering (2024)

Abstract: Robustness in Natural Language Processing continues to be a pertinent issue, where state of the art models under-perform under naturally shifted distributions. In the context of Question Answering, work on domain adaptation methods continues to be a growing body of research. However, very little attention has been given to the notion of domain generalization under natural distribution shifts, where the target domain is unknown. With drastic improvements in the quality and access to generative models, we answer the question: How do generated datasets influence the performance of QA models under natural distribution shifts? We perform experiments on 4 different datasets under varying amounts of distribution shift, and analyze how “in-the-wild” generation can help achieve domain generalization. We take a two-step generation approach, generating both contexts and QA pairs to augment existing datasets. Through our experiments, we demonstrate how augmenting reading comprehension datasets with generated data leads to better robustness towards natural distribution shifts.

## introduction in this work, we perform a systematic study of how "in-the-wild" generation can affect the distributional robustness of question-answering models trained on the popular stanford question answering dataset (squad) @xcite . synthetic data generation is a widely adopted method for domain adaptation in qa systems @xcite @xcite . however, domain adaptation methods have access to unlabelled/labelled data belonging to the target domain, and do not account for unseen natural distribution shifts. our work studies the effect of generated data on distribution shifts where the target domain is unseen. ## conclusion and future avenues we created a framework that enhances the performance of reading comprehension models by supplementing real datasets with a diverse dataset generated by contemporary, real-world generative models. our findings indicate that this training method yields superior results on test datasets and those with natural distribution shifts, due to the added robustness from training on the generated data as opposed to traditional methods. in the future, we want to explore a more extensive comparison against question generation methods and how this paradigm fits into fine-tuning larger models.