A Lightweight Method to Generate Unanswerable Questions in E nglish (2023)

Abstract: If a question cannot be answered with the available information, robust systems for question answering (QA) should know *not* to answer. One way to build QA models that do this is with additional training data comprised of unanswerable questions, created either by employing annotators or through automated methods for unanswerable question generation. To show that the model complexity of existing automated approaches is not justified, we examine a simpler data augmentation method for unanswerable question generation in English: performing antonym and entity swaps on answerable questions. Compared to the prior state-of-the-art, data generated with our training-free and lightweight strategy results in better models (+1.6 F1 points on SQuAD 2.0 data with BERT-large), and has higher human-judged relatedness and readability. We quantify the raw benefits of our approach compared to no augmentation across multiple encoder models, using different amounts of generated data, and also on TydiQA-MinSpan data (+9.3 F1 points with BERT-large). Our results establish swaps as a simple but strong baseline for future work.

## introduction question answering datasets in nlp tend to focus on answerable questions @xcite , but unanswerable questions matter too because: (1) real-world queries are unanswerable surprisingly often -e.g., 37% of fact-seeking user questions to google are unanswerable based on the wikipedia page in the top 5 search results @xcite ; and (2) identifying unanswerable questions is an essential feature of reading comprehension -but conventional extractive qa systems typically guess at plausible answers even in these cases @xcite . to aid in building robust qa systems, more datasets have begun to include unanswerable questions, e.g., the squad 2.0 dataset in english @xcite and the multilingual tydiqa context: the only indigenous mammals of bermuda are five species of bats, all of which are also found in the eastern united states:. . . ## our augmentation methods inspired by the crowdworker-written unanswerable questions in @xcite , we generate 2 types of unanswerable questions by modifying answerable ones with antonym and entity swaps. our generated data is then filtered based on empirical results presented in appendix b.1. we examine results for each augmentation method separately, but we also experimented with combining them in appendix b.2. examples of output from our methods are shown in figure 1 and in appendix d. we use spacy @xcite for partof-speech tagging and dependency parsing, and allennlp's @xcite implementation of @xcite model for named entity recognition. using nltk @xcite , we access wordnet @xcite for antonyms and lemmatization. ## experimental setup task. we evaluate on the downstream task of extractive question answering, i.e., we judge an unanswerable question generation method to be better if training with its data improves an extractive qa system's performance compared to other methods. given a question and some context (a sentence, paragraph or article), the task is to predict the correct answer text span in the passage, or no span if the question is unanswerable. performance is measured with exact match (em) and f1, computed on the answer span strings. datasets. we use squad 2.0 @xcite and the english portion of the tydiqa dataset @xcite ) that corresponds to minimal-span extractive question answering. squad 2.0 uses paragraphs as context whereas tydiqa uses full articles. to keep the tydiqa setting similar to squad 2.0, we modify the task slightly, discarding yes/no questions and questions for which there is a paragraph answer but not a minimal span answer. for both datasets, we train on original or augmented versions of the training set and report performance on the development set. all data statistics are shown in models. we experiment with base and large variants of bert @xcite , roberta @xcite , and albert @xcite , all trained with huggingface transformers @xcite ; see appendix a for further details. ## comparison with previous squad 2.0 augmentation methods we compare methods on their bert large performance using 2 strong baselines for unanswerable question generation: unansq @xcite and the state-of-the-art method crqda @xcite . we use publicly-released unanswerable questions for both methods, which only exist for squad 2.0. in theory, crqda can generate both answerable and unanswerable questions but we only use the latter for an even comparison and because only these are made available. as the results in ## beyond squad 2.0 and bert-large to more robustly evaluate our augmentation methods, we experiment with more models of multiple sizes (albert and roberta) as well as with an additional dataset (tydiqa-minspan). ## conclusion and future work our lightweight augmentation method outperforms the previous state-of-the-art method for english unanswerable question generation on 4 metrics: development set performance, unanswerability, readability and relatedness. we see significant improvements in squad 2.0 and tydiqa-minspan performance (over a no-augmentation baseline) across multiple encoder models and using different amounts of generated data. overall, we find that when it comes to unanswerable question generation, simpler is better. we thus hope that future work justifies its complexity against our strong baseline. although we have shown that entity-based augmentation creates data that is useful for models to learn from, it is still unclear why. several of our examples seem to contain false presuppositions, e.g., "when did christopher columbus begin splitting up the large bronx high schools?" @xcite term these "questionable assumptions," and find them to be challenging for models like macaw, gpt-3 and flan-t5. while @xcite studies what makes answerable multiple-choice questions hard for models, we still do not know what makes unanswerable questions hard, and how this relates to domain, model type and size. beyond unanswerable question generation and even question answering, we hope our work encourages nlp researchers to consider whether simpler approaches could perform competitively on a task before using sledgehammers to crack nuts.