A Pipeline for Generating, Annotating and Employing Synthetic Data for Real World Question Answering (2022)

Abstract: Question Answering (QA) is a growing area of research, often used to facilitate the extraction of information from within documents. State-of-the-art QA models are usually pre-trained on domain-general corpora like Wikipedia and thus tend to struggle on out-of-domain documents without fine-tuning. We demonstrate that synthetic domain-specific datasets can be generated easily using domain-general models, while still providing significant improvements to QA performance. We present two new tools for this task: A flexible pipeline for validating the synthetic QA data and training down stream models on it, and an online interface to facilitate human annotation of this generated data. Using this interface, crowdworkers labelled 1117 synthetic QA pairs, which we then used to fine-tune downstream models and improve domain-specific QA performance by 8.75 F1.

## introduction having enough relevant training data is a key factor for achieving strong performance in machine learning and nlp @xcite , but for many tasks, large domain-specific datasets are expensive and time-consuming to create manually. this is especially true for tasks like extractive question answering (qa), which both relies on domain-specific knowledge and requires skilled annotators. these difficulties have led to increased interest in synthetic data generation recently @xcite through various methods such as bootstrapping from smaller datasets, or through generative models which create entirely new data. we make the following contributions: synthetic nlp data generation synthetic data generation is an attractive option for dataset creation, especially for domain-specific tasks. various methods for bootstrapping from smaller datasets have been devised, such as back-translation @xcite and sibylvariant transformations @xcite . backtranslation produces paraphrases through round-trip translation, while sibylvariant transformations modify or combine texts in predictable ways to create new data with a different label. of particular interest are methods that use text generation models to create entirely new data, rather than simply paraphrasing or combining inputs predictably. a variety of these models have been used to generate new qa pairs @xcite , such as the t5 model @xcite and bert @xcite . synthetic data generation can be particularly useful when fine-tuning a model on a specific domain, for which manually-curated datasets may not exist. whilst high quality datasets such as squad 2.0 @xcite do exist for qa tasks, they tend to only have general content, e.g. from wikipedia. thus models trained on them often struggle on more domain-specific tasks @xcite , see also section 4.3 below). evaluation of synthetic qa pairs evaluating question generation (qg) models can be difficult due to the nature of the problem: a good question tends to have various qualities (grammatical, answerable, non-trivial to answer, etc.) that are difficult to capture in a single metric, especially one that correlates well with human judgements @xcite . nonetheless, several metrics such as bleu @xcite and bertscore @xcite have been proposed, though they rely on having reference questions available and often do not capture whether or not the question is answerable @xcite . however, @xcite show that the use of unanswerable questions when training qa models is important for real-world performance, making it a metric of interest. round-trip evaluation, such as the methods proposed by @xcite , allows for evaluating the generated data by checking how consistent downstream model results are when synthetic data is used as the model input, e.g. if the generated answer is found for a synthetic question when the question is input to a qa model. we adopt this approach and discuss it further in section 4.2. ## system overview figure 1 shows an overview of our system for creating domain-specific synthetic qa pairs which are used to train downstream models. the qg process (see section 3.2 for details) creates domain-specific qa pairs from unlabelled texts. this data is then annotated for grammaticality and correctness using the annotation tool, allowing for the creation of two new domain-specific datasets to fine-tune both grammaticality and qa models. we take a subset of a proprietary knowledge base as our set of input documents and use this to create our domain-specific qa dataset (which we call "syfter"). the knowledge base contains documents obtained by scraping online articles and is focused on business news, such as information about corporate structures, and is thus quite distinct in subject matter from our external domain-general data (squad 2.0, see section 3.2). ## experiments and results the grammaticality and question answering models are tested in both the setting of interest (combined domain-general and domain-specific data) as well as two baseline data settings (domain-general data only 12 , and domain-specific data only). this allows us to both measure how useful the synthetic data is as an addition to domain-general data and to also evaluate the feasibility of fine-tuning using only synthetic data, which would reduce time and expense significantly given its small size. the combined test sets for the grammaticality and qa models are produced by combining the appropriate domain-general data (cola or squad) with the domain-specific syfter data and then testing the model on this combination dataset. test dataset qa model exact match similarity squad 2.0 roberta 67.81% 81.89% syfter roberta 64.55% 77.27% to date we've delivered more than one billion pieces of protective equipment to the frontline. how many pieces of protective equipment have been delivered to the frontline? more than one billion as a major food sector player, bel fully assumes its duty to do everything possible to ensure the continuity of its operations. in both the domain-general and domain-specific settings, but do not evaluate the combined setting due to the nature of the evaluation (see section 4.2). ## conclusion we present a pipeline for using and evaluating synthetic qa data and an interface for annotating it, as well as annotation guidelines. the combination of domain-general and synthetic data allows our qa model to perform significantly better (+ 9 f1) on domain-specific documents than it did when trained solely on a similar amount of domain-general data. the pipeline is simple to apply to both current and future state-of-the-art models, enabling better performance in low-resource domains.