Synthetic Data Generation for Multilingual Domain-Adaptable Question Answering Systems (2022)

Abstract: Deep learning models have significantly advanced the state of the art of question answering systems. However, the majority of datasets available for training such models have been annotated by humans, are open-domain, and are composed primarily in English. To deal with these limitations, we introduce a pipeline that creates synthetic data from natural text. To illustrate the domain-adaptability of our approach, as well as its multilingual potential, we use our pipeline to obtain synthetic data in English and Dutch. We combine the synthetic data with non-synthetic data (SQuAD 2.0) and evaluate multilingual BERT models on the question answering task. Models trained with synthetically augmented data demonstrate a clear improvement in performance when evaluated on the domain-specific test set, compared to the models trained exclusively on SQuAD 2.0. We expect our work to be beneficial for training domain-specific question-answering systems when the amount of available data is limited.

## introduction recent advances in tackling the problem of question answering (qa) rely on large-scale, opendomain datasets @xcite , annotated by humans and composed primarily in english (e.g. squad 1.0 @xcite , and squad 2.0 @xcite ). despite some indications of poor robustness and generalisation @xcite , models trained on such datasets are capable of providing topicagnostic, general-purpose assistance to their users @xcite . nevertheless, most industrial applications of qa systems are domain-specific, and often need to be able to operate in multilingual environments. data collection and manual composition of datasets for each domain and language is most definitely a laborious task, not to mention that certain domains are of little academic or commercial interest and are only of use for some low-resource communities @xcite . moreover, while the current synthetic data generation systems focus on augmenting qa data in the squad format, foot_0 little research has been done on either the generation of synthetic data from natural plain text, or in multiple languages. furthermore, most machine reading comprehension (mrc) benchmarks focus primarily on the creation of questions with multi-word factoid answers (e.g. squad 2.0 pairs each factoid question with a wikipedia paragraph), as well as unanswerable questions @xcite . however, in a real-world scenario, a qa system should ideally be able to provide a response on semantically complex questions such as "i am an eu citizen living in the uk. what changes for me after brexit?", and questions containing grammar and spelling errors (e.g. questions asked by a non-native speaker, or containing mistakes caused by dyslexia). in this work, we introduce a domain-adaptable end-to-end pipeline for generic synthetic data generation that requires no manual textual preprocessing, and allows for the integration of mul-tilingual features. we utilise this pipeline to create domain-specific training sets in english (en) and dutch (nl) from the web scraped data of the single digital gateway and your europe portal, 2 which provides information on rules and procedures for citizens and businesses in the eu, in all european languages. we combine the obtained data with squad 2.0 in english and its machinetranslated-into-dutch version to fine-tune multiple instances of a bert (bidirectional encoder representations from transformers) model @xcite on the qa task. we then cross-evaluate the performance of these models on the relevant test sets, and observe improvements on the qa task when evaluated on the domain-specific test sets, while remaining competitive against models trained on the squad-only counterparts in both languages. ## related work existing approaches to synthetic data generation often view question and answer generation as dual tasks @xcite , where one task can improve the other and viceversa. roundtrip consistency @xcite is one of the methods that combines question generation and question answering models to, first, generate a question conditioned on a pre-selected answer span and its context, and then match against it an answer predicted by a qa system. if there is a match, the triplet (i.e. context, question and answer) is considered valid. cloze generation @xcite is a more intuitive approach: it logically splits a document in ratio of 20:80, with the introduction being the first 20% of the input text. it is assumed that the introduction contains answer candidates that are likely to occur in the remainder of the document. potential answer candidates are consequently selected by matching multi-word spans between introductory sentences and the rest of the text. these approaches, however, focus on potential answers that are primarily named entities or noun phrases @xcite @xcite . for our use-case, we are interested in finding answers of longer spans that might contain administrative procedures in a multilingual setting (e.g. answers to such questions as "how do i request an interna-2 https://ec.europa.eu/growth/singlemarket/single-digital-gateway_en og results tend to be scattered across different websites that often lack any guarantee of quality or reliability, and significant information gaps remain in many areas, leaving important questions unanswered mt resultaten zijn meestal verspreid over verschillende websites die vaak geen enkele garantie voor kwaliteit of betrouwbaarheid hebben, en er blijven op veel gebieden aanzienlijke informatielacunes, waardoor belangrijke vragen onbeantwoord blijven ## methodology we developed a synthetic data generation pipeline that converts plain text into question answering pairs via the following steps: passage detection, keyword filtering, question generation and question paraphrasing. ## experiments we fine-tuned the multilingual distilled version of bert @xcite (mdistilbert) on the qa task using the synthetic data obtained using the methods described in section 3 and the squad 2.0 datasets (we refer to section 4.1). full overview of the training data, its sources and size, can be found in tables 3 and 5 . as multilingual bert models are known to perform better on tasks 10 https://github.com/ramsrigouthamg/ paraphrase-any-question-with-t5-textto-text-transfer-transformer11 https://quoradata.quora.com/firstquora-dataset-release-question-pairs 12 https://ec.europa.eu/cefdigital/wiki/ display/cefdigital/etranslation in english @xcite , we performed separate experiments with english and dutch data, as well as experiments with the bilingual data combined. all models were tested on four test sets, two in each language. ## discussion of results in this section we compare the performance of the qa models trained on both non-synthetic (i.e. squad) and synthetic data, and models trained exclusively on non-synthetic data. as discussed in section 4.2, we present results for both english and dutch. we also evaluate the performance of a bilingual qa model. in more interestingly, in tables 9 and 10 , we present the results on the domain-specific (your europe) test sets for en and nl. we observe that models trained on non-synthetic data only (qa s, en , qa s, nl , qa s, en-nl ) demonstrate an overall lower performance compared to the models also trained on synthetic data (qa sg, en , qa sgp, en , qa sgp, nl and qa sgp, en-nl ). comparing scores achieved by qa sg, en and qa sgp, en we can also conclude that adding synthetic segments obtained via qp results in an increase in performance, consistent across all metrics. finally, from these results we also see that bilingual models trained on synthetic and non-synthetic data achieve better performance than their monolingual version (i.e. qa sgp, en and qa sgp, nl versus qa sgp, en-nl ). ## conclusion in this paper we presented a novel multilingual domain-adaptable pipeline for the generation of synthetic training data for qa models. our experiments demonstrate that models trained with synthetic data achieved improved performance on domain-specific test sets that included not solely factual, but semantically complex questions, both in english and dutch. as our pipeline incorporates two mt5 models fine-tuned on task-and languagespecific datasets, we demonstrate that it is possible to make use of mt and apply our approach to any language supported by mt5. one of the remaining challenges of this approach is the quality monitoring of the generated synthetic questions, especially for languages other than english. it would be useful to experiment with more advanced filtering methods than the method based on keyword extraction proposed in this work. for instance, a semantic similarity feature could potentially detect questions that might not include specific keywords, but also questions containing synonyms of extracted keywords or semantically close paraphrases. we also assume that it would be useful to introduce an additional feature to evaluate the chunks that are processed by our pipeline for synthetic data generation, as not every input paragraph or sentence would serve as an answer to a potential question in a real-world scenario.