Data Augmentation for Biomedical Factoid Question Answering (2022)

Abstract: We study the effect of seven data augmentation (DA) methods in factoid question answering, focusing on the biomedical domain, where obtaining training instances is particularly difficult. We experiment with data from the BIOASQ challenge, which we augment with training instances obtained from an artificial biomedical machine reading comprehension dataset, or via back-translation, information retrieval, word substitution based on WORD2VEC embeddings, or masked language modeling, question generation, or extending the given passage with additional context. We show that DA can lead to very significant performance gains, even when using large pre-trained Transformers, contributing to a broader discussion of if/when DA benefits large pre-trained models. One of the simplest DA methods, WORD2VEC-based word substitution, performed best and is recommended. We release our artificial training instances and code.

## introduction question answering (qa) systems aim to answer natural language questions by searching in structured @xcite @xcite or unstructured data, such as free-text documents @xcite . here we consider qa of the latter kind. fully fledged qa systems for document collections retrieve relevant documents, identify relevant passages, extract and aggregate answer spans etc. @xcite . there are also different types of questions, e.g., yes/no, factoid, list, how-to. thus, creating realistic datasets to train and evaluate complete qa systems for document collections is resource intensive, especially for systems targeting specialized domains. a prime example is the biomedical domain, the focus of this work, where obtaining realistic training (and test) instances requires medical expertise, which is costly and diffi-cult to obtain. consequently, biomedical datasets for full qa systems contain just a few thousand training instances @xcite or focus on simpler question types only, e.g., yes/no questions @xcite . a simplified form of qa for textual data is machine reading comprehension (mrc) @xcite @xcite @xcite @xcite @xcite , where the system is given a question and a particular (or a few) passage(s) and the answer must be found therein. in effect, mrc focuses on a particular core stage of a full qa pipeline, identifying answer spans, assuming that relevant documents and passages have already been identified. we also focus on this stage, adopting an mrc setting. large generic (non domain-specific) mrc datasets have been constructed via crowd-annotation @xcite @xcite , but crowd-annotation on that scale is difficult when biomedical expertise is required. an alternative is to automatically generate cloze-style mrc datasets. the last sentence or title of a random passage is treated as a question, some part (e.g., named entity) of the 'question' is masked, and the system is required to predict it. this approach has been used to generate large artificial cloze-style mrc datasets @xcite @xcite , including biomedical ones @xcite . these datasets could be used to augment real ones, but have mostly been used as artificial experimental setups only. when training examples for end-tasks are limited, as in realistic biomedical qa datasets, the currently dominant approach in nlp is to use pretrained transformers @xcite @xcite @xcite , possibly pre-trained on domain-specific corpora @xcite @xcite , and fine-tune (further train) them on the limited examples of the end-tasks. nevertheless, increasing the number of end-task examples typically improves performance. one way to achieve this is to employ data augmentation (da) @xcite , which adds artificial training instances to a training set, in our case the training set of the end task. it is unclear, however, which da methods improve most (if at all) the performance of pre-trained models per endtask @xcite . consequently, @xcite recommend exploring when da is effective for large pre-trained models. in this paper, we thoroughly examine the impact of da in biomedical qa, focusing on the factoid questions of the bioasq challenge @xcite in an mrc setting, i.e., we assume that relevant text passages, called snippets in bioasq, have already been identified. we first evaluate on bioasq three indicative off-theshelf pre-trained models, distilbert @xcite , biobert @xcite , albert @xcite , already fine-tuned on squad @xcite or squad-v2 @xcite , and we select albert as our weak baseline. we also fine-tune albert on bioasq, on top of its squad fine-tuning, to obtain a stronger baseline. we then obtain additional artificial training instances from an artificial cloze-style mrc dataset, or via back-translation, information retrieval (ir), word substitution based on word2vec or masked language modeling, question generation, or by extending the given passages with additional context. our main contributions are: (1) we present the largest (in terms of methods) experimental comparison of da methods for qa, focusing on biomedical qa, where obtaining training instances is particularly difficult and costly. (2) we show that da can lead to very large performance gains, even when using pre-trained transformers fine-tuned on large generic (squad) and/or small domainspecific (bioasq) end-task datasets, contributing to a broader discussion of if/when da benefits pretrained models. (3) we show that artificial clozestyle mrc datasets are useful for da. (4) we show that one of the simplest da methods, word2vecbased word substitution, is also the best and is, therefore, recommended. (5) we make our artificial training examples and code publicly available. foot_0 2 experimental setup 2.1 bioasq data in a squad setting we experiment with data from bioasq-8 (2021), phase b, task b @xcite , which contain english questions of biomedical experts. each question is accompanied by (i) the gold answer (often several alternative phrasings) and (ii) gold relevant passages, called snippets (usually a single sentence each) from biomedical articles; the gold snippets contain the gold answer or other relevant information. there are four question types: yes/no, factoid, list, and questions requiring a summary. we focus on factoid questions (e.g., "which gene is involved in giant axonal neuropathy?"). we convert the bioasq data to triples each containing a question, a single gold snippet, and the span of the gold answer in the snippet, much as in squad @xcite . if a question has multiple gold snippets, we produce equally many triples, discarding snippets that do not contain the gold answer. this conversion and considering only factoid questions allow us to use pre-trained transformers already fine-tuned on squad in a similar setting. 2 a disadvantage of the conversion is that our results are not directly comparable to those of bioasq. the goal of our work, however, is to study the effect of different da methods on a modern transformer-based qa baseline (and we show that fine-tuning it first on squad helps), not to compete against bioasq participants, who often use models tailored to the particular competition. from the 941 factoid questions of the original bioasq data, we obtained 3415 question-snippetanswer triples. we split these in training, development, test subsets (2848, 271, 296 triples, resp.), ensuring no question is in more than one subsets. ## data augmentation methods there are two alternatives when using the artificial training instances that da generates @xcite . in our case, we always start with albert, pre-trained on a generic corpus, and already finetuned on squad-v2. in the first alternative, the model is then further fine-tuned on the artificial instances, and is then finally fine-tuned on the endtask data (bioasq). in the second alternative, the artificial and the end-task data are mixed, and the model is fine-tuned on the mixed data. in each experiment below, we use the alternative (among the two) that leads to the best development prauc. ## limitations and future work a limitation of our work is that we consider only da in the input space, i.e., the artificial instances are in textual form, like the original ones, as opposed to, e.g., interpolating feature vectors @xcite @xcite . we also consider only offline augmentation, i.e., the artificial instances are generated once, before training, as opposed to artificial instances generated anew for each training epoch. these two limitations, which are common in da for nlp, allow generating model-agnostic training instances once and reusing them across training epochs and different models. this greatly reduces computation costs and allows sharing the augmented datasets. online da, however, exposes the model to many more synthetic data; and feature space da may act as layer-specific regularization. one could also exploit ideas from active learning @xcite to select the most informative, diverse, and representative artificial training instances among those that da generates. small subsets of the bioasq data could also be used to study the effect of da in few-shot learning.