Towards more equitable question answering systems: How much more data do you need? (2021)

Abstract: Question answering (QA) in English has been widely explored, but multilingual datasets are relatively new, with several methods attempting to bridge the gap between high- and low-resourced languages using data augmentation through translation and cross-lingual transfer. In this project we take a step back and study which approaches allow us to take the most advantage of existing resources in order to produce QA systems in many languages. Specifically, we perform extensive analysis to measure the efficacy of few-shot approaches augmented with automatic translations and permutations of context-question-answer pairs. In addition, we make suggestions for future dataset development efforts that make better use of a fixed annotation budget, with a goal of increasing the language coverage of QA datasets and systems.

## introduction automatic question answering (qa) systems are showing increasing promise that they can fulfil the information needs of everyday users, via information seeking interactions with virtual assistants. the research community, having realized the obvious needs and potential positive impact, has produced several datasets on information seeking qa. the effort initially focused solely on english, with datasets like wikiqa @xcite , ms marco @xcite , squad @xcite , quac @xcite , coqa @xcite , and natural questions (nq) @xcite , among others. more recently, heading calls for linguistic and typological diversity in natural language processing 1 code and data for reproducing our experiments are available here: https://github.com/navidrajabi/ emqa . ## problem description and settings we focus on the task of simplified minimal answer span selection over a gold passage: the inputs to the model include the full text of an article (the passage or context) and the text of a question (query). the goal is to return the start and end byte indices of the minimal span that completely answers the question. our models follow the current state-of-the-art in extractive question answering, relying on large multilingually pre-trained language models (in our case, multilingual bert @xcite ) and the task-tuning strategy of @xcite , which outperforms approaches like documen-tqa @xcite or decomposable attention @xcite . in all cases, we treat the official tydi qa development set as our test set, since the official test set is not public. 2 we provide concrete details (model cards, hyperparmeters, etc) on our model and training/finetuning regime in appendix a. to simulate the scenario of data-scarce adaptation of such a model to unseen languages, we will treat the tydi qa languages as our test, unseen ones. we will assume that we have access to (a) other qa datasets in more resource-rich languages (in particular, the squad dataset which provides training data in english), and (b) translation models between the languages of existing datasets (again, english) and our target "unseen" languages. in the experiments sections, we first focus on few-and zero-shot experiments ( §3) and then study the effects of language selection and budgetrestricted decisions on training data creation ( §4). evaluation we report f1 score on the test set of each language, as well as a macro-average excluding english (avg l ). in addition, to measure the expected impact on actual systems' users, we follow @xcite in computing a populationweighted macro-average (avg pop ) based on language community populations provided by ethnologue @xcite . ## is few-shot a viable solution? we first set out to explore the effect of the amount of available data on downstream performance. starting with baselines relying solely on englishonly squad, we implement a few-shot setting for fine-tuning on the target languages of tydi qa. 3 to our knowledge, this is the first study of its type on the tydi qa benchmark. the straightforward baseline simply provides zero-shot results on tydi qa after training only on english. we first study a monolingual few-shot setting. that is, we fine-tune the model trained on the english squad dataset, with only a small amount of data (10, 20, or 50 training instances) in the test language. due to space limitations, we only present results with 50 examples per language in we now turn to a multilingual few-shot setting. exactly as before, we assume a scenario where we only have access to a small amount of data in each language, but now we fine-tune using that small amount of data in all languages. for example, 10 training instances in each language result in training with 90 training examples over the 9 test languages. a sample of our experimental results are presented in simply adding 50 instances from each language we obtain an f1 score of 67.9 over the zero-shot baseline, an improvement of almost 7 percentage points which reduces the zero-full gap by 43.4%. we note that the total 450 training instances represent less than 1% of the full tydi qa training set! doubling that amount of data to 100 examples per language further increases downstream performance to an average overall f1 score of 71.7. going further to the point of adding 500 training instances per language (for a total of 4500 examples) leads to even larger improvements for an average f1 score of 76.7. that is, using less than 10% of the available training data we can reduce the average f1 score performance gap by more than 82%. for a few languages the gap reduction is even more notable, e.g., more than 92% for finnish. ## discussion we show that data augmentation through translation along with few-shot fine-tuning on new languages with a uniform budget allocation leads to a performance close to 98% of an approach using 10x more data, while producing more equitable models than other budget-constrained alternatives. the implications of our findings become clear with a counter-factual exploration. the gold passage portion of the tydi qa dataset includes around 87,000 annotated examples (50k for training across 9 languages and about 37k development and test samples). consider the scenario where, given this annotation budget, we maintain the same evaluation standards collecting 4k development and test examples per language, but we only collect 500 training examples per language. in that case, we could have created a much more diverse resource that would include at least 19 languages! now consider the expectation of the downstream accuracy in our counterfactual scenario: uniform budget allocation on 19 languages would lead to an average accuracy (f1 score) of around 78% (similar to our experiments). instead, under the (currently factual) scenario where we only have training data for 9 languages, the average accuracy for these 9 languages is around 80%, but the zero-shot expected average on the other 10 languages is 10 points worse -in that case, the overall average accuracy would be around 74%, 4 points lower than that of the egalitarian allocation scenario. hence, as long as the ideal scenario of collecting a lot of data for a lot of languages remains infeasible, we suggest that the community puts an additional focus on the linguistic diversity of our evaluation sets and use other techniques to address the lack of training data.