Building a S wedish Question-Answering Model (2020)

Abstract: High quality datasets for question answering exist in a few languages, but far from all. Producing such datasets for new languages requires extensive manual labour. In this work we look at different methods for using existing datasets to train question-answering models in languages lacking such datasets. We show that machine translation followed by cross-lingual projection is a viable way to create a full question-answering dataset in a new language. We introduce new methods both for bitext alignment, using optimal transport, and for direct cross-lingual projection, utilizing multilingual BERT. We show that our methods produce good Swedish question-answering models without any manual work. Finally, we apply our proposed methods on Spanish and evaluate it on the XQuAD and MLQA benchmarks where we achieve new state-of-the-art values of 80.4 F1 and 62.9 Exact Match (EM) points on the Spanish XQuAD corpus and 70.8 F1 and 53.0 EM on the Spanish MLQA corpus, showing that the technique is readily applicable to other languages.

## introduction the application of supervised machine learning approaches on reading comprehension tasks such as question answering has traditionally been unsuccessful due to a lack of large-scale datasets for training and the lack of powerful enough language models @xcite . in recent years however, important steps have been made in both areas with the introduction of large-scale datasets (squad @xcite , selqa @xcite ) and the paradigmshifting language model bert. while this has enabled impressive results for english question answering, there is still a lack of such large-scale datasets in other languages such as swedish. we therefore explore whether it is possible to train a model for swedish questionanswering using the existing english dataset. our two main approaches are to (1) fine-tune a multilingual bert model on the english squad (stanford question answering dataset) and see how well it generalizes to swedish, i.e. doing zero-shot learning, and to (2) machine-translate the english dataset into swedish and fine-tune a swedish bert model on it. we also evaluate various combinations of the two. as squad is based on retrieving the answer from a text, the main challenge with the translation to swedish consists in determining where the beginning and end positions of the answers are in the translated text, i.e. projecting the answer span onto the translated sentence. this is difficult as the translation may change the order of the words (e.g. "the 1973 oil crisis" → "oljekrisen 1973"), and the translation also changes depending on the context around it. we experiment with two approaches to the projection problem: one optimal transport-based solution that creates a word-alignment mapping between the english and the swedish sentence, and one deep learning-based solution that uses multilingual bert to find the position of the answer given translations of the english answer and a few words surrounding it, with varying amounts of surrounding words included; something we will refer to as a translation pyramid. we evaluate our model for swedish qa on our machine-translated versions of the squad dev set, but also apply the method on spanish to evaluate it on two human-made benchmarks, where we establish a new state-of-the-art for spanish qa systems. we make both the swedish and spanish datasets produced by our method freely available foot_0 . ← -----→ b i 2 related work @xcite trained a korean questionanswering model using a machine-translated squad in conjunction with a small manually annotated "seed" dataset of korean question-answer pairs, which is used to predict the translation certainty to remove poor translations from the dataset. their method for determining the answer span in the translated text is based on modifying the text to be translated by adding citation marks around the answer span. we found this to be unreliable (sometimes citation marks would be removed or shifted in the translation) and to compromise the translation quality. we therefore translate the original text as is, and instead propose a method for reliably finding the translated answer span afterwards. @xcite ) used an automatic method to translate squad into spanish. they use a custom-trained neural machine translation model to translate the dataset and then create word alignments using the methods described in @xcite to find the answer span in the translated text. this method is similar to our optimal transport method but uses a different way to find the word alignments. ## problem definition the extractive question answering task we want to solve is defined as follows. given a context paragraph c (for the squad dataset, these are paragraphs from wikipedia articles) and a question q, we want to extract the answer from the context paragraph c. this consists in predicting the start and the end position of the answer in c, so that our predicted answer becomes c start:end . the squad dataset essentially consists of tuples of (c, q, start, end) and the translation of the dataset requires not only retrieving the text translations c and q , but also finding the corresponding ground-truth start and end such that the answer is given by c start :end . this is what we refer to as the projection problem.