G erman Q u AD and G erman DPR : Improving Non- E nglish Question Answering and Passage Retrieval (2021)

Abstract: A major challenge of research on non-English machine reading for question answering (QA) is the lack of annotated datasets. In this paper, we present GermanQuAD, a dataset of 13,722 extractive question/answer pairs. To improve the reproducibility of the dataset creation approach and foster QA research on other languages, we summarize lessons learned and evaluate reformulation of question/answer pairs as a way to speed up the annotation process. An extractive QA model trained on GermanQuAD significantly outperforms multilingual models and also shows that machine-translated training data cannot fully substitute hand-annotated training data in the target language. Finally, we demonstrate the wide range of applications of GermanQuAD by adapting it to GermanDPR, a training dataset for dense passage retrieval (DPR), and train and evaluate one of the first non-English DPR models.

## introduction & related work research on non-english machine reading for question answering (qa) suffers from limited availability of annotated non-english datasets. with the english squad dataset @xcite as a role model, there are only a few resources of similar format, such as the french fquad @xcite , the korean korquad @xcite , and the russian sberquad @xcite datasets. as an alternative, there are machine-translated datasets for training @xcite . with further improving performance of translation models, machinetranslating datasets is a promising direction for the future as it saves the costs of manual annotations. however, translations can also be a source of errors (some translated answers contain non-negligible errors) and can have a negative impact on the variety of language use that might limit model performance. another line of research uses multilingual models such as mbert @xcite , xlm @xcite or xlm-roberta @xcite trained on english qa datasets for zero-shot language transfer to the target domain. while these models perform astonishingly well on qa in unseen languages, they do not perform as well as on english qa @xcite @xcite . furthermore, multilingual models are much larger than their monolingual counterparts, rendering them unsuitable for most production systems where memory consumption and query speed matter. besides qa, a highly related task called "passage retrieval" has been worked on recently. passage retrievers can be used for document retrieval, but also play an important role when scaling extractive qa models to large document bases (open-domain qa). dense passage retrieval (dpr) @xcite uses two separate, bert-based encoders @xcite for questions and passages. in addition to the positive passages (contexts) as contained in existing squadformatted datasets, the training of dpr leverages hard negative passages that are automatically selected for each query and an optimization trick called in-batch negatives. in-batch negatives optimization assumes positive passages of one annotated question to be negative passages for a different question. by adding hard negatives selected from the whole german wikipedia to german-quad, we create a training dataset in dpr format, which we call germandpr. our contributions can be summarized as follows: first, we present a new dataset of extractive question/answer pairs, germanquad, and show that training on this dataset achieves a new state of the art for german qa. we also present the first multi-way annotated german test set for a more fair evaluation of qa models. second, we evaluate the efficacy of a technique to improve the annotation process, where simple questions are reformulated. third, we present a new dataset for dense passage retrieval, germandpr, and demonstrate another application scenario of germanquad by converting it to a different format and enriching it with additional data. this conversion approach could also be used with other qa datasets. to the best of our knowledge, there is no other non-english dpr model published at the point of writing, presumably due to the lack of suitable training datasets. our annotations and trained models are released under the creative commons attribution 4.0 international license (cc-by 4.0). 1,2,3 2 germanquad dataset creation the creation of germanquad is inspired by insights from existing datasets as well as our domain labelling experience from several other projects. we combine the strengths of squad, such as high out-of-domain performance, with self-sufficient questions that contain all relevant information for open-domain qa as in the natural questions dataset @xcite . our training and test datasets do not overlap like other popular datasets @xcite and include complex questions that cannot be answered with a single entity or only a few words. ## germandpr dataset creation the squad format is not suitable for training dpr models. dpr requires additional data and a different data format. in dpr, each question/answer pair needs to be accompanied not only by the context that contains the answer (positive context) but also by other contexts that are semantically similar to the question but do not contain the answer (hard negative contexts). we take germanquad as a starting point and add hard negatives from a dump of the full german wikipedia. following the approach by @xcite , we use bm25 to find passages in wikipedia that are most similar to the question but do not contain the answer string. these passages are then added as hard negatives. to this end, we split each wikipedia article into passages as described for the creation of germanquad in section 2.1. after indexing all passages with elasticsearch, we use its bm25 implementation to retrieve three hard negatives for each question. however, we do not use the full ger-manquad dataset but only question/answer pairs that have been labelled by the annotators as short answers. the reason is that hard negatives must not contain the answer, which we take into account by checking for the exact answer string. this exact string matching cannot catch rephrased versions of the answer, which we found is problematic especially for long answers but rarely for short answers. figure 2 exemplifies the data format. the final germandpr dataset comprises 9275 question/answer pairs in the training set and 1025 pairs in the test set. for each pair, there are one positive context and three hard negative contexts. ## experiments & results the following experiments study the impact of the number of training samples on model performance for extractive qa, the benefits of reformulating previously annotated questions, and the model performance compared to related work and baselines for models trained on germanquad and germandpr. ## conclusion and future work only a few resources for non-english qa and passage retrieval are available. in this paper, we presented two new datasets, germanquad and ger-mandpr, to foster research in this area. we described the data annotation process in detail so that other researchers can reproduce the approach for other languages. our experiments showed that training qa models on germanquad achieves a new sota for german qa and that training dpr models on germandpr drastically outperforms a bm25 baseline. a promising path for future work is to research new evaluation metrics for qa. especially for longer answers in german, we find that current metrics penalise too much if a predicted answer contains some extra words although being semantically similar to the ground-truth answer. future metrics could take into account the semantic similarity of predicted and ground-truth answers.