MAUPQA : Massive Automatically-created P olish Question Answering Dataset (2023)

Abstract: Recently, open-domain question answering systems have begun to rely heavily on annotated datasets to train neural passage retrievers. However, manually annotating such datasets is both difficult and time-consuming, which limits their availability for less popular languages. In this work, we experiment with several methods for automatically collecting weakly labeled datasets and show how they affect the performance of the neural passage retrieval models. As a result of our work, we publish the MAUPQA dataset, consisting of nearly 400,000 question-passage pairs for Polish, as well as the HerBERT-QA neural retriever.

## introduction open-domain question answering (openqa) systems aim to provide answers to questions from a variety of topics, using a large collection of passages as a knowledge base. recently, the development of such systems has been accelerated by the release of several large-scale question-passage datasets, such as ms marco @xcite , triv-iaqa @xcite , and natural questions @xcite . these datasets enabled the training of neural passage retrieval models (e.g. dense passage retrieval, @xcite , which can select passages from a knowledge base that are the most likely to contain the answer to the question. however, the annotation of such datasets is a time-consuming and expensive process, which limits their availability for less popular languages @xcite . another limiting factor is the availability of real questions. datasets like ms marco or natural question consist of real questions asked by search engine users. for less popular languages (like polish), such a source of questions is not available. this leads to two alternatives: either to train a system on a small dataset (which might not be sufficient for the model to reach its full potential) or to create a dataset automatically. the first approach was recently described by @xcite who published the polqa dataset which consists of 7,000 trivia questions and 87,525 manually annotated passages. in this work, we experiment with the latter approach and show how different methods for automatic data collection can impact the performance of the neural passage retrieval models. our contributions can be summarized as follows: weakly-labeled datasets over the years, many techniques were developed for the automatic creation of weakly-labeled datasets. one general idea is to use a weak model to automatically label the unlabeled dataset @xcite . in the case of openqa, either simple lexical models like bm-25 (robertson and zaragoza, 2009) or more powerful neural models are used to retrieve relevant passages for given questions. to further improve the accuracy of retrieved examples the passages can be filtered out using cross-encoders @xcite or answers (if available, @xcite . however, the above method can only be used if the source of questions is available. if that is not the case, then questions can be automatically created. either using templates @xcite or trained models @xcite . another line of work takes advantage of existing datasets and translates them automatically to other languages @xcite . the quality of the machine translation model directly impacts the quality of the created dataset @xcite . polish openqa datasets few datasets exist for polish openqa. the first published dataset for passage retrieval was the czy wiesz? dataset @xcite . it is a collection of 4,721 questions from the did you know? section on polish wikipedia out of which only 250 questions were manually labeled with a relevant passage. @xcite later annotated an additional 1,070 questions with relevant passages. the polqa dataset @xcite is a recently introduced dataset for polish openqa. it consists of 7,000 trivia questions and 87,525 manually annotated passages (both positive and hardnegative). even though the number of questionpassage pairs is impressive for a less popular language like polish, the number of unique questions is still rather limited. ## maupqa dataset the maupqa dataset consists of seven parts. four of them are created from scratch (czy wiesz?, gengpt3, templates, wikidef), and the other three are based on existing resources (mkqa, mtnq, mfaq). ## evaluation we use the tevatron library @xcite to train the neural retriever. for each dataset, we finetune the herbert base model @xcite for 2,000 steps, with batch size 128 and learning rate 10 -5 . otherwise, we use default parameters. we experimented with training models for 5,000 steps but it didn't increase the performance. we use a single hard-negative per question when training on polqa dataset. for other datasets, we only use in-batch random negatives as they don't contain hard-negatives. for evaluation, we use accuracy@10 (i.e. is there at least one relevant passage within the top 10 retrieved passages) and ndcg@10 (i.e. score of each relevant passage within the top 10 retrieved passages depends descending on its position, @xcite ). each model is evaluated on the polqa development dataset. we use provided polish wikipedia dump as a knowledge base. ## results the baseline retriever trained using manually annotated polqa dataset achieves 60.8% accuracy@10 (see as expected, the best model is mtnq with an accuracy of 58.5%. it is the second largest dataset, similarly to polqa it contains mostly trivia questions, and is based on manually labeled questionpassage pairs. comparably large mfaq dataset obtains much lower performance (38.7%), probably due to domain mismatch as otherwise, its quality is higher than mtnq. the mkqa, which is a manually translated subset of nq dataset achieves surprisingly good results (51.5%). it is unexpected considering that only 21% of its passages are actually relevant. the second best result (54.2%) is achieved by the gengpt3 dataset. despite the diverse nature of the questions from different domains, and the relatively modest size of the dataset, it exhibits a remarkable level of quality that allows it to serve as a reliable source for training a passage retriever. the third best result (54.1%) is scored by czywiesz-v2 dataset. the other two datasets created based on wikipedia perform much worse, templates obtains accuracy of 45.9% and wikidef only 19.9%. it is also the lowest result of all datasets, probably due to its low diversity. none of the datasets is perfect and each of them has its own disadvantages. however, the retriever trained on all of them results in better performance than the manually annotated dataset (61.2% vs 60.8%). if we further fine-tune the retriever pretrained on maupqa, we obtain the state-of-the-art result for polish passage retrieval of 62.7%. we name this retriever herbert-qa and release it alongside the created datasets. ## conclusion in this work, we present maupqa, the largest polish qa dataset with almost 400k question-passage pairs. even though the dataset is created automatically it achieves competitive results on the polish we believe our work will benefit the polish nlp community, both by publishing a maupqa dataset, as well as the state-of-the-art passage retrieval model. our study also lays a path for other languages on how to construct similar datasets.