N or Q u AD : N orwegian Question Answering Dataset (2023)

Abstract: In this paper we present NorQuAD: the first Norwegian question answering dataset for machine reading comprehension. The dataset consists of 4,752 manually created question-answer pairs. We here detail the data collection procedure and present statistics of the dataset. We also benchmark several multilingual and Norwegian monolingual language models on the dataset and compare them against human performance. The dataset will be made freely available.

## introduction machine reading comprehension is one of the key problems in natural language understanding. the question answering (qa) task requires a machine to read and comprehend a given text passage, and then answer questions about the passage. in recent years, considerable progress has been made toward reading comprehension and question answering for english and several other languages @xcite . in this paper, we present norquad: the first norwegian question answering dataset for machine reading comprehension. the dataset consists of 4,752 question-answer pairs manually created by two university students. the pairs are constructed for the task of extractive question answering aimed at probing machine reading comprehension (as opposed to information-seeking purposes), following the methodology developed for the squad-datasets @xcite . the creation of this dataset is an important step for norwegian natural language processing, considering the importance and popularity of reading comprehension and question answering tasks in the nlp community. in the following we detail the dataset creation (section 3), where we describe the passage selection and question-answer generation, present relevant statistics for the dataset and provide an analysis of human performance including sources of disagreement. in order to further evaluate the dataset as a benchmark for machine reading comprehension, we perform experiments (section 4) comparing several pre-trained language models, both multilingual and monolingual models, in the task of question-answering. we also compare models against human performance for the same task. we further provide an analysis of performance across the source data domain and annotation time and present the results of manual error analysis on a data sample. the stanford question answering dataset (squad) 1.1 @xcite along with squad 2.0 @xcite which supplements the dataset with unanswerable questions are the largest extractive qa datasets for english. squad 1.1 contains 100,000+ questions and squad 2.0 contains 50,000 questions. several squad-like datasets exist for other languages. the french question answering dataset (fquad) is a french native reading compre-hension dataset of questions and answers on a set of wikipedia articles that consists of 25,000+ samples for version 1.0 and 60,000+ samples for version @xcite . the german germanquad is a dataset consisting of 13,722 question-answer pairs created from the german counterpart of the english wikipedia articles used in squad @xcite . the japanese question answering dataset (jaquad) consists of 39,696 question-answer pairs from japanese wikipedia articles @xcite . the korean question answering dataset (korquad) consists of 70,000+ human-generated questionanswer pairs on korean wikipedia articles @xcite . the russian sberquad consists of 50,000 training examples, 15,000 development, and 25,000 testing examples @xcite 2 . to the best of our knowledge there are no extractive question answering datasets available for the other nordic languages, i.e., danish or swedish. ## dataset creation we collected our dataset in three stages: (i) selecting text passages, (ii) collecting question-answer pairs for those passages, and (iii) human validation of (a subset of) created question-answer pairs. in the following, we will present these stages in more detail and provide some statistics for the resulting dataset as well an analysis of disagreements during human validation. @xcite reported that the absolute majority of available qa datasets target only one domain with rare exceptions. to provide some source variation in our dataset, considering our limited resources, we decided to create question-answer pairs from passages in two domains: wikipedia articles and news articles. ## experiments in this section, we assess the use of the norquad dataset as a benchmark for norwegian machine reading comprehension. given the small size of the dataset, compared to many other qa datasets, one important question to assess will be the level of performance that can be obtained with less than 5,000 question-answer pairs. to evaluate models, we use two metrics which are used to evaluate performance on most squadlike datasets: exact match and f1 score, as described in section 3.4. we split the norquad dataset randomly into three sets: training (80%), validation (10%), and test (10%). we split datasets to the abovementioned fractions separately for wikipedia and news datasets to observe if performance will differ depending on domain. the test sets consist of human validated question-answer pairs, hence we may compare models' results to human performance on the same data. the human validation process is described in section 3.4. we establish benchmarking experiments using a set of different pre-trained language models, outlined below. due to the small size of norquad, as compared to other squad-like datasets, we run all configurations five times with different random seeds and report the mean and standard deviation from these experiments. details on selected hyperparameters are located in appendix a.1 4.1 baseline models norwegian models. we compare two monolingual transformer models based on the architecture from bert @xcite : the norbert2, originating from the initiative started in norlm @xcite , and the nb-bert model from @xcite . ## error analysis in this section, we have a more detailed look at how the models performed on our dataset. we sampled 60 errors from the system output. the model in question is nb-bert. we found that in 57% of cases there is at least some overlap between the prediction and the annotated span. in 85% of instances the predicted answers are grammatically and semantically viable phrases, but are not factually the correct answer to the current question. within this category there are cases of questions where the answer is the same type of entity as the one asked for in the question (e.g. a person, but the wrong person), and in other cases the entity is of a different type (e.g. a number, instead of a person). in 13% of cases the predicted phrase could be considered a good answer to the question, but the annotator made a different selection. this means that the question must be considered ambiguous and that the system is not necessarily to blame for the error, but rather the annotation itself, because according to the annotation guidelines each question should have only one possible answer (see section 3.2.2). please see further discussion on this type of ambiguity in section 3.5 on analysis of disagreement between annotators. ## conclusion in this paper, we presented norquad-the first question answering dataset for norwegian. we collected passages from norwegian wikipedia articles and a collection of norwegian news texts and manually created over 4,700 questions. in the experiments, we fine-tuned several pre-trained language models with norquad and found that the best performing model achieved 69.68% for em and 81.27% for f1 score on the entire test set. averaged human performance on the test set was 78.13% for em and 91.14% for f1. the dataset and our experiments are available at https:// github.com/ltgoslo/norquad . furthermore, we presented human validation of the part of the dataset. we noticed that annotators and models performed better on the news dataset and we performed experiments to find out whether the time of annotation influences the performance. we measured performance of annotators and nb-bertthe best performing model. we found that both annotators and the model perform better on the second half of the datasets meaning that the annotators got better and more consistent in their question-answer pairs over time. while it is clear from our experiments that the number of question-answer pairs are sufficient to achieve a decent performance, improvements are certainly possible both in terms of size and data quality. to improve the dataset quality, one possible avenue for future work is to collect multiple answers for the questions as was done in squad @xcite . another possible extension in the future is the addition of unanswerable questions @xcite to the dataset.