A Collection of Question Answering Datasets for Norwegian (2025)

Abstract: This paper introduces a new suite of question answering datasets for Norwegian; NorOpenBookQA, NorCommonSenseQA, NorTruthfulQA, and NRK-Quiz-QA. The data covers a wide range of skills and knowledge domains, including world knowledge, commonsense reasoning, truthfulness, and knowledge about Norway. Covering both of the written standards of Norwegian – Bokmål and Nynorsk – our datasets comprise over 10k question-answer pairs, created by native speakers. We detail our dataset creation approach and present the results of evaluating 11 language models (LMs) in zero- and few-shot regimes. Most LMs perform better in Bokmål than Nynorsk, struggle most with commonsense reasoning, and are often untruthful in generating answers to questions. All our datasets and annotation materials are publicly available.

## introduction an essential part of developing language models (lms) is benchmarking -i.e., a systematic evaluation of models on standardized datasets to assess their generalization abilities and limitations, enabling a fair comparison across various criteria @xcite . one of the well-established benchmarking areas is question answering (qa), which tests the lm's ability to apply knowledge acquired from diverse domains to answer user questions @xcite @xcite . while there is a rich ecosystem of qa resources for typologically diverse languages @xcite , a significant gap remains for lesserresourced languages @xcite , including norwegian. existing norwegian qa datasets primarily focus on the machine reading comprehension task, limiting the evaluation scope of lm's abilities in norwegian language understanding and generation @xcite @xcite . furthermore, prior work relies on english-to-norwegian machine translation as the dataset creation method @xcite , which fails to capture the linguistic nuances and aspects of history, geography, and culture that are relevant to the end user. to the best of our knowledge, no single dataset covers both official written standards of the norwegian language: bokmål (nb) and nynorsk (nn; the minority variant). to address this gap, we introduce four new qa datasets in both norwegian nb and nn: noropenbookqa foot_0 , norcommonsenseqa foot_1 , nortruthfulqa 3,4 , and nrk-quiz-qa foot_4 . our datasets are designed to evaluate the lm's norwegian-specific & world knowledge, common sense reasoning abilities, and truthfulness in the form of multiple-choice and free-form qa. the 10.5k question-answer pairs are created by a team of native norwegian speakers through manual translation and localization of english-oriented datasets -openbookqa @xcite , commonsenseqa @xcite , and truth-fulqa @xcite -with a dedicated effort to also create novel norwegian-specific examples from scratch. nrk-quiz-qa comprises examples from more than 500 quizzes published by nrk, the national public broadcaster in norway. our main contributions are summarized as follows: (i) we create a collection of four qa datasets that target the least addressed qa directions for norwegian; (ii) we evaluate 11 publicly available lms that support norwegian in zero-and few-shot 2 related work ## datasets this section outlines our approach to adapting and localizing english-oriented qa resources to the specific contexts of norwegian society, culture, and knowledge. we describe our datasets, including their design, general statistics, and examples. ## results this section describes our empirical evaluation results, which are summarized in in contrast, norwai-mistral-7b, normistral-7b-warm, viking-13b, and mistral-7b perform consistently well in most evaluation configurations. notably, mistral-7b performs best on nortruthfulqa multiplechoice and noropenbookqa, which we attribute 22 github.com/ltgoslo/norqa to strong cross-lingual generalization abilities due to the high quality of the pretraining corpus. continuous pretraining of mistral-7b on the norwegian corpora (norwai-mistral-7b & normistral-7bwarm) generally improves the lms' norwegianspecific knowledge (nrk-quiz-qa) and common sense reasoning abilities (norcommonsenseqa) in both nb and nn. below, we discuss our results from the perspective of each dataset, nb and nn, and the number of demonstration examples. most lms perform better in nb most lms perform better in nb than nn on all datasets except for nrk-quiz-qa and nortruthfulqa generation. the accuracy δ-scores range from 5% to 8% on norcommonsenseqa (e.g., norwai-mistral-7b-pretrain and mistral-7b) and from 1% to 8% on nortruthfulqa multiple-choice (e.g., norgpt-3b and norwai-mistral-7b). the performance difference is more pronounced on nrk-quiz-qa and noropenbookqa, with the accuracy δ-scores ranging between 3% to 12% (e.g., norllama-3b and norwai-llama2-7b) and 1% and 18% (e.g., norgpt-3b with @xmath0. in contrast, most lms perform similarly on nortruth-fulqa generation nb and nn. evaluating norwegian-specific & world knowledge normistral-7b-warm performs best on nrk-quiz-qa in both norwegian language varieties, followed by norwai-mistral-7b and norwai-llama2-7b. norwai-mistral-7b-pretrain performs on par with norllama-3b and norgpt-3b, while the other lms pretrained from scratch (norbloom-7b/normistral-7b-scratch, viking-7b/13b) perform significantly better in most evaluation regimes. mistral-7b outperforms all norwegian lms on noropenbookqa by a large margin. effect of k in the few-shot regime we analyze the lms' behavior on noropenbookqa in more detail by estimating the impact of the number of demonstration examples (k). our key findings here are: (i) norllama-3b, norgpt-3b, viking-13b, normistral-7b-scratch, and norwai-mistral-7b-pretrain demonstrate more limited incontext learning abilities, showing only minor performance improvements as k increases; (ii) the highest number of demonstrations (@xmath1, and many lms achieve their highest scores with 4-shot learning (@xmath2; (iii) norbloom/normistral-7bscratch, norwai-mistral-7b-pretrain, and viking-7b demonstrate greater sensitivity to k in nn compared to other lms. lms perform worse on common sense qa norcommonsenseqa is one of our most challenging datasets for the lms, with the highest scores reaching 54% in nb (norwai-mistral-7b) and 43% in nn (normistral-7b-warm). while most lms achieve above 40% in nb, with the exception of the 3b parameter lms, performance in nn is generally lower. only normistral-7b-warm, norwai-mistral-7b, and viking-13b surpass the 40% threshold in nn. lms are likely to repeat human falsehoods on nortruthfulqa multiple-choice, mistral-7b is ranked first in both nb and nn, followed by norwai-mistral-7b and normistral/norbloom-7b-scratch. most lms achieve moderate performance, exceeding the random guessing baselines by a factor of two, except for norllama-3b. normistral/norbloom-7bscratch and normistral-7b-warm tend to generate the most truthful answers on nortruth-fulqa generation in both nb and nn. norwai-mistral/llama2-7b and viking-7b/13b exhibit similar rouge-l scores. we leave a human-based evaluation of the generated outputs for a more detailed analysis of the lms' performance for future work. ## conclusion and future work this paper introduces a collection of four new qa datasets for norwegian nb and nn created by native speakers and tailored to evaluate the lms' abil-ities with respect to the norwegian language and culture. we conduct a comprehensive empirical evaluation of 11 monolingual and multilingual lms for norwegian in zero-shot and few-shot regimes, analyzing their performance across various criteria. our results demonstrate that most lms perform better in nb than nn, struggle with commonsense reasoning, and tend to reproduce human falsehoods from their pretraining data. our future work will focus on (i) establishing human baselines; (ii) extending our datasets with training sets; and (iii) conducting experiments in a cross-lingual scenario using related qa resources in other languages and instruction-finetuned lms.