F o QA : A F aroese Question-Answering Dataset (2025)

Abstract: We present FoQA, a Faroese extractive question-answering (QA) dataset with 2,000 samples, created using a semi-automated approach combining Large Language Models (LLMs) and human validation. The dataset was generated from Faroese Wikipedia articles using GPT-4-turbo for initial QA generation, followed by question rephrasing to increase complexity and native speaker validation to ensure quality. We provide baseline performance metrics for FoQA across multiple models, including LLMs and BERT, demonstrating its effectiveness in evaluating Faroese QA performance. The dataset is released in three versions: a validated set of 2,000 samples, a complete set of all 10,001 generated samples, and a set of 2,395 rejected samples for error analysis.

## introduction recent nlp advancements, driven by the transformer architecture @xcite , have led to large-scale models that excel in understanding @xcite and generating @xcite human language. while many models are "massively multilingual" @xcite @xcite they often perform better on high-resource languages, leaving low-resource languages undersupported. furthermore, low-resource languages typically have limited access to native speakers who can serve as data annotators, making it difficult to create high-quality evaluation datasets. high-quality evaluation datasets are crucial for assessing and improving models for these languages, helping to measure performance and guide language technology development. extractive qa datasets @xcite are especially useful, as they simulate realworld applications like retrieval-augmented generation @xcite . creating these datasets traditionally requires substantial human effort, often involving multiple annotators for question generation and answer validation. standardising methods for creating these datasets can significantly advance technology for low-resource languages. our research addresses these challenges and makes the following key contributions: ## related work qa systems are divided into extractive and abstractive types @xcite . this work focuses on extractive qa, also known as reading comprehension, where text passages are paired with questions, and answers are directly extracted from the text. a well-known example of an extractive qa dataset is the stanford question answering dataset (squad), which includes over 100,000 qa pairs from wikipedia articles @xcite . in the case of icelandic, a language closely related to faroese, several qa datasets have been developed. @xcite introduced a cross-lingual open-domain qa system using machine-translated data, and the natural questions in icelandic, an extractive qa dataset, which demonstrates approaches applicable to other low-resource languages such as faroese @xcite . similarly, @xcite developed a method to gamify qa dataset creation. however, both approaches relied heavily on human question generation, which bottlenecked the dataset creation process. at the time of writing, few benchmark datasets exist for faroese. @xcite introduced named entity recognition foot_2 and semantic text similarity datasets foot_3 . the flores-200 dataset @xcite is another significant contribution to faroese benchmarks, being a multilingual parallel corpus covering over 200 languages, including faroese. additionally, @xcite introduced scala-fo, a linguistic acceptability dataset for faroese. despite these resources, a dedicated faroese qa dataset is still lacking, which this work aims to address. ## faroese setup we applied our methodology (section 3) to the faroese wikipedia foot_6 as the text source and gpt-4-turbo-2024-04-09 (openai, 2023) as the generative model, selected for its top performance on faroese tasks in the scandeval benchmark @xcite . to ensure non-trivial contexts, only articles with over 1,000 characters were included, i.e., 1675 articles in total and 655 articles used for the validated dataset. we set the model temperature to 1.0 and generated a maximum of 1,024 tokens, with a consistent random seed (4242) to maintain re-producibility. the system prompt we use is the following: you are a helpful faroese question answering dataset generator. the only language you know is faroese. while we did not explore faroese-language prompting or prompt variations in this study, such modifications could potentially improve the effectiveness of our approach. as our primary focus was developing a question-answering dataset for faroese, we leave prompt optimisation for future work. the following prompt was used for generating qa pairs: the following is a wikipedia article in faroese. ¡article¿ {article} ¡/article¿ generate 2 to 10 questions about the article, depending on the length of the article, all of which answered in the article. you also have to supply answers to the questions, and the answers have to appear exactly as written in the article (including same casing). the answers should only contain the answers themselves, and not the surrounding sentencekeep the answers as short as possible. the answers have to be different from each other. all your questions and answers must be in faroese. your answer must be a json dictionary with the key "results", with the value being a list of dictionaries having keys "question" and "answer". lastly, we use the following prompt to re-write the questions: the following is a faroese question. ¡question¿ {question} ¡/question¿ re-write the question, preserving the meaning, using synonyms or a different (valid) word order. your question must be in faroese. your answer must be a json dictionary with the key "question". in both prompts, we replace {article} and {question} with the actual wikipedia article and the generated question, respectively. ## evaluation we evaluated several models on the dataset. since we ensured that all answers appear exactly as in the documents, this allows us to evaluate both encoder models and decoder models on the dataset. we evaluate both faroese and massively multilingual models on foqa, the results of which can be found in we also evaluated the model used to generate the dataset, gpt-4-turbo-2024-04-09, on 10 full openai model id: gpt-4-1106-preview 11 full openai model id: gpt-4o-2024-05-13 12 full openai model id: gpt-4o-mini-2024-07-18 13 https://hf.co/meta-llama/llama-3.1-8b 14 https://hf.co/ai-sweden-models/ gpt-sw3-6.7b-v2 the corrected samples, before and after the correction. this was to test whether the model is biased towards its own generated questions, or whether it generalises to the corrected ones as well. surprisingly, the model ended up performing significantly better foot_13 on the corrected samples, rather than the samples it had generated itself. ## discussion and future work our evaluation of the foqa dataset reveals insights into the performance of various language models on faroese qa tasks. gpt-4-turbo and gpt-4o achieved the highest performance scores in our evaluation, though further research would be needed to understand whether this indicates genuine faroese language comprehension or other factors like strong general question-answering capabilities. this finding suggests promising directions for low-resource language processing, while highlighting the need for more detailed investigation into how these models handle faroese specifically. an important observation from our annotator indicates that most errors in the generated questions were grammatical in nature rather than contextual. this suggests a need for dedicated benchmarks specifically measuring grammatical correctness of llms in faroese, which would complement foqa's focus on qa capabilities. early question answering datasets like squad faced criticism that their questions were too simplistic, often directly mirroring the source text structure. later datasets like tydi qa @xcite and natural questions in icelandic @xcite addressed this by having annotators create natural questions first, which were later matched to source material. this approach prevented the tight coupling between question phrasing and source text that can make questions artificially easy. following this insight, we implemented question rephrasing in our methodology. however, we acknowledge that we did not specifically measure performance differences between original and rephrased questions, which would require separate evaluation sets. we found that encoder models like mdeberta-v3 @xcite , fobert and scandib-ert @xcite per-form significantly worse than the decoder models, but that could simply be explained by the fact that these models differ in sizes by several orders of magnitude. a controlled experiment will need to reveal whether architectural choices are the real cause for difference in performance or whether it is due to other reasons such as parameter count. a performance gap has been observed between encoder-type models and decoder-type models across other languages and @xcite suggests that certain architectures may be inherently better suited for specific language processing tasks. for future work, we propose evaluating larger open models, such as 70b parameter models and even larger ones like llama 3.1 405b @xcite . additionally, assessing the performance of claude 3.5 sonnet @xcite would be valuable, given its strong performance on icelandic nlp tasks foot_14 since icelandic is a language closely related to faroese. ## conclusion we introduced foqa, the first faroese extractive question-answering dataset, containing 2,000 qa pairs. all samples underwent initial validation by one annotator, followed by a second validation phase where the correct samples were split equally between the original annotator and a second annotator. our evaluation reveals significant performance gaps between decoder-based llms and encoder models, with gpt-4-turbo achieving the highest f1 score of 77.6, while encoder models like mdeberta-v3 and scandib-ert scored around 30. notably, our analysis of question types shows a diverse distribution across categories, with people-related questions comprising the largest portion at 33.95%. the dataset's manual validation process identified common error patterns, including grammatical gender mistakes and icelandicisms, providing valuable insights for future faroese language model development. the foqa dataset serves as valuable benchmark for evaluating faroese language understanding. additionally, our contributions include a semi-automated methodology for creating extractive qa datasets for low-resource languages.