W iki QA - IS : Assisted Benchmark Generation and Automated Evaluation of I celandic Cultural Knowledge in LLM s (2025)

Abstract: This paper presents WikiQA-IS, a novel question-answering dataset focusing on Icelandic culture and history, along with an automated pipeline for dataset generation and evaluation. Leveraging GPT-4 to create questions and answers based on Icelandic Wikipedia articles and news sources, we produced a high-quality corpus of 2,000 question-answer pairs. We introduce an automatic evaluation method using GPT-4o as a judge, which shows strong agreement with human evaluations. Our benchmark reveals varying performances across different language models, with closed-source models generally outperforming open-weights alternatives. This work contributes a resource for evaluating language models’ knowledge of Icelandic culture and offers a replicable framework for creating similar datasets in other cultural contexts.

## introduction recent advancements in natural language processing (nlp) have led to significant improvements in question-answering systems, particularly through large language models (llms) @xcite . while these models show impressive capabilities, they can generate incorrect or fabricated information, a phenomenon known as hallucination @xcite . this makes it crucial to systematically measure how much factual knowledge these models actually possess about specific domains, such as individual cultures or topics. current evaluation methods often lack domain-specific benchmarks, making it difficult to assess models' true understanding of particular cultural contexts. this paper presents an automated approach to generate and evaluate questions and answers, using icelandic culture and history as a case study. icelandic, despite its small speaker base, has a rich literary and historical heritage. however, creating comprehensive qa datasets for such domains is resource-intensive if done manually. while prior work on icelandic qa datasets has focused on language and reading comprehension @xcite @xcite @xcite , there remains a need for a dataset testing knowledge of culture and history in an open-ended fashion. our research introduces a method leveraging an llm to automate the generation of highquality questions and answers based on icelandic wikipedia articles, inspired by previous work extracting knowledge from wikipedia @xcite and work on automatic qa dataset creation @xcite . this approach addresses the challenge of creating largescale datasets for low-resource languages and extends the application of language models to cultural and historical knowledge evaluation. the main contribution of this paper is the wikiqa-is corpus foot_0 along with the pipeline used to generate the corpus and the automatic evaluation approach foot_1 . this research not only contributes to create benchmarks focusing on icelandic cultural knowledge but also offers a replicable framework adaptable to other languages and cultural contexts. ## discussion our study demonstrates the effectiveness of leveraging llms for creating specialized questionanswering datasets. the significant difference in retention rates between wikipedia-based questions (89.8%) and news articles (36.5%) underscores the importance of source material selection the performance analysis reveals a clear hierarchy among models, with closed-source models generally outperforming open-weights alternatives. this gap highlights ongoing challenges in democratizing advanced language understanding capabilities for specialized domains. our automatic evaluation method shows promise for efficient, large-scale assessment, though it may be influenced by the judge model's capabilities and biases. future work could explore expanding source materials to reduce potential biases, and develop more comprehensive categorization of questions to uncover specific areas of model strength and weakness. while our method provides valuable insights into models' cultural knowledge, it represents just one facet of measuring world knowledge, and complementary approaches could offer a more holistic assessment of cultural understanding.