Danoliteracy of Generative Large Language Models (2025)

Abstract: The language technology moonshot moment of Generative Large Language Models (GLLMs) was not limited to English: These models brought a surge of technological applications, investments, and hype to low-resource languages as well. However, the capabilities of these models in languages such as Danish were, until recently, difficult to verify beyond qualitative demonstrations due to a lack of applicable evaluation corpora. We present a GLLM benchmark to evaluate Danoliteracy, a measure of Danish language and cultural competency across eight diverse scenarios such as Danish citizenship tests and abstractive social media question answering. This limited-size benchmark was found to produce a robust ranking that correlates to human feedback at \rho \sim 0.8 with GPT-4 and Claude Opus models achieving the highest rankings. Analyzing these model results across scenarios, we find one strong underlying factor explaining 95% of scenario performance variance for GLLMs in Danish, suggesting a g factor of model consistency in language adaptation.

## introduction benchmarks shape technologies. by acting as normative guidelines for technology applications, benchmarks imply directions of research and development that ultimately impact users @xcite . gllms specifically have emerged as a technology with near-universal impact, including lowerresource languages such as danish @xcite . if the challenging task of general gllm evaluation is not extended to low-resource languages, practitioners start from scratch for each model use case, inhibiting practical adoption or possibly resulting in risky, undertested implementations. we take up the challenge of creating a gllm evaluation benchmark for danish, a north germanic language spoken by 6 million people, primarily in the nordic country of denmark. as depicted in figure 1 , our approach is to create a compilation of small-scale, diverse evaluation scenarios combined into one general benchmark to reveal model danoliteracy. by danoliteracy we refer to the level of gllm real-world knowledge (rwk), natural language understanding (nlu), and natural language generation (nlg) in danish. this paper presents the resulting danoliterate benchmark, describing evaluation methods, datasets and model results. we analyze these results with the goals of validating and exploring evaluation methodology. an important part of this analysis is to investigate the feasibility of such evaluation: does this small-scale, language-specific approach achieve a significant ranking of gllms? even if a non-spurious leaderboard can be discerned from the result, it is not enough to validate the benchmark which might actually show something orthogonal to danoliteracy. thus, as a benchmark validation tool, we additionally present a user survey, collecting the preferences of danish speakers when interacting with hidden pairs of gllms in an arena a/b test setup. the availability of a suite of meaningful benchmark results allows us to investigate gllm behavior: initially, we explore which specific models are most danoliterate and how different types of gllms compare. beyond that, we are particularly interested in capability consistency across tasks: if a gllm performs strongly in one danish usecase scenario, does this performance generalize to other danish scenarios across different domains and objectives? we hope so. if the answer is no, practitioners are without general results to trust, requiring a full model re-evaluation for each downstream use. however, if capability consistency is present, we should be able to find a single underlying axis that correlates with performance across diverse scenarios. such a general dimension of danoliteracy can be compared to the g factor of general human intelligence @xcite . if one significant, main factor is found, it implies a level of stability that can help guide the expectations of practitioners across gllm implementations in varying and even novel danish tasks. the contributions presented in this paper can be summarized as follows: ## conclusions based on the ability to robustly discover model groupings at different danish capability levels and correlate these rankings with human feedback, we conclude that a scenario compilation approach can meaningfully reveal gllm capabilities. we show that, in danish, open-weights gllms currently lag behind large, closed, multilingual, instruct-tuned models, such as gpt-4 and claude opus. for our evaluation setup, we observe one underlying factor in model capability across the diverse test scenarios. this observation is supported by similar structures in other danish, english and multilingual scenario compilations which we consider a positive result for low-resource evaluation: by using curated and language-specific scenarios, the general landscape of gllm capabilities for a given low-resource language can be meaningfully inferred even if resources limit the scale.