SnakModel : Lessons Learned from Training an Open Danish Large Language Model (2025)

Abstract: We present SnakModel, a Danish large language model (LLM) based on Llama2-7B, which we continuously pre-train on 13.6B Danish words, and further tune on 3.7M Danish instructions. As best practices for creating LLMs for smaller language communities have yet to be established, we examine the effects of early modeling and training decisions on downstream performance throughout the entire training pipeline, including (1) the creation of a strictly curated corpus of Danish text from diverse sources; (2) the language modeling and instruction-tuning training process itself, including the analysis of intermediate training dynamics, and ablations across different hyperparameters; (3) an evaluation on eight language and culturally-specific tasks. Across these experiments SnakModel achieves the highest overall performance, outperforming multiple contemporary Llama2-7B-based models. By making SnakModel, the majority of our pre-training corpus, and the associated code available under open licenses, we hope to foster further research and development in Danish Natural Language Processing, and establish training guidelines for languages with similar resource constraints.

## introduction the landscape of large language models (llms) has seen rapid expansion, with an increasing trend towards open-weight releases for a broader range of languages. notable english-centric examples include pythia @xcite , vicuna @xcite , @xcite , qwen @xcite , llama2 @xcite , llama3 @xcite , olmo @xcite , and phi @xcite . simultaneously, recent efforts have extended llms to multilingual settings, including models such as mt5 @xcite , bloom @xcite , aya @xcite , @xcite , and eurollm @xcite . as anglocentric and/or multilingual llms have nonetheless continued struggling to adapt to lowerresource settings-especially with respect to pragmatic and sociolinguistic factors @xcite @xcite -there is growing interest in language-specific llms, either tailored to a single language (see related work; section 2) or specialized for a small set of similar languages @xcite . however, the best practices for creating such language-adapted llms have yet to be established-especially for smaller language communities with resource limitations with respect to data, compute, or both. danish offers a particularly interesting testbed among these smaller languages. as a mid-resource language, which is typologically related to english and has largely overlapping character sets, it has sufficient textual data for llm adaptation, yet is far from the levels of its neighbors (e.g., swedish; @xcite . additionally, it lacks advanced resources like native instruction-tuning data or human-preference data, making it necessary to use translated datasets for which the downstream effects on model functionality are not yet well understood. linguistically, danish has also been shown to be more challenging to learn for humans than its neighbors due its phonological complexity @xcite , which results in downstream effects on discourse, such as additional conversational redundancy @xcite . with the goal to provide the danish community with a custom-adapted resource, as well as to establish better-grounded guidelines for creating llms in languages with similar linguistic characteristics and resource constraints, we present and analyze snakmodel-7b base/instruct , two llms designed specifically for the danish language. our base model builds upon llama2-7b, which we continuously pre-train on a diverse collection of danish corpora comprising 350m documents (sentences/paragraphs) and 13.6b words, before tuning it on 3.7m danish instruction-answer pairs. we evaluate our model against contemporary llama2-7b-based models on the danish part of the scandeval benchmark @xcite that encompasses both language and culture-specific tasks. by releasing not just the related artifacts (final model, intermediate checkpoints, pre-training data, code), but by also analyzing the effects of early decisions in the training and model design process on intermediate training dynamics and downstream performance, we aim to provide resources that are not just relevant for danish, but for llm adaptation in general. contributions. this work contributes: ## final results and analysis benchmark results. using our final model configurations, we present our results on the test split of the danish portion of scandeval in in the final rows, we show the mistral-based models (best results in orange). we evaluate in f 1 , macroaveraged f 1 (mf 1 ), micro-averaged f 1 (µf 1 ), bertscore (berts.; @xcite , and accuracy (acc.). llama2-7b models on the same danish instruction datasets as snakmodel-7b instruct , in order to isolate the effect of danish language modeling pre-training. finally, we include comparisons to the viking-7b model (siloai, 2024) and similarlysized models based on the mistral model suite @xcite ; danish-foundation-models-team, 2024). overall, snakmodel-7b instruct outperforms all other llama2-7b-based models, including those with access to the same set of danish instructiontuning data, with a final average benchmark score of 56.63. the performance improvements over the english model are particularly pronounced for sub-tasks based on natural danish data, including la (33.43 → 52.91), tm (38.69 → 71.05), and ct (57.05 → 71.88). while the mistral-7b-based models outperform snakmodel-7b instruct by up to 4.65% abs., this approximately matches the base model performance difference between mistral-7b-v0.1 and llama2-7b base which spans 7.28%. qualitative behaviors. since scandeval scores are largely computed using constrained generation, we would like to highlight some qualitative observations from when models generate text without constraint. first, we find that llama2-7b models fail to generate danish text consistently, even when explicitly prompted to do so (confirming the findings by @xcite . since they nonethe-less achieve non-trivial benchmark scores under constrained generation, we hypothesize, that they obtain some danish language functionality during their original, primarily english pre-training. our custom llama2-7b models to which we add danish instruction tuning (+inst da ) generate danish responses (even when prompted in english), highlighting that a relatively small amount of translated danish instructions is sufficient to bias models towards generating output in a new language. nonetheless, the fact that snakmodel-7b instruct , which is trained on non-translated danish text outperforms the models trained on translated data, highlights the importance of curating high-quality native-language data for the adaptation target. ## guidance for future work from our final evaluation, as well as our analysis of the training dynamics of snakmodel-7b instruct , we next consolidate some guidance for future work adapting english llms to languages with similar linguistic properties and resource constraints. data. as we found large overlaps across data sources, as well as large amounts of non-danish or irrelevant data (section 3), applying stringent pre-processing standards is important when working with smaller languages-especially when automatic filtering tools may be biased towards larger, related languages (e.g., swedish). training. our training dynamics analysis (section 4.3) showed that despite our total 13.6b word pre-training corpus, applying instruction tuning after 2,000-5,000 steps of danish pre-training (i.e., less than half of the corpus) may already be sufficient to obtain close-to-final performance. for instruction tuning itself, one epoch over translated data appears to be sufficient to amplify instructionfollowing functionalities in the target language. nonetheless, training on non-translated target language data is important to improve performance on more culturally specific tasks based on native data (i.e., la, tm, and ct). finally, our weight divergence analysis revealed that most parameter updates are consolidated in the embeddings, feed-forward up-projections, and language modeling head. as english and danish share a relatively similar syntactic structure, languages with more distinctive typologies may nonetheless exhibit larger changes to the self-attention parameters. for model adaption across a comparable typological distance as english and danish however, focusing training efforts on the aforementioned parameter types-in addition to employing existing parameter-efficient fine-tuning techniques (e.g., @xcite )-may therefore yield even higher efficiency gains. ## conclusion in this work, we introduced the snakmodel suite, which includes a 7b-parameter base and instruction-tuned llm for danish, in addition to its pre-training and instruction-tuning data, intermediate checkpoints, and evaluation. by analyzing design decisions related to data curation and training dynamics, we further consolidated guidelines for future work adapting llms to new languages, to foster research not just in danish, but in language communities with similar resource constraints.