Small Languages, Big Models: A Study of Continual Training on Languages of Norway (2025)

Abstract: Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Northern Sámi. To address this issue, we present a novel three-stage continual training approach that substantially improves the downstream performance together with the inference efficiency for the target languages. Based on our findings, we train, evaluate, and openly release a new generative language model for Norwegian Bokmål, Nynorsk, and Northern Sámi with 11.4 billion parameters: NorMistral-11B.

## introduction the development of large language models typically requires massive amounts of training data, which benefits wide-spread languages such as english, but poses a significant challenge for less widely spoken languages. norwegian, with its two written standards bokmål and nynorsk, 1 currently has approximately 24b words available in our filtered text collection -about three orders of magnitude less than english @xcite . the situation is even more challenging for northern sámi with only 40 million words available. 2 1 while bokmål is the main variety, roughly 15% of the norwegian population uses nynorsk. the two varieties are so closely related that they may be regarded as 'written dialects', but the lexical differences can be relatively large. 2 the sámi languages are a group of uralic languages, of which northern sámi is the most widely used variant. with the number of speakers estimated to be between 15,000 and 25,000 in total across norway, sweden and finland, it is still considered to be an endangered language. as the sámi people are recognized as an indigenous people in norway, sámi has status as an official language along with norwegian. to address this data scarcity, we propose a novel approach combining three key elements: knowledge transfer from existing models, data augmentation with related languages, and targeted upsampling. this method enables us to train an 11.4bparameter model that achieves state-of-the-art performance across norwegian language tasks while obtaining strong capabilities in northern sámi. the three main research contributions of this paper can be summarized as follows: ## training corpus our goal is to train a model for the official languages of norway. however, this task is made difficult by the uneven distribution of these languages and the fact that there is only about 24 billion words in these languages available in the publicly accessible high-quality corpora (see below). ## training and evaluation of normistral this section describes the training and evaluation pipeline of normistral-11b; a continually trained mistral-nemo-base-2407 language model. 3 the presented methods are evaluated later in section 5. ## results we report the aggregated evaluation results in bokmål, nynorsk and sámi performance we evaluate the models on all target languages: bokmål, nynorsk and northern sámi. relative to other models, the performance gains of normistral-11b stay consistent across these three languages. it is possible to estimate the difference in performance on nynorsk compared to bokmål when focusing on norsumm, a dataset that is perfectly balanced and parallel for the two variants of norwegian. the substantially higher score for bokmål indicates that the much smaller amount of nynorsk in the training corpus (even after upsampling) limits the downstream performance on this language variant. the results on the english-to-sámi translation suggest that our model was able to learn aspects of this language even though it made only 0.5% of the training corpus. however, any stronger claim about the level of understanding of sámi would require a substantially more robust benchmarking suite than what is currently available. ## methodological comparisons we have conducted an initial comparative study of different training methods before settling on the pretraining process from section 3 and training normistral-11b. the results are presented in architectural choice there are many promising improvements of the original gpt neural architecture @xcite -we considered two recent and well-studied architectures: bloom @xcite and llama @xcite , which is also used for training the mistral models @xcite . we adopted the training hyperparameters suggested by the respective papers and trained two models with 7 billion parameters on the same norwegian corpus and with the same norwegian tokenizer. from scratch vs. warm-starting the central research question of this paper is how to train a good large language model for relatively small languages. here we test our proposed three-stage continual pretraining and compare it against a model trained from scratch. for a fair comparison, we train two 7-billion-parameter models on the same norwegian corpus (the norwegian colossal corpus by @xcite , and with the same architecture and tokenizer. note that we do not consider existing methods that do not adapt the subword vocabulary -like simple continual training or adapter tuning (yong et al., 2023) -because they necessarily lead to inefficient inference ( hybrid masked-causal modeling interestingly, we do not observe an overall increase in performance after training with the 'dual' training objective, as opposed to the observations by charpentier and @xcite . however, we believe that this can be explained by continued training -the hybrid masked-causal training is used for a negligable number of steps compared to the fully-causal pretraining of the base mistral model. number of training steps finally, we compare the performance of model checkpoints saved at different points of training. we can make several observations from the results: 1 they confirm the data-scaling laws by @xcite as the model continues to improve even after (at least) four repetitions of the norwegian data; 2 tokenizer adaptation (the first two stages of our training method) is a simple and efficient way of adapting a model to a new language without losing performance; 3 the three-stage continual pretraining does not affect all downstream tasks equallywhile it usually leads to monotonical improvement, there are some tasks (norquad) that experience an initial decrease in performance. further investigation is needed to determine if this drop is significant and if it can be avoided by a more careful switch to a new language distribution at the start of training. ## related work norwegian language models there have been several prior efforts on creating language models for norwegian. when it comes to creating openly available generative decoder-only models for norwegian, most of the main efforts are listed in section 3.3 and used in our experiments. however, one other notable mention is nb-gpt-j-6b -a finetuned version of the english gpt-j-6b model. 5 released by the national library of norway in 2022, it was the first large generative language model trained for norwegian. there have also been several efforts on developing smaller transformer models, e.g., based on the bert encoder architecture @xcite and the t5 encoder-decoder architecture @xcite . the norbert family of models were first released by @xcite and have by now reached their third iteration of releases @xcite and come in several different sizes; ranging fron 15m parameters for the xs model to 323m for norbert3 large. @xcite also introduced the nort5 family of models, ranging 32m to 808m parameters. whereas the above-mentioned models where all trained from scratch for norwegian, kummervold et al. (2021) trained nb-bert (base and large) by fine-tuning the pre-trained mbert model on norwegian data, also reusing the tokenizer. a similar approach was followed for the north-t5 models. 6 language models for northern sámi as for northern sámi, @xcite has recently experimented with targeting this language. however, their models have not been published nor did they evaluate them on any downstream tasks; we are thus not able to compare them to our model. continual training techniques adaptation of pretrained language models to new domains by continual training has a long history @xcite . our three-stage continual pretraining is designed specifically for adapting language models to a new language -by entirely replacing the original tokenizer, we can get an efficient model (by compressing the textual input into a short sequence of tokens) without the need of any extra parameters. simple continual pretraining works well performance-wise but the training and inference computation cost is high @xcite . a substantially more efficient approach is to introduce a new tokenizer and replace the embedding layers (first two stages of our approach), as proposed by @xcite . similarly, @xcite only use the first and last stage of our method -they extend the vocabulary by 5 000 new tokens and then train the full model. on the other hand, kim et al. (2024) pursue a more careful approach, the most similar to our training method. they first extend the subword vocabulary with extra tokens and then meticulously train the new and old parameters in eight subsequent stages. ## conclusion we presented normistral-11b, a new large language model for norwegian bokmål, nynorsk, and northern sámi. we proposed a novel threestage continual pretraining approach that efficiently adapts existing models to other languages while maintaining high performance and increasing their inference speed. this approach involves training a new tokenizer, realigning embedding weights, and then training the full model. we also demonstrated the benefits of hybrid masked-causal pretraining, which allows the model to be used flexibly as either a causal or bidirectional model. our extensive evaluation shows that normistral-11b achieves the state-of-the-art performance across a wide range of norwegian tasks, while also showing promising results for northern sámi. this suggests that our approach could be beneficial for developing large language models for other smaller languages. to facilitate further research and development, we have released normistral-11b, the three 7b models trained for section 5, training code, and a new northern sámi corpus at limitations limitations of the base language model since normistral-11b is continually pretrained on the existing mistral-nemo-12b weights, the model is partially dependent on the training data of the original mistral model. the exact composition of this training data is not known, which to some extent limits more detailed studies of this model. specifically, the original model might have been trained on contaminated data, which could explain its high-scores on well-known evaluation tasks such as belebele. computational cost as mentioned in section 3, training normistral-11b took more than 55 000 gpu/hours. this is a significant amount. we have not yet estimated the co 2 footprint of the full training, but it was conducted on the lumi supercomputer which is powered exclusively with renewable electricity and deployed in one of the most ecoefficient data centers in the world. 7 evaluation of northern sámi knowledge finally, our evaluation for northern sámi is limited to english-sámi translation, which is obviously insufficient. unfortunately, we lack more advanced or diverse benchmarks for low-resource languages like this one. we hope to see further development in this direction by the nlp community. chen. 2022. lora: low-rank adaptation of large language models. in international conference on learning representations.