Poro 34B and the Blessing of Multilinguality (2025)

Abstract: The pretraining of state-of-the-art large language models now requires trillions of words of text, which is orders of magnitude more than available for the vast majority of languages. While including text in more than one language is an obvious way to acquire more pretraining data, multilinguality is often seen as a curse, and most model training efforts continue to focus near-exclusively on individual large languages. We believe that multilinguality can be a blessing: when the lack of training data is a constraint for effectively training larger models for a target language, augmenting the dataset with other languages can offer a way to improve over the capabilities of monolingual models for that language. In this study, we introduce Poro 34B, a 34 billion parameter model trained for 1 trillion tokens of Finnish, English, and programming languages, and demonstrate that a multilingual training approach can produce a model that substantially advances over the capabilities of existing models for Finnish and excels in translation, while also achieving competitive performance in its class for English and programming languages. We release the model parameters, scripts, and data under open licenses at https://huggingface.co/LumiOpen/Poro-34B.

## introduction neural language models based on the transformer architecture @xcite have led to substantial advances in natural language processing. encoder-only transformer models such as bert @xcite have advanced the state of the art in a broad range of classification tasks, while decoder-only models such as gpt @xcite have redefined what can be achieved by generative models, opening new areas of study in prompting and in-context learning. the success of these models is related in substantial part to their scaling properties: training larger models on more data leads to better results and even entirely new capabilities @xcite . studies refining our understanding of the optimal balance of model size and training steps have increased the demands on data @xcite , and many recent models optimize further for inference-time efficiency by training smaller models on more data @xcite . these developments have introduced increasing demands for textual data, with many recent models pretrained on a trillion tokens or more (e.g. @xcite @xcite @xcite . while such resources can still be assembled from internet crawls for a few of the languages best represented online, for the vast majority of human languages we have already run out of data for training the largest of language models @xcite . while it is standard to repeat training data, repetition can lead to reduced sample efficiency and degradation of performance @xcite : @xcite estimate that the value of repetition starts to diminish rapidly after four epochs and that repetition ceases to add information around 40 epochs. the availability of data is thus currently a limit for monolingual training for all but a few of the highest-resourced languages. multilingual training offers one obvious solution for increasing the amount of training data available, and a large number of multilingual transformer models have been introduced (e.g. @xcite @xcite . however, despite the intuitive appeal of augmenting training data with texts in other natural languages, multilinguality is frequently seen as a negative -commonly referred to as the curse of multilinguality @xcite . while there have been studies of the tradeoffs between monolingual and multilingual training @xcite as well as efforts to enhance models specifically for multilinguality @xcite and to introduce additional language capabilities to existing models @xcite @xcite , state-of-the-art generative models are still frequently trained near-exclusively on large languages such as english, with only limited efforts specifically focusing on optimizing performance for smaller languages. in this study, we explore how to lift data limitations to create stateof-the-art large generative models from scratch for smaller languages, drawing on the understanding emerging in recent studies on how to make the most of limited data and assure that multilinguality is a blessing rather than a curse. some key lessons from previous work include 1) limited multilinguality instead of a large number of languages @xcite 2) matching scripts (e.g., latin) @xcite and 3) matching language families @xcite , 4) incorporating a cross-lingual signal using translation pairs @xcite , 5) oversampling target language data up to four epochs @xcite and 6 ) augmenting natural language with programming language data @xcite . we chose to specifically target the finnish language, which is an interesting case for study as it is a uralic language with no large close neighbours in its language family, necessitating more distant transfer than, for example, between english and another germanic language. while the language is natively spoken by under six million people, its resources are still sufficient to consider a monolingual training approach for larger generative models. in a recent study, luukkonen et al. (2023) combined several web crawls and curated sources of finnish to create a dataset of approximately 40b tokens and introduced the monolingual fingpt models trained from scratch for 300b tokens. with approximately 8 epochs, the repetition of data is expected to show diminishing returns @xcite , and the largest of these models show signs of data limitations, with the 8b parameter model outperforming the 13b in benchmarks. we believe it should be possible to overcome these limitations by applying the lessons listed above. while we cannot match language families, we train for four epochs over the finnish data and augment it with both english and programming language data as well as an explicit cross-lingual signal from translation pairs. we pursue this approach to create poro 34b, training a 34b parameter model for a total of 1t tokens -25 times more than the available finnish data -and evaluate the model in detail on finnish, english, and programming language tasks. we find that the model not only achieves the goal of substantially advancing over the performance of existing finnish models, but is also competitive in its class of open models on english and code as well as remarkably strong in translation tasks. low-quality texts and boilerplate, filter toxic context, and deduplicate repeated texts. we illustrate the pretraining data distribution in figure 1 and describe the data briefly in the following. data sources are detailed in finnish for finnish pretraining data, we draw on the resources recently introduced by luukkonen et al. for creating the fingpt model family. we exclude the epub and lehdet resources provided by the national library of finland for that work as they could not be shared due to copyright limitations, but use the remaining sources of data, totalling to a 32b token monolingual corpus. the majority of the finnish data originates from web crawls (approx. 84%) complemented with news sources (approx. 2%), project lönnrot, the finnish equivalent of project gutenberg copyrightfree book corpus (approx. 0.5%), wikipedia (approx. 0.5%) and finnish online discussion forum contents from reddit and suomi24 (approx. 13%). following the rule of thumb proposed by @xcite , we upsample the 32b tokens of finnish so that four epochs over the data are made during training. consequently, approximately 13% of the total tokens seen in pretraining are finnish. english for english pretraining data, we primarily use slimpajama @xcite , a cleaned and deduplicated subset of the redpajama corpus foot_2 (together computer, 2023), from which we excluded data from the books category due to their copyright status. we supplemented this dataset with the project gutenberg public domain books data from the dolma corpus foot_3 @xcite . we train for one epoch over the 542b tokens of the english data, which thus represents slightly over half of the 1t total training tokens. programming languages to introduce data representing various programming languages (referred to hereinafter as "code" for short) into our pretraining, we make use of the starcoder corpus @xcite , a processed subset of the stack corpus foot_4 @xcite . the original corpus consists of 208b tokens, which we oversample 1.5x so that approximately a third of the tokens seen during pretraining represent code. cross-lingual data we introduce a cross-lingual signal into pretraining by including translation examples from opus @xcite . specifically, we use the english-finnish examples from the tatoeba dataset @xcite to generate instruction-formatted translation examples. the tatoeba training data was reformatted into a minimalistic instruction-following format by recasting each english-finnish translation pair into a document with the following format: 
translate into finnish: {{en}} 
{{fi}} where {{en}} and {{fi}} are the english and finnish texts (resp.) of the translation pair. we additionally reverse the translation order (i.e., finnish to english instead of english to finnish) for a total of two documents for each sentence pair. no weighting is applied to the approximately 8b tokens of cross-lingual data, which thus represents slightly under 1% of the pretraining tokens. ## methods in this section, we describe the method used to create the poro 34b tokenizer, the pretraining setup, and provide an estimate of the compute cost of pretraining the model. ## evaluation we thoroughly analyze the capabilities of the model for finnish, english and code, first briefly reporting perplexity results and then focusing on community-standard benchmarks for evaluating generative models. we then assess the quality of finnish text generated by the model and finally evaluate the model's translation capability from english to finnish (and vice versa). for comparison, we include results for the state-ofthe-art finnish language models, fingpt 8b and fingpt 13b @xcite , and a selection of similarly-sized general-purpose open source base language models trained on broadly comparable numbers of tokens for english 9 : llama 33b @xcite , mpt 30b (mo-saicml, 2023), and falcon 40b @xcite . we also provide results for star-coder base @xcite as a reference for performance on code tasks. ## discussion and conclusions in this study, we have considered the challenges that the availability of data poses for pretraining 14 we did not evaluate the gpt models and google translate on tatoeba because of the associated api costs. 15 we attempted to reproduce some of the flores-101 results reported by @xcite and obtained a slightly higher result for gpt-4 in eng-fin translation (37.5 instead of 35.33) and slightly lower results for m2m-12b and nllb-1.3b (31.4 and 26.6, respectively). for the sake of consistency, we present the results from that study without modification. our model architecture and the finnish datasets included follow those of the fingpt family of monolingual finnish models, which were constrained by the available finnish training data. the superior performance of our model in finnish evaluations demonstrates that multilingual training can lift such limitations, allowing further scaling of models focused on smaller languages. in future work, we hope to explore this effect more systematically to answer some of the many questions that remain open regarding the training of large generative models for smaller languages, including the impacts of covering multiple smaller languages and the effect of the size of data available in the target languages. a number of the choices made in training poro 34b were made with incomplete information regarding their specific impacts on the final model. for example, we opted to include a comparatively large amount of programming language data as well as instruction-formatted translation examples in the pretraining data, the latter on the assumption that this would provide a cross-lingual signal that would strengthen the ability of the model to benefit from data in a more distantly related language (english). while this approach is intuitively appealing and the performance of our model suggests that it has at a minimum not notably detracted from the capabilities of the model, we did not as part of this work have the resources to conduct ablation studies nor to explore alternative ways to incorporate cross-lingual information in pretraining. we aim to study these questions further in future work. we hope that our approach can serve as a template for the creation of larger models for other smaller languages and that the model introduced in this work can serve as both as a focus of research in its own right as well as a starting point for further pretraining, finetuning and alignment to create useful models, tools and methods not only for finnish but also other languages. we release the model weights as well as all relevant documentation and software fully openly at https: //huggingface.co/lumiopen/poro-34b .