Got Compute, but No Data: Lessons From Post-training a Finnish LLM (2025)

Abstract: As LLMs gain more popularity as chatbots and general assistants, methods have been developed to enable LLMs to follow instructions and align with human preferences. These methods have found success in the field, but their effectiveness has not been demonstrated outside of high-resource languages. In this work, we discuss our experiences in post-training an LLM for instruction-following for English and Finnish. We use a multilingual LLM to translate instruction and preference datasets from English to Finnish. We perform instruction tuning and preference optimization in English and Finnish and evaluate the instruction-following capabilities of the model in both languages. Our results show that with a few hundred Finnish instruction samples we can obtain competitive performance in Finnish instruction-following. We also found that although preference optimization in English offers some cross-lingual benefits, we obtain our best results by using preference data from both languages. We release our model, datasets, and recipes under open licenses at https://huggingface.co/LumiOpen/Poro-34B-chat-OpenAssistant.

## introduction foundational llms are language completion models that need to be finetuned after pretraining to be able to respond to user questions and follow instructions @xcite . this post-training process involves supervised finetuning where the model is trained to act as an assistant by training on a dataset of prompt-response pairs. preference optimization further aligns the model to human preferences such as helpfulness, harmlessness, and honesty @xcite . these methods have resulted in llms becoming more capable of answering complex questions involving reasoning, coding, math, and science (e.g., @xcite @xcite . the effectiveness of these methods, however, have not been demonstrated for smaller and less-resourced languages, such as finnish. one of the major challenges we face in posttraining in smaller languages is the scarcity of training data. the situation is even more challenging for commercial settings as most of the datasets available today are generated by llms with restrictive licenses. the availability of evaluation benchmarks for chat models in small languages is also an issue. popular benchmarks such as mt-bench @xcite and ifeval @xcite are designed for english models and have not been adapted for use in a multilingual setting. 1 in this paper, we discuss our experiences in post-training an llm in finnish and english. we use the llm that we want to finetune to machine-translate instruction and preference datasets into finnish. we use a commercial machine-translation service to translate a widely-used instruction-following evaluation (ifeval) benchmark into finnish. we experimented with different combinations of finnish and english data in instruction tuning and preference optimization. we also experimented with different methods to improve vanilla instruction tuning. ## related work the post-training of base llms, popularised in in-structgpt @xcite , can be broadly divided into two categories: instruction tuning and preference optimization. instruction tuning, also known as supervised finetuning (sft), trains a base llm to answer questions and follow instructions by training on a dataset of promptresponse pairs with a language modeling objective. preference optimization further improves the model's ability to follow conversations and teaches a model to generate responses that align with human preferences by showing the model samples of desirable and undesirable responses (or a ranking of responses). direct preference optimization (dpo; @xcite ) is a rewardfree preference optimization technique that optimizes directly on the preference data and does not require training a separate a reward model. it is a popular alternative to reward-based methods such as proximal policy optimization (ppo; @xcite ) because it is less computationally expensive and achieves promising results. post-training llms in a multilingual setting is an under-explored topic @xcite @xcite . previous studies have experimented with monolingual and multilingual instruction tuning of multilingual base llms @xcite . these studies show that monolingual instruction tuning transfers some instructionfollowing capability to the other languages in the model but is dependent on the amount of multilingual data that the base llm was trained on. a few studies have investigated multilingual preference optimization @xcite . lai et al. generated synthetic preference datasets for 26 languages and performed rewardbased preference optimization on bloom and llama 7b models. their results show that preference optimization offers a slight improvment over sft. @xcite out that these preference-optimized models still underperform compared to massively multilingual llms that are finetuned only with sft. the scarcity of instruction and preference datasets is a major challenge in post-training llms for smaller languages. previous efforts to assemble finetuning datasets through machine translation, crowd-sourcing, and synthetic data generation include @xcite @xcite . evaluating openended responses of chat models in small languages is also a challenge. previous studies have investigated using llm-as-a-judge in multilingual settings but these studies focused on standard nlp tasks such as summarization and questionanswering @xcite . ## experimental setup we use poro 34b as the base llm @xcite . poro is trained on 1t tokens of english, finnish, and code, with 129b tokens for finnish. we use the transformer reinforcement learning library (trl; @xcite ) for instruction tuning and preference optimization. we finetune all of the model parameters in our experiments. 2 we use 32 amd mi250x gpus in our experiments. for sft, we use a micro batch size of 4 and a gradient accumulation step of 1, resulting in a global batch size of 128. we use a learning rate of 2e-5 with a warmup rate of 0.1 and finetune for 3 epochs. for dpo, we use a global batch size of 64, learning rate of 5e-7, warmup rate of 0.1, and train for 5 epochs. ## datasets sft we use a curated version openassistant 2 (oasst2; @xcite ) containing the topranked english conversations. this dataset has 4,692 samples. 3 we translate oasst2 into finnish using poro with few-shot prompting. poro has been shown to produce higher-quality finnish translations compared to other open mt systems @xcite . for this reason, we did not experiment with translations from other open mt models and focus our efforts on the poro-translated dataset. we use heuristics to clean up the translations. after posttranslation cleaning, our oasst2 finnish data has 4,399 samples. ## evaluation we use the instruction following (ifeval) benchmark to evaluate instruction-following performance @xcite . ifeval has 541 prompts where a prompt contains verifiable instructions that can be checked with a deterministic program, circumventing the need of an llm or human as judge. examples of instructions include adding keywords to the response, formatting the response in json, or responding in a specified language. 5 we translate the ifeval prompts into finnish using deepl foot_6 . ifeval has 31 prompts that require the response language to be in a language other than english. we exclude these prompts for this work due to poro being constrained to only english and finnish. we report the results for the remaining 510 prompts only. ifeval reports strict accuracy and loose accuracy where loose accuracy accepts minor transformations in the responses. for the sake of clarity, we report only the strict accuracy in this work. we run evaluations through the lm evaluation harness @xcite . the translated ifeval is available at https://huggingface.co/ datasets/lumiopen/ifeval_mt . ## experiments multilingual sft finnish instruction data is more difficult to obtain compared to english; therefore, we want to investigate how the amount of finnish instruction data affects performance. we construct data mixes from the english and finnish oasst2 datasets such that we start with just the english data and gradually introduce more finnish samples into the mix starting from 10% of the finnish data and then going up to 100%. we call these data mixes en-fi-xpct (i.e., the data mix with just the english samples is called en-fi-0pct while the data mix with all the english and finnish samples is called en-fi-100pct). by default, we do not mask prompts during training (i.e., we incorporate the improving vanilla sft we investigate whether we can improve sft by adding noise to the word embeddings in the instruction data (nef-tune; jain et al. ( 2024 )). we also experiment with prompt masking where the loss is calculated only on the completion tokens. our baseline for these experiments is the sft model trained on the en-fi-100pct data mix. multilingual dpo we opt to use dpo for preference tuning as it has been found to fare better in ifeval, in addition to being more stable and requiring less compute @xcite . we tuned the β parameter of dpo with values {0.01, 0.05, 0.1} and found β = 0.05 to be optimal. we experiment with using either the english or finnish datasets and using both. as our baseline, we use the sft model trained on the en-fi-100pct data mix. ## results and discussion multilingual sft in figure 1 shows the accuracy by instruction category of the en-fi-100pct model. the model struggles most with the combination categorythis category includes combined instructions such as giving two responses that are separated by a given separator or repeating the prompt without modification before giving the response proper. the poor performance is probably because the instruction contains multiple steps that must be followed in order to give a correct response. for instance, if the model gives two responses but these responses are numbered instead of separated by an asterisk, the response is considered incorrect. the model also struggles with punctuation instructions such as avoiding the use of commas likely because texts without commas are rare in the dataset. overall, the results indicate that finetuning with a few hundred finnish instruction samples achieves results close to finetuning with ten times that amount. in terms of instruction types, the model struggles with multi-step instructions and unusual instruction types. previous work has indicated that carefully curating the instruction dataset is vital to a strong sft model @xcite . in future, we aim for a smaller but higherquality data by, for instance, removing highly similar prompts, diversifying tasks, and curating the sources of the samples. ## conclusions and future work in this work we share our findings from posttraining poro 34b in english and finnish. due to the scarcity of finnish post-training datasets we opted to machine-translate instruction and preference datasets using poro. to evaluate the results of our experiments, we translate ifeval, a widely-used instruction-following evaluation benchmark. we experimented with using different combinations of english and finnish data in sft and found that using all available data from both languages gave the best performance overall. using only 10% of the finnish instruction data (around 400 samples), however, still gives competitive performance. we contribute to finnish llm development by releasing our datasets, recipes, and model with open licenses at https: //huggingface.co/lumiopen/ poro-34b-chat-openassistant. in future we want to explore different ways of obtaining more finnish data by, for instance, generating synthetic instruction and preference datasets. we will use these synthetic datasets to further investigate other alignment and finetuning methods. additionally, we are interested on de-veloping an evaluation benchmark for open-ended conversations in finnish that takes cultural and linguistic nuances into account.