How Well do LLM s know Finno-Ugric Languages? A Systematic Assessment (2025)

Abstract: We present a systematic evaluation of multilingual capabilities of open large language models (LLMs), specifically focusing on five Finno-Ugric (FiU) languages. Our investigation covers multiple prompting strategies across several benchmarks and reveals that Llama-2 7B and Llama-2 13B perform weakly on most FiU languages. In contrast, Llama 3.1 models show impressive improvements, even for extremely low-resource languages such as Võro and Komi, indicating successful cross-lingual knowledge transfer inside the models. Finally, we show that stronger base models outperform weaker, language-adapted models, thus emphasizing the importance of base model in successful language adaptation.

## introduction large language models (llms) have recently made significant advances in multilingual settings. for instance, gpt-4 achieves 80.9% accuracy for latvian and 76.5% for icelandic on the 3-shot mmlu benchmark @xcite . for some time, strong multilingual capabilities were mainly limited to proprietary models, such as chatgpt foot_0 and claude foot_1 , whose weights, training details, and inference processes are kept private. these models outperformed open llms foot_2 like llama 2 models @xcite , on non-english tasks. however, open-weight llms have recently begun to close this gap @xcite , even though the officially supported languages of these models remain limited and the primary focus is on those with significantly more data available than for finno-ugric (fiu) languages. on the other hand, it has been observed that even models optimized solely for english, such as the llama 2 family models @xcite , demonstrate some understanding of a wide range of languages beyond their intended use @xcite . in experiments conducted by @xcite , the llama 2 7b chat model correctly answered 14% and 40% of basic open-ended questions in estonian and finnish, respectively, even though only 0.03% of the llama 2 training data was in finnish and less than 0.005% in estonian @xcite . this work evaluates the multilingual capabilities of open llms on five fiu languages: finnish, estonian, livonian, võro, and komi. among these, finnish and estonian are the most wellresourced, making it easier to adapt existing llms for these languages through continued pretraining @xcite . in contrast, võro, livonian, and komi are extremely low-resource languages, making language-specific adaptation considerably more challenging. the aim of this work is to clarify the capabilities of open llms in understanding fiu languages. while it is evident that open llms can understand these languages to some degree @xcite , their proficiency and comparative performance across models remain largely unexplored. we focus on llama models, which have demonstrated state-of-the-art performance and competitiveness with proprietary models @xcite and have been widely used in non-english adaption @xcite @xcite @xcite . another reason for focusing on llama models is that the newer llama 3.1 models are natively multilingual, potentially improving performance on unsupported languages as well. for further insights, we compare llama models with mistral nemo @xcite , another natively multilingual open model shown to be competitive with llama 3.1 model of the same size. we evaluate only base models rather than chatoptimized models, as most knowledge is acquired during pretraining @xcite . in other words, a stronger base model offers greater potential for developing a strong chat model. consequently, the performance of base models on different fiu languages can serve as a relative estimate of the chat model's quality. the evaluation is conducted using several existing benchmarks that include one or more finno-ugric languages. we examine both the zero-shot and few-shot capabilities of these models. additionally, we explore whether chain-of-thought prompting, which involves first translating the input to english, could improve results on finno-ugric languages. in summary, we seek to answer the following research questions: ## datasets the selection of benchmark tasks is determined by the availability of datasets for our target languages. in total, we evaluate the models on five tasks using nine datasets. these datasets primarily originate from cross-lingual benchmarks that include multiple languages. for our experiments, we use only the subsets that correspond to the selected target languages. a summary of the datasets, tasks and their language coverage is provided in machine translation (mt) our evaluation includes translation tasks from low-resource fiu languages to english. for this purpose, we use the flores-200 benchmark (nllb team, 2022), which includes estonian and finnish, and the flores-smugri dataset @xcite , which translates the first 250 sentences from flores-200 to ten low-resource fiu languages, including komi, võro, and livonian. to ensure consistency, we use only the first 250 sentences of flores-200 for estonian and finnish as well. multiple choice qa this task involves selecting the correct answer from a set of options, given a passage, a question, and possible answer choices. we use the belebele dataset @xcite , which augments paragraphs from the flores-200 benchmark with corresponding questions and answer choices. among its 122 languages, belebele includes estonian and finnish. @xcite further extend the dataset to cover võro, livonian, and komi, resulting in a total of 127 examples per language. for consistency, we use the same number of examples for estonian and finnish. ## methodology for tasks that do not require open-ended text generation (e.g., belebele, sib, xcopa), performance is evaluated by calculating the log likelihood of each possible answer choice and selecting the most likely one as the prediction. in contrast, tasks requiring open-ended text generation, such as flores, extractive qa, we use greedy decoding to generate predictions. we report the results both in zero-shot and fewshot setting where we add either 1, 3 or 5 inputoutput pairs to the prompt to provide the model with task-specific guidance. additionally, we investigate the impact of cot prompting, which guides the model to generate intermediate reasoning steps before producing the final answer. drawing inspiration from @xcite , the intermediate steps require translating the input into english, identifying the answer in english, and translating it back to the target language. cot prompting can also be used both in zero-shot foot_3 and few-shot settings. in the zero-shot setting, the prompt ends with "let's think step-by-step" @xcite , while in the few-shot setting, this is followed by explicit reasoning steps. figure 1 illustrates model input and output in one-shot setting with and without cot. we use regexes to extract answers from the generated text in tasks requiring decoding. although this approach may occasionally produce false negatives, the models generally adhere well to the output format in few-shot settings. we implement all evaluation strategies with lm-eval-harness framework @xcite and make the task configurations publicly available. foot_4 5 results ## conclusion we evaluated the llama 2 and multilingual llama 3.1 family models on five finno-ugric languages with varying amounts of available resources. our results show that llama 2 7b and 13b perform poorly on most languages, except for finnish, where they achieve moderate results. in contrast, the llama 3.1 family models demonstrate impressive performance, even for extremely lowresource languages like võro and komi. the comparison of zero-shot and few-shot prompting indicates that few-shot prompting is beneficial across all languages. however, increasing the number of examples does not always lead to better performance. similarly, few-shot cot prompting brings substantial benefits for tasks like commonsense reasoning but negatively affects others, such as qa. notably, the strongest model, llama 3.1 70b, benefits less from cot prompting on tasks where it helps weaker models, suggesting that strong cross-lingual capabilities reduce reliance on cot prompting. outstanding results in mt, xcopa, and belebele for estonian and finnish highlight the need for stronger benchmarks to better assess the capabilities and limitations of these models. the surprisingly strong results from llama 3.1 70b on komi and võro, despite extremely limited resources, demonstrate effective cross-lingual knowledge transfer and reduce the dependence on large target-language datasets for reasonable performance. finally, our comparison with mistral nemo suggests that the latter outperforms llama 3.1 8b in estonian and finnish. furthermore, our analy-sis of llama models versus llammas shows that a stronger, general-purpose base model consistently outperforms a weaker base model adapted to a specific language, emphasizing the critical role of the base model in successful language adaptation.