First Steps in Benchmarking L atvian in Large Language Models (2025)

Abstract: The performance of multilingual large language models (LLMs) in low-resource languages, such as Latvian, has been under-explored. In this paper, we investigate the capabilities of several open and commercial LLMs in the Latvian language understanding tasks. We evaluate these models across several well-known benchmarks, such as the Choice of Plausible Alternatives (COPA) and Measuring Massive Multitask Language Understanding (MMLU), which were adapted into Latvian using machine translation. Our results highlight significant variability in model performance, emphasizing the challenges of extending LLMs to low-resource languages. We also analyze the effect of post-editing on machine-translated datasets, observing notable improvements in model accuracy, particularly with BERT-based architectures. We also assess open-source LLMs using the Belebele dataset, showcasing competitive performance from open-weight models when compared to proprietary systems. This study reveals key insights into the limitations of current LLMs in low-resource settings and provides datasets for future benchmarking efforts.

## introduction the recent progress of large language models (llms) has made them very popular and widely used. being the most widely used natural language processing technique (nlp) today, llms differ in their performance depending on several key factors, such as, the quality and size of the training data, the model architecture, the computational resources used for training, and the specific tasks they are evaluated on. most of the language data used for training llms is in english and few other widely spoken languages, while other languages, especially lessand low-resourced, are represented by very small portions of data. for example, in recently developed eurollm multilingual language models for europe, english language data form 50% of training data, while low-resourced languages, such as latvian, lithuanian, estonian, finnish, and others are represented by about 1% of data @xcite . as a result, although many language models are multilingual and powerful in language transfer, they have generally demonstrated considerably less reliable results on low-resource languages @xcite . the fast growth of llms in size, language coverage, and overall quality, has made benchmarking critical for assessing llm performance and capabilities across various tasks. a wide range of benchmarks are available to evaluate different capabilities of large language models. they span multiple categories, including natural language understanding and generation, robustness, ethics, or biases of the models @xcite . llms have demonstrated impressive gains on natural language understanding (nlu) benchmarks, starting from glue @xcite and su-perglue @xcite with 10 tasks related to different nlu problems, followed by mmlu @xcite covering nearly 60 subjects and bigbench @xcite with more than 200 tasks, as well as many other benchmarks. however, many of these benchmarks focus on the english language, as well as some other widely spoken languages and only some attempts have been made to evaluate llm performance on low-resource languages. a recent study of llms for european languages @xcite has identified eight eu languages as low-resource (croatian, estonian, irish, latvian, lithuanian, maltese, slovak, and slovene). the aim of this paper is to conduct an initial assessment of natural language understanding and reasoning skills of different llms for the lowresource latvian language: we provide the datasets used in our experiments to facilitate further benchmarking of latvian 1 , ensuring that researchers have access to the resources necessary to replicate and build upon our work. by making these datasets publicly available, we aim to support the development of robust tools and methodologies for the latvian language, as well as foster collaboration and facilitate advancements in natural language understanding for low-resourced languages. ## related work latvian is an indo-european language of the baltic branch with about 1.5 million native speakers. taking into account its size, it is rather well supported by language technologies @xcite . however, in the context of llms the latvian language is a low-resource language @xcite . before 2024, only limited research has been conducted on the performance of bert family language models (e.g., znotin , š and bārzdin , š (2020), vīksna and skadin , a (2020)). a widely used latvian dataset to assess llm performance on different natural language processing tasks (ner, pos-tagging, dependency parsing) is fullstack-lv dataset @xcite . comparison of several bert family models that 1 https://github.com/lumii-ailab/ vti-data include latvian (mbert @xcite , lvbert (znotin , š and bārzdin , š, 2020), and lit-lat bert (ulčar and robnik-šikonja, 2021)) has been performed by @xcite . the evaluation showed that the litlat bert model has the best performance in named entity recognition, part-of-speech tagging, and word analogy tasks, whereas lvbert demonstrated the best score for the dependency parsing task. until 2024, there were no datasets available to assess the natural language understanding and reasoning skills of llms in latvian and compare them across different models or languages. for example, mbert's performance has been evaluated using the xnli dataset @xcite ) -an evaluation corpus for language transfer and crosslingual sentence understanding in 15 languages, but it does not contain any latvian samples. similarly, the dataset for the evaluation of multilingual llms developed by okapi @xcite , in which the english part was translated with the help of chatgpt, covers 26 languages, but does not include any of the the baltic languages (the "smallest" language is danish with 6 million speakers, followed by slovak with 7 million speakers). latvian is mentioned as one of the languages on which the gpt-4 model was evaluated with mmlu benchmark @xcite . the prompts were machine-translated from english into latvian. when comparing gpt-4's 3-shot accuracy on mmlu across different languages, english reaches 85.5% (only 70.1% for gpt 3.5), while latvian -80.9% @xcite . different approach has been chosen by dar ' gis et al. (2024), who used standardized latvian high school centralized graduation exams as a benchmark dataset. they showed that several opensource models have reached competitive performance in nlu tasks, narrowing the gap with gpt-4, while keeping notable deficiencies in natural language generation tasks (specifically in generating coherent and contextually appropriate text analyses). recently meta has released the belebele benchmark @xcite . this benchmark was used to evaluate three masked language models (xlm-v, infoxlm and xlm-r) and several llms (gpt3.5-turbo, falcon, and llama). the accuracy of these models for the latvian language varies from 37.6% for falcon 40b 5-shot in-context learning model to 74.1% for translate-train-all xlm-v large model. finally, european llm leader-board that includes latvian has been recently published on huggingface. 2 this leaderboard provides a comparison of more than 15 open-source multilingual llms across several machine-translated benchmarks -arc, gsm8k, hellaswag, mmlu and truthfullqa. ## evaluation of bert family models while today llms offer broad multilingual capabilities, they may not always be the best solution for low-resourced languages, thus in some cases bert-based models still remain relevant as a costeffective, adaptable, and open-source alternative for research and real-world applications in underrepresented languages. although several bert models include latvian, their nlu capabilities have not been assessed due to the absence of necessary evaluation datasets. ## evaluation of commercial llms as next, in spring 2024, we evaluated the performance of several commercial models on the latvian language to assess their capabilities in handling low-resource languages. ## evaluation of open llms we continue to explore the impact of machine translation on benchmarking using the recently released multilingual belebele dataset, which includes latvian. we compare the performance of several popular open-weight llm families (gemma, llama, mistral, and qwen) using both the original and machine-translated versions of the dataset. ## conclusion in this study, we provided an initial assessment of several large language models' performance in latvian across different natural language understanding tasks. results of our evaluation of multilingual commercial and open-source models highlights the disparities in model accuracy when applied to lowresource languages. our findings indicate that for the low-resource language latvian, the top-performing llms can achieve similar results on both the original (human-created) and machine-translated datasets. however, machine translation proved less effective for high-resource language benchmarks, such as english, where it significantly impacted model accuracy. while machine translation offers a feasible route to generate benchmarks for low-resource languages, it is not without its pitfalls. the choice of translation method and the inherent properties of the language models significantly influence the outcomes of benchmarking exercises. additionally, the benchmarking of open-source llms against proprietary systems reveals a narrowing performance gap. despite these advances, significant challenges remain, including the lack of comprehensive evaluation datasets tailored to latvian. by introducing adapted versions of the copa foot_6 and mmlu foot_7 datasets and evaluating models on the belebele dataset, this paper lays the groundwork for further research in benchmarking. 11 future work should focus on creating robust, high-quality datasets specifically for low-resource languages and exploring novel architectures that can better generalize across linguistic diversity.