Localizing AI: Evaluating Open-Weight Language Models for Languages of B altic States (2025)

Abstract: Although large language models (LLMs) have transformed our expectations of modern language technologies, concerns over data privacy often restrict the use of commercially available LLMs hosted outside of EU jurisdictions. This limits their application in governmental, defense, and other data-sensitive sectors. In this work, we evaluate the extent to which locally deployable open-weight large language models support lesser-spoken languages such as Lithuanian, Latvian, and Estonian. We examine various size and precision variants of the top-performing multilingual open-weight models, Llama 3, Gemma 2, Phi, and NeMo, on machine translation, multiple-choice question answering, and free-form text generation. The results indicate that while certain models like Gemma 2 perform close to the top commercially available models, many LLMs struggle with these languages. Most surprisingly, however, we find that these models, while showing close to state-of-the-art translation performance, are still prone to lexical hallucinations with errors in at least 1 in 20 words for all open-weight multilingual LLMs.

## introduction since the fall of 2022, openai and other big tech companies have transformed llms from an obscure technology little known outside the academic circles to a major household name. key to this was the llms' ability to perform tasks specified in free-form instructions, making them excel as nlp tools foot_0 . furthermore, these models can learn during inference from relevant examples provided as inputs, making them adaptable to new requirements or even tasks. @xcite showed that in such a setting, gpt-3, provided with relevant translation examples, outperforms machine translation systems of major companies, including google, deepl, and modernmt. however, data privacy concerns often constrain the use of commercially available llms hosted outside eu jurisdiction, limiting their application in governmental, defence, and data-sensitive private sectors. fine-tuning and operational deployment of adapted models can incur prohibitive costs in the case of commercially available llms, emphasizing the need for sovereign ai solutionslocally deployable alternatives that ensure security, control, and compliance. recently, many powerful alternatives to the commercially available online llms have emerged @xcite @xcite @xcite . although many of these llms officially support only a handful of languages with a large speaker base, their training data often incorporate texts from many other languages. therefore, in practice, these languages receive some degree of support. however, the extent to which these languages are supported, to the best of our knowledge, still needs to be evaluated. in this work, we aim to answer the question of to what extent, if at all, several popular open-weight models support lithuanian, latvian, and estonian. all three languages have relatively small speaker bases and thus are unlikely to be focal points of major multilingual openweight llms. we examine variants of meta's llama 3, google's gemma2, mistral's nemo, and microsoft's phi3 in their performance in multiplechoice question answering (mcqa) and machine translation (mt). we also manually assess the text quality generated by these models by identifying the rate of incorrect words when answering openended questions. we find that while some models like gemma 2 nearly match the performance of top commercial models, many llms struggle with these languages. surprisingly, even those models that approach state-of-the-art translation capabilities are still susceptible to lexical hallucinations. ## experimental setting we evaluate llms on multiple-choice question answering, machine translation, and text generation quality in open-ended question answering. while our experiments focus on the model performance for three languages-lithuanian, latvian, and estonian-each with a speaker base under 3 million, we also include results for czech and english for comparison purposes. we have chosen to evaluate models that consistently appear on various leaderboards. specifically, we assess the 8.03b and 70.6b parameter versions of llama 3 and llama 3.1, as well as the 3.21b parameter version of llama 3.2 (dubey et al., 2024) from meta; the 9.24b and 27.2b parameter versions of gemma 2 @xcite by google; the 3.8b and 14b versions of phi 3 by microsoft @xcite ; and the 12.2b parameter nemo by mistral ai @xcite . to provide context for our experiments, we include online models by ope-nai such as gpt-3.5 turbo and gpt-4o @xcite and deepl machine translation systems. in experiments assessing the quality of lithuanian text generation, we incorporate the lithuanian language-specific fine-tuned versions of llama 2 @xcite with 7b and 13b parameters, developed by neurotechnology -lt-llama 2 @xcite . we run llms on our local hardware using the default inference parameters of the ollama platform foot_1 , which offers several levels of precision for model quantization: 4bit, 8bit, and full-precision -16bit. by default, we use 4bit precision in all our experiments, albeit at the cost of some performance degradation. we also evaluate the performance drop due to quantization by contrasting the results of quantized models with their fullprecision counterparts. machine translation for mt experiments, we use the flores-200 benchmark dataset @xcite , which comprises parallel sentences in over 200 languages foot_2 . we use the devtest subset from flores-200, which contains 1,012 sentences. we test llms in a zero-shot inference scenario. we use the following english prompt to request text translations from the specified source and to the specified target language: "{lang a }: {sentence langa } translate the above {lang a } text into {lang b } {lang b }: " the translation and evaluation are performed at sentence-level; the inference is conducted in a single run for each test sentence. for automatic evaluation of mt quality, we use comet foot_3 @xcite as it has been shown to have a higher correlation with human judgments than bleu @xcite and to be more suitable for unrelated system comparison @xcite . ## results and discussion mt evaluation results (see quantisation impact on mt quality analysis (see mcqa results (see quantization impact on mcqa analysis unveils a similar picture as the analysis above for mt: llama models are more sensitive to quantization, while gemma 2 are more robust. as a result, gamma 2 models show little performance degradation when much more efficient 4bit models are used. it's worth noting, however, that the accuracy drops because quantization differs depending on the amount of data each language has. less spoken languages like lithuanian, latvian, and estonian are affected more than english and czech, for which overall results are better. for example, the llama 3.1 70b model loses 0.06, 0.12, and 0.08 accuracy for lithuanian, latvian, and estonian, respectively, but only 0.01 and 0.02 for english and czech. lesser-spoken languages like lithuanian, latvian, and estonian have less support in openweight models compared to more populous languages such as czech. these differences are more pronounced in smaller and lower-quality models, especially in tasks where models generate text in lesser-spoken languages (e.g., mt from english into lithuanian). comparatively good results for czech suggest that these disparities are related to the amount of data each llm has seen for each language during training, rather than factors such as the structural complexity of the language. ## conclusions our findings demonstrate that certain open-weight llms, such as the gemma 2 family, achieve performance comparable to top-tier commercial products, such as general-purpose models like ope-nai's gpt-4o and specialized machine translation services like deepl. this progress enables local, secure, and private solutions, supporting the development of sovereign ai for many language tasks in governmental, defence, and other data-sensitive private sectors. nevertheless, unless specifically fine-tuned for languages like lithuanian, most multilingual models are still surprisingly prone to lexical hallucinations, highlighting the need for 1) high-quality language data for languages of the baltic states and 2) research on language-specialized llms.