Evaluating Open-Source LLM s in Low-Resource Languages: Insights from L atvian High School Exams (2024)

Abstract: The latest large language models (LLM) have significantly advanced natural language processing (NLP) capabilities across various tasks. However, their performance in low-resource languages, such as Latvian with 1.5 million native speakers, remains substantially underexplored due to both limited training data and the absence of comprehensive evaluation benchmarks. This study addresses this gap by conducting a systematic assessment of prominent open-source LLMs on natural language understanding (NLU) and natural language generation (NLG) tasks in Latvian. We utilize standardized high school centralized graduation exams as a benchmark dataset, offering relatable and diverse evaluation scenarios that encompass multiple-choice questions and complex text analysis tasks. Our experimental setup involves testing models from the leading LLM families, including Llama, Qwen, Gemma, and Mistral, with OpenAI’s GPT-4 serving as a performance reference. The results reveal that certain open-source models demonstrate competitive performance in NLU tasks, narrowing the gap with GPT-4. However, all models exhibit notable deficiencies in NLG tasks, specifically in generating coherent and contextually appropriate text analyses, highlighting persistent challenges in NLG for low-resource languages. These findings contribute to efforts to develop robust multilingual benchmarks and improve LLM performance in diverse linguistic contexts.

## introduction the dream that artificial intelligence (ai) can perform many tasks in a similar manner to humans became closer with the release of chatgpt by ope-nai in november 2022. today, several large language models (llm) have been made available by global companies and are widely used by society and industry for various text generation tasks, such as question answering, text summarization, translation, etc. however, llms have shown considerably less reliable results for low-resource languages @xcite . the reason for this is the fact that most of the language data used for training llms is in english and few other widely spoken languages, while low-resource languages are represented by very small portions of data. benchmarking is a crucial step in evaluating llm performance and capabilities across various tasks. it involves setting standardized tests or tasks to measure the llms' performance. a lack of benchmarks that enable comprehensive multilingual evaluation is one of the reasons why research on llms and machine learning models for nlp is still mostly focused on english and some other widely spoken languages. the aim of this paper is to conduct an initial evaluation of open-weights llm capabilities in latvian, both in natural language understanding (nlu) and in natural language generation (nlg). the evaluation was performed using high school centralized graduation exams, overseen by the national centre for education. high school exams serve as an excellent benchmark dataset because they offer a relatable point of reference, allowing for comparison not only between different models but also between the performance of llms and the expected achievements of high school graduates. ## related work recent advances on llms have led to impressive gains on nlu benchmarks, starting from glue @xcite and superglue @xcite with 10 tasks related to different nlu problems, followed by mmlu @xcite which covers nearly 60 subjects (including stem: science, technology, engineering and mathematics), and bigbench @xcite with more than 200 tasks, as well as many other benchmarks. again, many well-known benchmarks are available only in english and other widely spoken languages. google researchers addressed the need for a highly multilingual benchmark when the first transformer-based llms appeared by introducing the cross-lingual transfer evaluation of multilingual encoders (xtreme) benchmark @xcite which is used to evaluate cross-lingual generalization capabilities of multilingual representations. although xtreme covers 40 typologically diverse languages, spanning 12 language families, baltic languages are not included in this benchmark. similarly, the dataset for the evaluation of multilingual llms developed by okapi @xcite , in which the english part was translated with the help of chatgpt, covers 26 languages except the baltic languages (the "smallest" language is danish with 6 million speakers, followed by slovak with 7 million speakers). the development of test sets for specific languages involves significant human resources. therefore a widely used strategy is to apply machine translation, with or without manual postediting. recently, this approach was chosen to translate the mmlu and copa @xcite datasets into latvian. foot_0 evaluation of openai chatgpt 3.5 turbo and google gemini 1.0 pro on the machine translated mmlu dataset shows that performance of these llms for latvian is worse when compared to english @xcite . it should be noted, that this dataset is not manually post-edited, and machine translation most likely has introduced some errors which can result in generating wrong answers. bakanovs (2024) experiment on a small subset of the social science domain shows that post-editing improves results by 3 percentage points for chatgpt-3.5 turbo and by 9 percentage points for gemini 1.0 pro. finally, gpt-4 has been evaluated by openai on several benchmarks @xcite , such as mmlu, hellaswag, ai2 reasoning challenge, winogrande, humaneval, and drop. when comparing gpt-4's 3-shot accuracy on mmlu across different languages, english reaches 85.5% (only 70.1% for gpt 3.5), while latvian -80.9%. with respect to educational tests and exams, openai has reported that "gpt-4 exhibits human-level performance on the majority of professional and aca-3 test setup all tests were run using the ollama toolkit on a computer with 8x interconnected nvidia a100 80gb gpus. the most popular open-source llm families were chosen to be tested: llama, qwen, gemma, and mistral. a non-quantized instruct-fp16 version was chosen for each model, except for llama3.1 405b because the model was too large, therefore its 5-bit k-quantized version was used instead. the emphasis in this article is on open-weights models. openai's gpt-4o model is added just for a reference as the most popular closed-source commercial model. in the gpt-4 technical report @xcite , latvian is classified as a lowresource language. although it could be argued that latvian is not as low-resource as many other languages, especially w.r.t. to the number of native speakers, it is considered as low-resource also by european researchers @xcite . ## centralized high school exams in latvia, centralized exams are a crucial component of the educational system, designed to standardize knowledge assessment across the country and to ensure that high school graduates meet national academic standards. these exams are taken at the end of the 11th or 12th grade and are required to obtain a high school diploma. students in latvia must take a certain number of centralized exams, though they have some flexibility in choosing which subjects to be examined in, depending on their future academic and career aspirations. it is not expected for a student to be able to pass the exams in all subjects. the mandatory exams include latvian language and literature, mathematics, and a foreign language of choice (usually english, but alternatives such as german, russian, or french are available). beyond these core subjects, students can opt to take additional exams in subjects like biology, chemistry, physics, history, geography, or informatics. higher education institutions in latvia typically use these scores as part of their admission criteria, often alongside other considerations such as entrance exams or interviews. this makes the performance on centralized exams a significant factor in a student's educational trajectory. these exams are designed and administered by the national centre for education. 2 the exams are intended to assess not just rote memorization, but also critical thinking, problem-solving abilities, and application of knowledge. the structure of the exam and the types of tasks vary from year to year. exams of 2023 were chosen for the initial version of this benchmark, since a lot of the exercises contained multiple-choice questions (mcq). in addition, models were also tested on text analysis task from the latvian language and literature exam. ## multiple-choice questions a set of 72 latvian mcqs was created, covering physics, geography, chemistry, biology, latvian language and literature exams. questions containing pictures and complex formulas were omitted. the models were tested with the zero-shot learning approach. the prompt started with a question, followed by answer options and concluded with the instruction: "atbildi formātā 'atbilde ir x', kur x ir pareizās atbildes burts." ("answer in the form 'answer is x' where x is the letter of the correct answer."). the results are shown in the first criterion evaluated was validity -how many of the generated answers matched the expected format. many models achieved 100% validity, indicating that instructions were understood and the zero-shot approach works well for this kind of task. for an answer to be valid it must contain the phrase "atbilde ir" followed by a letter a-z. the answer may also contain extra text (usually, an explanation) before or after the phrase. the second criterion evaluates consistency. each prompt was sent to each model 10 times with a different seed value each time to reduce the chance of a lucky guess. the model must choose the same option for the same question every time. this criterion was evaluated on a per-question basis. to count an answer to a question as consistent, all answers to the same question must be valid, and the chosen option must be the same in every attempt. the final measure binds it all together. for a question to be counted as correctly answered, all responses must be valid, consistent, and correct. such a strict requirement was used to measure the true expected correctness rate. the questions in biology and geography had higher correctness scores overall. the chemistry scores were lower because some of the questions contained chemical reaction equations, and some physics questions required not only reasoning, but also calculations. the non-quantized fp16 models had very similar correctness compared to 5-bit k-quantized models. for such tasks, the quantized models would be more appropriate due to their significantly smaller memory and compute footprint. ## text analysis and writing skills one of the tasks in the latvian language and literature exam in 2023 was to read two texts (each about 600 words) and write a text analysis (500-600 words) comparing both texts, following the principles of text composition and including the specified content components: the same task was given to the largest model from each of the llm families. the result was evaluated by an expert in linguistics using the same guidelines and criteria as students were evaluated on the exam. the results are evaluated according to 15 criteria divided into four categories: knowledge and understanding, argumentation, language quality, and creativity. for each criterion, students can get 1 to 4 points, except for language quality for which 0 points can be assigned as well. the overall score can range from 11 to 60 points. in our evaluation, the same scoring method was used to strictly comply with the official guidelines. the results of the evaluation of the text analysis and writing skills are shown in similarly, knowledge and understanding was based on the number of facts mentioned in the text, therefore shorter analysis had a disadvantage in this category. the text generated by qwen2 was very difficult to understand with many illogical sentences, which led to a low score in other categories. the text generated by mistral had many agreement errors, such as subject-verb, noun-adjective, tense, gender, and singular/plural disagreement. demonstrating author's individuality was one of the conditions to get top scores in originality (part of creativity), which was lacking in all of the analysis. there was also a lack of comparison to nowadays, which was a condition to get top scores in the knowledge category. all of the models analyzed the two texts mostly separately, using the specified content component subsections. a cohesive, fluent analysis with introduction, discussion, and conclusion was expected instead. in this task, zero-shot learning did not work well. for such tasks, examples or more detailed instructions provided in the prompt would probably lead to better results. experiments were also conducted using llms as evaluators. each model was asked to assess the text analysis generated by every other model ten times. the results were not promising. the scores varied a lot between the runs, and the average scores by any model did not correlate with human evaluation. the worst category is language quality assessment. most of the errors found by the models were not actually errors, and many actual errors were missed. ## conclusion and further work the experiments validated the use of latvian high school centralized graduation exams as a source for natural language understanding (nlu) tasks. this gives us motivation to continue the work on expanding the size of the data set. the performance gap for latvian between the best open-weights llms and gpt-4o is minor. the biggest surprise was the performance of the relatively small gemma2 27b model. the quantized version is small enough to be run on a consumer grade gpu, making it perfect for large-scale nlu tasks, such as classifying or tagging documents even without fine-tuning. this opens up huge possibilities for nlp in latvian, especially in digital humanities. the performance in natural language understanding did not correlate with the performance in natural language generation (nlg) in text analysis tasks, showing the importance of evaluating both tasks separately. despite nlg shortcomings discussed, in human evaluation the best open-source llms achieved score above 66% (40 out of max 60 points) compared to average 56% score reported for the actual human graduation exam. unfortu-nately, nlg tasks are hard to evaluate, since human evaluation requires a lot of resources. even the best models showed no correlation between their assessment of other llm's on the nlg tasks, and the human evaluation. this makes the current generation of llms not well suited for nlp tasks like error detection and correction, text normalization and data denoising in latvian. the high out-of-vocabulary (oov) word density score, measured against the large latvian thesaurus database @xcite was a good indicator of poor language quality, but a low number of oov words is not an indicator of high nlg score, because most errors were grammatical errors. finding a good automatic nlg evaluation methodology is still an open research question. the dataset created and used in this evaluation is available as open data via a github repository. 3 overall, the open-weights models show promising performance on latvian, suggesting that finetuning such models for low-resource languages might achieve competitive results with much lower costs compared to training language-specific llms from scratch.