LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriama (2025)

Abstract: This paper evaluates the language understanding capabilities of various large language models (LLMs) through an analysis of 112 translated and human-edited questions from the Multitask Language Understanding (MMLU) dataset, focusing specifically on two underrepresented languages: Latvian and Giriama. The study compares the performance of six state-of-the-art (SOTA) models, with OpenAI’s o1-preview model demonstrating superior performance across all languages, significantly outperforming non-proprietary models in Latvian and all other models in Giriama. Human editing of automated translations from English to Latvian yielded only a small, statistically insignificant improvement in performance estimates, suggesting that machine-translated benchmarks may be sufficient for comparing model performance in languages with established digital resources like Latvian. However, automated translation to Giriama proved infeasible, and model performance in Giriama remained poor, highlighting the persistent challenges LLMs face with low-resource languages. These findings underscore the need for more comprehensive datasets and improved machine translation capabilities for underrepresented languages, while emphasizing the importance of localized benchmarks and human evaluation in addressing cultural and contextual limitations in AI models.

## introduction the potential benefits of advanced artificial intelligence (ai) are vast, but to ensure these advantages are globally accessible, it's crucial that ai systems perform well across multiple languages. previous research has highlighted a significant disparity between the performance of frontier large language models (llms) in english compared to other languages, particularly those with limited resources @xcite @xcite . recently, there has been growing interest in assessing the capabilities of llms, with studies such as helm @xcite , @xcite , lambada @xcite evaluating various model functions. however, these evaluations mostly focus on english, leaving a gap in assessing llms' multilingual performance. as new language technologies based on llms rapidly emerge, evaluating their multilingual effectiveness is crucial @xcite . as ai models continue to evolve, it's essential to monitor how this language gap is narrowing. users working with models in various languages could greatly benefit from comparative performance analyses across different linguistic contexts. however, evaluating model performance in non-english languages presents challenges, for example manual translation is time-consuming, and this has forced the nlp community to focus on a selection of tasks and languages only. moreover, it has become standard practice to machine translate the training set but use human translation for test sets @xcite . while automated translation of benchmarks is cost-effective, it raises concerns about quality. conversely, human translations, though potentially more accurate, can be prohibitively expensive. driven by these considerations this study aims to address the following key questions: in our work, we utilize the massive multitask language understanding (mmlu) benchmark, which covers 57 subjects ranging from stem to humanities and social sciences. our goal is to enhance the understanding of llms performance in low-resource languages, with a specific focus on latvian and giriama, and to contribute to the development of ai systems that are both linguistically and culturally inclusive. ## evaluation metrics we evaluated the performance of six llms on four distinct language tasks: english, latvian, machinetranslated latvian, and giriama using an accuracy score. a uniform temperature setting of 0.5 was applied across all models, except for the o1-preview, for which only a fixed temperature of 1 was supported. for each model, accuracy was computed as the proportion of correct outputs from a test set comprising 112 samples. to account for uncertainty in the performance estimates, we employed the wilson score interval. this method provides a more accurate estimation of confidence intervals for binomial proportions p such as model accuracy by considering the sample size n and desired confidence level (typically set at @xmath0.96 for a 95% confidence interval). the wilson interval is preferred over traditional intervals like wald due to its robustness, particularly with smaller sample sizes, ensuring more reliable confidence bounds around the accuracy metric. we tested statistical significance using a twoproportion z-test, comparing each model's performance against the highest-performing model in its respective task category. this approach allowed us to ascertain whether differences in accuracy were statistically significant or occurred due to random chance. the evaluation process leveraged the uk aisi inspect framework (ai safety institute, 2024), which provided a standardized structure for implementing and automating our assessment. ## conclusion our evaluation of six frontier llms across english, latvian, and giriama reveals several critical insights about the current state of multilingual ai capabilities: