B ench MAX : A Comprehensive Multilingual Evaluation Suite for Large Language Models (2025)

Abstract: Existing multilingual benchmarks focus primarily on language understanding tasks. There is a lack of benchmarks to measure comprehensive critical capabilities of large language models (LLMs) across diverse languages, including instruction following, reasoning, code generation, and long context understanding. To bridge this gap, we develop BenchMAX, a multi-way multilingual benchmark that covers 10 diverse tasks, to evaluate LLMs’ general abilities across many languages. To ensure high data quality, each sample is post-edited by three native annotators after machine-translating from English into 16 languages. Extensive experiments on BenchMAX reveal uneven utilization of core capabilities across languages, emphasizing the performance gaps that scaling model size alone does not resolve. BenchMAX serves as a comprehensive multilingual evaluation platform, providing a promising test bed to promote the development of multilingual language models. The dataset and code are publicly accessible.

## introduction large language models (llms; @xcite @xcite have displayed remarkable proficiency across a wide range of tasks, mainly because they excel in instruction following, reasoning, long context understanding, code generation, and so on @xcite @xcite @xcite . inherently, these capabilities are languageagnostic. the numerical outcome remains consistent regardless of whether one learns the arithmetic expression 1 + 1 = 2 in english or chinese. similarly, when it comes to coding tasks, using english or chinese instructions does not alter the fundamental logic of the code. however, numerous empirical studies have shown that llms' multilingual performance is quite unbalanced when handling the same tasks @xcite @xcite across different languages. however, current benchmarks @xcite @xcite do not support comprehensive testing of the language-agnostic abilities of llms, particularly in low-resource languages, for several reasons. tasks like xwinograd @xcite and @xcite , based on multiple-choice formats, do not fully evaluate the generative capacities of llms. additionally, the limited language overlap across existing benchmarks poses challenges in assessing llm performance in diverse languages. recently, p-mmeval @xcite is proposed as a multilingual multitask benchmark, with the majority of its tasks still following a multiple-choice format. while it includes assessments like mgsm @xcite and mifeval that cover partial language-agnostic capabilities, this narrow focus still leaves a significant gap between research evaluation and real-world applications. to tackle this problem, we develop a comprehensive, multi-way, and challenging multilingual evaluation suite, called benchmax, to help the community better analyze and improve the languageagnostic capabilities of llms. covering 17 languages 3 , benchmax not only includes a broader range of language families but also emphasizes the diversity of writing systems across languages. meanwhile, benchmax highlights diverse advanced capabilities including instruction following, code generation, long context understanding, reasoning, tool use, and translation. for evaluating each capability, we include one or two related tasks as shown in figure 1 . domain translation, a byproduct of data construction, poses a new challenge for llms by necessitating fine-grained control and domain-specific terminology understanding over the translation process. to ensure high quality, we design an annotation framework to optimize the dataset quality with human efforts and llm feedback. the process involves translating data from english to other languages using machine translation systems, postediting each sample by three native annotators with multiple iterations across most tasks, and picking the final translation version using a strong llm that involves swapping sample positions for debiasing @xcite . popular multilingual llms are evaluated on benchmax, revealing that language notably influences language-agnostic capabilities of existing llms. interestingly, simply increasing the parameters can boost average performance on these tasks but does not universally reduce the performance gap across languages. moreover, compared to general translation, domain translation not only poses new challenges for llms but also requires new evaluation metrics. the main contributions can be summarized as follows: ## related work prior to the era of llms, most multilingual benchmarks are designed to evaluate discriminative models and take the form of classification tasks, such as xnli @xcite , xcopa @xcite , xcsqa (talmor et al., 2019), etc. @xcite however, due to their limited complexity and lack of format diversity, these tasks become less practical. recently, mgsm @xcite has become the most frequently used dataset in papers and reports from leading llm teams @xcite @xcite , which measures the mathematical reasoning capability across 11 languages. another widely used multilingual benchmark is the translated version of mmlu @xcite @xcite , which contains knowledgeintensive tasks. however, due to the lack of a unified dataset version, scores are often difficult to compare between studies. moreover, recent analyses have revealed that mmlu contains numerous ground truth errors @xcite , obscuring the accurate evaluation. more recently, include @xcite has been proposed to evaluate multilingual regional knowledge, lacking assessment of language-agnostic capabilities. to address these limitations, our work focuses more on language-agnostic capabilities and includes more tasks such as reasoning and code generation. furthermore, our benchmark incorporates a broader range of tasks to evaluate llms multilingual capabilities more comprehensively compared to previous aggregated benchmarks, such as seaeval @xcite and p-mmeval @xcite . importantly, all translated samples except the long context data in our benchmark are post-edited by native annotators. ## benchmark construction we extend the evaluation of the critical capabilities of llms into multilingual scenarios. to ensure sufficient linguistic diversity, we select 16 non-english languages ( § 3.1). meanwhile, 10 diverse tasks evaluating 6 crucial capabilities are chosen to facilitate comprehensive assessment ( § 3.2). sub ## conclusion we introduce benchmax, a comprehensive, highquality, and parallel multilingual benchmark comprising 10 tasks assessing crucial capabilities across 17 diverse languages. the multilingual task data is initially translated from english using machine translation and subsequently refined through multiple iterations of post-editing by native speakers, ensuring high data quality. through extensive experiments, we find that the language-agnostic capabilities of current leading llms remain uneven across different languages. while increasing model size consistently enhances multilingual performance, the performance gap between english and other languages persists, highlighting the need for further efforts to achieve balanced multilingual capabilities.