m H uman E val - A Multilingual Benchmark to Evaluate Large Language Models for Code Generation (2025)

Abstract: Recent advancements in large language models (LLMs) have significantly enhanced code generation from natural language prompts. The HumanEval Benchmark, developed by OpenAI, remains the most widely used code generation benchmark. However, this and other Code LLM benchmarks face critical limitations, particularly in task diversity, test coverage, and linguistic scope. Current evaluations primarily focus on English-to-Python conversion tasks with limited test cases, potentially overestimating model performance. While recent works have addressed test coverage and programming language (PL) diversity, code generation from low-resource language prompts remains largely unexplored. To address this gap, we introduce mHumanEval, an extended benchmark supporting prompts in over 200 natural languages. We employ established machine translation methods to compile the benchmark, coupled with a quality assurance process. Furthermore, we provide expert human translations for 15 diverse natural languages (NLs). We conclude by analyzing the multilingual code generation capabilities of state-of-the-art (SOTA) Code LLMs, offering insights into the current landscape of cross-lingual code generation.

## introduction llms have transformed software development with their ability to generate programming code from simple natural language instructions. llms are trained on extensive datasets that include diverse code samples, aiding programmers in code development and debugging. they also make programming more accessible to beginners. however, assessing the performance of these models across different coding tasks is still a major challenge. comprehensive testing is essential to verify that these models are both effective and adaptable, 1 github.com/mraihan-gmu/mhumaneval-benchmark rather than only performing well under specific conditions. the most widely used benchmark for evaluating these models is openai's humaneval @xcite , which includes a collection of 164 tasks generated by human experts. each task includes an english prompt, a canonical solution provided by the authors, and three test cases. although this benchmark is commonly used, it has significant limitations, such as limited test coverage and minimal support for non-english and non-python prompts. while recent variations @xcite of humaneval address some of these issues, most do not include prompts in nls other than english and, in particular, in low-resource nls. consequently, current benchmarks fail to provide key insights into the multilingual capabilities of llms in the context of code generation. figure 1 demonstrates one such example. while the widely used gpt3.5 @xcite model performs perfectly for the original prompt "write a python code snippet that detects whether a year is a leap year or not.", it fails when the same prompt is given in a low-resource language (nyanja, in this case). in this work, we introduce mhumaneval, a novel multilingual code generation benchmark including coding prompts in 204 nls and expert human translations for 15 nls. mhumaneval further includes canonical solutions in 25 pls, including 4 new pls that are not covered by any prior benchmarks. the primary contributions of this paper are as follows: the paper addresses two research questions (rqs): finally, we also report secondary findings related to the translation quality of machine translation (mt) methods on coding prompts. ## related work the most widely used benchmark dataset for evaluating code llms is the aformentioned humaneval @xcite . another key benchmark is deepmind's mbpp @xcite , which includes 974 tasks with 3 test cases each. despite their popularity, these benchmarks have significant limitations, such as inadequate test case coverage, limited number of pls, and small task sets that do not represent real-world scenarios. other benchmarks, like concode @xcite (java), axibench @xcite (java), cseprompts @xcite (python) and codeapex @xcite (c++) focus on a single pl. to broaden pl coverage, @xcite combine both humaneval and mbpp and add 17 more popular pls besides python, such as c++, java, ruby, and php. however, all prompts remain in english, with only 3 test cases per task. similarly, the authors of babelcode @xcite include 14 pls and a more extensive test suite. to address test case coverage, @xcite introduce two datasets, humaneval+ and mbpp+, with significantly more test cases per task, ensuring both node and edge coverage. notably, code llm performance decreases with the additional test cases, highlighting the initial benchmarks' limitations. nevertheless, these benchmarks also use english prompts exclusively. few studies explore non-english coding prompts and evaluate code llms on them. the recent benchmark, humaneval-xl @xcite , extends coverage for both nls and pls. this benchmark includes coding prompts in 23 nls and solutions in 12 pls. the original prompts from hu-maneval @xcite are translated into 23 different nls using gpt-4 @xcite , with the quality of these translations assessed using a thresholded bertscore @xcite . while humaneval-xl explores multilingual prompts for code generation 1 ). we describe mhumaneval in detail in this paper and we evaluate sota models on this dataset. ## the mhumaneval benchmark the mhumaneval benchmark is curated based on prompts from the original humaneval @xcite dataset. it includes a total of 33,456 prompts, significantly expanding from the original 164. the curation process can be divided into several key steps, as illustrated in figure 2 and elaborated upon in the following subsections. ## experiments experimental setup for machine translation, gpt-4o and google translate are accessed via api keys, eliminating the need for gpu hours. we use recommended hyperparameters without tuning, setting max _ tokens to 1000 and temperature to 0.7. the nllb model runs on a single nvidia a100 gpu with 40 gb of memory. in code generation, proprietary models are accessed through api keys, negating the necessity for gpu hours. we adhere to recommended hyperparameters, setting max _ tokens to 1000 and temperature to 0.7. wizardcoder and aya utilize full precision (fp32) models on four nvidia a100 gpus, each with 40 gb memory, following model size type ref. gpt4o -base (achiam et al., 2023) claude-3.5-opus -base (anthropic, 2024) gpt3.5 175b base (brown et al., 2020) deepseek-coder-v2 236b finetuned (dai et al., 2024) wizardcoder 33b finetuned (luo et al., 2023) aya 33b finetuned (üstün et al., 2024) ## insights and analysis upon curating the mhumaneval benchmark and completing the model evaluations, we now present some key analyses and gained insights based on the obtained results. ## conclusion this study introduces mhumaneval, a comprehensive multilingual code generation benchmark for assessing llms across 204 languages. we curated high-quality prompts for each language and evaluated various models. our analyses, including ablation studies, provided insights into llms' multilingual code-generation capabilities, addressing the rqs posed in section 1: rq1: how do the code generation capabilities of llms vary when prompts are provided in english, or other high-, mid-, and low-resource nls? llms generally demonstrate optimal performance when prompted in english. for prompts in other languages, performance varies based on the language's resource level. high-resource languages tend to yield superior results compared to mid-and low-resource languages. the extent of performance variation is contingent upon the specific language of the prompt and the model's prior exposure and training in that language. this variation is likely influenced by the model's training data and the relative abundance of resources available for each language. rq2: how does the performance of multilingual llms compare to specialized, fine-tuned code llms in code generation tasks on the mhumaneval dataset? while code-finetuned language models excel at generating code from english prompts, multilingual models demonstrate strong proficiency across various nls. notably, even without specific code fine-tuning for different nls, they achieve decent results in code generation. this phenomenon suggests that multilingual models can generalize coding capabilities across nls, leveraging their understanding of multiple nls to support diverse linguistic contexts in programming. while we draw some insightful conclusions from curating and evaluating mhumaneval, to facilitate further research, we are making it publicly available. we plan to expand coverage to more nls and pls in future updates. despite the high cost of human translation, we included human annotations for 15 nls, including some low-resource and rare ones. currently, our dataset includes 164 prompts per language, following the humaneval benchmark, with plans to increase this number. we will also explore strategies to enhance low-resource language performance, such as transfer learning and diverse training datasets. comparative studies between general-purpose multilingual llms and specialized code llms will help optimize multilingual code generation.