MUG -Eval: A Proxy Evaluation Framework for Multilingual Generation Capabilities in Any Language (2025)

Abstract: Evaluating text generation capabilities of large language models (LLMs) is challenging, particularly for low-resource languages where methods for direct assessment are scarce. We propose MUG-Eval, a novel framework that evaluates LLMs’ multilingual generation capabilities by transforming existing benchmarks into conversational tasks and measuring the LLMs’ accuracies on those tasks. We specifically designed these conversational tasks to require effective communication in the target language. Then, we simply use task success rate as a proxy for successful conversation generation. Our approach offers two key advantages: it is independent of language-specific NLP tools or annotated datasets, which are limited for most languages, and it does not rely on LLMs-as-judges, whose evaluation quality degrades outside a few high-resource languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and low-resource categories, and we find that MUG-Eval correlates strongly with established benchmarks ( r > 0.75) while enabling standardized comparisons across languages and models. Our framework provides a robust and resource-efficient solution for evaluating multilingual generation that can be extended to thousands of languages.

## introduction large language models (llms) have demonstrated remarkable capabilities in many languages, but evaluating their multilingual generation abilities remains a significant challenge, particularly for low-resource languages. these challenges are particularly pronounced for low-resource languages, which often lack robust natural language processing tools, comprehensive reference corpora, or established benchmarks. consequently, evaluation resources for these low-resource languages predominantly derive from massively multilingual evaluation benchmarks @xcite @xcite . extending and evaluating natural language generation tasks presents considerable complexity, especially in the absence of languagespecific resources. recent approaches @xcite have employed llms-asjudges, but they face an inherent limitation-the reliability of judgments depends on the evaluator llm's performance in the target language. while this limitation may be less pronounced for highresource languages @xcite , the applicability of such approaches to low-resource languages remains unclear and has not been rigorously validated. conventional evaluation approaches for in this paper, we propose mug-eval, a framework for evaluating the multilingual generation capabilities of llms, particularly for languages where direct evaluation proves challenging or infeasible. our methodology creates information-gap scenarios that require successful communication in the target language to complete tasks, such as providing hidden information to one agent while another must discover it through questioning. we implement three tasks in mug-eval by adapting existing benchmarks into conversational and multilingual settings-easy twenty questions @xcite , mcq @xcite , and code reconstruction @xcite -where task completion rates serve as proxies for different aspects of generation ability: reasoning, instruction following, and programming ( §3.1). our approach builds on the insight from @xcite : instead of directly assessing llm-generated text quality, we can indirectly measure how well the llm comprehends what it has itself generated. we evaluate 8 llms across 30 languages from high-, mid-, and low-resource categories as defined by @xcite . our experiments demonstrate that mug-eval has strong discriminative power, enabling precise comparisons both across languages and across models ( §4.1). the framework shows high internal consistency among its three tasks and correlates strongly (pearson's @xmath0.75) with established benchmarks including belebele @xcite , multiq @xcite , and global-mmlu (singh et al., 2025) ( §5.1). additionally, our analysis of mcq conversation reveals that when native-language references are unavailable, english is not always the optimal substitute language, particularly for lowresource languages ( §5.2). our primary contribution lies in proposing mug-eval foot_0 , a novel language-agnostic framework for evaluating multilingual generation in large language models through self-comprehension tasks, without relying on language-specific nlp tools or human annotations. to demonstrate the utility and effectiveness of this framework, we structure the paper as follows. we begin by reviewing the landscape of multilingual generation evaluation, identifying critical gaps in existing methodologies that motivate our approach ( §2). we then present the design of mug-eval, introducing three conversational tasks that recast generation evaluation as a communication-based task ( §3). we evaluate eight large language models in 30 linguistically diverse languages, demonstrating strong correlations with established benchmarks while offering unprecedented scalability ( §4). through detailed analysis, we uncover cross-linguistic performance patterns and validate the effectiveness of mug-eval as a robust, language-agnostic evaluation framework ( §5), and conclude with directions for future work in multilingual llm evaluation ( §6). ## related work task 1: easy twenty questions find the target word 彗星 target word candidate words questioner llm answerer llm question generation question answering それは動物ですか no それは食べ物ですか それは天体ですか yes それは惑星または衛星ですか no is it a celestial body? is it a planet or satellite? no 彗星 judge by string match task 2: mcq conversation choose the answer for the question ini adalah akuisisi terbesar dalam sejarah ebay. perusahaan berharap untuk ... dan memperoleh popularitas di area tempat skype memegang posisi kuat, seperti ... peppermint, billboard comet is it an animal? is it food? what was acquired by ebay? it is the biggest acquisition in ebay's history. the company hopes to ... and gain popularity in areas where skype holds a strong position, such as ... did ebay acquire microsoft? did ebay acquire apple? did ebay acquire skype? this is a function called `triangle_area` which takes and `h`. it returns a value which is the area of a trian the formula: half of `a` multiplied by this is a function called `triangle_area` which takes two parameters `a` and `h`. it returns a value which is the area of a triangle, calculated using the formula: half of `a` multiplied by `h`. chrf (popović, 2015) assess generation quality by comparing outputs against reference texts, usually requiring human-generated target texts as ground truth. these metrics are widely adopted in benchmarks such as mega @xcite , @xcite , multi-if @xcite , and benchmax @xcite . however, such reference-based approaches are limited by their reliance on high-quality parallel data, which is scarce in many languages. moreover, they struggle in cross-lingual comparisons due to their sensitivity to lexical and syntactic features. to address these limitations, reference-free methods-particularly those using llms as evaluators-gained attention ( dang drawing inspiration from 20q @xcite and @xcite , our framework builds on tasks that inherently require both comprehension and generation, foregrounding successful communication as the central evaluation criterion. designed to be language-agnostic, reference-free, and llmindependent, it offers a more equitable and scalable multilingual evaluation across an unlimited spectrum of languages. this section provides detailed descriptions of each task and evaluation procedures. additional details, including prompts and generation parameters, are provided in the appendix b.2. ## experiments models. we evaluate eight multilingual large language models to assess their generation capabilities across diverse languages. our selection includes four open-weight models: llama 3.3-70b (llama team, 2024), llama 3.1-8b, qwen2.5-72b (qwen team, 2024), and qwen2.5-7b, alongside four closed-source models: gpt-4o (openai, 2024), gpt-4o-mini, gemini 2.5 flash @xcite , and gemini 2.0 flash @xcite . all models are accessed via api endpoints, with gpt-4o variants served through azure openai services and the remaining models through openrouter. detailed model information is provided in the appendix b.1. languages. we test our framework on 30 languages grouped by resource availability following singh et al. (2024)'s classification, with 10 languages selected from each resource category. we include high-resource languages arabic (arb), chinese (zho), english (eng), french (fra), german (deu), hindi (hin), italian (ita), japanese (jpn), portuguese (por), and spanish (spa); midresource languages bengali (ben), greek (ell), hebrew (heb), indonesian (ind), korean (kor), lithuanian (lit), malay (zsm), romanian (ron), thai (tha), and ukrainian (ukr); and low-resource languages amharic (amh), hausa (hau), igbo (ibo), ## conclusion a fundamental limitation in multilingual evaluation is the reliance on ground-truth references or llm-based judgments, which are often unreliable or infeasible for low-resource languages. to address this, we introduce mug-eval, a languageagnostic evaluation framework based on three conversational task completion between llms that assess both generation and comprehension. we evaluate 8 llms across 30 languages using mug-eval. our framework demonstrates strong internal consistency and aligns well with established multilingual benchmarks, while remaining reference-free and cost-effective. our results highlight a few implications. first, mug-eval enables fine-grained performance comparisons even in lowresource settings due to its task diversity and saturation characteristics. second, we find that substituting native-language passages with english often degrades performance-especially for low-resource languages-underscoring the need for evaluation methods that go beyond english-centric assumptions.