Findings of the WMT 25 Multilingual Instruction Shared Task: Persistent Hurdles in Reasoning, Generation, and Evaluation (2025)

Abstract: The WMT25 Multilingual Instruction Shared Task (MIST) introduces a benchmark to evaluate large language models (LLMs) across 30 languages. The benchmark covers five types of problems: machine translation, linguistic reasoning, open-ended generation, cross-lingual summarization, and LLM-as-a-judge.We provide automatic evaluation and collect human annotations, which highlight the limitations of automatic evaluation and allow further research into metric meta-evaluation. We run on our benchmark a diverse set of open- and closed-weight LLMs, providing a broad assessment of the multilingual capabilities of current LLMs. Results highlight substantial variation across sub-tasks and languages, revealing persistent challenges in reasoning, cross-lingual generation, and evaluation reliability. This work establishes a standardized framework for measuring future progress in multilingual LLM development.

## introduction we are witnessing rapid development of multilingual large language models (llms). however, as pointed out by recent works @xcite @xcite , multilingual benchmarks lack comprehensiveness, scientific rigor, and consistent adoption across research labs, undermining their value in guiding multilingual llm development. among common problems are benchmark contamination @xcite , label noise @xcite , reliance on non-native (machine-)translated instances @xcite , and inconsistent evaluation pipelines. for instance, some leading llm descriptions report multilinguality solely through translated mmlu. there is a mist surrounding multilingual evaluation that we aim to see through with this year's mist shared task. we introduce a novel multilingual evaluation benchmark that systematically assesses several key capabilities of llms across 30 diverse languages using the following sub-tasks: we benchmark several of the most commonly used open-and closed-weight systems on our tests. these tests provide a multi-faceted evaluation framework that highlights the strengths and limitations of current llms across diverse linguistic phenomena while drawing on the rigorous principles established within the mt evaluation research. in addition to automatic metrics, we conduct human evaluation for all sub-tasks without definitive answers, which is then used to assess llmas-a-judge systems. test sets, system outputs, and human judgments are released with a permissive license. 1 ## data and methodology in this section, we describe the datasets and preparation steps used for each of the sub-tasks in our benchmark. for every sub-task, we curated or adapted data across up to 30 languages. the highlevel statistics are in ## benchmarked models for this shared task, we defined two categories for model participation: constrained with several restrictions on model size and licensing; and unconstrained without any limitations. the same way as the general machine translation task @xcite . specifically, the constrained category is restricted to models with fewer than 20b parameters and requires that models be shared as open weights. unfortunately, our shared task did not obtain any (valid) participating systems. however, we collected and benchmarked outputs of popular models. the selection process was to identify the strongestperforming system per category for each of the popular model families. this approach ensured that both constrained and unconstrained models were consistently represented; the resulting model list thus reflects a broad yet balanced selection of models, enabling multilingual assessment of the current llm landscape across languages and problems. the list of all systems is in ## results in this section, we present the results and key insights for each sub-task and benchmarked model. although automatic evaluation was applied to all prompts and outputs, human evaluation was not conducted for all tasks, systems, or languages, due to budget constraints and annotator availability. nonetheless, human evaluation often proved more reliable than automatic metrics, so we release all annotations for future work on meta-evaluation. ## conclusion we introduced the wmt25 multilingual instruction shared task, where the main contribution is a unified benchmark spanning five evaluation tasks: machine translation, linguistic reasoning, open-ended generation, cross-lingual summarization, and llm-as-a-judge. the benchmark covers up to 30 languages evaluated both automatically and by humans, and emphasizes robust evaluation of multilingual llm capabilities. we release all prompts, outputs, and human annotations to facilitate reproducibility and research. across languages, the best systems achieve well below half of the attainable lr points, indicating that current models struggle with structured, language-agnostic reasoning rather than knowledge recall. in oeg human evaluation, score spread is widest for naturalness compared to instruction following and coherence, echoing user reports that non-english outputs often sound robotic or translationese. high-resource or typologically close pairs exhibit tight spreads among top systems, while lowresource and script-variant directions show large gaps and instability. automatic scores enable broad coverage, but human annotations exposed language/script biases, instruction-following failures, and cases where metrics or judges disagree, underscoring the value of our released human-rated subsets.