IRB - MT at WMT 25 Translation Task: A Simple Agentic System Using an Off-the-Shelf LLM (2025)

Abstract: Large Language Models (LLMs) have been demonstrated to achieve state-of-art results on machine translation. LLM-based translation systems usually rely on model adaptation and fine-tuning, requiring datasets and compute. The goal of our team’s participation in the “General Machine Translation” and “Multilingual” tasks of WMT25 was to evaluate the translation effectiveness of a resource-efficient solution consisting of a smaller off-the-shelf LLM coupled with a self-refine agentic workflow. Our approach requires a high-quality multilingual LLM capable of instruction following. We select Gemma3-12B among several candidates using the pretrained translation metric MetricX-24 and a small development dataset. WMT25 automatic evaluations place our solution in the mid tier of all WMT25 systems, and also demonstrate that it can perform competitively for approximately 16% of language pairs.

## introduction machine translation (mt) is an important yet unsolved nlp task with significant practical applications @xcite . large language models (llms) have become a basis for state-of-the-art mt solutions, and the best approaches rely either on commercial llms or on open-weights models adapted using translation-specific data @xcite . however, commercial cloud-based models may introduce cost constraints and dependency issues, while the adaptation of open-weight models commonly requires substantial computational resources and specialized training datasets. recent developments in llms yielded smaller yet capable models such as gemma3 @xcite and qwen3 @xcite , which are multilingual and support instruction following and reasoning. in parallel, research in multi-agent systems led to task-independent workflows, such as we hypothesize that the combination of capable smaller llms and agentic workflows has the potential to create a resource-effective translator with solid performance. our participation in the wmt25 translation task @xcite is oriented toward testing this hypothesis in the controlled environment of the "constrained" track, which allows only openly available datasets and models below 20b parameters. we evaluate our approach on the wmt25 general machine translation task, which assesses mt systems across four domains (news, social media, speech, and literary) with document-level context and multi-modal resources including video, image, and speech data @xcite . the task comprises 16 language pairs covering major language groups including morphologically rich, lowresource, and diverse script languages. we also participate in the multilingual subtask, which extends evaluation to 15 additional target languages. overview of the dataset statistics can be found in as the first step in designing our system we tested several multilingual generative models and encoder-decoder models specialized for translation, evaluating them on a subset of pairs from the wmt24++ dataset @xcite . gemma3-12b @xcite proved to be the best solution in terms of metricx-24-xl metric (juraska et al., 2024) so our final system is based on this model. we enhance the model with a version of the self-refine workflow @xcite based on a prompt adapted for machine translation. our system uses as input only the text modality, and works with paragraph-sized text segments. wmt25 evaluations using a number of automatic translation metrics @xcite show that our system achieves mid-level performance when compared with all the participating systems that include team-submitted solutions (both constrained and unconstrained), as well as benchmarks added by the organizers (individual llms and commercial solutions). the system achieves competitive performance for five language pairs (@xmath0, @xmath1, @xmath2, @xmath3, @xmath4. we make the code of the system freely available. 1 2 related work several multi-agent llm systems for machine translation (mt) have been proposed recently, often inspired by human collaborative problemsolving and professional translation workflows @xcite @xcite @xcite . these systems aim to address the limitations of single-model mt systems, including in handling linguistic nuances, context, and idiomatic expressions. they consist of autonomous llm-based agents assigned to specialized tasks, organized in a workflow and sometimes embedded in an iterative loop. such multi-agent systems can demonstrate superior performance compared to non-agentic baselines @xcite @xcite , . for example @xcite reported large improvements over conventional zero-shot prompting and even outperformed top-performing wmt 2024 systems in some cases. furthermore, human evaluations frequently show a preference for translations produced by these multiagent systems @xcite . while effective, the proposed systems rely on powerful llms (such as gpt-4o and gemini 1.5 pro) and often involve complex and computationintensive workflows, which entail latency and computational overhead. in contrast, our system relies on a smaller open-weights model and a simple onestep self-refine workflow @xcite . this makes it resource efficient with translation time comparable to zero-shot inference with a single llm. the wmt25 translation evaluation dataset encompasses 31 language pairs with diverse characteristics, enabling assessment across different language families, resource levels, and translation scenarios @xcite . the list of language pairs and the statistics of associated sub-datasets is displayed in the general mt task comprises 16 language pairs covering both large and small languages. the task includes both english-centric and non-english language pairs, with english-to-target directions covering arabic (egyptian), bhojpuri, chinese (simplified), czech, estonian, icelandic, italian, japanese, korean, maasai (kenya), russian, serbian (latin), and ukrainian. additionally, the task features non-english source languages with czechto-german, czech-to-ukrainian, and japanese-to-chinese pairs. the dataset exhibits significant variation in size and text complexity. russian dominates with 7,804 texts, followed by hindi with 5,087 texts, while smaller language pairs like italian contain only 87 texts. the dataset contains texts from four domains: news articles, transcripts of video speech associated with audio data, social media posts associated with printscreen images, and literary texts. a significant number of texts from the general mt subtask does not belong to any domain. the multilingual (sub)task extends evaluation to 15 additional target languages: bengali, german, greek, persian, hindi, indonesian, italian, kannada, lithuanian, marathi, romanian, serbian (cyrillic), swedish, thai, turkish, and vietnamese. english is the only source language, and all the texts belong to one of the four domains described above. as the statistics in ## system the goal of our submission was to examine how a lightweight, simple, and computationally efficient agentic workflow for mt compares against a range of other approaches . to enable a fair comparison with systems utilizing a similar level of resources, we submitted our solution to the "constrained" track @xcite which allows only open-weights models with a combined size below 20b parameters. we always apply our translation methods on the level of a paragraph, i.e., no document-wide context is used. additionally, only text is used as input, i.e., image and speech data provided for parts of the dataset is not used. ## results irb-mt submitted translations for all of the 31 language pairs of the generalmt and multilingual subtasks @xcite . automatic nonhuman evaluations show that irb-mt is a midtier constrained system that outperforms baseline gemma3-12b in most cases, which demonstrates the benefit of the self-refine approach. out of the 16 pairs for which human evaluation was performed, irb-mt's performance was high enough for it to be selected for human evaluation in the case of 13 pairs (81.25% of pairs ( kocmi et al., 2025b) ). to further analyze the relative performance of irb-mt we rely on the autorank metric that aggregates the rankings of multiple translation quality metrics, in order to mitigate biases of individual metrics @xcite . other evaluated translation systems consist of systems submitted by participating teams, and additional organizerchosen systems included for comparison @xcite . we label these two groups "team" and "benchmark" systems, respectively. benchmark systems include open-weight and cloud-based llms, and commercial mt systems. the autorank statistics in as for the unconstrained track irb-mt compares relatively favorably with the team systems, often placed close to the middle of the list. however, this probably has most to do with the fact that, surprisingly, submitted unconstrained systems generally perform worse than the submitted constrained systems @xcite . unconstrained benchmark systems consist of midsized llms (above 20b parameters), commercial llms, and commercial translation systems. these systems mostly outperform irb-mt for large languages and for most european languages, while irb-mt compares more favorably and sometimes competitively on mid-and lower-resourced languages and non-european languages. as the table human evaluation @xcite of selected translation systems was performed by applying the esa annotation method @xcite and, for two language pairs, by applying the mqm method @xcite . the evaluated systems were clustered based on the statistical significance of performance differences @xcite . when compared to other human-evaluated systems, irb-mt is located at or below the median for the majority of language pairs. out of 11 pairs for which the systems were esa-annotated, for 6 pairs irb-mt either has a score above 66%, or it is not significantly different from systems scoring above 66% @xcite . according to esa annotation guidelines the score of 66% is the threshold for translations with "most meaning preserved and few grammar mistakes" @xcite . we take this as an argument for our system's ability to generate good translations for a non-trivial percentage of language pairs. for the challenging english-arabic pair, irb-mt outperforms all of the constrained systems and compares favorably to 50% of the unconstrained systems @xcite . ## conclusion and future work we proposed a simple lightweight "self-refine" workflow for machine translation, based on the multilingual gemma3-12b llm with instructionfollowing and reasoning capabilities. our approach was included in the wmt25 @xcite evaluation with automatic metrics, performed on a large set of translation systems. the results place our system, on average, in the mid-tier, but there is a significant performance variations across language pairs, with the tendency of better per-formance on mid-and low-resource languages. while the irb-mt fails to come close to the topperforming systems, human esa annotations show that it often produces good translations @xcite . future research on the improvement of our approach should tackle the issue of performance variability. human or llm-assisted examination of language pairs with the lowest scores would reveal the type of errors and suggest improvements. although the agentic workflow by-and-large outperforms the base gemma3 model, there is a significant variation in the gap between the two. analyzing the language pairs and texts for which irb-mt fails to improve upon the base system could lead to the refinement of the agentic system. other directions for further improvement of irb-mt include: a larger thinking budget, an iterative improvement loop, and a more granular agentic system with specialized roles. refinement of the agentic structure could combine elements of the existing mt workflows @xcite @xcite @xcite . the key challenge is to boost performance without relying on overly complex and long workflows, or on too large llms. examining different combinations of the translator llm and the refiner llm could also lead to a performance boost, and to insights into the models' behavior. in general, it would be interesting to examine how close can purely agentic approaches come to the data-driven approaches based on llm adaptation and fine-tuning, and how does the compute-vsperformance tradeoff look like.