IRB - MT at WMT 25 Terminology Translation Task: Metric-guided Multi-agent Approach (2025)

Abstract: Terminology-aware machine translation (MT) is needed in case of specialized domains such as science and law. Large Language Models (LLMs) have raised the level of state-of-art performance on the task of MT, but the problem is not completely solved, especially for use-cases requiring precise terminology translations. We participate in the WMT25 Terminology Translation Task with an LLM-based multi-agent system coupled with a custom terminology-aware translation quality metric for the selection of the final translation. We use a number of smaller open-weights LLMs embedded in an agentic “translation revision” workflow, and we do not rely on data- and compute-intensive fine-tuning of models. Our evaluations show that the system achieves very good results in terms of both MetricX-24 and a custom TSR metric designed to measure the adherence to predefined term mappings.

## introduction when translating texts from specialized technical domains such as medicine, finance, or law, it is important to translate technical terms accurately and consistently @xcite . to this end, the translation systems can be provided with an existing list of terms and their translations. while the llms have emerged as state-of-the-art models for mt @xcite , they are rarely evaluated on specialized domains that require strict adherence to terminology. the goal of the wmt25 terminology translation task @xcite is to determine how well do the modern mt systems tackle this challenge. recently, a number of capable multilingual llms that support instruction following were made available to the community @xcite @xcite . in parallel, research in multi-agent workflows showed that embedding individual llms in multi-step workflows leads to performance gains. these workflows can be generic, such as self-refine @xcite , or task-oriented workflows in which agents are assigned natural task-specific roles @xcite . our goal was to propose a resource-efficient solution based on smaller instruction-and reasoningcapable multilingual llms with open weights, embedded in an agentic workflow for performance improvement. we hypothesize that such a workflow could lead to solid performance for a number of language pairs, as the multilinguality of modern llms facilitates translation, and their instructionfollowing capabilities enable the implementation of complex terminology-and revision-related instructions. such a system does not require datasets and compute for model adaptation and fine-tuning -only relatively modest inference-time compute to run the agents is needed. participation in the wmt25 terminology task enables us to compare such a system with various other approaches. wmt25 terminology task datasets are divided into track1 and track2, consisting of texts from the information technology and financial domains, respectively @xcite . track1 datasets contain paragraph-level texts and cover en-de, en-es, and en-ru language pairs. track2 datasets contain long document with documentlevel terminology mappings and cover en-zh and zh-en pairs. predefined @xmath0: "proper" terminologies covering technical terms, and "random" terminologies with random words. the idea is to measure the influence of the predefined terminology on the performance. for the same reason, an additional "no terminology" setup is included in the task. our system, named meguma, is an agentic translation system that operates in three phases: 1) translation with individual llms; 2) translation revision based on reasoning llms; 3) the selection phase using a custom terminology-aware translation quality metric. revision agents act as senior translators who revise the work of junior translators @xcite . in the revision phase, a reviser llm examines a number of phase one translations and composes the final translation. variants of gemma3, qwen3, and eurollm models are employed as base translators, while gemma3 and qwen3 models set up for "thinking" mode are used for revision. the final translation is selected from all the generated translations using a combination of two translation measures: metricx @xcite , and a custom tsr metric that measures the adherence to predefined @xmath1. the final system achieves good results. for track1 all metricx scores (range between 0 and 25, lower is better) are below 2.00, while the tsr scores are above 0.85 (indicating that 85% of source terms are correctly translated). for track2 all metricx scores are below 2.5 (below 2.0 for the "proper terms" setup), and the tsr scores range between 0.69 and 0.80 except for the random terms setup with scores between 0.40 and 0.50. our contributions are: a new agentic system for terminology-aware machine translation, a new tsr metric for approximating the adherence to predefined terminology, and a ttq metric that aggregates metricx and tsr in order to combine translation quality and the adherence to predefined terminology. we make the code and the models' output freely available. 1 ## datasets three different collections of datasets were released for the terminology mt task: 1) dev, a set of three datasets for sentence-and paragraphlevel terminology translation task; 2) track1, a set of three datasets for sentence-and paragraph-level terminology translation task; and 3) track2, a set of ten test datasets for document-level terminology translation task. each of these two tracks focus on a different domain, with track1 dealing with the information technology domain, while track2 consists of texts and terminology from the finance domain @xcite . the datasets in dev and track1 are quite similar, with the same language pairs and one unique dataset per pair: english to german, english to spanish, and english to russian. track2 contains 1 https://github.com/igrubi/irb-mt-wmt2025 ten unique datasets -five datasets for english to traditional chinese translation and five datasets for traditional chinese to english. each text from dev, track1, and track2 datasets is associated with a set of terms and their translations. these term mappings define the terminology-aware translation tasks. for dev and track1 a term dictionary is provided at the sentence/paragraph level, and it contains source_term : target_term entries. for track2 each document is provided with a dictionary of source terms mapped to a list of viable target terms. each track1 and track2 has three variants of term mappings: 1) no terms; 2) proper terms; and 3) random terms. these three variants differ in their inputs. the no terms variant uses only input texts, the proper terminology variant adds specialized dictionaries for domain-specific terms, and the random terminology variant uses dictionaries with randomly selected words from the texts to compare the impact of accurate versus arbitrary terminology on system performance @xcite . while the dev and track1 datasets contain 500 smaller texts with an average of approximately 10 whitespace-separated tokens (see the appendix, to reduce the required computational resources and increase the quality of translations and evaluations, we convert track2 translation datasets from the document-level to the paragraph-level format of track1. this process requires two steps: text splitting and term splitting. following a data analysis showing that track2 texts contain paragraphs separated by two or more newline characters, we first use regular expressions to split texts into paragraphs. however, since the term mappings are provided on the document-level, another step is needed to create paragraph-level term dictionaries containing only the source terms that occur in a paragraph. to achieve this, for each paragraph we examine all source terms in the document-level term dictionary and check if a source term is present in the paragraph text. however, we need to prevent subterms (substrings of longer terms) to be detected in places where their super-term exists. to this end we employ these steps: 1) lowercase the text and all terms; 2) sort the list of all document terms by decreasing length (longest terms at the start of the list); 3) iterate through the term list and check if a term is present in the paragraph text. if a term is detected, we save it to the paragraph-level dictionary and remove all occurrences of the term from the paragraph text. this way a sub-term will not be included after a super-term. with this procedure we create a paragraph-level term dictionary that contains only a relevant subset of document-level terms. with this procedure we only reduce the set of dictionary keys to the paragraph-level. the dictionary values remain the same as in the original track2 datasets, i.e., each source term is still mapped to a list of viable target terms. this is the key difference between the generated track2 paragraphlevel term dictionaries and the track1 dictionaries, and we tackle these two cases differently with translation and revision agents. finally, in order to recreate the whole translated document later, we give each paragraph a unique identifier that contains information on the exact location of the paragraph within the document. after the final submission we discovered that there was a bug in the function that we used for splitting chinese terms. namely, when stringmatching a term in a text, we required the term to be surrounded by word-separating whitespace. while this is a proper approach for european languages, it is not for traditional chinese because often there is no such separation between words. as a result, we lost a good number of the paragraph terms for the datasets where the source language is chinese. for details, see the appendix, subsection a.3, difference between ## the system our translation system produces an output in three phases: 1) individual translator llms generate initial translations, 2) reviser llms generate improved translations from the initial ones, and 3) all of the candidate translations are pooled, and the best one is selected based on a custom quality metric. in the development phase we benchmarked a number of translator-llm candidates on a development set consisting of a subset of pairs from both the dev, track1 and track2 datasets. most of the evaluated llms demonstrated good results and were included into the final system. we start the detailed description of the complete system with metrics, used both for the benchmarking of llm and for the selection of final system outputs. ## results the final system produces high-quality translations with low metricx scores (tables 3 and 5 ) and high tsr scores (tables 4 and 6 ). tsr scores are significantly higher for track1 than for track2-this could be caused by both the language differences (european languages vs. chinese) and by differences in the complexity of the terminology dictionaries (single-choice vs. multi-choice terms). when compared to the translation produced by all of the llm agents used (both translators and revisers) the final system's translations have the best metricx and tsr scores for almost all track1 and track2 datasets. the only exceptions are: 1) translations from english to german for the "proper term" case where a standalone trans-eurollm has the best metricx score (track1, see the appendix, the final system uses a metric-guided approach based on the custom ttq metric used to select the best output from a pool of translations. the selection step is an important part of the proposed translation pipeline since it significantly increases translation quality scores, raising them above the scores of the best individual translators and the scores of the best reviser agents (see the appendix, section e). the reason why the selector chooses the best translation from the pool of translations of all agents and not only the reviser agents is shown in tables 7 and 8 . these tables show the frequency with which an agent's output was chosen for the final solution. although the reviser agents have a higher chance of being chosen by the selector (17.13% vs. 6.89% in track1, and 14.3% vs. 9.42% in track2), in total about half of all samples are selected from individual translator agents (48.2% in track1 and 56.5% in track2). in addition, there is an interesting trend in the results. metricx scores are increasing (i.e. translation quality drops) for random and proper term cases compared to the no-term case (tables 3 and 5 ; for more information, see the appendix, subsections e.1.1, e.2.1, e.2.2 and e.2.3). we hypothesize that forcing a predefined translation of a set of terms reduces the general translation quality because translators need to balance between two different objectives: 1) general translation and 2) terminology constraint. however, this hypothesis requires more comprehensive research to be proven. in track1, our system has an average chrf2++ of 67.2 (6th place) and a high average term-acc of 97.4 (4th place). in terms of pareto optimality between chrf2++ and term-acc, our system is near-optimal, with only two systems having pareto dominance over it: o3-term-guide and duterm. in track2, our system has an average chrf2++ of 54.3 (3rd place) and a competitive average term-acc of 79.5 (2nd place), with only one system, commanda_wmt, having pareto dominance over it. in our final submission there is an error for the proper and random cases of track2 translations from traditional chinese to english. the error originates in the data preparation process (more information can be found at the end of subsection 2). we estimated the drop in scores as the result of this error (see the appendix, subsections e.2.4 and e.2.8). the expected drop is around 0.30 metricx scores and around 0.03 and 0.11 tsr scores for random and proper term cases. expectedly, this error effects the official results of our system: there is a large gap between the term-acc score for en-zh (96.6) and the term-acc score for zh-en (62.4) @xcite . to examine the impact of the error, we evaluated foot_1 the debugged version of the system and obtained the zh-en term-acc of 96.6 (see e.4 for more details). our repository foot_2 contains details of the error, reproducibility instructions, and the outputs of both error-containing and error-free versions of the system.