Findings of the WMT 25 Terminology Translation Task: Terminology is Useful Especially for Good MT s (2025)

Abstract: The WMT25 Terminology Translation Task releases new resources in high-stakes domains and investigates the capabilities of translation systems to accurately and consistently translate specialized terms. This year, we feature new domain and language coverage over previous editions, introducing two distinct tracks: (1) sentence-level translation in the information technology domain for English→German, English→Russian, and English→Spanish, and (2) document-level translation in the finance domain for English↔Traditional Chinese with a document-level one-to-many dictionary. Participants are challenged to translate texts under three modes: no terminology, proper terminology, and random terminology, allowing for a causal analysis of terminology utility. Evaluation combines overall quality, terminology accuracy, and terminology consistency. This shared task attracted broad participation, with 13 teams submitting 20 systems in Track 1 and 4 teams participating in Track 2. The results show that providing proper terminology consistently boosts both overall translation quality and term accuracy, whereas reliance on random terminology yields smaller gains. Despite the near-saturation of sentence-level benchmarks, document-level finance translation still fallsshort, indicating an urgent need for long-form evaluation and more robust metrics tailored to professional domains.

## introduction time flies. since the 2023 edition of the wmt terminology translation task @xcite , rapid advances in machine translation (mt) and large language models (llms) have achieved near-human quality for general-domain translation in several languages @xcite . nonetheless, it remains an open question whether these powerful models or techniques can successfully address terminology translation, where the need for accurate and consistent conversion of terminologies poses extra difficulty in addition to general translation quality. in professional fields like finance, medicine, and law, the correct use of specialized, agreed terms is critical for accuracy and clarity in communications, making terminology translation a research problem of high commercial value @xcite . the field has seen various efforts in modeling @xcite , evaluation @xcite , and translator tool development @xcite @xcite , but recent progress seems to slow down even for high-resourced languages. for example, at the 2024 conference on machine translation, only two papers were dedicated to terminology translation @xcite . it is in this context of exciting general domain progress versus modest attention to terminology translation that we organize the wmt25 terminology translation task. the task comes with two primary objectives: (1) to provide an understanding of the current landscape of terminology-aware translation, and (2) to announce and release new, humanannotated datasets to facilitate future research. we organize two tracks covering five translation directions and both sentence-and document-level translation. particularly, the document track features large document-level one-to-many dictionaries that are more realistic in production. in terms of evaluation, we run multiple metrics that target different facets of terminology conversion, all while taking into account the general quality. moreover, to estimate the added value (causal effect) of a provided terminology dictionary, systems are evaluated in three translation conditions: without a dictionary, with a proper terminology dictionary, and with a dictionary of random words. the main findings from the wmt 2025 terminology translation task across submissions are: ⋆ most systems involve an llm in some way. top systems achieve close to perfect terminology accuracy in the sentence-level track, indicating llm's suitability for the task, and a saturation in the sentence benchmark. we need to move to document-level benchmarking, which is also more aligned with the practical use. ⋆ contrary to the expectations based on previous research, there is little trade-off between general translation ability and terminology accuracy for most participating systems. ⋆ incorporating some terminologies always benefits overall translation quality; using proper terms is more useful than random terms for topperforming systems. our task is organized into two tracks, each focused on different domains and input sizes (of both texts and dictionaries), with a unified evaluation protocol. this allows this edition of the terminology shared task to cover wider scope of fields and input types (contrary to @xcite , where only medical domain and sentence-level data were provided), as well as tackling the domains in a more controlled manner (contrary to @xcite , where every translation direction had its own domain of texts, from nlp abstracts to web novels). ## data the data for both tracks foot_0 is provided in the jsonl format, where each instance has the following entries. see ## metrics and evaluation for both tracks, we run reference-based evaluation using gold translations and corresponding terminology dictionaries. we use three types of metrics in our shared task evaluation: overall quality (string match using bleu and chrf2++; document-level mqm with llm-as-a-judge), terminology accuracy, and terminology consistency. this choice of metrics is motivated by two factors: (1) we wish to measure different aspects in translating terminologies, and (2) modern automated metrics can be less robust than a simple string-matching chrf, especially in domains that were not part of the metrics' training data @xcite . new this year, we rank submissions based on the pareto efficiency measured by the quality metric and the terminology. we provide terminology consistency scores as an analysis, and use the document-level automqm scores for a separate ranking in track 2, as this is not fully empirically validated. ## participants and system descriptions this year, apart from our baseline, we see 20 systems in track 1 and 4 systems in track 2. their descriptions are provided below. for an easier navigation over the variety of approaches, we label them with the main features and components of particular submissions, namely: canddict term j ,cand j ←canddict term j ,cand j + 1 5: algdict i,term j ← candj 6: pseudrefdict ← {} # pseudo-reference choice 7: for term @xmath0: pseudrefdictterm k ←assignpseudoref(term k ) 9: for srci, tgt i , algdict @xmath1: for @xmath2: algorithm 3: terminology consistency (track 1: sentence level) with pseudo-reference initialization of the most frequent terms. input x is a list of sourcetranslation pairs ⟨(src 1 , tgt 1 ), . . .⟩. nmt nmt model (encoder-decoder) llm llm (decoder-only model) multiple models (agents, preprocess-ing+postprocessing, etc.) • training data: dataug data augmentation datcur data curation (filtering big corpora, enriching training data with annotation, etc.) • model update techniques: *ft fine-tuning, continuous pre-training, supervised fine-tuning, etc. *po various types of preference optimization: grpo, ppo, dpo, etc. duterm nmt llm dataug *ft this is a two-stage algorithm for terminology translation @xcite . it uses a terminology-aware nmt model fine-tuned from nllb 3.3b @xcite , and prompts gpt-4o for postediting. to construct the nmt training data, they first extract bilingual terminology dictionaries from wmt25 dev sets, which are then supplemented with terminologies generated by the llm. then they use an llm to synthesize parallel sentences containing one or more terminologies. specifically, the terms in both source and target sentences are bounded with special tags for identification. after filtering the training data for quality with comet-qe and other rules, the nmt undergoes terminology-aware fine-tuning. given the source, the nmt's translation and term pairs, they prompt gpt-4o to refine the translation for better fluency while keeping the constraints. 2025 ) presented an llm-based translation system using a pipeline approach that combines prompting with modular preprocessing and postprocessing components. in a preparatory stage, the llm analyzes the source text to extract key terms and idioms, which are then matched with entries from bilingual dictionaries; user-provided glossaries can also be incorporated to enforce consistent terminology. after translation, additional post-processing steps may be applied. for example, a custom seq2seq grammatical error correction model is used to improve icelandic translations. the system participated in both terminology tracks: for track 1, it employed its standard pipeline with terminology mapping, while for track 2, the backbone model was switched from claude 3.5 to gpt-4.1, as the former refused to translate some test examples. ismt-tiu (tiuterm-v0, tiuterm-v1) llm icl the team submitted two systems, both relying on in-context learning of llms, with fewshot examples retrieved with bm25 from the dev set. tiuterm-v0 is llama-3-8b @xcite . they use a novel approach of fine-tuning terminology translation using grpo @xcite . specifically, they introduce a terminology adherence reward, which penalizes outputs that do not contain the correct terminology. the training data are based on pseudo-terminology mined heuristically using named entities, noun phrases, and adverbial constructions. the reward during training is joined with a general mt quality reward using a quality estimation model. ## shared task results main results are presented in ## analysis apart from reporting general translation quality, terminology accuracy, and terminology consistency, we analyze the terminology incorporation and metrics, hoping to provide insights to the community: ## conclusions we now conclude the third iteration of the wmt terminology translation task. in comparison to the 2021 and 2023 editions, this time we featured both sentence and document translation tracks with brand new data and domains. the former track ensured continuity, while the latter approximated reallife use cases better. we introduced an llm-based document-level automqm and used pareto optimality to rank participants, but we kept the three inference modes from 2023 for a causal analysis. we attracted more than 20 submissions, three times more than the previous edition. the overwhelming majority used llm-based solutions with different types of training techniques. this goes in line with a general trend in the machine translation field towards llm-based solutions highlighted by @xcite . top-scoring systems in the sentence-level track reached good overall translation quality and nearly perfect term accuracy; the document track remains a more challenging task with respect to both metrics. the term consistency, on the contrary, shows a more stable behavior in both tracks, with overall higher scores for document-level mt. in terms of the inference modes, better systems benefit more from proper terminologies, while lower-scoring systems are less sensitive to dictionaries. finally, we see high correlations between term-based metrics, but not between them and the overall quality, which highlights the necessity to keep at least one terminologyspecific metric for this task. outlook. the lessons from the shared task also hint at the possible directions for its future iterations: we are open to collaborations, and we especially welcome resources that can be used towards test sets or human evaluation. stay tuned!