Findings of the WMT 2023 Shared Task on Machine Translation with Terminologies (2023)

Abstract: The WMT 2023 Terminology Shared Task investigates progress in machine translation of texts with specialized vocabulary. The participants were given the source text and segment-level terminology dictionaries for three language pairs: Chinese→English, English→Czech, and German→English. We evaluate 21 submissions from 7 teams on two main criteria: general translation quality and the effectiveness of translating specialized terminology. Systems took varied approaches — incorporating terminology at inference time or weakly supervised training that uses terminology access. While incorporating terminology dictionaries leads to improvement in the translation quality, incorporating an equal amount of information from the reference leads to similar results. This challenges the position of terminologies being the crux of meaning in translation, it can also be explained by inadequate metrics which are not terminology-centric.

## introduction general-purpose machine translation models often show limitations when applied to specialized tasks, like translating specialized vocabulary. this gap is critical in medicine, science, and law, where language precision is paramount -medical inaccuracies, juridical misunderstandings, and technological malfunctions can lead to serious problems. the translation of technical terms is not a mere exercise in lexical fidelity -it supports effective communication in highly specialized fields. terminology correctness and consistency has already been long in focus from the modelling @xcite , evaluation @xcite @xcite and translators' perspective @xcite @xcite . we shed light into recent advancement in this area by assessing mt systems with segment-level ## task description we focus on how translation quality improves with the incorporation of segment-level terminology on @xmath0, @xmath1, and @xmath2. participants are given source sentences along with a segment-level terminology dictionary (source and hints in for the purposes of this study, we define terminology as low-frequency words or phrases that occur typically within a particular domain, such as computer science paper abstract. we scan the source and references for such phrases and provide this segment-level annotation, together with the source, to the participants in the form @xmath3. given that the participants are given a part of the reference, y , this raises the following question: is the improvement in translation quality due to the information that a particular terminology x is translated as y or merely because a part of the reference is leaked to the model? to better attribute any performance gains, we therefore also test a different mode, where we give the participants "terminologies" where x ′ and y ′ are still aligned spans and translations of each other, but sampled randomly. that is, they are treated as terminology but are, in fact, random phrases (random in by comparing performance across these modes, we isolate the model's inherent translation ability and its ability to make use of the terminology. ## data for mt training, the participants were restricted to only the parallel or monolingual datasets enumerated in the wmt general track @xcite . 2 the inclusion of pre-trained models was permitted, provided that such usage was explicitly declared. any employment of terminology-specific datasets that were not part of the specified resources was expressly disallowed. for the terminologytargeted evaluation, we repurposed one dataset and created two new ones. from all of them, we provided 100 segments to the participants as a sanitycheck development set. see examples for all language pairs in ## participants and system descriptions we received a total of 21 per-language submissions from 7 teams. we provide short descriptions of their systems, based on the submitted details. adaptterm @xcite . the terminology-enriched mt system builds on @xcite ; @xcite . it consists of: 1. using an llm to generate bilingual synthetic data based on the provided terminology; 2. fine-tuning a generic model, opus, with a mix of the terminology-based synthetic data generated by #1 and a randomly sampled portion of the original generic data; and 3. generating translations with the fine-tuned model from #2, and then fixing translations that do not include the required terms with an llm. lingua custodia @xcite . this submission includes all three language directions. they use two strategies to extract synthetic terminology from the training data. the first one relies on the invariable n-grams between the source and the target sentence, while the second one extracts parallel sentences that appear inside another training sample as one terminology item. then, they train a transformer-based model with annotated data using the extracted terminology, identical to alam et al. @xcite . in addition, after the text annotation, they further apply several annotated data filters to reduce some bias introduced by the automatic annotation. the final trained model can be used directly to translate a text with any new terminology. opuscat @xcite . a standard transformer system is finetuned with parallel data where parts of the source sentences have been annotated with their corresponding translations in the target sentences, causing the system to learn to copy the annotated target parts from the source sentence into the target sentence. the translations are generated using a series of models, with different fine-tuned terminology models acting as backoff models to the base transformer model, in cases where the base transformer output does not contain the specified terminology. uedin (bogoychev and chen, 2023). their primary system, twoshot, is 2-shot decoding where we enforce terminology constraints via terminology hints in the source and if this does not work we use alignment-based methods to identify the mistranslated terminology word on the target side and penalize it, giving the decoder a chance to generate the hinted word. system tag is decoding with terminology hints while llm is an unconstrained contrastive system. bjtu-lb (no description paper). they train the in-context learning ability of the model, and then concatenate the term translation pairs in front of the sentence to be translated as the context. the model can generate different translation results according to different contexts. varco-mt @xcite . the forcegen is a transformer-based model that is tailored to ensure the appearance of given terminology in the generated output. by modifying the input format and decoding process, it incorporates a copy mechanism on the source side, allowing it to copy the target terminology from the provided terminology pairs. during the generation process, it uses a force decoding technique, which compels the model to actively generate the target terminology as needed. the tssnmt is a novel transformer-based nmt model that uses a shared encoder to process both input text and terminology. the model then employs cross-attention mechanisms between the two encoder hidden states and passes them through a gate, enabling the model to autonomously decide which pieces of information (input or terminology) to focus on during translation. huawei. did not submit system description. the translations are also on a subset not used for final evaluation. we include the results in the analysis sections in gray (huawei) for completeness but urge the readers not to draw any comparisons to other systems. ## evaluation our evaluation is focused on: (1) general translation quality, (2) quality of translation of specific terminologies, and (3) efficiency in using segmentlevel terminology dictionaries. standard metrics. following recent trends in mt evaluation @xcite , we use chrf @xcite and comet @xcite for the general translation quality evaluation. 3 while the latter one is generally touted as more robust and correlated more with human judgement, in this case we are also concerned in exact match of n-grams, which is captured by chrf. term success rate. in the terminology success rate we compare the machine-translated terms with their dictionary equivalents. one would be tempted to check for the presence of the reference terminology translation in the output by the regular expression match. however, this is sensitive to minor orthographic variants. therefore, we use fuzzy search with threshold of 90% to scan for terminology matches, yielding a number between 0 (no terminology translated correctly) to 1 (all terminology translated correctly). term consistency. this metric looks at whether technical terms are translated uniformly across the entire text corpus. we aim for high consistency, measured by the low occurrence of multiple translations for the same term within the text. we use the approach suggested by @xcite . given the source sentences, outputs, and source terms assigned to each sentence, we firstly make word alignment for the source sentences and outputs, and extract the aligned translated terms for each source term occurrence. then, we automatically choose the "pseudo-reference" terminology translations, based on which translation of which source term occurred in the text first. in the last step, we compare two sets -the real outputs and the pseudo-references for each term occurrenceby means of f 1 score on a scale of 0 (no consistent terminology) to 1 (all terminology translated consistently). ## related work similar to the previously shared task on translation using terminologies @xcite , our terminology hints are mined semi-automatically. we also extend this line of work by contrasting random and proper terminologies. the focus on terminologies in translation is an important one. both zouhar et al. (2020) and @xcite show that the ordering of the system diverges when comparing performance on terminologies versus general performance. ## conclusion this iteration of machine translation with terminologies focused on evaluating the efficiency of using segment-level terminology dictionaries. i.e. it is not enough that the system performs well but it should also perform better when given this additional information. indeed, the improvement between base and proper terminology enriched translations ranged across systems between 0 and 10 chrf points. this helps isolate which terminologyenhancement methods are the most useful.