Automated Evaluation for Terminology Translation Related to the EEA Agreement (2025)

Abstract: This paper presents a submission to the WMT25 test suite subtask, focusing on terminology in official documents related to the EEA Agreement. The test suite evaluates the accuracy of MT systems in translating terminology for the English-Icelandic translation direction when applied to the EEA documents. We focus on the use of terminology in four domains of the agreement; science, technology, economics, and society. We find that the manual evaluation confirms that our test suite can be helpful in selecting the best MT system for working with these domains. Surprisingly, an online system that does not achieve very high scores on the general translation task, according to preliminary results, is most adept at translating the terminology our test suite evaluates.

## related work semenov and bojar (2022) measured consistency, unambiguity and adequacy in automatically translated legal texts by counting the correct occurrences of the exact term. @xcite found that the herfindahl-hirshman index (hhi) can be used to measure the consistency of terminology in a translated corpus, since the hhi works on a small amount of data. an overall translation consistency index (c t ) for a source term is calculated as follows: p is the number of translation having the source term t, and each frequency share is calculated as the ratio of its frequency f i to the total translation occurrence within a product k j . the score ranges between 0 and 10, with 10 being perfect consistency. alam et al. (2021) introduced a benchmark for evaluation of quality and consistency of terminology translation in a shared task at wmt21, with creating new evaluation datasets that were annotated by professional translators for their terminology consistency. they found that general translation quality does not have to be sacrificed for terminology compliance. @xcite evaluated the efficiency of using segment-level terminology dictionaries in a shared task at wmt23, and concluded that an improvement in mt performance when using a terminology dictionary ranged between 0 and 10 chrf points. two participants in the wmt 2024 test suite subtask @xcite focused specifically on @xmath0: friðriksdóttir (2024) focuses on various aspects of genderinclusive translation, including lgbtqia+ terminology and whether translations are current and culturally appropriate, as the terminology in that domain has been updated repeatedly. @xcite focus on idiomatic expressions and proper names in their test suite. their evaluation is keyword-based, checking if content words in the idioms are translated correctly and whether the proper names have correct translations. ## methodology we built an automated keyword-based evaluation regarding the eea agreement. the aim was to test the ability of mt systems when it comes to the translation of terms in the agreement, as disambiguation is important when it comes to the translation of legal texts. to test this, we ran the automated evaluation and confirmed our method with manual evaluation. we manually collect sentences from eu regulations and directives relevant to the eea agreement. for each sentence, we tag terms and find the standard icelandic translation for each term on the official icelandic website for the eea agreement. 1 the icelandic translations are used for automatic evaluation and a subset of the translations from each mt system is manually evaluated 1 https://gagnagrunnur.ees.is/ in order to understand whether the automatic evaluation is close to human judgment. ## results results of the automatic evaluation are presented in ## conclusions and future work we evaluated 34 mt systems using our test suite, automatically and manually. the evaluation shows that while a few of the systems translate the majority of the terms correctly, they are all quite far from perfect. our automatic evaluation orders the systems in a similar way to the manual evaluation, indicating that an automatic approach such as this one can be useful to help translators find the most useful system for the task. a larger set of sentences and terminology would improve our test suite, especially if we include terminology from other subdomains. given the large amount of published documents relating to the eea agreement it is almost, if not entirely, impossible to test for every single term that appears in those documents. we plan to look into the frequency of terms in order to reconstruct the test suite in a way that may be more indicative of real-world usage, on the one hand giving terms that appear often in the agreement more weight, but on the other make sure that a representative part of terms that rarely appear is also included. to build further on the test suite, we also plan to look into results for each subdomain and see if mt systems perform better for a specific domain. another interesting area is to test the translation of neologism, as acts about new developments, especially scientific and technological ones, often call for new terminology. finally, looking into the translation of terms that have more than one allowed icelandic translation, based on context and subgroups, could help us understand problems that translators might miss and special attention has to be paid to. this test suite can be adapted to other languages with relative ease, which allows further work on other language directions. the same way, or whether for some mt systems correct translations may be fortuitous incidents. our selection of terms was not always systematic and we do not always consider all terms in a given sentence. we were not able to check every category under each subgroup in this test suite due to the size limitations.