S em E val-2025 Task 2: Entity-Aware Machine Translation (2025)

Abstract: Translating text that contains complex or challenging named entities—e.g., cultural-specific book and movie titles, location names, proper nouns, food names, etc.—remains a difficult task for modern machine translation systems, including the latest large language models. To systematically study and advance progress in this area, we organized Entity-Aware Machine Translation, or EA-MT, a shared task that evaluates how well systems handle entity translation across 10 language pairs. With EA-MT, we introduce XC-Translate, a novel gold benchmark comprising over 50K manually-translated sentences with entity names that can deviate significantly from word-to-word translations in their target languages. This paper describes the creation process of XC-Translate, provides an overview of the approaches explored by our participants, presents the main evaluation findings, and points toward open research directions, such as contextual retrieval methods for low-resource entities and more robust evaluation metrics for entity correctness. We hope that our shared task will inspire further research in entity-aware machine translation and foster the development of more culturally-accurate translation systems.

## introduction background. the emergence of multilingual large language models (llms) and the wide availability of massive multilingual datasets have significantly advanced the field of machine translation (mt) @xcite @xcite . these developments have led to mt systems that not only perform exceptionally well in high-resource languages but also support a growing number of low-resource languages @xcite @xcite . nevertheless, the research community still faces several unresolved challenges in mt. among these, the translation of text that contains entities is still a hard task, especially with particular categories of entities, e.g., movies, books, food, locations, and sometimes even people, to name a few. indeed, word-for-word, or literal, translations of their names may not be suitable due to culture-specific references, which can vary depending on social, geographical, and historical contexts, among other factors @xcite . ## entity-aware machine translation task definition. we introduce entity-aware machine translation (ea-mt), a new shared task that evaluates how well systems handle entity translation across 10 language pairs. more formally, ea-mt is defined as follows: given a source sentence s e in english that contains an entity mention e, the goal is to produce a translation t in a target language l target that correctly adapts the entity to its culturallyappropriate equivalent e in that language. for each sentence s e in english, we have: ## xc-translate: a novel gold benchmark for entity-aware mt in this section, we introduce xc-translate, a novel gold benchmark for evaluating the performance of mt systems in translating text containing entities with names that are significantly different across a set of 10 diverse languages. creating xc-translate represents a core contribution of this semeval task, as it was specifically designed to address the challenges of entity-aware machine translation. while we encourage readers interested in the full technical details to refer to our dedicated publication @xcite , we provide an overview of the benchmark creation process in this section. to create xc-translate we employ a four-step process: ## evaluation metrics to evaluate translation quality, we employ a combination of two metrics: comet and m-eta. dealing with multiple references. xc-translate contains multiple references for each sentence, which is a often considered a best practice in mt tasks to account for the fact that there are multiple valid ways to convey the same meaning in a target language. when evaluating the translations produced by the systems, we use the best reference approach, which selects the best reference translation for each system output based on the highest score of the evaluation metric, instead of using the average score across all references. comet. comet @xcite ) is a neural metric that evaluates overall translation quality by comparing system outputs against human references. unlike traditional lexical overlap metrics such as bleu, comet leverages contextualized embeddings to capture semantic similarity between translations. we use comet-22 @xcite , which has shown strong correlation with human judgments across multiple languages. however, while comet measures general translation quality, it may not specifically capture entity translation accuracy. ## overview of participating systems in total, 54 participants registered for the task on our codabench competition and submitted 322 runs on the test set, each run containing the predictions of a single system on at least one language pair. among these, 25 teams submitted the final results of 53 systems for the official leaderboard and 18 teams submitted system description papers. we provide an overview of the systems at ea-mt in as shown in figure 2 , we can observe a clear trend toward the use of large language models (llms) for entity-aware machine translation, as 86.8% of the systems are based on llms and only 13.2% of the systems are based on traditional nmt models. moreover, retrieval-augmented generation (rag) is one of the most common techniques used by the participants, as 26.4% of the systems use this technique to improve their performance. there is still a significant number of participants (47.2%) who opted to fine-tune their models on a training dataset, which is a considerable proportion since only open-source models can be directly fine-tuned. not surprisingly, the most used llms for this task align with the most popular and best-performing llms in the general nlp community, i.e., gpt-4o, qwen-2.5, and llama-3. ## results, analysis, and discussion the number of submissions to the official leaderboard allows to provide a birds-eye view of the performance of different systems on the task, and analyze a few interesting trends, which are discussed in depth in appendix b. ## conclusion and future work in this paper, we presented the entity-aware machine translation (ea-mt) shared task, which was part of semeval-2025. the goal of ea-mt is to evaluate the ability of machine translation systems-traditional nmt and modern llmsto translate text that contains challenging entities, e.g., entities that are affected by cultural and linguistic differences across languages. to this end, we created xc-translate, a benchmark for entityaware machine translation that contains over 50k sentences in english with their translations into 10 languages, with a total of over 100k manuallycreated and manually-verified translations. we also proposed a new evaluation metric, m-eta, which focuses exclusively on entity translation correctness. finally, we analyzed the results of the official leaderboard and discussed the key trends in the systems submitted to the ea-mt shared task. in general, xc-translate has shown that state-of-theart llms still struggle with entity translation, and that different approaches are needed to bridge the gap between llms and human translators. in the future, we plan to extend xc-translate with additional language-pairs-including pairs where the source language is different from english-and domains, and to introduce new challenges, such as code-switching, low-resource languages, and larger coverage of emerging entities. limitations xc-translate. xc-translate is a benchmark for entity-aware machine translation that contains over 50k sentences in english with their translations into 10 languages, with a total of over 100k manuallycreated and manually-verified translations. however, xc-translate is limited in several ways. first, xc-translate only provides translations from english to 10 languages, which limits its applicability to other language pairs. we avoided reversing the translation direction to avoid the risk of introducing noise in the translations and dealing with known issues in back-translation, including the increase of translationese artifacts. second, xc-translate only contains translations of entities that are present in wikidata and feature at least one alias in the source and target languages. this means that xc-translate may not be representative of entities belonging to domains that are not well covered by wikidata. third, while we strived to cover as many languages as possible, we focused on the most widely spoken languages. increased attention to low-resource languages is needed to ensure that xc-translate is representative of the diversity of languages spoken around the world. fourth, xc-translate only includes questions (see section 3). this means that xc-translate may not be representative of other types of text, such as narratives or technical documents. m-eta. m-eta is a specialized metric that focuses exclusively on entity translation correctness. we introduced m-eta to address the limitations of other metrics, such as comet and bleu, which do not specifically capture entity translation accuracy. however, m-eta also has limitations. first, m-eta only considers the correctness of entity translations and does not account for other aspects of translation quality, such as fluency and coherence. therefore, m-eta should be used in conjunction with other metrics to provide a comprehensive evaluation of translation quality. second, m-eta relies on the availability of valid aliases for entity names in the target language, which is a manually-intensive process and may not be feasible for all entities. systems. the systems submitted to the ea-mt shared task are based on a variety of approaches, including traditional nmt, retrieval-augmented generation, and large language models. although the prevalence of llms in the submitted systems is a promising trend, it also raises concerns about the reproducibility and generalizability of the results. many of the systems are based on proprietary llms, which limits their accessibility and reproducibility. additionally, systems based on closedsource models are difficult to analyze and understand, making it challenging to identify potential biases in the models, which can lead to unfair treatment of certain groups or individuals. finally, the reliance on large-scale pre-trained models raises questions about the environmental impact of training and deploying these models. therefore, solutions based on smaller and open-source models may still be competitive and more sustainable in the long run if we consider other factors, such as reproducibility, latency, and energy consumption. neural machine translation. in findings of the association for computational linguistics: acl 2023, pages 1697-1710, toronto, canada. association for computational linguistics. andrea zenotto, vincenzo catalano, and ruben tetamo.