UA lberta at S em E val-2025 Task 2: Prompting and Ensembling for Entity-Aware Translation (2025)

Abstract: We describe the methods used by our UAlberta team for the SemEval-2025 Task 2 on Entity-Aware Machine Translation (EA-MT). Our methods leverage large language models with prompt engineering strategies suited to this task, including retrieval augmented generation and in-context learning. Our best results overall are obtained with ensembles of multiple models, leveraging named entity knowledge in the dataset. Finally, we provide proof-of-concept experiments showing that lexico-semantic knowledge can be used to identify high-quality translations. We further demonstrate that our methods can function even without gold named entity translations, by using an alternative knowledge base such as BabelNet.

## introduction this paper describes our work on semeval 2025 task 2: ea-mt: entity-aware machine translation @xcite . the ea-mt task is closely related to the well-studied task of machine translation (mt), with two key distinctions: (1) all input sentences contain a named entity, such as a location or the name of a film, and (2) the evaluation metric places a strong emphasis on the correct translation of named entities. this variant of mt is particularly challenging because named entities are often not translated literally or word-for-word @xcite . the ea-mt datasets are designed to emphasize the correct translation of named entities. each instance consists of a source english sentence, its translation in one of the target languages, a unique instance id, the entity's wikidata id, the type of the entity (e.g., "film"), and the named entity translation (net). the data is split into four parts: training, sample, validation, and test. (as our methods are unsupervised, we do not make use of the training data.) figure 1: our mt system ensembling template. a highest-scoring candidate translation is selected, with a tie-breaker applied if necessary. this paper represents a further extension of our work on a comprehensive theory of lexical semantics, grounded in discrete, language-independent concepts @xcite . we have applied theory-driven techniques based on multilinguality and lexical knowledge bases in our approaches to prior semeval tasks @xcite @xcite @xcite . the most directly relevant was the task of idiomaticity detection @xcite , as neither idiomatic phrases nor named entities can be translated word by word (literally). for this ea-mt task, we build upon prior work on ensembling mt systems (figure 1 ). @xcite experiment with various methods for ensembling the output translations of large language models (llms), specifically motivated by the hallucination problem. vernikos and popescu-belis (2024) leverage quality estimation systems to ensemble llms. however, these prior works do not specifically focus on the correct translation of named entities. contrariwise, we incorporate nets into both llm prompts and ensembling systems. in this paper, we posit and test four hypotheses: (1) retrieval augmented generation (rag; @xcite and in-context learning @xcite can improve the performance of llms on the ea-mt task. (2) translation quality can be increased by ensembling the outputs of multiple mt systems via favoring nets retrieved from a knowledge base. (3) the literalness of a sentence translation correlates with the quality of the trans-lation. (4) in the absence of gold named entities, a multilingual wordnet can serve as a source of named entity translations. we pursue two directions in developing our systems: prompt engineering and ensembling, which can also be used in tandem. prompt engineering involves employing techniques for instructing llms such as in-context learning and rag, which we use to develop a prompt template for llms with a particular focus on nets. our second principal direction, ensembling, consists of selecting the best translation among those produced by different mt systems given the same input. we use validation set performance, named entity identification, and literalness metrics as sources of information for our various ensembling strategies. our results indicate that both prompt engineering and ensembling yield improvements over baseline methods. we show that a net-based ensembler outperforms its individual component systems. both rag and in-context learning improve the performance of llms on ea-mt. surprisingly, using literalness to select the best translation yields no consistent improvement. in the official evaluation, our best system ranks 4th overall among 27 teams, and achieves the highest reported score for arabic. notably, we outperform all submitted systems on the comet metric. ## methods in this section, we describe our methods for the ea-mt task. ## systems, tools, and resources our methods require multiple candidate sentence translation candidates, as well as named entity translations (nets). for the former, we use an llm and two commercial mt systems. ## results we present the evaluation results of our methods across all 10 english-to-target language pairs. the principal metric for this task is the harmonic mean of comet and m-eta. we begin by reporting the results for our official submissions, followed by an analysis of our findings explored on the provided validation set. in addition to those already described, we report the results of three baselines: gpt as run by the shared task organizers (gpt-st), gpt prompted without nets (gpt-prompt), and an ensembler which randomly selects one of the candidate translations (random ensembler). ## conclusion after extensive experimentation on the ea-mt datasets, we conclude that three of four hypotheses formulated in section 1 are well-supported by our empirical results. our work provides new evidence supporting the use of external knowledge bases in semantic tasks, and prompts further exploration of word-level analysis, especially as it corresponds to literalness. finally, we note that our highestscoring methods were ranked at or near the top of the official leaderboards for several languages and categories.