FII the Best at S em E val 2025 Task 2: Steering State-of-the-art Machine Translation Models with Strategically Engineered Pipelines for Enhanced Entity Translation (2025)

Abstract: Entity-Aware Machine Translation (EAMT) aims to enhance the accuracy of machine translation (MT) systems in handling named entities, including proper names, domain-specific terms, and structured references. Conventional MT models often struggle to accurately translate these entities, leading to errors that affect comprehension and reliability. In this paper, we present a promising approach for SemEval 2025 Task 2, focusing on improving EAMT in ten target languages. The methodology is based on two complementary strategies: (1) multilingual Named Entity Recognition (NER) and structured knowledge bases for preprocessing and integrating entity translations, and (2) large language models (LLMs) enhanced with optimized prompts and validation mechanisms to improve entity preservation. By combining structured knowledge with neural approaches, this system aims to mitigate entity-related translation errors and enhance the overall performance of MT models. Among the systems that do not use gold information, retrieval-augmented generation (RAG), or fine-tuning, our approach ranked 1st with the second strategy and 3rd with the first strategy.

## introduction entity-aware machine translation (ea-mt) aims to improve mt accuracy for named entities, including proper names, dates, and domain-specific terms @xcite . these are crucial in fields like technical documentation, legal texts, and medical literature @xcite ), yet translating them remains challenging despite modern advancements. early rule-based mt struggled with named entities due to rigid linguistic rules @xcite . statistical machine translation (smt) in the 1990s improved overall quality but still faced issues with proper nouns and domain-specific terms @xcite . phrase-based smt (pb-smt) in the 2000s enhanced phrase-level translations but remained inconsistent with named entities and long-distance dependencies @xcite . neural machine translation (nmt) and transformer-based models like bert and gpt @xcite have enhanced fluency and contextual awareness. yet, challenges remain in entity preservation, cultural adaptation, and low-resource language support @xcite @xcite . for semeval 2025 task 2 @xcite ) on ea-mt, we developed two systems to improve entity-centric translation across ten languages. our approach combines: the remainder of this paper is structured as follows. section 2 provides a background on the evolution of mt, from early rule-based systems to state-of-the-art transformer models. section 3 details the system architecture design for ea-mt, outlining the experimental setup and datasets used. section 4 presents the results and the comparative evaluation. finally, section 5 discusses conclusions and future directions for entity-aware mt research. from the 1960s to the 1980s, early machine translation (mt) systems were rule-based, offering structured translations but struggling with named entities, proper nouns, and idiomatic expressions @xcite . the 1990s introduced statistical machine translation (smt), leveraging probabilistic models to improve flexibility, yet still facing challenges with rare terms and domain-specific terminology. the 2000s saw phrase-based smt (pb-smt), enhancing contextual coherence but retaining difficulties with named entities @xcite . neural machine translation (nmt) emerged in the 2010s, using deep learning to improve fluency and entity handling, though challenges persisted in low-resource languages and domain adaptation @xcite . today, transformer-based models like gpt and bert push translation accuracy forward, excelling in contextual understanding but still struggling with cultural adaptation and low-resource languages @xcite . large language models (llms), such as gpt-1 https://github.com/deliagrigorita/fii-the-bestsemeval2025 4, now rival leading nmt systems, though performance varies across language pairs (manakhimova et al. 2023). despite advances, translating low-resource languages remains a challenge, necessitating refined techniques like back-translation and transfer learning @xcite . hybrid methodologies integrating rule-based, statistical, and neural approaches continue to be explored for further improvements @xcite . ## results in this section, we present the evaluation results of our two proposed strategies. the evaluation was conducted using two main metrics: comet and m-eta across ten target languages: arabic (ae), german (de), spanish (es), french (fr), italian (it), japanese (jp), korean (kr), thai (th), turkish (tr), and traditional chinese (tw). the final evaluation score is calculated as the harmonic mean of the comet and m-eta scores, ensuring a balanced assessment that accounts for both overall translation quality and entity preservation. the first strategy demonstrated varying levels of performance across languages, as shown in while both strategies performed well in handling named entities in translation, the second strategy generally produced higher final scores across most languages. improvements in m-eta and comet scores were particularly noticeable for italian (it it), french (fr fr), and chinese (zh tw). however, variations still exist among different languages, indicating that certain language pairs may require further refinement. future work will explore the potential benefits of merging these two strategies to leverage their strengths and further enhance translation performance. foot_0 ## conclusion in this work, we explored entity-aware machine translation (ea-mt) by proposing two approaches aimed at improving the translation of named entities across multiple languages. our first approach relied on the mbert model for named entity recognition (ner) combined with wikidatabased entity translations, while our second approach leveraged large language models (llms) with structured prompt engineering to enhance translation accuracy. our experiments demonstrated that accurately recognizing and preserving named entities is crucial for high-quality translation. we identified several challenges, such as inconsistent entity annotations in the dataset and grammatical disruptions caused by placeholder-based translations. to mitigate these issues, we refined our methodology by incorporating normalization techniques and utilizing wikidata as a reliable source for entity translations. the second approach, which integrated llms for translation while maintaining entity integrity, proved to be more effective in producing fluent and semantically accurate translations.