H ausa NLP at S em E val-2025 Task 2: Entity-Aware Fine-tuning vs. Prompt Engineering in Entity-Aware Machine Translation (2025)

Abstract: This paper presents our findings for SemEval 2025 Task 2, a shared task on entity-aware machine translation (EA-MT). The goal of this task is to develop translation models that can accurately translate English sentences into target languages, with a particular focus on handling named entities, which often pose challenges for MT systems. The task covers 10 target languages with English as the source. In this paper, we describe the different systems we employed, detail our results, and discuss insights gained from our experiments.

## introduction the quality of translations produced by machine translation (mt) systems has improved considerably @xcite . however, despite these advancements, translations into the target language still contain errors, often due to the challenges associated with translating named entities @xcite . entity-aware machine translation is a type of mt that considers specific entities, such as names, locations, and organizations, to enhance translation accuracy and fluency @xcite . several approaches have been proposed to improve these models' ability to translate named entities more effectively, accounting for the need for transliteration in some cases while generating equivalent translations in others. in this paper, we describe our submissions to se-meval 2025 task 2: entity-aware machine translation shared task @xcite . our systems include sequence-to-sequence entity-aware supervised models trained to improve the translation of english sentences into french, german, spanish, italian, japanese, and arabic. a pretrained nllb @xcite model was fine-tuned for bilingual translation with the provided training data in each of these languages. furthermore, we investigated the performances of a closed-source large language model (llm), gemini @xcite , on these languages, in addition to chinese, korean, thai, and turkish. our results indicate that gemini, in a zero-shot setup, achieved the best overall performance, with only a few languages showing improvements when examples from the training data were incorporated in few-shot setups. recent advancements also include entity-aware multi-task training on pre-trained models such as t5, which improves ne translation quality and also increases the number of named entities generated in german translations of english texts @xcite . @xcite introduced the amff and camff frameworks, which utilize attention mechanisms to improve named entity recognition (ner) by incorporating multilevel contextual features. @xcite introduced entity-aware techniques (eat) that integrate semantic parsing to help mt models recognize and accurately translate named entities. @xcite evaluated ne translation in speech translation systems, highlighting persistent challenges despite improvements in recall and precision. while these methods improve entity translation, our approach fine-tunes the pre-trained nllb model using both the provided training data and extracted named-entity translations from wikidata. additionally, we evaluate the closed-source gemini model without fine-tuning, assessing its performance in both zero-shot and few-shot setups. our work focuses on achieving a balance between overall translation quality and ensuring the accurate translation of critical named entities. ## proposed approaches the team adopted four approaches for this task. the first two approaches included fine-tuning the nllb pre-trained model in a bilingual setup for @xmath0. the two other approaches were the use of two different prompt templates to evaluate the performance of google's gemini closed-source model. the team combined the traditional fine-tuning technique and promptengineering strategies to assess their relative effectiveness in preserving entity integrity during bilingual translation. ## conclusion prompt engineering for large language models like gemini proves effective for entity-aware machine translation. comparative studies with fine-tuning the nllb-200 model show a consistent performance advantage for gemini's zero-shot and fewshot prompting across target languages, highlighting its ability to preserve entity integrity in translation. while our results reveal nuances in performance across language families, with european languages exhibiting stronger overall scores and an intriguing performance from turkish, the overarching trend favors prompt-based methodologies.