CHILL at S em E val-2025 Task 2: You Can’t Just Throw Entities and H ope— M ake Your LLM to Get Them Right (2025)

Abstract: In this paper, we describe our approach for the SemEval 2025 Task 2 on Entity-Aware Machine Translation (EA-MT).Our system aims to improve the accuracy of translating named entities by combining two key approaches: Retrieval Augmented Generation (RAG) and iterative self-refinement techniques using Large Language Models (LLMs).A distinctive feature of our system is its self-evaluation mechanism, where the LLM assesses its own translations based on two key criteria: the accuracy of entity translations and overall translation quality. We demonstrate how these methods work together and effectively improve entity handling while maintaining high-quality translations.

## introduction entity-aware machine translation (ea-mt) focuses on accurately translating sentences containing named entities, such as movies, books, food, locations, and people. this task is particularly important because entity names often carry cultural nuances that need to be preserved in the translation process @xcite . the challenge lies in the fact that named entities typically cannot be translated through simple word-for-word or literal translations. for instance, the movie "night at the museum" is translated as "박물관이 살아있다" (meaning "the museum is alive") in korean, rather than the literal translation "박물관에서의 밤." this example illustrates why literal translations of entity names can be inappropriate. this issue becomes particularly critical in domains such as journalism, legal, or medical contexts, where incorrect entity translations can significantly compromise factual accuracy and trustworthiness. the semeval 2025 task 2 @xcite ) addresses this challenge by requiring participants to develop systems that correctly identify named entities and transform them into their appropriate target-language forms. in this paper, we present our system, which combines retrieval-augmented generation (rag) and self-refine @xcite approaches. our system first retrieves entity information (labels and descriptions) from wikidata @xcite ids provided by the task organizers. this information is then incorporated into prompts for a large language model (llm). however, we discovered that merely providing entity information to the llm does not guarantee accurate entity-aware translations. to address this limitation, we implemented the selfrefine framework where the same llm model evaluates the initial translation based on two criteria: entity label accuracy and overall translation quality. in summary, our approach integrates both rag and self-refine frameworks to achieve optimal translation results. we also present case studies demonstrating concrete improvements achieved through our feedback mechanism. ## dataset the dataset for this task was provided by the organizers in multiple stages: sample, training, validation, and test sets as shown in the test set, which contained approximately 5,000 sentences for each language direction (totaling 49,606 sentences), was released without groundtruth target references. the official evaluation was conducted using withheld references that were later made available by the organizers. ## system description this section provides a detailed description of our system for entity-aware machine translation. our approach combines retrieval-augmented generation (rag) with self-refinement to ensure accurate entity labeling and high-quality translation. ## conclusion in this paper, we present an effective approach to entity-aware machine translation that combines retrieval-augmented generation (rag) with a self-refinement mechanism. our system features a two-criteria feedback system that identifies and corrects both entity label inaccuracies and translation errors. when tested against the baseline gpt-4o model, our system demonstrates significant improvements across all language pairs in the task dataset. the experimental results highlight two key findings: (i) the integration of rag with entity information from external knowledge substantially improves translation accuracy. (ii) self-refinement mechanism consistently enhances translation quality across all language pairs through iterative feedback and correction. our case studies reveal that the system effectively addresses both entity-specific challenges and general translation issues. these results suggest that combining knowledge retrieval with self-refinement is a promising direction for entity-aware machine translation. looking ahead, future work could explore incorporating entity retrieval methods without using gold entity. jingyi zhang, masao utiyama, eiichro sumita, graham neubig, and satoshi nakamura. 2018. guiding neural machine translation with retrieved translation pieces. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pages 1325-1335, new orleans, louisiana. association for computational linguistics.