RAG thoven at S em E val 2025 - Task 2: Enhancing Entity-Aware Machine Translation with Large Language Models, Retrieval Augmented Generation and Function Calling (2025)

Abstract: This paper presents a system for SemEval 2025 Task 2 on entity-aware machine translation, integrating GPT-4o with Wikidata-based translations, retrieval augmented generation (RAG), and function calling. Implemented in RAGthoven, a lightweight yet powerful toolkit, our approach enriches source sentences with real-time external knowledge to address challenging or culturally specific named entities. Experiments on English-to-ten target languages show notable gains in translation quality, illustrating how LLM-based translation pipelines can leverage knowledge sources with minimal overhead. Its simplicity makes it a strong baseline for future research in entity-focused machine translation.

## introduction and background the aim of the ea-mt task, also referred to as semeval 2025 -task 2 @xcite , is to develop systems capable of accurately translating english sentences containing challenging named entities into one of the target languages: arabic, german, spanish, french, italian, japanese, korean, thai, turkish, or chinese. named entities, which often denote proper names-such as those of people, organizations, landmarks, locations, events, or even titles of books, movies, tv series, and products-pose significant translation challenges that require deep domain and cultural expertise. examples of input sentences a system solving this task might receive, as well as the desired french and italian translations, are provided in our methodology is designed to replicate the practical challenges encountered by data scientists, including the constraints under which solutions are typically developed. to this end, we intentionally limit our approach to in-context learning without fine-tuning, while augmenting the standard retrieval augmented generation (rag) pipeline with additional tools, taking inspiration from prior lang sentence en i watched the movie "the shawshank redemption" last night. fr j'ai regardé le film "les évadés" hier soir. en i bought a new book called "the catcher in the rye". it ho comprato un nuovo libro chiamato "il giovane holden". our empirical results outlined in section 3 demonstrate that, even under these constraints, a well-optimized rag system can serve as a robust baseline, achieving state-of-the-art performance in certain contexts. to aid future development in this area, we release all our code and configuration under the terms of an opensource license foot_0 . ## results and analysis our findings indicate that incorporating wikidata ids leads to a significant performance boost-improving absolute scores by at least 20 points. this suggests that leveraging this external data source effectively addresses the challenge. furthermore, the fact that the top 10 models on the final leaderboard foot_2 all utilized the gold data reinforces this conclusion. notably, although our model ranked second among the non-finetuned approaches, the performance gap was minimal (91.72 vs. 91.62). in contrast, experiments that combined wikidata with wikipedia data and employed wikidata aliases resulted in inferior performance, as detailed in to isolate the impact of gold data, we conducted additional experiments without its use. as evidenced in ## conclusion in this work we introduce a family of modular, llm-centric translation pipelines that combine gpt-4o with wikidata-driven entity linkage, automatic function calling, and retrieval-augmented generation to tackle the entity-aware machine translation task. controlled experiments with and without gold entity identifiers show that symbolic priors can be exploited to close the entity gap: the gold-aware configuration reaches an average m-eta + comet score of 91.62, ranking 4 th overall, 2 nd among non-finetuned approaches and best overall for chinese, while the non-gold variant attains 1 st place among all submissions that forgo labeled data. achieved without task-specific finetuning, these results suggest that lightweight retrieval-reasoning hybrids may serve as strong baselines for future research in multilingual, entityaware machine translation and motivate their adoption as reference systems in forthcoming studies and shared tasks.