Sakura at S em E val-2025 Task 2: Enhancing Named Entity Translation with Fine-Tuning and Preference Optimization (2025)

Abstract: Translating name entities can be challenging, as it often requires real-world knowledge rather than just performing a literal translation. The shared task “Entity-Aware Machine Translation” in SemEval-2025 encourages participants to build machine translation models that can effectively handle the translation of complex named entities.In this paper, we propose two methods to improve the accuracy of name entity translation from English to Japanese. One approach involves fine-tuning the model on entries, or lists of entries, of the dictionary. The second technique focuses on preference optimization, guiding the model on which translation it should generate.

## introduction the translation of named entities is a challenging aspect in machine translation (mt) field, as translating proper names, locations, etc. is not always straightforward. for instance, "カールじいさんの空飛ぶ家" (meaning "carl grandpa's flying house") is the japanese version of the name of the film "up" (see entry "q174811" 1 in wikidata). machine translation (mt) systems would not be able to translate the title without having explicit knowledge of such name entity. the "semeval-2025 task 2: entity-aware machine translation" @xcite is a task 2 that challenge the participants to develop mt models capable to translate sentences containing complex named entities. this task becomes even more difficult when translating into japanese due to the variations in script. japanese writing uses three scripts, i.e. kanji, hiragana and katakana, each with its own set of rules and purposes. some words might be writ-1 https://www.wikidata.org/wiki/q174811 2 https://sapienzanlp.github.io/ea-mt/ ten in kanji, while others may require hiragana or katakana. for instance, in the previously-mentioned film title, カール (carl) is written in katakana, which is typically used for foreign words or names; じいさ ん (grandfather) appears in hiragana, reserved for native japanese words and grammatical elements; and 空飛ぶ家 (flying house) is in kanji, employed for more complex or meaningful words. typically, to integrate new knowledge into an large language model (llm) techniques like supervised fine-tuning (sft) are employed. however, doing sft only with dictionaries may lead to overfitting, as the model might become too focused on single-word translations. we participated in the entity-aware machine translation (team sakura) to address these challenges. in this paper, we describe and compare different methods of integrating these dictionaries into the training process. ## related work several efforts have been made to influence the generation process so that the models produce words that are closer to the desired ones. many techniques involve fine-tuning the model with biased data. this can be done through data selection @xcite @xcite or synthetic data generation @xcite . dictionaries are also used during decoding by either adding lexical constraints @xcite or incorporating the dictionary directly into the prompt @xcite . in this paper, we explore entity translation as an llm alignment @xcite problem. our goal is to promote outputs that are closer to human expectations. specifically, we apply preference optimization, a machine learning technique designed to improve models by focusing on preferences. rather than relying on a single ground truth target for predictions, it fine-tunes the model using preference data. the objective is to learn the relative desirability of different outcomes, rather than simply predicting a label. a policy π represents the model's strategy for choosing between different possible outcomes. therefore, given an input x and two outputs y w and y l (with the first output being more desirable than the second) the goal is to find a policy π θ so that it favors π θ (y w |x) over π θ (y l |x) an approach to achieve this is direct preference optimization (dpo) @xcite which involves adjusting a policy π θ compared to a reference π ref in order to increase the log-ratio for the preferred outcome, i.e. r @xmath0, and decrease for the non-preferred, i.e. r @xmath1, minimizing the loss ## proposal our goal is to identify an effective method for integrating a dictionary of named entity translations into the knowledge of an mt model. generally, to adapt a model for specific translation tasks, it is fine-tuned using in-domain data. however, finetuning with dictionaries could lead to the model producing shorter sentences, which might hurt the overall translation performance. ## results and analysis the results of the models are shown in ## conclusion our system demonstrated competitive results in both m-eta and comet metrics. the leaderboard 8 shows that its performance can be comparable to some of the larger models. in this paper, it has been shown that fine-tuning the model on the dictionary can improve translation accuracy. however, this comes at the cost of reduced quality. therefore, we proposed two alternatives that achieve a balanced trade-off between translation quality and entity translation accuracy. the first approach involves fine-tuning with lists of named entity pairs, which helps mitigate the quality decline while improving m-eta scores. the second alternative utilizes preference optimization, which also results in improved m-eta scores, while maintaining a similar level of translation quality. in this study, we utilized the rakutenai-7b-chat model, originally developed for japanese and english. consequently, our experiments focused on these languages only. nonetheless, we believe the proposed approach can be generalized to other language pairs. furthermore, we want to investigate whether this is applicable to bigger models, like those presented in the leaderboard of the workshop. one limitation of this work is that we added the dictionary knowledge as lists of one, five, and ten words. in the future, we would like to explore what sizes are optimal to achieve the best performance. in addition, we want to explore whether adding a combination of these would lead to better results. another way to further explore this work is to generate synthetic sentences from the dictionary instead of sticking to the list of words.