silp_nlp at S em E val-2025 Task 2: An Effect of Entity Awareness in Machine Translation Using LLM (2025)

Abstract: In this study, we investigated the effect of entity awareness on machine translation (MT) using large language models (LLMs). Our approach utilized GPT-4o and NLLB-200, integrating named entity recognition (NER) to improve translation quality. The results indicated that incorporating entity information enhanced translation accuracy, especially when dealing with named entities. However, performance was highly dependent on the effectiveness of the NER model.

## introduction machine translation (mt) of named entities (nes), such as person or place names, remains a major challenge even for advanced models. this is largely because nes appear less frequently in training data compared to other words or phrases. additionally, new and unseen nes like organization or product names are constantly being created, and even common nouns can function as nes in certain contexts. on the other hand, named entity recognition (ner) has achieved reasonably high accuracy for many languages, with precision scores around 80-90% @xcite . however, since most mt models rely solely on parallel training data and contextual cues for translation, they often struggle with rare nes that appear infrequently or not at all during training. in such cases, the models may "hallucinate," generating words or phrases that are statistically similar to the rare ne in embedding space but do not provide the correct translation. this can result in the creation of entirely new, unintended words instead of accurate translations. in this translation work from the source language to the target language, we incorporated en-tity aware (ea) in the prompt to achieve better results. we utilized the universal ner model @xcite for entity extraction. ## data this shared task focuses on translating from english to various target languages. each target sentence in english is accompanied by a wikidata id, which represents the entity type. the goal of this task is to translate the source sentence into the tar-get sentence, with a primary focus on accurately translating the entities corresponding to their respective types. training and validation data include source sentences along with their corresponding wikidata ids, which indicate the entity types present in the sentences and target sentences @xcite . the testing data consists of source sentences and their associated wikidata ids that correspond to the entity types found within those sentences. ## evaluation metric the evaluation of this task is based on three metrics for the comprehensive assessment: ## results and analysis tables 3 and 4 present the comparative results of various machine translation systems under entityaware (ea) and non-entity-aware configurations. the evaluation is based on three key metrics: m-eta, comet, and a composite overall score, averaged across all languages. additionally, in contrast, the fine-tuned nllb-200 model exhibits relatively stable performance across both settings. the improvement in the overall score from 20.44 to 20.50 with ea is minimal, despite a small rise in comet (from 85.2 to 86.0) and m-eta (from 11.61 to 11.64). this suggests that nllb-200, while effective in standard translation tasks, is less sensitive to explicit entity markup, possibly due to its architecture and training design, which average across all languages system m-eta comet overall the language-specific results in for fine-tuned nllb-200, while the m-eta scores are generally lower, entity awareness does show marginal improvements across italian, arabic, and french, though the gains are not consistent across all languages. in summary, the results highlight the effectiveness of entity-aware translation in improving multilingual translation quality, particularly in transformer-based generative models like gpt-4o. while both models benefit from entity incorporation, the gains are significantly more pronounced in autoregressive generative models than in encoderdecoder models like nllb-200. this underscores the importance of integrating named entity tagging as an auxiliary signal, especially in settings where entity fidelity and factual grounding are critical. and gpt-4o-mini) and the fine-tuned nllb-200 model. our results demonstrate that incorporating explicit entity information significantly enhances translation quality in gpt models. the models exhibited particularly strong gains in languages with rich named entity structures, such as spanish, french, and german. in contrast, the nllb-200 model showed only marginal improvements, indicating that while it may implicitly learn entity representations, it does not benefit substantially from direct entity injection using the current architecture. these findings highlight the suitability of gptstyle autoregressive models for prompt-based entity-aware enhancements, whereas encoderdecoder architectures like nllb-200 may require structural changes to leverage entity information more effectively. for gpt-based models, future research could explore advanced prompting strategies, such as multiturn dialogue prompts, chain-of-thought reasoning, or contextual expansion around named entities. additionally, integrating retrieval-augmented generation (rag) or few-shot in-context learning using entity-rich examples could further boost translation fidelity. for nllb-200 and similar encoder-decoder models, future work may focus on architectural enhancements, such as entity-aware attention mechanisms, adapter layers for entity embedding, or multi-task learning frameworks that jointly train on translation and ner objectives. in conclusion, this work emphasizes the importance of integrating structured entity knowledge into multilingual translation systems and opens up promising directions for enhancing translation quality in low-resource and entity-rich contexts.