Team ACK at S em E val-2025 Task 2: Beyond Word-for-Word Machine Translation for E nglish- K orean Pairs (2025)

Abstract: Translating knowledge-intensive and entity-rich text between English and Korean requires transcreation to preserve language-specific and cultural nuances beyond literal, phonetic or word-for-word conversion. We evaluate 13 models (LLMs and MT systems) using automatic metrics and human assessment by bilingual annotators. Our findings show LLMs outperform traditional MT systems but struggle with entity translation requiring cultural adaptation. By constructing an error taxonomy, we identify incorrect responses and entity name errors as key issues, with performance varying by entity type and popularity level. This work exposes gaps in automatic evaluation metrics and hope to enable future work in completing culturally-nuanced machine translation.

## introduction machine translation (mt) has progressed significantly with the introduction of the transformer paradigm @xcite . supervised machine translation models have improved their performance in challenging scenarios such as longdocument translation and stylized translation @xcite . despite the success of these models in general translation, they still struggle to translate named entities which are culturally-naunced or language-specific @xcite , in other words, entities which are rooted in social, geographic, historical, and political contexts @xcite . however, the emergence of self-supervised large language models (llms) and their zero-shot translation capabilities have shown to be a promising avenue to address these problems. their ability to learn in-context enables new capabilities, such as terminology constrained translation unlike foundation approaches @xcite . in this paper, we conduct a thorough analysis of * these authors contributed equally. english-to-korean translation through the following: we hope this focused work on english-korean motivates similar work in other domains, to further understand culturally-nuanced and language-specific translation. ## related work while mt has continued to improve from rnn-based models @xcite to transformer-based models @xcite @xcite @xcite , entity translation remains a significant challenge due to the need for both direct wordfor-word conversion (transliteration) and contextual adaptation (transcreation) @xcite . for example, the english query, "what is the rotten tomatoes score of john wick?" should result in "rotten tomatoes" being translated to " 로튼 토마토 ", the movies and tv review site, instead of "썩은 토마토 ", the literal translation meaning rotting fruit. while llms are a promising avenue to address these problems, they are primarily trained on large-scale multilingual corpora with an english-centric bias. this results in them struggling to capture the nuanced sociocultural and historical contexts necessary for effective transcreation @xcite . efforts to enhance entity translation through retrieval-augmented generation (rag) have been introduced, such as leveraging knowledge graphs (kgs) and structured databases. for example, kg-mt proposed using multilingual knowledge graphs to improve cultural adaptation in mt by providing contextually appropriate entity translations @xcite . the complexities of english-korean entity translation stem from fundamental linguistic and cultural differences between the two languages. transliteration is complicated by phonetic variations and word structure differences, while transcreation requires adapting names, idioms, and references to maintain cultural and linguistic naturalness @xcite . existing research has primarily focused on western-centric language pairs, leaving english-korean entity translation an area in need of further investigation @xcite . given these challenges, improving llm-driven mt requires context-sensitive modeling and culturally aware translation strategies. this study aims to bridge these gaps by evaluating stateof-the-art llms and multilingual mt models on entity-dense and knowledge-intensive texts, combining automatic evaluation metrics with human assessment to gain insights into translation quality. ## experimental setup to comprehensively evaluate the performance of llms on the task of mt, we consider 13 models including the most popular and best performing llms from openai (gpt-4, gpt-4o, o1, o1-mini), anthropic (claude 3.5 sonnet, 3.5 haiku), google (gemini 1.5 flash, 1.5 pro), meta (llama3-8b), grok (grok-2), and deepseek (r1-7b) and recent multilingual mt models (nllb-200 and mbart-50) @xcite @xcite @xcite @xcite . with these models, we conduct an automatic evaluation with several metrics (section 3.1) and an in-depth human evaluation (section 3.2). for this evaluation, we use the task dataset provided by @xcite that was prepared from xc-translate @xcite . xc-translate is a multi-reference, human-curated dataset that is challenging due to its focus on translating cross-cultural texts containing entity names. ## results and discussions the automatic evaluation results presented in to complement our automatic evaluation, we conducted human assessments to gain a more nuanced understanding of translation quality across models. our analysis reveals that 459 out of 650 evaluated samples contain translation errors, with grok 2 exhibiting the lowest error rate. among these mistranslations, 266 cases involve incorrectly translated entities, which annotators identified by highlighting discrepancies between the english source and korean target texts. we further constructed a comprehensive error ontology adapted and expanded from @xcite with annotator-provided explanations, illustrated in ## further analysis we conduct a deeper, comprehensive analysis from our evaluation results to understand fine-grained insights in machine translation. in section 5.1, we investigate whether entity popularity and typology influence translation quality. section 5.2 explores the relationship between automatic metrics and human evaluations, providing insights into their complementary nature and potential discrepancies. ## conclusion and future work in this paper, we evaluated 13 large language models and multilingual machine translation systems on their ability to handle culturally-nuanced and language-specific translation tasks across knowledge-intense and entity-dense questions from english to korean. our comprehensive assessment combined three automatic evaluation metrics with complementary human evaluations to thoroughly understand model performance. while our findings demonstrate that llms generally outperform traditional multilingual machine translation models, significant challenges remain, particularly regarding the appropriate transliteration versus transcreation of text. we hope this work encourages future research expanding beyond entity-dense and knowledge-intensive content to explore additional language pairs and text genres, ultimately informing targeted improvements in machine translation capabilities.