The Challenge of Translating Culture-Specific Items: Evaluating MT and LLM s Compared to Human Translators (2025)

Abstract: We evaluate state-of-the-art Large Language Models (LLM’s) ChatGPT-4o, Gemini 1.5 Flash, and Google Translate, by focusing on the translation of culture-specific items (CSIs) between an underrepresented language pair: the Flemish variant of Dutch and Serbian. Using a corpus derived from three Flemish novels we analyze CSIs in three cultural domains: Material Culture, Proper Names, and Social Culture. Translation strategies are examined on a spectrum that goes from conservation to substitution. Quantitative analysis explores strategy distribution, while qualitative analysis investigates errors, linguistic accuracy, and cultural adaptation. Despite advancements, models struggle to balance cultural nuances with understandability for the target readers. Gemini aligns most closely with human translation strategies, while Google Translate shows significant limitations. These findings underscore the challenges of translating CSIs—particularly Proper Names—in low-resource languages and offer insights for improving machine translation models.

## introduction recent advancements in machine translation (mt) have significantly enhanced its quality and broadened its applicability, even in the domain of literary translation, an area often considered resistant to automation due to its reliance on nuance, creativity, and cultural context. existing studies have reported varying levels of success for mt tools, with accuracy rates ranging from 44% @xcite to 20% @xcite . several researchers have investigated the potential of machine translators pre-trained on literary texts @xcite ; @xcite , showing that tailored systems can improve automatic evaluation metrics for prose translations when compared to baseline models. beyond improvements in output quality, recent scholarship has also investigated how mt and computer-assisted translation (cat) tools might be adapted to support the specific demands of literary translation. hadley (2023), for instance, argues that these technologies can serve as productivity aids rather than replacements for human creativity. he identifies a range of functionalities, such as sentence length control, rhyme pattern identification, and syllable counting, that could be incorporated into cat tools to assist translators working with poetry or stylistically marked texts. similarly, @xcite provide empirical evidence that the tool puncat, designed to support the translation of puns, can stimulate and broaden the translator's pool of creative solutions, thus enhancing problem-solving in areas of high linguistic density and ambiguity. in parallel, the impact of mt on translator creativity and reader experience has also been part of several studies. their findings revealed that while human translations exhibit a higher degree of creativity, there is no statistically significant difference in the overall reading experience between human and post-edited machine translations @xcite . furthermore, large language models (llms) have introduced tools capable of tackling complex tasks, such as creative writing @xcite and poetry @xcite , expanding their potential applications. the growing capabilities of mt tools and llms have led to their widespread use in various fields, including literary translation. according to a recent study conducted by the european council of literary translators' associations @xcite , more than half (54%) of literary translators from 34 member countries occasionally use mt tools in their work, primarily for translating short passages or sentences (62%). notably, some publishers have begun offering literary translators assignments to revise machine-translated texts, with approximately 7% of translators in serbia reporting such requests @xcite . this trend is further supported by publishers' emerging plans to release books translated entirely by artificial intelligence @xcite . this trend emphasizes the growing importance of evaluating mt tools from a practical, user-oriented perspective. despite these advancements, one of the greatest challenges in both human and machine translation remains the accurate handling of culture-specific items (csis). these elements are particularly complex due to their dual role: they function within the narrative structure while carrying connotations and references to concepts often absent in the target culture. this duality makes csis a critical focus for evaluating mt systems. understanding how mt tools and llms manage culturally bound elements provides valuable insights into their performance, particularly in literary translation, where maintaining the integrity of csis is crucial. this need becomes especially apparent in light of findings by @xcite , who who studied the use and perceived usefulness of translation technologies by dutch literary translators. her research shows that many literary translators consider mt and cat tools largely inadequate for capturing essential literary features such as style, humor, irony, and metaphor, as well as broader aspects such as context and cultural background. these perceptions underscore the persistent gap between current technological capabilities and the nuanced demands of literary translation. therefore, examining the treatment of csis by mt and llm systems not only provides a means to evaluate current performance but also reveals areas in need of targeted development, contributing to the creation of more culturally aware and context-sensitive translation technologies. this study also addresses the challenges posed by low-resource languages, such as serbian, which lack sufficient training data for mt systems. serbian ranks among the least technologically developed european languages, alongside maltese, irish, luxembourgish, and bosnian, as highlighted by the ele (european language equality) project @xcite . furthermore, in the field of machine translation research, studies on low-resource languages often focus on their pairing with dominant global languages, such as english. by examining an underrepresented language pair, this research contributes to a deeper understanding of mt performance in less-studied linguistic contexts and offers insight into existing gaps that can inform future improvements in ai tool development. this article investigates how contemporary mt tools, including chatgpt-4o, gemini 1.5 flash, and google translate, handle csis, and how their use of translation strategies compares to those of human translators. in this study, the term machine translation (mt) is used as an umbrella term encompassing both neural machine translation (nmt) systems and large language models (llms). these tools were chosen to facilitate a comparison between nmt and llm-based approaches. nmt systems, such as google translate and deepl, rely on large-scale parallel corpora and are expensive to develop and maintain, which limits their coverage of lessresourced language pairs. at the time of writing, google translate is the only major nmt service that supports translation between dutch and serbian, restricting access to high-quality nmt for serbian-speaking users. in contrast, llms are trained on vast multilingual datasets, including monolingual and non-parallel corpora, which allows them to perform translations across a broader range of language pairs, even in lowresource scenarios. their growing adoption by professional translators, as indicated in recent surveys such as the ceatl report (2024), further underscores their relevance to translation practice. by analyzing strategy distribution, mistranslation rates, and error patterns across cultural categories, the study evaluates the differences between these models, identifies which tool aligns most closely with human translation, and highlights the most common types of errors in each approach. ## methodology this study is based on the research into the translation of csis in flemish literature @xcite , extending its scope to include a comparative analysis of machine translation and human translation strategies. specifically, it investigates the translation of csis across three cultural categories-material culture (mc), proper names (pn), and social culture (sc)-as defined by @xcite . material culture (mc) includes items such as food, drink, and towns/housing, reflecting tangible aspects of everyday life. proper names (pn) encompass street names, brand names, and horeca (hotel, restaurant, and café) names, which often require specific adaptation to the target culture. social culture (sc) comprises job titles, sports and games, and leisure activities, highlighting culturally embedded practices and societal roles. these categories were selected due to the clear divergence in translation strategies applied to each. as demonstrated in budimir (2021), translators predominantly employed orthographic adaptation and literal translation when rendering pns, thereby opting for the conservation of these elements. in contrast, description and localization were more frequently used for items related to mc and sc, where translators tended to adapt the elements to the expectations of target readers. to further support this categorization, a chi-square test was conducted to assess whether there were significant differences in the strategies used by the human translators within each category. the test revealed no statistically significant differences in the distribution of translation strategies between the translators (@xmath0.066 for pn, @xmath1.256 for mc, and @xmath2.438 for sc). this outcome supports the assumption that it is primarily the nature of the csi-and not the individual translator-that influences strategic choices, thereby reinforcing the relevance of these categories for analyzing patterns of translation behavior across different cultural domains. the research adopts a mixed-method approach. quantitative analysis examines the distribution of translation strategies employed by machine translation models and human translators, while qualitative analysis explores errors, linguistic accuracy, and cultural nuances. the methodology comprises three phases: (1) corpus formation, (2) extraction of translation equivalents, and (3) classification of translation strategies. ## discussion the present analysis of translation strategies and error patterns highlights the key challenges of machine translation of csis and especially into morphologically complex languages like serbian. when compared to human translation, machine translation exhibits significant problems in handling csis. the study of human translation strategies reveals clear and consistent patterns when dealing with proper names, including street names, brand names, and horeca terminology @xcite . orthographic adaptation is commonly applied when the name includes people's names, particularly historical figures, well-known fictional or real people, or toponyms. in contrast, literal translation is used when the name consists of nouns and adjectives. this systematic approach ensures that cultural and semantic nuances are preserved in the target text while maintaining the readability and cultural familiarity for the audience. machine translation, however, fails to follow such patterns, often producing random and inconsistent results. another notable issue is the predominant use of repetition without adding contextual information. while repetition can sometimes suffice when the meaning of the csi can be inferred from context, this is often not the case. the preservation of the communicative function is a crucial aspect of translating csis @xcite . simply repeating a term without adaptation or explanation can fail to convey its intended cultural significance, leaving the target audience disconnected from the original message. for instance, retaining pastries such as mastellen and pistoletten in their original forms does not evoke any cultural or semantic associations apart from the act of eating. this approach neglects the cultural connotations and traditional significance attached to these items in the source culture. such csis require additional strategies, such as adaptation or explanatory supplementation, to ensure that their cultural and communicative essence is effectively conveyed @xcite . without added context, the meaning and significance of these items are lost to the serbian audience, reducing the overall effectiveness of the translation @xcite . ## conclusion this study has highlighted several key findings regarding the performance of machine translation (mt) systems in translating culture-specific items (csis) between flemish dutch and serbian. first, while models such as gemini and chatgpt demonstrate a promising use of generalization and localization strategies for material and social culture csis, they often fail to apply nuanced approaches required for complex or less common csis. proper names, in particular, pose significant challenges due to the rich inflectional demands of serbian and the need for orthographic adaptation. from a strategy perspective, the analysis reveals that gemini exhibits the most balanced distribution of approaches, incorporating literal translation, orthographic adaptation, generalization, and localization more effectively than other models. nevertheless, even gemini struggles with systematic cultural adaptation and fails to match the nuanced strategies consistently employed in human translation. chatgpt's use of the combination strategies shows potential for improving contextual clarity, yet its tendency to omit morphological adaptations in serbian remains a limitation. meanwhile, google translate, while heavily reliant on repetition, exhibits the highest error rates and demonstrates limited cultural sensitivity in handling csis. these findings underscore the irreplaceable role of human translators in effectively handling csis, particularly in literary and culturally rich texts. human translators not only bring cultural and contextual understanding to the task, but also excel at preserving the communicative function of csis, a dimension often overlooked by mt systems. for example, while mt models tend to rely on repetition or overgeneralization, human translators adapt csis dynamically, ensuring that their cultural essence and intended meanings resonate with the target audience. furthermore, the implications of this study extend to translator training and workflow design. as mt systems become more prevalent, human translators are increasingly assuming roles as post-editors. this shift emphasizes the importance of equipping translators with the skills needed to identify and address the shortcomings of mt outputs, such as the failure to capture cultural nuances or apply morphological adaptations. by integrating human expertise with mt capabilities, translation workflows can achieve greater efficiency while preserving linguistic and cultural fidelity. recent studies have increasingly emphasized the potential of cat and mt tools to enhance translator efficiency, particularly when dealing with complex or culture-bound elements that require extensive background research and strategic decision-making. hadley (2023) highlights how such tools can alleviate cognitive load by supporting specific aspects of literary translation, such as managing sentence length, rhythm, or poetic form. similarly, @xcite demonstrate that specialized tools like puncat can aid in resolving linguistically dense challenges, such as puns, by expanding the translator's pool of potential solutions. these developments suggest promising avenues for future tool design. in the context of csi translation, where human translators often invest significant time in interpreting meaning and selecting appropriate strategies, mt systems could be further adapted to present a range of contextually informed suggestions. experimenting with prompt engineering, designed to generate multiple culturally and linguistically relevant options for each csi, may prove especially beneficial in supporting informed and efficient human decision-making. in this regard, hybrid humanmachine approaches and the development of culturally aware translation tools are crucial. @xcite provide a compelling framework for advancing mt by integrating cultural databases and csi annotations, as well as introducing innovative metrics to evaluate cultural and contextual fidelity. building on such approaches could significantly enhance mt performance, particularly for texts with rich cultural content. one concrete avenue for such improvement involves addressing the persistent errors in orthographic adaptation, particularly when translating into morphologically rich languages like serbian. these errors could be mitigated through the integration of language-specific orthographic rules, culturally adapted name databases, and targeted post-editing support within llm systems. such refinements, combined with the insight and flexibility of human translators, would allow for more accurate and culturally resonant translations of proper names and other csis.