Translating Across Cultures: LLM s for Intralingual Cultural Adaptation (2024)

Abstract: LLMs are increasingly being deployed for multilingual applications and have demonstrated impressive translation capabilities between several low and high-resource languages. An aspect of translation that often gets overlooked is that of cultural adaptation, or modifying source culture references to suit the target culture. While specialized translation models still outperform LLMs on the machine translation task when viewed from the lens of correctness, they are not sensitive to cultural differences often requiring manual correction. LLMs on the other hand have a rich reservoir of cultural knowledge embedded within its parameters that can be potentially exploited for such applications. In this paper, we define the task of cultural adaptation and create an evaluation framework to evaluate the performance of modern LLMs for cultural adaptation and analyze their cross-cultural knowledge while connecting related concepts across different cultures. We also analyze possible issues with automatic adaptation. We hope that this task will offer more insight into the cultural understanding of LLMs and their creativity in cross-cultural scenarios.

## introduction recent progress in nlp is largely driven via llms, which have shown great promise in a variety of tasks including text generation, language understanding, question answering, code generation, and even machine translation. though llms have not achieved state-of-the-art performance for machine translation @xcite , their instructionfollowing ability makes them suitable for tasks involving more creativity and customization during generation. many translation applications require literal translations for which specialized transformer-based models trained on parallel data rachel green: wow, this is so cool, you guys. the entire city is blacked out! monica geller: mom says it's all of manhattan, parts of brooklyn and queens, and they have no idea when it's coming back on. rachel green: wow, you guys, this is big. monica geller: pants and a sweater? why, mom? who am i gonna meet in a blackout? power company guys? eligible looters? could we talk about this later? ok. phoebe buffay: can i borrow the phone? i want to call my apartment and check on my grandma. what's my number? rachel green: wow, this is so cool, you guys. the entire city is blacked out! monica geller: mom says it's all of mumbai, parts of thane and navi mumbai, and they have no idea when it's coming back on. rachel green: wow, you guys, this is big. monica geller: salwar kameez and a dupatta? why, mom? who am i gonna meet in a blackout? electricity board guys? eligible neighbors? could we talk about this later? ok. sl emphasis tl emphasis word-for-word translation literal translation faithful translation semantic translation communicative translation idiomatic translation free translation adaptation figure 2: newmark (1988)'s v diagram of translation methods. sl: source language, tl: target language in this study, we steer clear of this debate and explore this task purely from an nlp perspective particularly investigating the power of large language models. we define a specific version of the task along with clear goals and an evaluation framework for assessing the effectiveness of these adaptations considering factors such as localisation, preservation, naturalness, and appropriateness. the motivation behind this work stems from the need to transcend the constraints of literal translation and explore freer forms of translation such as adaptation. due to the rising creativity, multilinguality, cross-cultural knowledge and instruction-following ability of modern language models, they have the potential to generate culturally resonant adaptations of the source text. we limit our study to cultural adaptation with english as the source and target language i.e. intralingual adaptation. as @xcite argues, although language and culture are interconnected, they are not synonymous. for example, english, being the lingua franca for many parts of the world, can carry views and concepts from different parts of the world. by sticking to english, we can specifically evaluate how well cultural aspects are adjusted in adaptation without the added complexity of translating between languages. as llms become more multilingual (in generation and understanding), their ability can better be evaluated for interlingual adaptation and related aspects of this study can be applied there. we can also view interlingual cultural adaptation as a combination of intralingual cultural adaptation and machine translation. we explore the following research questions and contribute along these: rq 1) how do we define what constitutes adaptation in terms of modifications to the source text i.e. what is changed during adaptation and for what purpose? rq 2) based on the goals of adaptation, what are the optimal criteria/aspects for evaluation? rq 3) given the evaluation, how proficient are modern language models at adaptation? what strategies do they employ, and to what extent do they adapt based on provided instructions? rq 4) what insights does this offer into their parametric cross-cultural knowledge? recent works on evaluating cultural awareness in llms have centred primarily around measuring cultural value alignment @xcite @xcite . while this is important, it does not necessarily indicate that llms are aware of culturespecific items or concepts from different cultures. more research is needed to assess whether llms truly understand these culture-specific items and concepts and can use them coherently in text. our research aims to address this question. jiang and joshi (2023) created a ranking-based statistical qa task that compared cultural concept popularity across countries. @xcite examined the cultural dominance of concrete (e.g., holidays and songs) and abstract (e.g., values and opinions) cultural objects in llm responses. @xcite introduced the idea of automatic cultural adaptation by adapting named entities across cultures and languages, however, it focused on simpler entities in standalone sentences. @xcite constructed resources for cultural adaptation of recipes and also evaluated their method against llm-based adaptation. more recently, @xcite created chinese-english menu corpora and defined an evaluation for the task of adapting restaurant menus. ## task definition for the task of adaptation, we use a corpus of dialogs from a tv show and adapt it to the target culture. we choose adaptation of dialogs instead of standalone sentences as done by @xcite since they provide richer context and are more representative of a true use case of adaptation. the original corpus of dialogues is denoted as we obtain an adapted version of these dialogues denoted as d @xmath0, c), f (d o 2 , c), . . . , f (d o n , c)}, where f represents the language model that adapts the original dialogues to the target culture and c is the specific cultural context or prompt representing the cultural context for adaptation. each dialogue d consists of a number of utterances i.e. @xmath1, u 2 , . . . , u m }. each utterance u @xmath2: text(u i ), where speaker(u i ) is the speaker or participant name and (u i ) is the textual content for utterance u i . our task is to evaluate how well dialogues in the adapted set d a are culturally aligned to the target culture while maintaining the intent and essence of the original dialogue. section 4 provides details on the exact aspects along which we assess these adaptations. ## annotating cultural references corpus description: we choose the 'friends dialogs' corpus for this study. we filter the data to choose dialogues with utterances between 1 and 15. the corpus includes 1110 conversations (or dialogs) containing 11812 utterances by 363 speakers. the reason for choosing such a corpus is that 'friends tv show' is deeply rooted in american culture offering a distinct contrast that highlights the need for adaptation when targeting a new cultural context, specifically india 1 in our study. here, adaptation ensures that the message is not only understood but also embraced and valued in a different cultural environment. culture is a multi-faceted concept. many scholars have tried to define culture. one such theory which is very relevant here and is also mentioned in cultural translation studies is hall's 1 we choose country as a proxy for culture @xcite . while a country such as india has many subcultures, still, many aspects and items are still universal and are relatable to a national audience. those remain the key focus of our study and our annotation task. hall's triad: technical formal informal music, art, food and drink, dress, architecture, institutions, visible behaviour. language appropriacy, rituals, customs, ways/styles (of discourse, dress) orientations action communication environment time space power individualism competitiveness structure thinking iceberg model of culture (or the triad of culture) which divides aspects of culture into three levels: visible (above the waterline), semi-visible and invisible (see figure 3 ) which are referred as the technical, formal and informal level of culture, respectively. as @xcite describes, these levels also relate to how we grasp culture: technical culture can be taught by an expert, formal culture through trial-and-error while informal culture is learned unconsciously. at the tip of the iceberg i.e. the technical level, the goal of translation is to transfer the terms and concepts of the source text to the target text with minimal loss. the terms and concepts are usually referred to as "culture-bound" terms, or "culturemes". hall's second level, i.e., the formal level of culture focuses on rituals, customs, and accepted or appropriate ways of doing things. this level follows the 'skopos theory' @xcite i.e translation should be oriented towards achieving the desired function in the target culture, rather than being faithful to the source text. hall's third level, i.e., the informal level cannot be taught or learned but is acquired 'out-of-awareness' or unconsciously. this is what makes a translation more artistic rather than mechanistic. rq 1: how do we define what constitutes adaptation in terms of modifications to the source text i.e. what is changed during adaptation and for what purpose? in this study, we mainly focus on the first two levels of culture. in order to evaluate whether an adaptation navigates different levels of culture, we need to annotate culturerelated references in the source text and look at how they are being adapted in the corresponding adaptation. we call these items adaptable items or culture-specific items(csi) used by @xcite . items which can undergo adaptation include references to concepts and realities which are foreign to the target culture, socially sensitive and taboo topics, colloquialisms, slang, idioms, figures of speech, humour, or content which can be considered offensive in the target culture. we manually annotate these items in our corpus of dialogues. we also categorise these items into the following categories : 1) ecology ((flora, fauna, winds etc.) 2) material culture (artefacts, food, clothes, houses, towns, transport etc.) 3) social culture (work and leisure) 4) institutions, organizations and ideas (political, social, religious, social, artistic, administrative, ideas etc.) 5) gestures and habits (name of regular behaviour and movements), as proposed by @xcite . additionally, we introduce four more categories which reflect the need for adaptation: 6) slang or figure of speech, 7) offensive content, 8) socially sensitive or taboo topics and 9) humour (since 'friends' is a sitcom). we use the descriptions from newmark (1988) plus descriptions of the other four categories as our annotation guidelines. while @xcite demonstrates an automated approach to annotating culture-bound items, however, for our use case, we observed that it only identifies a fraction of items which can undergo adaptation. also, csi are culture-specific not due to their origin but also due to their foreignness to the target culture. for example, sausage is common in both the usa and the uk but still foreign in indian culture, so manually annotating these items based on the foreignness to the target culture is desired. this is especially important due to the "mcdonaldization of society" @xcite where cultural boundaries are becoming blurred and the notion of foreignness is constantly evolving due to migration and cultural exchange. therefore, we also annotate the degree of foreignness to the target audience to provide a more accurate depiction and expectation since items that are more foreign to the target culture should be more likely to undergo adaptation. we define 3 foreignness levels: 1,2 and 3 for our annotation. foreignness level 1 consists of items/behaviours which have traceable foreign origin/usage however they are common (in terms of familiarity, integration and perception) in the target culture. for example, pizza, chocolate, cricket and coffee are fairly common in india. we omit items in foreignness level 1 from our analysis. for items with foreignness level 2, they may be recognized in the target culture, but their usage or significance is somewhat foreign or less familiar. for example, sushi, tacos, k-pop and beer are not very common and not fully assimilated or mainstream in india. items in foreignness level 3 are largely unfamiliar or perceived as distinctly foreign within the target culture. some examples include kimono, rodeo, thanksgiving etc. which are largely unknown to the indian audience. human annotation: we hired three human annotators from india for our study (both annotation and human evaluation (section 7.2)), who were able to understand different aspects and sensitivities of indian culture expressed through english. the annotators were aware of the source ("usa" as a proxy) culture and good at identifying what aspects of it are foreign to india and to what extent. they were instructed to annotate foot_0 for these cultural items in the corpus, their categories and foreignness level. recent studies @xcite have indicated that deliberation can enhance the quality of answers and even a small number of debates can outperform the wisdom of large crowds @xcite . therefore, in this study, annotations were carried out using deliberation through verbal discussion until a consensus was reached. some examples of csi for different categories or foreignness levels are given in ## evaluation of cultural adaptation in order to design an evaluation framework for adaptation, we need to understand the motivation and goals behind it. in the following section, we mention some goals of adaptation which are aspects along which we assess the quality of cultural another world, thanksgiving, days of our lives, bridesmaids, halloween etc. institutions, organisations and ideas alan alda, mattress king, wendy's, fica, fortunata fashions etc. gestures and habits "you licked and you put", "honk honk", "cha-ching", "step-ity step and jazz hands" etc. offensive content "go to hell", dumb ass, bitch, "climb out of my butt", "third nipple" etc. socially sensitive and taboo topics porn, naked, lust, have sex, undressing etc. humour knock-knock jokes, "get him something like a wrecking ball, or a vile of smallpox" etc. ## prompting for cultural adaptation in this study, based on our goals, we use a simple prompt which includes our goals and exemplars to guide the llm for expected adaptations of the dialogs. the adaptation prompt is given in you have to adapt the given dialogue to align with indian culture and audience while keeping the response in english. adapt culture-specific references/items (do not change character names) which are foreign to indian culture to align with indian cultural context, norms, and sensitivities, while maintaining the correctness, coherence and keeping original intent intact. also adapt very foreign humour, slang or figure of speech unfamiliar to indian english audiences, offensive and socially sensitive or taboo content while making sure that the intensity of emotions like humour don't get affected. ensure that code-mixing is avoided, and output remains in english. every utterance in the original dialogue should have a corresponding utterance in the adapted version, don't add or delete utterances or don't change speakers. ## results and analysis rq 3: given the evaluation, how proficient are modern language models at adaptation? what strategies do they employ, and to what extent do they adapt based on provided instructions? 7.1 edit level analysis % csi edited as shown in figure 5 , %csi edited is lowest for llama-2 70b (45%). this suggests a lower extent of adaptation. llama-3 8b (82.8%) and llama-3 70b (79.7%) seem to perform equally well in editing csi. items from the 'ecology' category have the highest percentage of items edited due to items that are easier to edit. a higher fraction of items with foreignness level 3(very foreign) were edited compared to items with foreignness level 2 indicating that adaptation is prioritizing changing of more foreign items preferably into localized, more relatable items/expressions. ## conclusion in this paper, we explored the task of cultural adaptation within the realm of nlp. we defined the cultural elements likely to undergo transformation during adaptation. we curated a corpus of dialogues, annotating culture-specific elements across various categories and levels of foreignness, and defined the goals and aspects of cultural adaptation employing both edit-level analysis and broader, more contextual dialogue-level analysis for evaluation. we assess the performance of several opensource llms for cultural adaptation and analyse how these aspects tie together. we found that while modern language models are able to localise context to a target culture to a significant extent, they often struggle with reasoning over these cultural artefacts resulting in a lack of coherence within the context of dialogue which often leads to loss of original message. the ability of llms to localize text for a specific target culture provides a good starting point for adaptation experts to take ideas from and further refine and enhance.