Cultural Transcreation with LLM s as a new product (2024)

Abstract: We present how at Unbabel we have been using Large Language Models to apply a Cultural Transcreation (CT) product on customer support (CS) emails and how we have been testing the quality and potential of this product. We discuss our preliminary evaluation of the performance of different MT models in the task of translating rephrased content and the quality of the translation outputs. Furthermore, we introduce the live pilot programme and the corresponding relevant findings, showing that transcreated content is not only culturally adequate but it is also of high rephrasing and translation quality.

## introduction as defined by díaz-millón and olvera-lobo (2021:358), transcreation is "a type of translation characterized by the intra-interlingual adaptation or re-interpretation of a message intended to suit a target audience (...) paying special attention to the cultural characteristics of the target audience". while transcreation has multiple uses and areas to which it can be applied to, our focus is on the cultural dimension, particularly in the field of machine translation in the domain of customer support, exploring in a near future the extrapolation to other domains, such as marketing. our goal is to enable companies to move to markets distant from their culture while feeling confident that they will be able to effectively communicate with their customers. our approach involves prompting an llm for rephrasing the source text produced by the customer support (cs) agents in english, before the text is translated with an mt model into the target languages (tl) of japanese (ja), korean (ko) and mandarin chinese (zh). our product can adapt messages in the source to the culture of the target audience without depending on the knowledge of the agent, making the final message culturally appropriate and, thus, improving communication between both parties, as shown in figure 1 . ## prompts and mt quality the first step in our transcreation task was to construct the rephrasing prompts for each of the tl. this was achieved through compiling the findings of our research on cultural and linguistic aspects of the tg, specifically regarding cs communication etiquette, as well as the input of native speakers, who are linguistic experts, and non-native speakers, which have lived in these countries for a period of time and learned the language there, into language specific guidelines. as languages which live in the confucianist cultural sphere of influence, our object languages share characteristics such as conveying politeness by showing deference and honoring the interlocutor @xcite . however, the degree and form through which these should be applied differ between them and, thus, different prompts were built depending on the tl. the next step was ensuring that the content rephrased in the source using our prompts could produce high-quality translations. in order to test this, we translated a total of approximately 1000 rephrased segments distributed across three language pairs (lps): ~400 segments for en-ja and en-ko and ~200 for en-zh, with six different mt providers (azure, aws, deepl, google, chatgpt and gpt-4) and annotated the outputs so the quality of the translations could be evaluated through multidimensional quality metrics (mqm) framework scores @xcite . the resulting average scores were of around 88 mqm for both en-ja and en-ko, with en-zh scoring higher with an average of 94.3 mqm, reflecting that different mt engines can be successful in the task of translating rephrased content with no adaptation or customization. in addition, by running an automatic quality estimation metric (qe) @xcite on the translation of the original message and their rephrased versions for the three lps, we could see that the transcreated messages score higher on average, thus indicating that cultural transcreation (ct) has the potential to improve translation quality. for the purpose of testing our ct product, we have integrated it into a cs platform in the form of a widget with a "rephrase" button which calls an api endpoint attached to the ct service after a pre-processing step, including data anonymization. the service relies on an llm (gpt-4), to rephrase the message according to the prompts, and returns the final rephrased message to the agent in seconds, ready to be translated and sent to the recipient. ## live pilot programme three customers were selected for our pilot evaluation and the data produced during this period, namely the original text, the rephrased text, and the target text (mt version sent to recipients), are being continuously assessed and analyzed quality-wise. during the first three weeks of the pilot, around three hundred cs emails (based on traffic volume per language) were rephrased: 58.7% for ja, 30.3% for zh and 11% for ko. in terms of ct quality, almost all the rephrased texts achieved the target-culturalization, and only about 8% were not culturally aware, but no critical level of errors were found. the more frequent rephrasing failures in all three languages were unnatural expressions, inappropriate word substitutions, and errors in the format of greetings and closings.