Mitigating Translationese with GPT -4: Strategies and Performance (2024)

Abstract: Translations differ in systematic ways from texts originally authored in the same language.These differences, collectively known as translationese, can pose challenges in cross-lingual natural language processing: models trained or tested on translated input might struggle when presented with non-translated language. Translationese mitigation can alleviate this problem. This study investigates the generative capacities of GPT-4 to reduce translationese in human-translated texts. The task is framed as a rewriting process aimed at modified translations indistinguishable from the original text in the target language. Our focus is on prompt engineering that tests the utility of linguistic knowledge as part of the instruction for GPT-4. Through a series of prompt design experiments, we show that GPT4-generated revisions are more similar to originals in the target language when the prompts incorporate specific linguistic instructions instead of relying solely on the model’s internal knowledge. Furthermore, we release the segment-aligned bidirectional German-English data built from the Europarl corpus that underpins this study.

## introduction there has been a surge of interest in the impact of translationese on the performance of natural language processing (nlp) applications. translationese has been shown to have tangible effects on © 2024 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. the outcomes of various cross-lingual tasks, potentially leading to biased results and decreased or artificially inflated performance, especially in evaluating machine translation (mt) models @xcite , but also in the natural language inference tasks when using translated datasets and cross-lingual transfer scenarios @xcite . while translationese is viewed as an inalienable property of translated language, preferences may lean toward translation variants that are closer to target language patterns provided that the meaning and usefulness of the message in the source language (sl) are retained. the task of reducing translationese by making translations less deviant from the originally authored text in the target language (tl) is a newly recognised and relevant nlp problem. at the same time, only a few studies actively address it, including dutta @xcite who remove translation bias in latent representation space, as well as @xcite and @xcite , debiasing translations at the surface text level. our work is the first to explore the utility of linguistically informed prompts to harness the generative capabilities of large language models (llms) in the task of translationese mitigation. this approach is inspired by the successful application of llms to a range of text adaptation tasks including simplification @xcite , style transfer @xcite , and translation (post-)editing @xcite . to the best of our knowledge, only @xcite uses llms to address translationese reduction. we extend this line of research. specifically, we focus on exploring the impact of linguistic knowledge, made available to the llm via prompts, on the outcomes of translationese reduction. the key research question is what type of information is required in the prompts to effectively guide the model through the rewriting process. we propose two approaches: (i) a self-guided approach, which probes the ability of the llm to solve the task independently using its internal knowledge versus (ii) a feature-guided approach, which relies on detailed linguistically-informed instructions to edit the input. the instructions are based on the observed deviations of each individual segment from the expected tl norm. we define the expected tl norm as the type of language that can be expected in the target culture in a comparable communicative situation. it is represented by the average feature values from the register-comparable corpus of tl documents produced by native speakers of the tl (hereinafter referred to as originals). the contributions of this work are as follows: active attempts to level out translationese bias include a method that can be applied in the translate-train foot_0 cross-lingual setup @xcite . they created a mapping from the original to the translated language, projecting original and translated text into a shared multilingual embedding space and minimising the distance between the mapped representations of the originals and translations. to mitigate translationese effects in translated data, dutta @xcite extended the iterative null space projection algorithm @xcite originally designed to mitigate gender attributes, to debias translationese artefacts, and not directly on the text itself, which makes them less interpretable. @xcite reduced translationese deviations at the surface level of text using abstract meaning representation (amr) proposed by @xcite ) as an intermediate form to abstract away from translationese artefacts. in another line of research, @xcite reframed the task as a self-supervised monolingual translationbased style transfer task, aiming to make humantranslated text closely resemble original texts in the tl. however, whether current out-of-the-box llms are able to mitigate translationese from text without removing traces of other variables remains unexplored. apart from the key related works in translationese mitigation, we elaborate on other contemporary studies that have used llms for manipulating text, sometimes with goals related to refining translations or removing undesired information from text representations. @xcite benchmarked the capabilities of llms to translate, and @xcite and @xcite to evaluate translations. along the same line, @xcite extensively analyses the translation output of llms to demonstrate that gpt-enabled translation achieves high quality when utilised for the translation of high-resource languages. however, it still falls short in terms of translation quality for underrepresented languages. likewise, @xcite investigated these differences in terms of the literalness of translations produced by standard nmt and chatgpt-3. contemporaneously to the present work, @xcite propose a simple way to refine translations iteratively with llms based on automatic post-editing that imitates human corrections. ## prompt generation our experiments are designed to explore the effectiveness of including various types of information in prompts that influence the generative behaviour of an llm in the task of translationese mitigation. the study is based on a bidirectional german-english subset of europarl data. each translation direction is aligned at the segment level, meaning that depending on the syntactic arrangement of the same content the source or the target side of the parallel data can have more than one sentence. we experimented with two prompting approaches: self-guided and feature-guided, each with two modes (min and detailed). the full prompt examples for each of these four prompting setups appear in appendix c. the four setups vary in the degree of independence in decision-making given to the model and in the level of linguistic instruction. below we provide a description for each setup. the instructions for all segments were precompiled based on the threshold calculations and formatted as a newline-separated list appended to the task statement, source segment, and target segment (i.e. ht). the two variations of this setup were only different in how detailed the description of each instruction was. ## conclusion in this paper, we explore the potential of using llm-prompts to reduce translationese-related differences between translated and non-translated texts. we evaluate four types of prompts based on either a high-level explanation of the translationese mitigation task or on a micro-managing approach to prompting where the model received segment-tailored instructions to increase or reduce the frequency of prominent translationese predictors. our findings demonstrate that gpt-4 was able to edit human translations to make them less distinguishable in an automatic classification setup from non-translations in both self-guided and feature-guided llm-rewriting modes. the best results were seen for english on the prompts containing feature-guided instructions with a linguistic description of special terminology, showing that the prompting approach benefited from including linguistic knowledge. for german the results were less straightforward but the advantages of detailed task information and specific linguistic instructions were visible. the inferior results on the re-translation task provide further evidence in favour of linguistic features for the translationese reduction task. in our experiments, prompting was more effective in the german-to-english translation direction even though the difference between translated and non-translated documents in german was more detectable to start with (as indicated by 5% higher classification results). we can tentatively explain this result by the language of instruction (english), which might prime the model for better performance when generating english output. future work may need to extend this research by including tasks with instructions in german, especially when the model rewrites translations into german. finally, we have seen that even though rewritten translations exhibited some new individual deviations from non-translations on some individual features, they did not coalesce into patterns picked be a classifier. this conclusion is supported by high results from content preservation metrics and from the manual analysis for accuracy and fluency of translations. while our translationese classification-based evaluation shows that llmrewriting is effective, in our paper we focus on the tip of the iceberg, i.e. the segments from 200 most contrastive documents in our data set. furthermore, manual evaluation and, to some extent automatic evaluation, show that content preservation under llm-rewriting needs more attention, and we will focus on this in our future research. appendix c. examples of prompts by approach and mode 1. self-guided approach: the model has to decide on itself whether a segment contains translationese or not. the same instruction was passed for each pair of segments. here is an original english text: "'in six short months, the presidency has conspired to undermine the stability pact, has shown contempt for the european union's policy towards russia and offended canada do not add any meta-phrases or quotation marks. do not copy the original text. your task is to reduce translationese in a human translation by re-writing it in a more natural way where possible. translationese refers to any regular linguistic features in the translated texts that make them distinct from texts originally produced in the target language, outside the communicative situation of translation. these features are typically detected by statistical analysis and are explained by the specificity of the translation process. human translators are known to simplify the source language content and to make it more explicit. translations can exhibit a tendency to conform to patterns which are typical of the target language, making the output less varied than in comparable non-translations in the target language. the more obvious sign of translationese is interference, which can be defined as over-reliance on the intersection of patterns found in source and target languages. translationese is manifested in the inflated frequencies of specific linguistic items such as function words (especially connectives and pronouns), unusual frequencies of some parts of speech (especially nouns and adverbs) or grammatical forms (especially forms of verbs), in reduced lexical variety and unexpected lexical sequences, in less natural word order, in longer and more complex sentences as well as lack of target language specific items and structures. here is an original english text: "'in six short months, the presidency has conspired to undermine the stability pact, has shown contempt for the european union's policy towards russia and offended canada."' this is its human translation into german: "'in sechs kurzen monaten ist es dem ratsvorsitz gelungen, den stabilitätspakt zu unterminieren, die politik der europäischen union gegenüber russland zu missachten und kanada zu beleidigen."' if you can detect any translationese deviations in this translation, revise this translation to make it sound less translated and return the revised version. if no translationese is detected, return the input translation. do not add any meta-phrases or quotation marks. do not copy the original text. here is an original english text: "'in six short months, the presidency has conspired to undermine the stability pact, has shown contempt for the european union's policy towards russia and offended canada."' this is its human translation into german: "'in sechs kurzen monaten ist es dem ratsvorsitz gelungen, den stabilitätspakt zu unterminieren, die politik der europäischen union gegenüber russland zu missachten und kanada zu beleidigen."' re-write this translation following the instructions: use pronouns instead of nouns as verbal arguments where possible. avoid constructions with indirect objects. do not add any meta-phrases or quotation marks. do not copy the original text. your task is to reduce translationese in a human translation by re-writing it in a more natural, less translated way. translationese refers to any properties of translations that make them statistically distinct from texts originally produced in the target language. here is an original english text: "'in six short months, the presidency has conspired to undermine the stability pact, has shown contempt for the european union's policy towards russia and offended canada."' this is its human translation into german: "'in sechs kurzen monaten ist es dem ratsvorsitz gelungen, den stabilitätspakt zu unterminieren, die politik der europäischen union gegenüber russland zu missachten und kanada zu beleidigen."' revise this translation following the instructions which reflect deviations of this segment from the expected target language norm: use pronouns instead of nouns or proper names as verbal arguments where possible. avoid constructions with indirect objects. an indirect object of a verb is any nominal phrase that is an obligatory argument of the verb but is not its subject or direct object. the prototypical example is the recipient (dem kind) with verbs of exchange: die frau gibt dem kind einen apfel. do not add any meta-phrases or quotation marks. do not copy the original text. your task is to re-translate a human translation to make it more natural in the target language if necessary. here is an original english text: "'in six short months, the presidency has conspired to undermine the stability pact, has shown contempt for the european union's policy towards russia and offended canada."' this is its human translation into german: "'in sechs kurzen monaten ist es