Using Translation Techniques to Characterize MT Outputs (2025)

Abstract: While current NMT and GPT models improve fluency and context awareness, they struggle with creative texts, where figurative language and stylistic choices are crucial. Current evaluation methods fail to capture these nuances, which requires a more descriptive approach. We propose a taxonomy based on translation techniques to assess machine-generated translations more comprehensively. The pilot study we conducted comparing human machine-produced translations reveals that human translations employ a wider range of techniques, enhancing naturalness and cultural adaptation. NMT and GPT models, even with prompting, tend to simplify content and introduce accuracy errors. Our findings highlight the need for refined frameworks that consider stylistic and contextual accuracy, ultimately bridging the gap between human and machine translation performance.

## introduction rapid advancements in neural machine translation (nmt) and generative pre-trained transformer (gpt) models have significantly improved the quality of machine-generated translations in recent years. in many cases, these models achieve an output product that closely resembles human translation @xcite , making it increasingly difficult to describe or evaluate their performance using traditional metrics. although early claims suggested that machine translation (mt) had reached parity with human translation @xcite , subsequent studies have challenged these assertions, underscoring the persistent difficulties of evaluating machine-generated output in a way that captures their full complexity @xcite . these debates further emphasize the limitations of current assessment methods, particularly their inability to account for the contextual and stylistic nuances @xcite that professional translators consider essential. one of the most pressing challenges in this context is the translation of creative texts, such as literature and marketing content. unlike technical or informational texts, which often follow predictable structures and terminology, creative texts rely heavily on figurative language, including irony, metaphor, and ambiguous phrasing. these elements often lead to overly literal, word-forword translations that do not convey the intended meaning in the machine-translated text @xcite . although current gpt models offer notable improvements by considering broader contextual relationships in sentences @xcite , they still struggle with the complexities of creative expression. current evaluations are usually based on automatic metrics such as bleu @xcite or comet @xcite , or on manual evaluations that produce a list of errors and their severity, such as the mqm taxonomy @xcite . however, these metrics mainly focus on the traditional accuracy and fluency paradigms, which do not account for any stylistic variation. recent research has even shown that the inclusion of machine-translated texts in test data can significantly affect the results of evaluation outcomes. for example, @xcite found that mt systems may appear to perform better or worse depending on the nature of the test data. thus, we need to explore alternative approaches to describe and assess these texts in accordance with the contexts in which they are intended to be used. in contexts where both the conveyed information and the expressive or persuasive function of the text are essential, human translators frequently employ a range of techniques to help the target audience grasp the subtle nuances of the original text. these strategies ensure that not only the content, but also the intended impact of the text is effectively conveyed in the translated version. if texts translated using nmt and gpt models are employed in the same scenarios where human translators apply these techniques, it is worth considering whether these techniques can also serve to describe and, consequently, evaluate the quality of machinegenerated translations. in this context, translation studies provide a rich theoretical framework that can offer more nuanced descriptive criteria. specifically, we develop a taxonomy partially based on the translation techniques defined by @xcite . their framework categorizes the translation techniques employed by human translators, which can serve as a benchmark for describing machine translations at a deeper level. using translation techniques such as modulation, amplification and explicitation, our proposed method aims to capture the complexity of translation beyond literal equivalence, helping us describe machine translation outputs or machine translationese. this approach enables us to assess how well mt models handle pragmatic and linguistic challenges, including idiomatic expressions, register changes, and cultural adaptation, thus providing a more comprehensive understanding of their strengths and weaknesses. the remainder of this paper is structured as follows. section 2 reviews related work on translationese regarding mt, highlighting some of the key concepts of its characterization. section 3 introduces the proposed framework based on translation techniques and its theoretical underpinnings. section 4 presents the setup and methodology used to conduct a pilot study of this framework, followed by the results in section 5. finally, section 6 concludes the paper and outlines future research directions. ## human translation vs machine translation the study of differences between translated texts and non-translated texts has long been a central focus of translation studies research, with early research identifying distinct linguistic features that describe what has been called translationese. toury 2012 differentiates between the law of interference, which refers to the elements of the source text that are retained in the translation, and the law of growing standardization, which relates to the tendency to apply the norms of the target language and culture to the translation product. thus, any final translation is the hybrid result of the application of both laws. @xcite makes a distinction between s-universals and t-universals. s-universals are features that can be traced back to the source text. t-universals, on the other hand, are features that should be studied by comparing translated texts to non-translated texts in the target language, using a comparable corpus. they include features such as simplification, untypical patterning, and underrepresentation of target-language-specific items. @xcite suggests there are several translation universals, which are linguistic features that tend to characterize translated texts regardless of the language pair or direction of translation. these include simplification, where translations exhibit reduced structural and lexical complexity; explicitation, the tendency to render implicit information more explicit; normalization, which aligns translations more closely with conventional target language norms; leveling-out, which results in reduced variation across different text types; and interference, where source language structures influence the target text. corpora have been used extensively to study translationese. for example, corpas @xcite argues that translated texts include lower lexical diversity, shorter sentence structures, and increased explicitation. these tendencies emerge due to the translator's dual commitment to preserving source meaning while ensuring readability in the target language. empirical studies using comparable corpora have consistently shown that translationese manifests across languages, regardless of the specific translation directions @xcite . human translations and machine translations have also shown divergences at the morphosyntactic level. @xcite conduct a large-scale fine-grained comparative analysis across three language pairs and show mt is consistently more conservative than human translations, as it shows less morphosyntactic diversity, more convergent patterns, and more one-to-one alignments. as mt technology advances, researchers have begun to investigate whether similar patterns can be detected in mt-generated texts and post-edited (pe) translations @xcite . some studies suggest that pe texts inherit certain traits from raw mt output, such as reduced lexical diversity and terminological consistency that align more closely with machinegenerated texts than with human translations. for instance, @xcite identify a loss in lexical richness in mt output, which could subsequently influence the characteristics of postedited texts. @xcite finds that post-edited documents have lower lexical variety and lower lexical density than human translations. moreover, sentence length and parts-of-speech in post-edited texts are more similar to the source language than those in human translations. a study by @xcite examines translation relations to identify differences between nmt and human translations. the findings reveal that nmt systems tend to rely more heavily on literal translations compared to human translators, especially in the use of semantic-level translation techniques. the advent of large language models (llms) and gpts has introduced the concept of generatese, referring to the distinct linguistic patterns produced by these models during text generation tasks, including translation. he et al. ( 2024 ) investigate whether llms can mimic human translation strategies by analyzing source sentences and inducing translation-related knowledge such as keywords and topics. their research shows that while llms can exhibit humanlike translation strategies, there are challenges to reducing errors such as hallucinations and mistranslations, which are often associated with generatese. comparative analyses between human translations and machine-generated texts have highlighted notable differences. a study by @xcite proposes an iterative prompting approach for llms to self-correct translations. interestingly, while this method reduces string-based metric scores, neural metrics suggest comparable or improved quality. these refined translations achieve better fluency, although other challenges related to generatese still remain. other studies also suggest that llms generate translations that deviate more from the source text than those produced by nmt models @xcite . ## framework of translation techniques translation techniques play a fundamental role in translation studies, serving as essential tools to analyze and understand the procedures by which translators achieve equivalence between source and target texts, and have long been studied by translation scholars @xcite @xcite @xcite . these techniques provide a framework for systematically identifying and categorizing the choices translators make during the translation process to address linguistic, cultural, and contextual challenges. their significance extends to various aspects of translation theory and practice, contributing to improving translation quality and the development of pedagogical approaches. translation techniques allow for a structured approach to evaluating translation choices by offering a set of predefined categories that describe how equivalence is achieved at the micro-textual level. this systematic analysis helps identify patterns in translator behavior, and to compare different translations of the same text. by distinguishing techniques, we can better understand how translators navigate linguistic and cultural differences. however, there is no consensus in academia on the classification and nomenclature of translation techniques. @xcite were the first to publish a classification of translation techniques with a clear methodological purpose. they defined seven basic procedures operating on three levels of style and classified them between literal and oblique. @xcite suggests three types of translation techniques: additions, subtractions and alterations. these techniques are used to adjust the form of the message to the characteristic structure of the target language, to produce semantically equivalent structures, to generate adequate stylistic equivalences, and to produce an equivalent communicative effect. @xcite uses the term procedures to classify translation techniques proposed by comparative linguists. these include: recognized translation, where an already accepted term is used even if it is not the most precise; functional equivalence, which replaces a term with a culturally neutral expression plus a qualifier; and naturalization, which adapts a source language word to the phonetic and morphological norms of the target language. he also introduces translation labels for provisional translations, often literal in nature. additionally, newmark allows for combining multiple procedures (doubles, triples, etc.) and includes synonymy as a separate category. @xcite modify and expand previous classifications. they isolate the concept of technique by focusing on the notion of functionality, situating it in relation to the text and the context. for our framework, we take into account previous research on mt-generated content (sanchez-gijón, 2024; @xcite and we make an effort to group the different phenomena in order to simplify corpus annotation. we simplify the original set of 18 translation techniques and add naturalness, which should be understood as a habitual use of the language, free of grammatical errors, fluid in style, and without expressions that are strongly influenced by other languages (do campo bayón and sánchez-gijón, 2024). below we define the translation techniques and illustrate them with some examples from the annotated segments of the pilot study detailed in section 4, for the catalan-english language pair: ca: ja devia tenir un senyal vermell a la cintura, però així que el vent m'havia sortit per la boca la cinta tornava a fer-me el martiri. [i must have already had a red mark on my waist, but as soon as the wind had left my mouth, the ribbon went back to tormenting me.] en: i pictured the red weal round my waist, but the moment i started rushing and getting out of breath, the elastic sliced into me again. ca: la meva reina, va dir [my queen, he said.] en: he said, my darling. it includes generalization and linguistic compression, conveying the same message with fewer details. example: ca: la cinta de goma a la cintura estrenyent, estrenyent (...) [the rubber band around my waist, tightening, tightening.] en: the elastic cutting deep into my waist (...). ca: (...) i a cada banda de la cara la medalleta de l'orella. [and on each side of the face, the little medal on the ear.] en: (...) and little medal-like ears. ca: tan petita i ja té promès? [so young and you already have a fiancé?] en: 'aren't you too young to have a fiancé?' ca: (...) i vinga riure [and he kept on laughing] en: (...) and he laughed till he cried. ca: (...) la meva mare morta i sense poderme aconsellar [my mother dead and unable to advise me] en: (...) my mother dead and gone and not around to give me advice • fluency and accuracy errors these errors occur when the translated text contains unnatural phrasing, awkward constructions, or inaccuracies that may hinder comprehension or misrepresent the source text. they can include grammatical mistakes, stylistic inconsistencies, or mistranslations that affect the quality of the final output. these are the usual elements included in traditional evaluations and are incorporated in our annotation process to better understand the output translations in relation to usual evaluation techniques. by applying these categories, we aim to gain deeper insights into the decision-making process of translators and the impact of various strategies on the final translated text. ## experimental setup and methodology as an initial step following the selection of the translation techniques to be used for annotation, we decided to conduct a pilot study using one of the most renowned works in catalan literature, la plaça del diamant by mercè rodoreda and its translation into english by peter bush in 2013. the novel was automatically segmented into sentences, and the first 60 segments were selected for the annotation process. we annotated the published translation into english and the translations produced by three mt engines. we used a nmt model (microsoft translator) and a gpt model (chatgpt), as research shows these models translate broader contextual relationships across sentences better than nmt models @xcite . moreover, we used chatgpt with a specific set of prompts to assess whether prompting techniques could improve the translation results for this type of text @xcite . we opted not to randomize the selection of segments, as the application of translation technique categories often relies on contextual references that extend beyond individual segments. maintaining sequential order allowed us to preserve the coherence of the text and ensure that context-dependent techniques could be accurately identified and applied. a relatively small number of segments was chosen for this pilot study, as its primary objective was twofold: first, to evaluate the relevance and applicability of the selected translation techniques; and second, to compare the results of the published human translation against raw machine translation (mt) outputs generated by nmt and gpt-based models with or without prompting techniques. for each segment in the source language, four translations were annotated: (1) human translation, (2) microsoft translator translation, (3) chat-gpt translation without additional prompts, and (4) chatgpt translation with specific prompts. for this version of chatgpt, we introduced the following prompts in english, which described both the step-by-step actions followed by professional translators as well as some considerations regarding the context and the text type: translate a literary text from catalan (ca) to english (en) while considering the cultural differences between the ca and en readers. follow a professional translator's strategy by considering the text's function, the cultural and social differences between the two audience groups, the author's style, and the text genre. assume the en reader is unfamiliar with ca culture, particularly regarding life in the 1950s in barcelona and surrounding areas. steps to follow: analyze the original text: understand the text's purpose, the author's style, and specific cultural references unique to 1950s barcelona. 2. identify cultural and social differences: note key cultural elements that may need context or adaptation for an en audience. 3. translation strategy: adjust cultural references as needed to make them understandable without losing the text's authenticity. maintain the original author's style and tone while ensuring it is accessible to an en audience. keep the genre conventions in mind to ensure the translated text aligns with expectations typical to that genre in english literature. adapt for en readers: provide additional context where necessary to enhance under-standing of cultural nuances without altering the narrative. 4. review and revise: ensure the final translation feels natural to an en reader and accurately represents the original text's nuances. output format: provide the translated text in a natural and fluent english format, maintaining the original length as closely as possible while ensuring cultural clarity. each segment was annotated by two different annotators with previous experience in similar tasks. for the segments in which both annotators were not in agreement, a third annotator assessed the proposals and made a final decision. ## results in figure 1 we can see the results of the annotation process. human translations include a higher number of translation techniques than any of the mt-produced translations, except for the simplification technique. in fact, this is one of the techniques that reduces source language information without any substitution or modification. however, human translation incorporates more omissions, which can be linked to the compensation process undertaken while translating, as in many other segments human translations incorporate techniques used to add more explicit information. it is also clear from the results that the nmt model (microsoft) does not use any of the translation techniques and thus produces more literal translations. from the annotated techniques, we can highlight the use of naturalness in the human translation, which is the most frequently applied. in the search to produce a text that engages the target reader and has the same impact as the source reader, the translator makes decisions that move away from word-to-word translation and incorporate a creative component. moreover, human translations also include increased use of the adaptation of the content (for example, with names of people and places) and amplification of certain elements to highlight them in the translation. in figure 2 we can see the results for accuracy and fluency for all output translations. all outputs contain a considerable high number of inaccuracies or translations which do not convey the meaning of the source text. once again, the nmt model produces the highest number of fluency and accuracy errors, which are highly reduced in the case of chatgpt. an interesting result is that the inclusion of prompts increases the number of accuracy errors. this could be linked to the effort made by chat-gpt to create more literary and creative content when the instructions explicitly indicate it. the creation of this type of translations seems to have as a side-effect the increased number of hallucinations or errors in the translations. ## conclusion and future work the improved quality of the translations produced by the nmt and gpt models makes it increasingly difficult to distinguish them from human-produced texts. current evaluation metrics fail to account for the stylistic and contextual nuances that are crucial in human translation. the challenge is particularly evident in the translation of creative texts, where figurative language plays a key role in meaningmaking. to address these limitations, we proposed a framework based on translation techniques, inspired by established models in translation studies. our pilot study comparing human, nmt and gptproduced translations of la plaça del diamant reveals significant differences in translation strategies. human translators employ a wider variety of techniques, such as amplification, naturalness, and adaptation, that contribute to more natural, culturally appropriate, and stylistically coherent translations. in contrast, nmt and gpt models, even with targeted prompts, tend to simplify content, favoring more literal renderings that sometimes fail to capture the expressive function of the source text. while prompting techniques can make gpt translations appear more creative, they also introduce a higher number of accuracy errors, suggesting a higher introduction of hallucinations. these findings reinforce the need for refined evaluation frameworks that move beyond traditional metrics to incorporate a deeper analysis of textual adaptation and stylistic effectiveness. by systematically categorizing translation strategies, our approach provides a more comprehensive way to assess how well machine translations handle complex linguistic and cultural challenges. future research should build on this framework by expanding corpus size, and exploring automated annotation methods to improve scalability. ultimately, integrating translation techniques into mt evaluation can offer a more human-centric perspective, bridging the gap between computational advancements and the nuanced decision-making process of professional translators.