Machine Translation Meets Large Language Models: Evaluating C hat GPT ’s Ability to Automatically Post-Edit Literary Texts (2024)

Abstract: Large language models such as GPT-4 have been trained on vast corpora, giving them excellent language understanding. This study explores the use of ChatGPT for post-editing machine translations of literary texts. Three short stories, machine translated from English into Dutch, were post-edited by 7-8 professional translators and ChatGPT. Automatic metrics were used to evaluate the number and type of edits made, and semantic and syntactic similarity between the machine translation and the corresponding post-edited versions. A manual analysis classified errors in the machine translation and changes made by the post-editors. The results show that ChatGPT made more changes than the average post-editor. ChatGPT improved lexical richness over machine translation for all texts. The analysis of editing types showed that ChatGPT replaced more words with synonyms, corrected fewer machine errors and introduced more problems than professionals.

## introduction in recent years, there has been a noticeable shift in the perception of the use of computerassisted translation technologies for literary translation. advances in the quality of machine translation (mt) and the development of sophisticated computer-assisted translation (cat) tools have contributed to this changing landscape @xcite . following the emergence of neural machine translation, a growing body of research has examined the use of machine translation (mt) and post-editing (pe) for literary texts. researchers have looked at various aspects related to the use of mt and pe in the context of literary translation, such as perceived usefulness of mt @xcite @xcite , ethical issues (taivalkoski-shilov, 2019; @xcite , translation quality @xcite @xcite , the impact on the translation process @xcite , and the reader's reception of the final product @xcite . further automation of the translation process can be accomplished by implementing automatic post-editing (ape), which refers to methods that improve the output of machine translation systems by applying automatic editing operations @xcite . technological advances are rapidly evolving and the potential of using ai systems based on large language models (e.g. chat-gpt) is currently being explored for a variety of applications @xcite , one of which is the post-editing of machine translation output @xcite . this study explores chatgpt's ability to automatically post-edit literary texts that were machine-translated from english into dutch by a neural machine translation system. we evaluate chatgpt's performance by comparing its automatically post-edited texts to versions that were post-edited by professional literary translators. ## related research over the past decade, a number of studies have been carried out to examine the usefulness and suitability of machine translation and post-editing for literary translation. researchers often compare raw (unedited) machine translations of literary texts with their (published) human-translated counterparts. the mt systems used are either generic systems @xcite or mt systems adapted specially for literary translation @xcite @xcite . in order to gain valuable insights into the strengths and limitations of mt for literary translation, error classification schemes such as mqm @xcite or scate @xcite are often used. these classification schemes typically distinguish between accuracy and fluency errors. accuracy errors refer to the failure to transfer meaning correctly from source to target, whereas fluency errors refer to the failure to produce grammatically correct, idiomatic and fluent translations. existing error classification schemes have been adapted to suit the specific characteristics of literary texts @xcite . despite the high quality of the current generation of transformer-based neural mt systems, they still produce errors in both accuracy and fluency. this is certainly the case with more creative use of language, which is typical of literary texts. in addition, machine-translated texts exhibit different linguistic characteristics (e.g. less lexical variety, less cohesion, syntactically less diverse texts) than human translations @xcite . the involvement of professional translators in the translation of literary texts is therefore essential. in the context of literary translation, post-editing can be applied by having human translators work on the raw machine translation suggestions @xcite @xcite @xcite . human translators then correct the errors and polish the machine's raw output, transforming it into a high-quality, publishable literary translation by ensuring that the translated texts capture the nuances, cultural references, and literary techniques present in the original work. in their study, @xcite compared three successive versions of a dutch translation of an english novel: the raw mt output, the postedited version and the revision of the post-edited text. they manually annotated the errors in the mt and categorised the editing changes in accordance with a linguistic typology. the study showed that most mt errors were corrected in the postediting process, and that the post-editor mainly made lexico-semantic and stylistic changes. fortyfour percent of the post-editing changes involved the correction of mt errors, 24% were preferred changes and 9% were labelled as 'undesirable'. they also used different automatic metrics to measure the (dis)similarity between the different versions, focusing on different aspects. the amount of editing was assessed by translation edit rate @xcite and charcut @xcite . semantic similarity was measured by the neural metrics comet @xcite and bertscore @xcite , which calculate the distance between vector representations of sentences and tokens. astred @xcite , a metric that compares the edit distance between the dependency structures of two sentences, taking into account word alignment information, was used to quantify syntactic changes. another feature that has been widely studied in previous research on literary machine translation is lexical richness. @xcite showed that mt systems are not able to achieve the same level of lexical richness as human translated texts. @xcite also observed a decrease in lexical richness from human translation to machine translation, suggesting a certain homogenisation of the lexicon used by nmt systems. @xcite investigated whether the level of lexical richness increases during postediting and revision, but in their study they found similar levels of lexical richness in the mt, pe and revised translation. large language models such as gpt-4 or llama are trained on unprecedentedly large corpora. llama-3 for example has been pre-trained on approximately 15 trillion tokens of text gathered from publicly available sources foot_0 . due to the size of the training set, they have a comprehensive understanding of language. as they are trained on a much larger data set than other end-user applications such as automatic speech recognition or machine translation, researchers propose a combination of the two. @xcite used llama to correct errors produced by the whisper automatic speech recognition system @xcite , a task similar to the post-editing of machine translation output. @xcite explore the use of gpt-4 for automatic post-editing of nmt output in different language pairs. they experimented with wmt-22 general mt translation task datasets and wmt-20 and wmt-21 news translation task submissions annotated with mqm. translation quality was assessed using neural evaluation metrics. their results show that gpt-4 effectively improves translation quality compared to the best systems from wmt-22 across a number of language pairs and generates meaningful edits to translations. but they also show that gpt-4 can produce hallucinated edits, suggesting caution in its use as an expert translation post-editor. research on the use of chatgpt for automatic post-editing is very scarce and has not yet been applied to challenging text types such as literary texts. in this study, we extend the work of @xcite and use chatgpt 4.0 to automatically post-edit more creative texts. we are not only interested in whether automatic post-editing with chatgpt improves the quality of the neural machine translation output. we also want to know how chatgpt's post-editing ability compares with that of professional literary translators. using automatic and manual evaluation methods we seek an answer to the following research questions • rq1: does chatgpt make more or less changes to the machine-translated texts than professional literary translators? ## discussion we conducted an experiment comparing the postediting capabilities of chatgpt with those of experienced professional literary translators working on english-dutch literary texts. we used a data set collected in the dual-t project, in which 23 professional english-dutch literary translators post-edited the neural machine translations of three short stories by the same author. we then asked chatgpt 4.0 to create post-edited versions of the three texts. this collection of post-edited literary translations allows us to compare the results of human and automatic post-editing. we formulated four research questions and used a combination of automatic and manual evaluation methods to compare all post-edited texts. the charcut and ter results show that chatgpt makes more changes to the machine-translated texts than the 'average human post-editor' (rq1). chatgpt achieved the highest charcut scores for two texts and made the most lexico-semantic changes in all texts compared to the human posteditors. chatgpt improved lexical richness over the machine translation for all texts. the obtained bertscore values indicate that the meaning of the text is less preserved in chatgpt's versions compared to those of the 'average professional literary translator' (rq2). when analysing the types of changes made by post-editors, we clearly see that posteditors mainly make lexico-semantic and stylistic changes, as was the case in @xcite . looking more closely at the types of changes made by individual post-editors, we can see that there is a great deal of variation between different post-editors. a high degree of individual variation between professional translators during revision has been observed in previous studies @xcite and can be attributed to the individual style of professional translators. the only striking difference between chatgpt and professional literary translators is that it replaces words with synonyms more often than human post-editors (rq3). with only 18 accuracy errors and 29 fluency errors, the neural machine translation system did an excellent job. most errors were solved during post-editing. looking at the mt quality labels, we can conclude that chatgpt solves fewer errors in the machine-translated texts and introduces more problems compared to professional literary translators (rq4). this study aimed to provide insights into the capabilities and limitations of chatgpt for automatic post-editing of literary machine translation. overall, chatgpt proved to be a more aggressive post-editor than the professionals, making too many changes to the machine-translated text, despite being explicitly instructed not to do so in the prompt ("do not edit the translation if the translation is faithful to the meaning of the source text and faithful to the style of the original author"). it also corrected fewer errors, introduced more problems and deviated more from the meaning of the target text. nevertheless, chatgpt corrected most of the errors and provided meaningful edits. while fully automatic post-editing with chat-gpt is not yet feasible, and probably not desirable from an ethical point of view, ai tools based on large language models can generate highquality post-editing suggestions. as such, they can certainly complement the toolkits of professional translators. a promising direction that deserves further investigation is to have human translators work directly on texts that have been automatically post-edited by ai. this could help to leverage the strengths of both human and machine skills.