Extending CREAMT : Leveraging Large Language Models for Literary Translation Post-Editing (2025)

Abstract: Post-editing machine translation (MT) for creative texts, such as literature, requires balancing efficiency with the preservation of creativity and style. While neural MT systems struggle with these challenges, large language models (LLMs) offer improved capabilities for context-aware and creative translation. This study evaluates the feasibility of post-editing literary translations generated by LLMs. Using a custom research tool, we collaborated with professional literary translators to analyze editing time, quality, and creativity. Our results indicate that post-editing (PE) LLM-generated translations significantly reduce editing time compared to human translation while maintaining a similar level of creativity. The minimal difference in creativity between PE and MT, combined with substantial productivity gains, suggests that LLMs may effectively support literary translators.

## introduction post-editing of mt has become an increasingly common service, given the cost-efficiency and good quality compromise that this practice offers. however, while several studies have confirmed that post-editing mt boosts productivity in terms of translation speed @xcite , the benefits diminish significantly when dealing with poor-quality mt outputs @xcite . this challenge is particularly pronounced for literary texts, where the final quality often suffers not only in terms of translation accuracy but also in the preservation of creativ-ity, as discussed by @xcite . recent llm advancements have demonstrated significant improvements in handling context issues and figurative language to generate highly accurate and fluent translations. unlike nmt systems that often tend towards generating translations that are either too literal or inaccurate, llms leverage large training data to generate context-aware translations less literally. nevertheless, the extent to which they may support literary translators, without sacrificing creativity, remains underexplored. in this study, we collaborated with four professional translators to evaluate the feasibility of postediting literary translations generated by llms, focusing on three key aspects: editing time, translation quality, and creativity. we compare the performance of gpt-4, gpt-3.5, and a literary-adapted mistral-7b model. we also developed a custom research tool called unior-pet @xcite to collect detailed statistics on the editing process of a literary sci-fi novel. our findings reveal that post-editing llmgenerated translations between well-supported languages significantly reduces editing time compared to human translation while maintaining a similar level of creativity. as the difference in creativity scores between human and post-edited llm translations appears to be minimal, our findings suggest that llms can serve as valuable tools for literary translators. ## related work research on post-editing has traditionally centered on technical and commercial texts, where terminological consistency and turnaround time are often prioritized @xcite . however, trans-lating creative works such as literature introduces unique challenges. nmt models have been shown to struggle with creative phraseological challenges, such as translating idiomatic expressions, where they often produce overly literal outputs. corpas pastor and noriega-santiáñez (2024) highlighted these limitations, particularly in the context of literary texts. in contrast, @xcite demonstrated that llms are capable of generating less literal and more contextually appropriate translations, especially when translating idiomatic expressions that tend to be generated with a higher level of abstraction, defined by the authors as "figurative compositionality". further studies on idiomatic expression translation, particularly for the english-italian language pair, have confirmed the high-quality results achieved by generalpurpose llms @xcite . their findings suggest that these models could address some of the shortcomings observed in nmt systems when translating literature, making them a promising tool for literary translation. a study conducted by guerberof-arenas and toral (2022) concluded that nmt was unable to handle the complex demands of translating literature or supporting literary translators effectively, resulting in low-quality outputs and diminished creativity. their findings revealed the limitations of such models in preserving creativity during translation, becoming a constraint for the translator's creativity when used. human translation (ht) consistently outperformed mt and pe in creativity, as evidenced by the annotation of units of creative potential. these findings align with the study by @xcite , that showed how the features found in post-edited translations align more closely with the ones found in the mt output than in the ht. however, more recent advances in llms may shift this paradigm. as demonstrated by karpinska and iyyer (2023) and @xcite , llms excel at leveraging training data to deal with context-related issues, which is critical for translating creative works that require discourse-level coherence and contextual understanding. techniques such as in-context learning @xcite and prompt engineering allow llms to maintain higher degrees of fluency, consistency, and stylistic fidelity compared to nmt systems. finally, their ability to adapt to specific linguistic patterns and translation memories in real time, as shown by @xcite , further enhances their applicability in the creative translation domain, suggesting that llms could potentially overcome the creativity gap identified in nmt outputs, supporting professional translators in producing high-quality creative translations with context-aware terminology and accurate lexicon. drawing on @xcite , in this study we consider creativity as a process that requires both originality and effectiveness @xcite . this implies that in order for a product to be creative, it needs not only to be novel but also of value, and therefore acceptable, for the context in which it is created. in section 5, we will use the annotations of units of creative potential to reflect the original units introduced by the translators (novelty), and translation quality metrics as a proxy for the translation acceptability. ## methodology we collaborated with four professional translators who specialize in literary and editorial translations to translate and post-edit excerpts from the novel "oryx and crake" by canadian author margaret atwood @xcite from english into italian. the novel was selected for its extensive use of playful and thought-provoking neologisms, vivid imagery, and richly detailed language, which present significant challenges in the translation process @xcite @xcite ## results and analysis thanks to the use of unior-pet and pet, we were able to collect significant data on each translation version providing foundation for a comparative analysis of the different models. more specifically, we have calculated quality metrics with bleu @xcite , chrf @xcite and comet @xcite ), which we average and normalize by time, as well as aggregated editing times. finally, we compute human-targeted translation edit rate @xcite . ## creativity annotation to evaluate creativity in the post-edited translations and conduct a model-wise comparison, we annotated units of creative potential in the st and creative shifts in the target texts (tt), that were originally generated by the three llms, and then post-edited by four translators. annotation process. our annotation framework follows the methodology proposed by @xcite , where units of creative potential (ucps) are defined as units that could invite creative deviations during post-editing, aimed at preserving or enhancing the creativity found in the st, and creative shifts reflect the actual creative units introduced by translators during postediting. annotations were performed by two linguists with expertise in translation studies, who are native speakers of the target language and proficient in english. after annotating 10% of the dataset, inter-annotator agreement (iaa) was calculated to ensure the reliability of the annotations. the initial agreement, measured with cohen's kappa, was equal to @xmath0.35 for type agreement and @xmath1.85 for span agreement, due to disagreements primarily on the type of creative shift to assign, rather than the identification of the creative shifts themselves. following a collaborative resolution process, we refined the annotation guidelines and calculated agreement on the final annotations, reaching a type agreement equal to @xmath2.57 and a similarly high span agreement, equal to @xmath3.86. ## discussion taken together, our results show that a larger and more advanced model (gpt-4) generated translations that required fewer edits and resulted in a higher-quality post-edited translation, as resulted from the lower editing time and the higher qualityto-time ratio. the creativity score is also the highest, suggesting an interesting correlation between original mt quality and creativity in post-editing. the domain-adapted mistral-7b model also displayed promising performance, obtaining a qualityto-time ratio higher than the one obtained by the larger gpt-3.5, requiring more edits but a significantly lower editing time, while obtaining a similar creativity score. in this case, we find that mistral's creativity comes at the cost of increased postediting effort. ht, despite requiring a significantly higher editing time, is the most accurate translation variant according to comet scores and it presents a high creativity score that is very similar to the post-edited texts. in overall, we find that the creativity score does not differ significantly between the four models, as both the number of identified creative shifts and the quality metrics are similar across all translation variants. these findings are in contrast with what was found in the original study, where the difference between the two modalities (ht and pe) was substantial and ht was found to be notably more creative than their post-edited variant. we speculate that the higher and more fluent mt quality given by llms may be of less constraint to the translator in the post-editing process, leading to equally creative translations. ## conclusion in this study, we investigated the potential of llmbased post-editing in the literary domain, comparing a literary-adapted mistral model with gpt-4 and gpt-3.5. by collaborating with four professional literary translators, we collected detailed data on editing times, error rates, and post-editing efficiency, using our custom-built tool unior-pet. we demonstrate the contributions that llms can make in literary post-editing workflows, bridging the gap between productivity and creativity. our findings highlight two important benefits granted by the adoption of llms. first, we demonstrate that, in the context of our study, creativity does not present a significant difference between human translation and post-edited llm translations. the marginal difference in creativity between the four translation variants suggests that the post-edited outputs may preserve creativity effectively. this may be due to the more fluent and higher-quality outputs given by the original mt versions, that represent less of a constraint to the translators, compared to nmt outputs. second, we observe a clear productivity gain in post-editing compared to human translation, even when post-editing translations generated by a smaller model. given that the creativity gap is relatively small across translation variants, the productivity gains may offset the minor differences in creativity, achieving similarly creative translations with significantly less effort and time. finally, we reinforce the potential of fine-tuning techniques for literary mt workflows, demonstrating that even by adopting a small literary-adapted model, it is possible to achieve a good balance between translation quality and efficiency.