‘Can make mistakes’. Prompting C hat GPT to Enhance Literary MT output (2024)

Abstract: Operating at the intersection of generative AI (artificial intelligence), machine transla-tion (MT), and literary translation, this paper examines to what extent prompt-driven post-editing (PE) can enhance the quality of ma-chine-translated literary texts. We assess how different types of instruction influence PE performance, particularly focusing on lit-erary nuances and author-specific styles. Situated within posthumanist translation theory, which often challenges traditional notions of human intervention in translation processes, the study explores the practical implementation of generative AI in multilin-gual workflows. While the findings suggest that prompted PE can improve translation output to some extent, its effectiveness var-ies, especially in literary contexts. This highlights the need for a critical review of prompt engineering approaches and empha-sizes the importance of further research to navigate the complexities of integrating AI into creative translation workflows effective-ly.

## introduction ever since chatgpt was released in november 2022, the world of language automation for translation purposesup to that point dominated by neural ## materials and methods to address the research question about the effectiveness of prompt templates in improving the quality of mt output, a detailed methodological strategy was developed. this strategy aims to assess systematically how structured prompts influence the performance and accuracy of prompted pe results produced by gpt-4. ## results in the first stage, a detailed qualitative assessment was undertaken to set a standard for translation quality. this encompassed a systematic evaluation of outputs from 7 distinct mt systems. the assessors could award a maximum of 28 points to each text, aligning with the 28 specific items scrutinized during the assessment process. as can be inferred from the results presented in a positive aspect of itemized evaluation is that it provides insight into the average quality of output, but also reveals that there are various textual aspects where improvements can be observed. for example, when analyzing items concerning 'accuracy' (corresponding to 5 items in total in the st), only the clmt (2/5) and deepl (1/5) systems were noted for correctly rendering items pertinent to this criterion. this shows that the qualitative analysis can be said to serve as a guidepost for targeted improvements, particularly in facets of the translations that directly impact textual accuracy. having established a baseline quality for unedited mt output, the study analyzed the impacts of three differentiated editing instructions. under the first condition, chatgpt was tasked with comprehensive pe (full pe) of the mt outputs while considering the source content. analysis of the data shows a general improvement in translation quality: on average, each text now correctly represents 8.29 items, marking an increase compared to the original mt output (1.29 items more than with the raw mt). roughly 30% of source items were more accurately rendered, indicating a modest enhancement in overall quality. these results suggest that pe prompting appears to be reasonably effective, and that further specification of prompts could indeed provide additional improvements. however, this improvement could also be said to present a complex picture. notably, the quality enhancement is not uniform across texts: almost half of the fpe texts show quality levels similar to those of the original mt outputs (deepl, google, gpt-3.5). substantial improvements can be primarily attributed to gains in performance observed in the systran version, which jumped from two to seven correctly resolved items. gemini and gpt-4 also showed some improvement, enhancing its score by three and two additional items. conversely, there was one instance of a decrease in quality: after fpe, a correctly resolved item is lost in the clmt output (score: 10). the detailed breakdown into subcategories reveals even more nuanced results. while the 'accuracy' category demonstrates room for significant improvement, fpe versions outperform the raw mt results slightly in this respect, increasing from three to five correctly interpreted items in total (5/30). what seems noteworthy is that the clmt output slightly regressed in terms of accuracy. in contrast, 'fluency' showed a rather marked improvement after fpe, with the number of instances in which fluency-related problems were satisfactorily resolved more than doubling (from ten to twenty-three correct instances after full pe). despite these gains, a trade-off is observed in the 'style' category, which experienced a serious decline post-fpe. whereas the raw mt outputs had initially provided satisfactory solutions for style-related items 36 times, this number suddenly dropped to 31 following comprehensive pe. this trend is hardly unexpected, as research on pe guidelines shows that style improvement is rarely explicitly addressed (see @xcite . the shifts in output quality for our three subcategories underscore the inherent challenges and compromises involved in balancing the intricate elements of accuracy, fluency, and style in the process of enhancing mt texts with the aid of genai technology. the second condition of the experiment focused specifically on stylistic aspects of the st, as chatgpt was tasked with full pe of the outputs while remaining mindful of the literary nuances of the st. this directive was expected to enhance tt quality by making the instructions more explicit, and, more importantly, tailored to the literary purpose of the text. in theory at least, this instruction would enhance the system's in-context learning performance @xcite . however, it is not superfluous to add that no specific guidelines were provided regarding the unique literary attributes that were to be preserved or highlighted, thus, leaving chatgpt to interpret these stylistic nuances autonomously. as can be inferred from still, the overarching trend points to a diminution in quality. this decrease becomes even more pronounced when analyzing the remaining versions. an already limited success in conveying accuracy seen in previous conditions further regresses, with almost all items (2 in total) being misrepresented under condition 2. the odd exceptions are observed in the google version and the gemini version: each version managed to capture one single item for accuracy. moreover, contrary to expectations, the 'style' category, the primary focus of this condition, witnesses a substantial downturn: initially, the raw outputs collectively presented 36 correct solutions, yet, under condition 2, this tally decreases to 27. it can be safely assumed that this reduction stems from gpt's unique interpretation of 'literariness', which seems to stray from the traditional (highly intricate) balance between form and content found in literary style, instead veering towards a more embellished, often overwrought rendition. this interpretation tends to produce what can be considered a 'pastiche' version of the st rather than a faithful literary rendition. gpt, rather than representing the literary style specific to the st, applies lexical choices it presumably understands as 'literary'. in doing so, it shows its inability to source beyond the overwhelming amount of stylistically unremarkable (clichéd, hackneyed) non-literary understandings of literature it can find. nevertheless, a rather interesting observation emerges in the category 'fluency', where the literary tone of voice appears to foster fluency: this is evidenced by an increase from ten to eighteen correct translation solutions. this suggests that while attempts to infuse a literary style clearly compromises accuracy and literary authenticity, the unintentional result is an improvement in the overall fluency displayed in the texts. in an attempt to refine the approach to stylistic fidelity, the third condition of the experiment was construed around an even more structured and detailed prompt. the prompt was divided into three stages, providing a scaffolded approach. the task involved: 1) collecting online information about donald barthelme's unique literary style; 2) analyzing the st to identify barthelme's stylistic elements in "i wrote a letter…"; and 3) utilizing this understanding during the pe process to maintain the original literary qualities (typical of barthelme's writing) in the subsequent versions. this third instruction was aimed at guiding chatgpt towards a deeper engagement with the literary characteristics of the st, moving beyond a highly superficial interpretation of 'literariness'. surprisingly, the results presented in the breakdown of results further underscores the challenges introduced by the author-tailored instruction. unlike the previous conditions, where some degree of improvement was noted in at least one category (accuracy under fpe, fluency under flpe), precise and clear instruction with a focus on barthelme's literary style had a detrimental effect on performance in all categories. again, this decline can be attributed to several factors. firstly, a noticeable increase in omissions can be found in the target output, with chatgpt tending to exclude significant portions of the text (mostly toward the end of the text), resulting in a blatant loss of content, as well as a distortion and simplification of barthelme's short story. similar issues are observed in other studies focusing on llms, particularly in chain-of-thought settings (e.g. @xcite ). llm's are prone to not only omitting key elements but also inventing non-existent off-target content or twisting the existing information in incomprehensible ways. this phenomenon, referred to as 'edit hallucinations', compounds the distortion and simplification observed in barthelme's short story. furthermore, our tailored approach seemed to encourage an over-the-top form of pastichea kind of pastiche of the pastichetransitioning from a general literary imitation to an unsatisfactory mimicry of barthelme's literary style. particularly, the nuanced balance between the mundane and the absurd that is characteristic of barthelme's story is completely lost on gpt. in the latest iterations, this stark imbalance manifested in versions that simply veer towards the grotesque, stripping away the subtlety and nuanced banality, the hallmarks of barthelme's narrative style. this misinterpretation, particularly evident in hyperbolic renditions of the texts, highlights the difficulties in capturing the intricate interplay of tones and themes inherent to barthelme's oeuvre using llms. ## discussion the findings from this study clearly reflect the challenges of prompt engineering as a means to optimize mt output through pe instructions. reflecting on the improvement brought about by fpe, it becomes evident that while prompted pe can indeed enhance translation outputa finding that is consistent with observations made in raunak 2023 et al.its effectiveness seems limited and is markedly inconsistent. the experiment's venture into more tailored instructions, under the condition 'full literary pe' , brought to light the complexities of encoding stylistic nuances in language models. the decline observed in output quality under this condition prompts a critical reassessment of approaches to 'literariness' in transformer architectures. the third condition's attempt to incorporate author-specific nuances into pe widened the divide separating algorithmic interpretation from literary sensibility even further. the nuanced implications of these findings beckon a reevaluation of our expectations from prompt engineering and language models, particularly within the context of mt output optimization. both within and beyond the academic realm, there is significant emphasis on the importance of prompt engineering and the refinement of prompts and prompt templates. while it is acknowledged that llms display unpredictable responses to similar prompts, there seems to be a need for precise and refined prompts and templates (see @xcite ). however, it appears that refinement, particularly in the form of instructions tailored to a literary context, currently leads to weaker output. this issue is primarily due to the tendency to beautify texts, a tendency associated within translation theoy with @xcite . the question now is whether this tendency can be suppressed through radically different or more refined instruction or specific settings (e.g. system instructions as provided through custom gpt's). ## conclusion the project detailed in this paper is situated at the intersection of on the one hand posthumanist translation theory, which in itself reconsiders notions of human intervention in translation, and the practical application of genai in multilingual workflows on the other. with this project, we have sought to explore the potential of prompted pe, a form of semi-automatic pe, as a substitute for human pe or an intermediary step to refine mt outputs and add an additional step to translation automation in workflows. our exploratory study scrutinized seven mt versions of a literary short story through the pe process, revealing that prompted pe, under specific conditions, yields marginal improvements. it was striking that more specific instructions, targeted toward literary translations, led to weaker performances. this outcome was quite intriguing as the view is widely held that prompt specificity is a driver of performance in ai-driven tasks, such as language translation @xcite . still, it is crucial to acknowledge the preliminary nature of these findings. as with much research in the nascent field of generative ai and translation, our study faces limitations that underscore a great need for further exploration. from a fundamental point of view, different takes on 'literariness' and 'style' can be applied to measure the creative prowess of genai (see boase-beier 2020). for a more comprehensive understanding of the 'literariness' of pe outputs, future research should also include a greater variety of literary genres and styles. additionally, there is a great need to expand research on the effects of prompted pe across a broader spectrum of languages (as in @xcite . finally, adverse effects of prompted pe might be mitigated when using different prompting strategies than the ones used in this study. to counteract observed 'pastiche effect', examplebased prompts, laying down clear criteria for the tone and the expected levels of faithfulness to the original, can be explored. another avenue for future research in the domain of literary translation is investigating the effects of customizing gpt's using domain-specific language resources such as translation memories (see @xcite . recent advancements in language automation have illuminated the potential of ai integration into linguistic workflows, not in the least in creative text domains. however, amidst the hype surrounding genai, the intrinsic complexity of creative tasks (e.g. literary translation) often gets overlooked or oversimplified in research in computational linguistics and translation studies. despite the critical acclaim for ai's creativity and the benefits of human-language prompting, our research has shown that it is and will always remain crucial to ensure a tight alignment between creativity and fidelity in the context of creative translation. appendix 1. source text items and corresponding analytical criteria item criterion 1 , asked him style -colloquialism 2 towaway zones accuracy 3 and i didn't like it fluency 4 cost me …, plus style -colloquialism 5 tiny little cars style -colloquialism 6 you ever notice …? you ever seen…? no you haven't. style -colloquialism [fluency] 7 , and to keep some mental health warm …, style -colloquialism [fluency] 8 a bucket of ribs accuracy 9 which i would gladly carry up there… fluency -idiom 10 i cabled him style -absurdism [accuracy] 11 and, by the way, what was the apartment situation up there? style -colloquialism [fluency] 12 it was bad, fluency -idiom 13 he replied by platitudinum plate style -absurdism [accuracy] 14 but what could he do? fluency -idiom 15 root cellar accuracy 16 'cause of me being a friend of the moon. style -colloquialism 17 pretty nice place fluency 18 the space shuttle hurry-up fund accuracy 19 drumming fiercely on a hollow log with a longitudinal slit tuned to moon frequencies style -absurdism [accuracy] 20 employment, medical coverage, retirement benefits, tax shelterage, convenience cards, and christmas club accounts accuracy 21 that's a roger, fluency -idiom 22 he moonbeamed back style-absurdism [accuracy] 23 by means of curly little algol circuits i had knitted myself on my apple computer style(absurdism), [accuracy] 24 that ticktacktoe was about as far as they'd got in that direction style -absurdism [accuracy] 25 via flights of angels with special instructions styleabsurdism [accuracy] 26 it looked to me like he had things pretty well in hand up there fluency 27 part-time if need be? stylecolloquialism 28 a shower of used-car asteroids with blueand-green bumper stickers styleabsurdism [accuracy]