Prompting C hat GPT for Translation: A Comparative Analysis of Translation Brief and Persona Prompts (2024)

Abstract: Prompt engineering has shown potential for improving translation quality in LLMs. However, the possibility of using translation concepts in prompt design remains largely underexplored. Against this backdrop, the current paper discusses the effectiveness of incorporating the conceptual tool of “translation brief” and the personas of “translator” and “author” into prompt design for translation tasks in ChatGPT. Findings suggest that, although certain elements are constructive in facilitating human-to-human communication for translation tasks, their effectiveness is limited for improving translation quality in ChatGPT. This accentuates the need for explorative research on how translation theorists and practitioners can develop the current set of conceptual tools rooted in the human-to-human communication paradigm for translation purposes in this emerging workflow involving human-machine interaction, and how translation concepts developed in translation studies can inform the training of GPT models for translation tasks.

## introduction translation quality is a pivotal topic in the field of machine translation. the development of large language models (llms) and the popularization of chatgpt since its public launch in november 2022 have attracted scholarly interests in im-proving the quality of translation outputs generated by llms. efforts to improve the quality of these translations have involved both fine-tuning and prompt engineering. despite these efforts, the performance of popular llms in executing translation tasks remains suboptimal, particularly when compared with professional translations used in the language service industry @xcite . therefore, the task of enhancing the performance of llms in conducting translation tasks continues to be an ongoing effort. compared to fine-tuning, prompt engineering provides greater accessibility for ordinary users with translation needs, especially those who operate on the user interfaces of llms such as chat-gpt. most research on prompt engineering for translation purposes draws on concepts such as zero-shot learning rooted in natural language processing (nlp) by feeding sample translations in the context window. in comparison, the possibilities for integrating translation concepts and strategies have received little attention. from the perspective of advancing translation studies, consolidating the synergy between humans and machines in achieving translation goals at a professional level is crucial. as @xcite rightly notes, "translation as an event can no longer be restricted to translating as an act, given that ai and other communicative modalities will increasingly be drawn into and embedded within the workflow." for the development of translation research, since most translation concepts are anchored in human-to-human communication, it becomes essential to evaluate their efficacy in the emerging workflow with human-machine communication involved, thereby strengthening the disciplinary foundation of translation studies in this novel context. for translation practice, enhancing our understanding of prompt engineering for translation could inform the approach we take for translator training in the changing landscape. as highlighted in a recent work trend report by microsoft (2023), 82% of leaders from various sectors stated that their employees will need new competenciessuch as ai delegation via prompts -to prepare for the expansion of ai. in the background, this research investigates the effectiveness of incorporating the notion of "translation brief" and the translator/author dichotomy into prompt design, as an attempt to explore the potential of using conceptual tools rooted in translation studies for improving the quality of llmgenerated translations. in this study, chatgpt is chosen for its popularity among general users and its user-friendly inter-face that accommodates individuals with limited computing expertise. based on two sets of experiments, this research seeks to answer two questions specific to the scope of the current study: 1) compared to a basic translation command, does a prompt containing information included in a typical translation brief help improve the quality of translation outputs? 2) drawing on the persona feature of chatgpt, does assigning the role of "translator" make a difference to the translation quality, with the basic instruction and the role of "author" as reference points? ## literature review in the guidance for prompt design in chatgpt published by openai, 1 six strategies are listed for creating effective prompts. even though there is an improvement in the content of this guideline when compared to its earlier version where only three generic strategies (i.e., show and tell, provide quality data, check your settings; accessed april 2023) were suggested, openai has not yet published any specialized guidance on prompt design for translation purposes in chatgpt. nonetheless, scholarly efforts have been made to address this issue. in the literature, most of the research focuses on prompting gpt models or other llms through apis @xcite @xcite . however, a small number of studies have also explored prompt engineering for translation tasks specifically through the user interface of chatgpt, drawing on different linguistic concepts. within this niche area, 1 https://platform.openai.com/docs/guides/completion/promptdesign two main threads have emerged: one is centered around specific translation problems, and the other features a more holistic approach. starting with those targeting specific translation problems, @xcite is the only one in the literature to date. drawing on the default model (gpt-3.5) of chatgpt, the author utilizes the "in context learning" capability of chatgpt (i.e., remembering what has been mentioned in the chat) to "teach" it how to translate attribute clauses. specifically, a translation strategy commonly adopted by translators to render attributive clauses from japanese into chinese was used by the author to design a set of prompts: "what is the noun modified by the attributive clause in the following sentence?", "place the noun modified by the attributive clause in the subject position of the attributive clause. and then separate [source sentence] into two sentences", and finally "translate the following sentence to chinese: [separated source sentence]". although this prescriptive application of a standalone translation strategy fails to take into consideration the dynamic context of handling attributive clauses, this paper presents a very interesting attempt to bring translation strategies into the horizon of prompt engineering. turning to the literature which investigates translation at a contextual level, key concepts tested in this group include "domain", "task", "part of speech", "discourse", and "pivot language" -all of them are well-established topics in translation studies but they have been used in a rather ambiguous way in these works. for instance, @xcite propose the concept of "task-specific prompts" (i.e., "you are a machine translation system") in their experiment, without concrete instructions on what to expect from a so-called "machine translation system". the rationale behind this design, according to the authors, rests in the assumption that chatgpt has been fine-tuned as a conversation system instead of a machine translation system, and this might have limited the translation ability of chatgpt. nonetheless, the effectiveness of altering a fine-tuned chatbot into a machine translation system with a single prompt line in the user interface remains questionable. additionally, the authors test the efficacy of "domainspecific prompts" (e.g., information about the topic or genre of the st, such as bio-medical or newsstyle) by providing chatgpt with both right and wrong domain information of the st. this design of using wrong domain information, from the perspective of translation studies, requires careful justification. the results, measured via automated machine translation quality evaluation metrics, suggest that providing task and correct domain information can indeed enhance chatgpt's translation performance. another case in point is @xcite . the authors introduce language direction, domain information, and part-of-speech information to their prompt design. similar to the definition of "domain" in @xcite , the authors include information about genre (i.e., news, e-commerce, social, and conversational texts) in their tests. these prompts were run through five different settings to test their efficacy. the results from automatic metrics further validate the use-fulness of domain-related information in prompt engineering for translation tasks. notably, although the outcome of introducing part-of-speech information in prompts was not promising, it suggests an intention to include grammatical segmentations into prompt design, which echoes the problem-oriented approach to enhancing translation quality, as mentioned above in @xcite . an interesting observation made by the authors regarding language direction lies in the disparity between high-resource languages and low-resource languages: domain information appears to enhance machine translation quality for high-resource languages but fails to demonstrate a comparable impact on low-resource languages. to understand the issue related to high versus low resource languages, @xcite propose a strategy called "pivot prompting". this notion, bearing similarities to the concept of relay translation, involves instructing chatgpt to translate the st into a high-resource language prior to translating it into the target language. even though the basic prompts were generated by chatgpt itself without further tweaks, the idea of relay translation turned out to be useful in improving translation quality between distant languages, as the results reported by the authors suggest. regarding the topic of context and discourse in translation, whilst all studies mentioned above focus on prompt design for translation at the level of single sentences or small sentence clusters, @xcite take a step forward to the document level. they put forward the concept of "discourse-aware prompts", introducing discourse as an evaluation criterion for assessing the quality of prompts in chatgpt. to identify the best discourse-aware prompt, the authors evaluate a set of basic prompts generated by chatgpt with two discourse-oriented metrics: one focuses on terminology consistency and another on the accuracy of zero pronoun translation. as can be seen from the design, discourse here is used in its micro sense as document-level coherence. macro discoursal information, such as the function of the st and target audience, is not taken into consideration when designing the prompts. the most relevant research to date, drawing on a contextualized approach inspired by translation concepts, is reported by @xcite . there are two sets of experiments in this research. first, the author adopts two concepts -purpose of the translation and target readers -for prompting chatgpt (gpt-4) to translate, simulating a real-life translation commission for chatgpt. instead of providing information about the purpose and target readers, the author designed a prompt that asks chatgpt to find the information itself: "translate the following japanese [source text] into english. please fulfil the following conditions when translating. purpose of the translation: you need to fill in. target audience: you need to fill in. overall, the current landscape of prompt design in chatgpt features important attempts to enhance its capability in executing translation tasks. however, a critical issue with these endeavors lies in the fact that the concepts being used in the prompts (e.g., "news-style") are too general to be informative, and some of the approaches (e.g., the out-of-context application of prescriptive translation strategies) bear striking resemblances to what happened in the early days in translation studies. the design of prompts shows that these research efforts have touched upon some key conceptual tools for translation, revealing the potential benefit that translation concepts can bring for enhancing llms' performance in generating professional level translations. ## research design building on the effectiveness of introducing contextual and domain-specific information as demonstrated in the literature, this paper investigates prompt design in light of two conceptual tools rooted in translation research: first, "translation brief" as featured in the functionalist approach to translation; second, the "author-translator" dynamic given the persona-matching feature of chat-gpt. ## findings and discussion results from the automatic evaluation metrics and human grading forms provide complementary insights into the quality of the generated tts, indicating the efficacy of each prompt. this section starts with the results of the two automatic metrics, before delving into human evaluation results. ## conclusion this study explores the efficacy of integrating concepts developed in translation studies into prompting chatgpt for translation tasks. by evaluating the outputs generated by chatgpt under four different prompts, it seeks to provide insights into the effectiveness of giving a translation brief to chat-gpt and assigning chatgpt the personas of an author and a translator. findings show that assigning the persona as a translator allowed chat-gpt to achieve the best performance among the four prompts, and that the translation generated by chatgpt using the translation brief prompt received the lowest ranking. this indicates that the classical settings of a translation brief, aiming at human-only workflow, might not work as well as one would expect in a human-machine workflow. however, it would be necessary to revisit the conceptual tools developed in translation studies, considering the development of translation technology and the changing landscape in the industry, so as to further consolidate the relevancy and credibility of translation studies as a discipline. similarly, training gpt models using aligned source and target texts, paired with translation briefs, and exploring other concepts developed in translation studies could be potentially beneficial. there are some limitations of the current research. for instance, when testing the consistency of the prompts based on the translation outputs generated by chatgpt, involving multiple raters, and conducting an inter-rater reliability test would be helpful. additionally, a reader centered human evaluation metrics and interviews with human evaluators would have been a good complement to the information based solely on the textual analysis of evaluators' comments extracted from the grading form. in addition, using document-level quality evaluation metrics might also strengthen the discussion of the results. as mentioned in the introduction, this research only provides partial insights into the two general research questions, based on the data collected in this experiment. to further develop this line of research, different prompts conveying information about translation concepts could be examined, across various genres, assessed with a human evaluation scale closer to the reality of translation reading by a larger number of human evaluators. this approach would generate more data, allowing for replication and statistical testing to enhance relia-bility. additionally, with the development of generative ai, research into other llms for translation purposes could offer valuable comparative insights for both practitioners and researchers in the field. thinking forward, as @xcite rightly note, although gpt models have promising potential in machine translation, their performance remains underexplored compared to commercial machine translation systems. llms are developing rapidly as we write. by extending the scope of translation studies from human-to-human communication to human-machine communication, translation researchers can help to co-shape the future of machine translation and theorize the practice of translation in the new era.