Towards Making the Most of C hat GPT for Machine Translation (2023)

Abstract: ChatGPT shows remarkable capabilities for machine translation (MT). Several prior studies have shown that it achieves comparable results to commercial systems for high-resource languages, but lags behind in complex tasks, e.g, low-resource and distant-language-pairs translation. However, they usually adopt simple prompts which can not fully elicit the capability of ChatGPT. In this report, we aim to further mine ChatGPT’s translation ability by revisiting several aspects: temperature, task information, and domain information, and correspondingly propose two (simple but effective) prompts: Task-Specific Prompts (TSP) and Domain-Specific Prompts (DSP). We show that: 1) The performance of ChatGPT depends largely on temperature, and a lower temperature usually can achieve better performance; 2) Emphasizing the task information further improves ChatGPT’s performance, particularly in complex MT tasks; 3) Introducing domain information can elicit ChatGPT’s generalization ability and improve its performance in the specific domain; 4) ChatGPT tends to generate hallucinations for non-English-centric MT tasks, which can be partially addressed by our proposed prompts but still need to be highlighted for the MT/NLP community. We also explore the effects of advanced in-context learning strategies and find a (negative but interesting) observation: the powerful chain-of-thought prompt leads to word-by-word translation behavior, thus bringing significant translation degradation.

## introduction recently, the emergence of chatgpt foot_0 has brought remarkable influence on natural language processing (nlp) tasks. chatgpt is a large-scale language model developed by openai, based on instruct-gpt @xcite , that has been trained to follow instructions with human feedback. chat-gpt possesses diverse abilities of nlp, including question answering, dialogue generation, code debugging, generation evaluation, and so on @xcite @xcite @xcite . we are particularly interested in how well chatgpt can perform on the machine translation task. previous studies @xcite on translation tasks have found that chatgpt performs competitively with commercial translation products (e.g., google translate and microsoft translator) on high-resource languages, but has limited capabilities for low-resource and distant languages. however, they only adopt simple prompts and basic settings regardless of the significant influence of the prompts' quality @xcite , which may limit chatgpt's performance. in this paper, we aim to further elicit the capability of chatgpt by revisiting the following three aspects and correspondingly propose an optimal temperature setting and two simple but effective prompts: task-specific prompts (tsp) and domain-specific prompts (dsp). temperature. temperature is an important parameter to ensure chatgpt generates varied responses to human queries. basically, decoding with higher temperatures displays greater linguistic variety, while the low one generates grammatically correct and deterministic text @xcite . however, for tasks with a high degree of certainty, such as machine translation, we argue, a diverse generation may impede its translation quality. we evaluate the performance of chatgpt at different temperatures to verify its effect and find the optimal temperature setting for the following experiments. ## evaluation setting we provide a brief introduction of the evaluation setting, which mainly includes the used models, test set, and evaluation metrics. ## zero-shot translation in this section, we explore the performance of chat-gpt from three aspects: temperature, task information, and domain information, and correspondingly propose an optimal temperature setting and two simple and effective prompts to improve chatgpt's performance. ## few-shot machine translation in this section, we simply explore the effects of advanced in-context learning (icl) strategies, specifically, we investigate chatgpt's few-shot icl and chain-of-thought (cot) abilities on mt tasks. ## conclusion in this paper, we investigate how to further mine chatgpt's translation ability from three perspectives, namely temperature, task, and domain information, and correspondingly propose an optimal temperature setting and two simple but effective prompts. we empirically demonstrated that there is a high correlation between temperature and chatgpt's performance, and a lower temperature usually can achieve better performance. experimental results across various language pairs and domains proved the effectiveness of our proposed prompts. we further explore the effectiveness of advanced in-context learning strategies for chat-gpt, we find that the few-shot in-context learning method can consistently improve chatgpt's performance, while conventional chain-of-thought (cot) prompting will degrade its performance because of its word-by-word translation behavior. in future work, besides the aforementioned explorations (ebmt-inspired prompts designing, statistical mt-inspired chain-of-thought designing), we would like to investigate how to further elicit the ability of chatgpt by designing more effective prompts (e.g., design human-like cot to navigate the llms, and better demonstration selection algorithms in few-shot icl) and investigate the ability of chatgpt for more mt settings (e.g., document translation).