Exploring Prompt Engineering with GPT Language Models for Document-Level Machine Translation: Insights and Findings (2023)

Abstract: This paper describes Lan-Bridge Translation systems for the WMT 2023 General Translation shared task. We participate in 2 directions: English to and from Chinese. With the emergence of large-scale models, various industries have undergone significant transformations, particularly in the realm of document-level machine translation. This has introduced a novel research paradigm that we have embraced in our participation in the WMT23 competition. Focusing on advancements in models such as GPT-3.5 and GPT-4, we have undertaken numerous prompt-based experiments. Our objective is to achieve optimal human evaluation results for document-level machine translation, resulting in our submission of the final outcomes in the general track.

## introduction recently, large-scale language models, such as gpt-3.5 and gpt-4 (gpt), have emerged as powerful tools in the field of natural language processing. these models have showcased their impressive capabilities in a wide range of tasks, including text generation, question answering, language translation, and more. language models like gpt-3.5 possess the ability to understand and generate coherent, contextually relevant text, capturing the nuances of language usage and producing high-quality outputs. in particular, machine translation is an area where these language models have shown tremendous promise. traditional machine translation models @xcite used the conventional transformer architecture @xcite since gpt-3.5 has the potential to revolutionize the translation process by leveraging its massive size and language understanding capabilities. with the advent of large models, the machine translation field has faced new challenges, and utilizing large models for machine translation is a novel attempt.by ef-fectively incorporating prompts and context, gpt-3.5 can produce translations that exhibit fluency, accuracy, and adherence to the source text. this study focuses on experimenting and evaluating different prompt engineering techniques to further enhance the translation performance of gpt-3.5. by providing more refined and contextually specific prompts, we aim to observe the model's ability to adjust and refine its translations, resulting in improved translation quality. additionally, we explore the impact of temperature adjustments on the generated translations, allowing us to fine-tune the level of randomness in the output and achieve more deterministic and accurate translations. furthermore, we investigate both sentence-level and document-level approaches, examining the effectiveness of gpt-3.5 in handling translations at different granularity levels. these approaches aim to leverage the model's language understanding capabilities to not only produce accurate sentencelevel translations but also ensure coherence and consistency at the overall document level. by delving into these aspects and evaluating the performance of gpt-3.5 in the context of the wmt competition, we aim to contribute to the broader understanding of the capabilities, strengths, and limitations of state-of-the-art language models in the field of machine translation. the inspiration for this study stems from the outstanding performance exhibited by these largescale language models, especially in addressing real-world challenges such as major wildfires. gpt-3.5-4k and gpt-3.5-16k, with their increased model capacities, have demonstrated remarkable capabilities in generating high-quality text across various domains. motivated by these advancements, our study aims to harness the power of these models and explore their potential in the specific domain of machine translation. by leveraging the robustness and adaptability of gpt-3.5-4k and gpt-3.5-16k, we conduct rigorous experimentation to thoroughly evaluate their translation capabilities. we delve into the nuances of different parameter adjustments, including prompts and temperature, to optimize and enhance the models' performance specifically for translation tasks. by strategically fine-tuning these parameters, we aim to unlock hidden potential and push the boundaries of their translation capabilities. real-world challenges, such as major wildfires, require timely and accurate translation of critical information across languages. the effectiveness of machine translation plays a pivotal role in communicating vital updates and ensuring efficient information dissemination during such situations. by investigating the translation capabilities of gpt-3.5-4k and gpt-3.5-16k, we strive to contribute insights that can improve translation efficiency and aid in overcoming language barriers in emergency situations. with this study, we aim to shed light on the immense potential of large-scale language models, such as gpt-3.5, in addressing real-world challenges through machine translation. by harnessing their capabilities and understanding their performance in various scenarios, we hope to pave the way for more effective and accurate translation systems that can assist in critical situations. ## methods we have designed three prompt schemes: p1: translate this sentence from src to tgt, do not write any explanations p2: translation request -sentence-by-sentence translation. language pair: src to tgt. instructions: 1. each sentence of the document will be provided individually in the "original sentence" section. 2. in the "translation" section, please provide the corresponding translation for each sentence, considering the context and aiming for faithful translation while minimizing unaligned translations. 3. avoid including any explanations in the translation. original sentence: p3: translation request -sentence-by-sentence translation. language pair: src to tgt. instructions: 1. each sentence of the document will be provided individually in the "original sentence" section. 2. in the "translation" section, please provide the corresponding translation for each sentence, considering the context and aiming for faithful translation while minimizing unaligned translations. 3. avoid including any explanations in the translation. 4.please review the translations for verifying that they remain faithful to the original text and provide revised versions accordingly if necessary. if no revisions are needed, provide the translations as they are. in our study, we conducted several experiments to evaluate the performance of gpt-3.5. the following were the approaches we employed: changing the temperature can control the randomness of the generated translations. ## result we conduct experiments to quantify the impact of each component in our system. the evaluation conduct on test set on wmt22 using sacrebleu @xcite and comet @xcite . as shown in based on our previous results, we have chosen gpt-3.5-16k as the final model for our submission. for the wmt23 en-zh/zh-en track, we set the temperature to 0 and utilized p1 as the prompt. adopting a multi-turn dialogue approach, we submitted our final results with the systerm name "lan-bridgemt". figure 1 and figure 2 show the results of our systerm. foot_0 additionally, for other language pairs in the general wmt competition, we opted to submit the results generated by our lanmt @xcite engine. this decision was made to assess the engine's performance and determine its scoring capabilities directly in the online evaluation environment. by taking these approaches, we aim to showcase the effectiveness of gpt-3.5 and demonstrate the performance of our lanmt engine in the respective wmt tracks. these submissions reflect our overarching goal of participating in and contributing to the advancement of machine translation research and development. ## conclusion in this study, we evaluated the translation performance of gpt-3.5 using various experimental approaches. our findings indicate that incorporating multi-turn dialogue prompts improves the translation quality of gpt-3.5, highlighting the importance of context in guiding the model's translations. furthermore, we observed that gpt-3.5-16k, compared to gpt-3.5-4k, demonstrates superior translation capabilities in commit scores, indicating its enhanced ability to understand and fulfill user instructions. however, there are marginal differences in the other two metrics, bleu and chrf. additionally, we found that lower temperature values result in improved translation quality, indicating the usefulness of controlling randomness in the generated translations. however, it is important to note that gpt-3.5 may struggle with understanding ambiguous prompts and lacks the ability to autonomously adjust and optimize its translations without explicit instructions. these findings contribute to our underfigure 2: score for en-zh translation task