DUTNLP System for the WMT 2023 Discourse-Level Literary Translation (2023)

Abstract: This paper describes the submission of DUTNLP Lab submission to WMT23 Discourse-Level Literary Translation in the Chinese to English translation direction under unconstrained conditions. Our primary system aims to leverage a large language model with various prompt strategies, which can fully investigate the potential capabilities of large language models for discourse-level neural machine translation. Moreover, we test a widely used discourse-level machine translation model, G-transformer, with different training strategies. In our experimental results, the method with large language models achieves a BLEU score of 28.16, while the fine-tuned method scores 25.26. These findings indicate that selecting appropriate prompt strategies based on large language models can significantly improve translation performance compared to traditional model training methods.

## introduction the dutnlp lab is actively participating in wmt23 discourse-level literary translation, focusing on chinese to english translation direction. as observed, prompting large language models (llms) has led to outstanding performance across a range of natural language processing (nlp) tasks @xcite @xcite . so our research involves experimenting with various prompts and in-context learning strategies, utilizing large language models. additionally, we conduct experiments to explore the impact of sentence length and data preprocessing methods on translation results. our research is primarily anchored in the gpt-3.5-turbo model @xcite , renowned for its outstanding language generation capabilities * corresponding authors spanning various domains, from writing to conversations. this model excels at producing natural and fluent text with simple prompts, making it accessible even to individuals without extensive technical knowledge. intriguingly, for crafting effective prompts to stimulate the machine translation capability of the large model, we take inspiration from gpt-3.5-turbo. we actively interact with it to derive prompts that can boost translation performance, resulting in the identification of three candidate translation prompt templates. our evaluation of these prompts in the discourse-level translation task indicates their overall effectiveness, with minor performance variations. recognizing the substantial impact of data quality on translation performance, we employ cleaner development corpora for our main experiments. when utilizing large pre-trained models, we conduct a data filtering process through off-the-shelf tools and manual rule-based approaches. further details will be seen in session 2. given the inherent randomness and flexibility in translations generated by large models, aligning the output with the source text can be challenging. to tackle this challenge, we develop scripts to identify segments with alignment errors and subsequently apply manual corrections for rectification. to sum up, our contributions can be outlined as follows: this paper is structured as follows: section 2 describes the data pre-processing strategies, followed by the details of our method in section 3. section 4 presents the experimental results and analysis, and we draw conclusions in section 5. ## data processing contrary to the conventional fine-tuning approach on large language models, our method utilizes a large pre-trained language model combined with prompts. in other words, our primary experiment do not require further model training. therefore, we conduct experiments using only a small portion of the development dataset. since the data quality significantly impacts our final translation performance, we adopt both traditional data processing methods and manual rules for filtering. the pre-processing strategies are as follows: translation prompt tp1 translate the following sentences "[insert text here]" from [src] to [tgt]. tp2 these sentences "[insert text here]" are in [src] and can be translated to [tgt] as follows: tp3 please provide translations of these sentences "[insert text here]" into [tgt]. chinese. chinese text is separately segmented by jieba tool. ## method to unlock the full potential of large language models, we introduce an innovative approach by seeking guidance from gpt-3.5-turbo for the creation of effective machine translation prompts @xcite . specifically, we pose the following query: 'provide ten concise prompts or templates that can prompt translation.' the obtained results are shown in figure 1 . upon observation, we note that the generated prompts are reasonable and similar. consequently, we consolidate them into three sets of candidate templates, as illustrated in in previous studies concerning discourse-level machine translation, it is evident that factors such as varying discourse lengths @xcite and different segmentation granularities @xcite can significantly impact translation performance. consequently, we design a series of comparative experiments to investigate these aspects. specifically, we segment the document texts into sizes of k and analyze the effects of different text lengths on machine translation performance in our experimental results. during the segmentation of document text, our goal is to achieve an equitable distribution of text segments and prevent a situation where only a few isolated sentences remain at the end of a document. to address this, we devise a text segmentation algorithm that preserves the data while also ensuring that the number of text portions between segments is as uniformly distributed as possible. the aim is to minimize variance in sentence counts, as illustrated below. the main strategy is as follows: for a document containing n lines of text, it undergoes slicing based on a specified size of m lines, where the quotient is denoted as p and the remainder as q. if there is a remainder (q ̸ = 0), it indicates the need to slice the text into n/p + 1 segments. this results in a new quotient, k, and a new remainder, t. consequently, the last t segments are allocated a line count of k + 1, while the rest of the segments maintain a line count of k. in traditional machine translation experiments, it is well-recognized that varying segmentation granularities can significantly influence translation quality, particularly in languages like chinese where clear word boundaries are often absent @xcite . therefore, we conduct additional experiments to assess the impact of segmentation granularity on translation performance. our experiments involve three different segmentation granularities for model input in both chinese and english datasets: unsegmented, chinese segmented using the 'jieba' tool, and chinese-english segmented using the 'moss' tool. finally, we compare the performance of our system with commonly used document-level machine translation models. detailed findings will be presented in the subsequent section. ## conclusion we have presented our experimental study on gpt-3.5-turbo for machine translation, covering translation prompts and robustness. results, we have noted that the utilization of the large language model with prompts achieves a significant improvement, nearly 3 points higher than the baseline. it even surpasses the currently widely used mbart+fine-tune approach for discourselevel machine translation. we also attempt to enhance translation performance by incorporating incontext information, but this lead to a negative impact. our future work may include investigating the impact of historical context on translation results and iterative refinement of translation. simultaneously, we will focus on the recognition and translation of discourse phenomena for large language models.