Document-Level Machine Translation with Large Language Models (2023)

Abstract: Large language models (LLMs) such as ChatGPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document-level machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs’ ability on discourse modeling. The study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of ChatGPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evaluating on a number of benchmarks, we surprisingly find that LLMs have demonstrated superior performance and show potential to become a new paradigm for document-level translation: 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of human evaluation; 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evaluation of LLMs (We release our data and annotations at https://github.com/longyuewangdcu/Document-MT-LLM).

## introduction in the past several years, machine translation (mt) has seen significant advancements with the introduction of pre-trained models such as bert @xcite , gpt-2 @xcite , and figure 1 : an example of translating a documentlevel text from english to chinese using gpt-4 (date: 2023.03.17). we highlight the discourse phenomena using figures and lines, which are invisible to gpt-4. t5 @xcite . these models have demonstrated impressive performance on mt @xcite @xcite . however, most of the existing work has focused on sentencelevel translation, which can result in translations that lack coherence and context. recent years have seen a growing interest in document-level translation, which is a crucial task that involves translating entire documents @xcite @xcite while modelling specific discourse phenomena @xcite @xcite @xcite . the most popular large language model (llm) -chat-gpt 3 shows the ability of maintaining long-term coherence and consistency in a conversation by conditioning on previous conversational turns. ad-ditionally, the model is trained on a large dialogue dataset, which allows it to learn the patterns and conventions of human communication, further improving its ability to document-level understanding and generation (as shown in figure 1 ). in this paper, we are particularly interested in how llms such as chatgpt perform for modeling document-level text, encompassing discourse phenomena such as entity consistency, referential expressions, and coherence. taking document-level mt as a testbed, we conduct an empirical study from three in-depth perspectives: while chatgpt has demonstrated remarkable abilities in long-text nlp tasks, we are specifically interested in how it performs on documentlevel translation. consequently, we conduct a systematic comparison of commercial mt products and advanced document-level approaches, utilizing both automatic and human evaluations to assess their discourse awareness. we conduct experiments on a variety of document-level translation benchmarks, covering three language pairs (i.e. chinese⇒english, english⇒german and english⇒russian) and seven domains (i.e. news, social, fiction, q&a, ted, europarl, and subtitle). we adopt a comprehensive set of evaluation methods to measure the performance of the models on documentlevel translation, including general-purpose metrics, discourse-specific metrics, and human evaluation. the main contributions are: ## comparison of translation models in this section, we compare various systems and methods for the document-level translation task. in the following experiments, we use the p3 prompt for chatgpt and the same document-level window size for mt models as the default setting. ## analysis of large language models we analyze the ability of llms to capture discourse knowledge from two perspectives: (1) probing the discourse knowledge encoded in llms, and (2) examining the impact of different training techniques on discourse modeling. ## conclusion and future work we provide a comprehensive evaluation of llms (such as gpt-3.5 and gpt-4) for document-level machine translation. our evaluation covers three main aspects: (1) the effects of discourse-aware prompts, (2) comparison of advanced translation models, and (3) analysis of discourse modelling abilities. with the release of the gpt-4 model, the discourse-aware performance has been significantly improved, making it a promising paradigm for document-level translation. despite its prowess in generative tasks, it struggles with discerning subtle distinctions for ranking. in our future work, we plan to explore more document-level evaluation method @xcite @xcite , latest long-text benchmarks @xcite @xcite and other mt scenarios @xcite @xcite . furthermore, we intend to delve into a more detailed analysis and comparison in future work. for instance, we will employ appropriate significant test methods to account for multiple comparisons (e.g. nonparametric kruskal-wallis test, bonferroni correction) and conduct a power analysis @xcite @xcite . about annotation consistency, we further apply the krippendorff's alpha coefficient and check the confidence interval @xcite .