Benchmarking and Improving Long-Text Translation with Large Language Models (2024)

Abstract: Recent studies have illuminated the promising capabilities of large language models (LLMs) in handling long texts. However, their performance in machine translation (MT) of long documents remains underexplored. This paper aims to shed light on how LLMs navigate this complex task, offering a comprehensive evaluation of their capabilities and limitations in long-text MT. First, we collect and construct an instruction-based benchmark dataset, specifically designed for the finetuning and evaluation of LLMs, encompassing multilingual, multi-domain, and document-level parallel data. Second, we conduct a comprehensive comparison between MT and LLM models concerning document-level translation. Our analysis uncovers that LLMs exhibit shortcomings in long-text domains, and their performance diminishes as document size escalates. By exploiting various extrapolation strategies, we enhance the capacity of LLMs to translate longer texts. We release data, code, and models at https://github.com/longyuewangdcu/Document-MT-LLM.

## introduction recently, significant progress has been made in the field of large language models (llms) such as chatgpt @xcite . these models have demonstrated remarkable capabilities in processing extensive texts, opening up new avenues for a variety of natural language processing (nlp) tasks @xcite . specifically, research on long-text llms primarily focuses on two aspects: 1) maximum capacity: how long text a model can handle @xcite ; 2) efficient utilization: how efficiently that model can model long texts @xcite . even though llms can process long texts, it does not necessarily imply they can effectively model such texts. the effectiveness and necessity of extrapolating llms for various nlp tasks remain ambiguous. in this paper, we predominantly delve into the use of llms for long-text machine translation (mt). diverging from other long-text tasks such as text summarization and completion, which involve either long input or output, long-text mt presents unique challenges. it necessitates not only comprehending extensive input but also preserving context and coherence over extended text sequences @xcite . this amalgamation of requirements intensifies the complexity of the task, positioning long-text mt as a rigorous benchmark for evaluating the potential and prowess of llms. to facilitate this study, we first construct an instruction-based benchmark dataset designed for finetuning and evaluating the document-level translation capabilities llms. this dataset comprises a total of 410k documents with a document length 160∼54k tokens, foot_1 spanning across seven domains (i.e. news, ted, subtitles, parliamentary, social, q&a, novels) and three language pairs (i.e. chinese-english, german-english, russian-english). leveraging this dataset, we conduct a comprehensive comparison between mt and llm models (e.g. deepl, gpt-4, llama). the experiments reveal that 1) all models experience a performance decline in the novels domain due to its exceedingly long nature; 2) the performance of llms drastically deteriorates as the document length increases from 2k to 8k. to further investigate the impact of enlarging document length on translation quality, we examine a variety of extrapolation strategies during finetuning stage. our findings indicate that a mixture of document lengths with multi-task learning can effectively enhance llms' performance. the main contributions are: • to facilitate future research on long-text transla- id instruction instruction-based example 1 give the tgt translation for these src sentences: d 2 please provide the tgt translation for these sentences: d 3 provide the tgt equivalent for the following src sentences: d 4 translate the following src sentences into tgt: d 5 translate the following sentences from src to tgt: d 6 convert the following sentences from src to tgt: d 7 render the following src sentences into tgt: d 8 what is the tgt version of these src sentences? d 9 what do the following sentences mean in tgt? d 10 what are the translations of these src sentences in tgt? d 11 how do these src sentences translate to tgt? d 12 give the tgt translation for this src c: d 13 please provide the tgt translation for the c: d 14 provide the tgt equivalent for the following src c: d 15 translate this c from src to tgt: d 16 translate the following src c into tgt: d 17 convert the following c from src to tgt: d 18 render the following src c into tgt: d each instance consists of 4 parts: system prompt, instruction, input and response. 19 what is the tgt version of this src c? d 20 what do the following c mean in tgt? d 21 what is the translation of this src c in tgt? d 22 how do this src c translate to tgt? d in #12∼22 prompts, we randomly select one word from the candidates @xmath0, "text", "paragraph", "passage"]. ## preliminary background recent years have seen a growing interest in document-level translation by leveraging traditional neural models @xcite @xcite , pre-training @xcite as well as llms @xcite . surprisingly, llms have shown potentials in modeling exceptionally long texts with complex discourse structures for document-level mt @xcite @xcite . more recently, there have been sev-eral length extrapolation methods aimed at efficiently extending the pre-trained context length of llms @xcite . however, they were rarely tested on challenging generation tasks such as long-text translation. our preliminary experiments on document translation using simple extrapolation methods @xcite show a significant performance degradation on wmt2023 english-german testset (llama-2 vs. +extrapolation: 39.1 vs. 35.6 d-bleu), which is consistent with the findings in multi-document qa task @xcite . thus, this work mainly investigate finetuning strategies. pseudo-document given a full length of document-level parallel corpus, previous works usually concatenate n consecutive sentences into a pseudo-document @xcite @xcite until it matches either the evaluation methods the d-bleu (documentlevel bleu) metric @xcite has emerged as the de facto standard for evaluating documentlevel translation models such as mcn @xcite , g-trans @xcite , and mr-doc2doc @xcite . besides, @xcite demonstrates that d-bleu are more attuned to discourse than sentence-level bleu. in this work, we utilize the commonly-used d-bleu as the default instead of sentence-level metrics since all translations are at the document level. as a supplement, we also report results using human evaluation @xcite and the blonde automatic metric @xcite in section 5.4 and the appendix §a.3, respectively. note that, evaluating document-level translation is another challenging issue, which potentially beyond the scope of this work. ## instruction-based benchmark dataset according to @xcite , we construct instruction-based datasets for long-text translation training and testing, spanning various domains and languages. in addition to leveraging existing parallel datasets, we create two challenging testsets. ## benchmark results we provide a comprehensive view of how mt and llm systems handle the complexities of long-text translation across various languages and domains. ## context extrapolation on translation in practice, it is unrealistic to keep increasing the context window of llms in the training process due to resource limits and unsatisfied performance @xcite . thus, it is interesting to investigate the potential of llms in translating documents that are noticeably longer than the predefined context window. in this section, we focus on curating the training data of long texts and explore various strategies for context extrapolation. we employ llama (7b) as testbed, which is pre-trained and fine-tuned with a maximum length of 2,048 token @xcite . following the llama-mt configurations detailed in section 4.1, we pre-process each training subset into pseudo-documents with defined length. then we randomly select 5k instances from each training subset (consider language directions), culminating ∼200k instances for sft phase. about evaluation, we utilize chinese⇒english wmt2022 news and mzprt novels as representative datasets and report d-bleu score. we have examined different methods for utilizing data to improve the effectiveness of modeling text at the document level. we maintain the overall bilingual token count used during instruction finetuning. note that three methods in section 5.1, 5.2, and 5.3 build upon each other.