Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning (2024)

Abstract: Large language models (LLMs) exhibit outstanding performance in machine translation via in-context learning. In contrast to sentence-level translation, document-level translation (DOCMT) by LLMs based on in-context learning faces two major challenges: firstly, document translations generated by LLMs are often incoherent; secondly, the length of demonstration for in-context learning is usually limited. To address these issues, we propose a Context-Aware Prompting method (CAP), which enables LLMs to generate more accurate, cohesive, and coherent translations via in-context learning. CAP takes into account multi-level attention, selects the most relevant sentences to the current one as context, and then generates a summary from these collected sentences. Subsequently, sentences most similar to the summary are retrieved from the datastore as demonstrations, which effectively guide LLMs in generating cohesive and coherent translations. We conduct extensive experiments across various DOCMT tasks, and the results demonstrate the effectiveness of our approach, particularly in zero pronoun translation (ZPT) and literary translation tasks.

## introduction with the increasing scale of parameters and training corpus, large language models (llms) have gained remarkable abilities to handle a variety of tasks via in-context learning @xcite , which allows language models to perform tasks with a few given demonstrations and humanwritten prompts as context. one particular area where llms have shown outstanding potential is machine translation (mt) @xcite @xcite . however, most existing studies @xcite have focused on sentence-level translation, which makes * equal contribution † corresponding author llm-based document-level machine translation underexplored @xcite . to bridge this gap, we adapt llms to documentlevel machine translation (docmt) in this paper. as a complex task different from sentence-level translation, one major challenge of docmt with llms is that the length of demonstrations for in-context learning is limited. for sentence-level mt, @xcite and @xcite show that using 32 or more randomly sampled bilingual parallel sentence pairs as prompts can effectively enhance the translation abilities of llms. however, for docmt, the length of the text segments to be translated or to be used as demonstrations inherently increase @xcite . consequently, the limited input window of large language models poses a significant obstacle @xcite . unlike sentence-level translation, the constraints of docmt hinder the seamless integration of such demonstrations. moreover, the lengthier nature of document-level inputs exacerbates issues related to inference speed. as the length of the demonstrations increases, the computational resources required for processing also escalate, potentially impeding real-time translation capabilities. previous efforts to docmt with llms, e.g., the method presented in @xcite , involve fine-tuning llms using two strategies: parameter-efficient fine-tuning and full parameter fine-tuning. such a framework may not be able to sufficiently overcome the limitation of the length of demonstration for in-context learning in llms. and it also reveals a significant issue: contextindependent translations, which can result in translations that lack coherence and disambiguation @xcite . to address these issues, we propose a novel method called cap, which stands for context-aware prompting. it consists of three essential steps. firstly, by combining token-and sentence-level attention scores, we obtain sentences with the highest sentence-level attention scores as context for each sentence. this step facilitates llms in generating coherent translations. secondly, to address the limitation of the length of demonstration for in-context learning in llms, we summarize the context and retrieve parallel sentences from the datastore, which are most similar to the generated summary. finally, we use the retrieved example pairs as demonstrations, prompting llms to produce document translations. in summary, the contributions of this work are as follows: translation oriented llms llms have demonstrated remarkable proficiency across a wide range of nlp tasks @xcite @xcite @xcite . recent research has shown that in-context learning can significantly enhance their performance when following general language instructions. specifically, there is a growing body of work exploring the translation capabilities of llms @xcite @xcite . however, it is important to note that these efforts have primarily focused on sentencelevel machine translation and have not delved into docmt @xcite . latest noteworthy studies in docmt are conducted by @xcite and @xcite . @xcite investigate the document-level translation capabilities of gpt-3.5-turbo, making it the most closely related work to ours that explores the docmt ability of llms. @xcite explore fine-tuning llms using two strategies: parameter-efficient fine-tuning and full parameter fine-tuning, to enhance the docmt ability of llms. such a framework may not be able to sufficiently overcome the limitation of the length of demonstration for incontext learning in llms. we observe that the existing translation-oriented llms tend to struggle with effectively translating sentences in the middle and ending of a document. existing methods that explore docmt with llms are rarely considering the issues of (1) the length limitation of demonstrations for in-context learning and (2) maintaining coherence and resolving ambiguities. in this work, we mainly concentrate on mitigating the length limitation of in-context learning for docmt and addressing the issues of incoherence and ambiguity in docmt with llms. ## cap: context-aware prompting in this section, to address the challenges associated with utilizing llms for docmt via in-context learning, we propose a context-aware prompting method. as shown in figure 1 , our approach consists of three steps. firstly, we take into consideration multi-level attention, subsequently opting for the sentences with the highest sentence-level attention scores as the context. secondly, we input the context into an llm to obtain its summary, then retrieve k example pairs from the datastore, which are most similar to the summary. thirdly, we use the retrieved example pairs as demonstrations in the prompt, guiding the model to generate accurate and coherent translations through a few-shot approach. ## experiments in this section, we conducted extensive experiments on various docmt tasks to examine the effectiveness of the proposed method. ## analysis in this section, we conducted experiments to further analyze the effectiveness of our approach. ## conclusion in this paper, we have presented a novel approach that guides the model to generate more accurate, cohesive, and coherent document-level translations via in-context learning. by selecting relevant contexts using our carefully designed method, we can extract topic information from the context summaries. these summaries serve as a guide for selecting the most beneficial examples from the datastore to enhance the in-context learning procedure of llms, hence mitigating the length limitation issue of in-context learning for docmt. experimental results on various benchmarks demonstrate the effectiveness of our approach, particularly in zpt and literary translation tasks, where our method outperforms the baseline significantly.