Towards Demonstration-Aware Large Language Models for Machine Translation (2024)

Abstract: Tuning-based large language models for machine translation (aka large translation model, LTM) have demonstrated significant performance in the field of machine translation. Despite their success, these models often face difficulties in leveraging demonstrations to further improve their performance. To tackle this challenge, we introduce a novel approach that integrates demonstration-aware training and inference strategies within the framework of tuning-based LTMs, hereby referred to as demonstration-aware LTMs. During training, we enrich the model’s learning process by incorporating both sentence- and document-level demonstrations derived from its original training dataset. During inference, the model synergizes its own contextual translations with retrieved high-quality demonstrations, leading to more precise and contextually appropriate outputs. Empirical results reveal that our demonstration-aware LTM not only mitigates the negative impacts traditionally associated with demonstrations but also secures substantial improvements in translation accuracy, particularly in domain-specific and document-level translation tasks. Source code and scripts are freely available at https://github.com/ChenLi0620/Demo-Aware-LLM-MT.

## introduction large language models (llms) @xcite have demonstrated significant advancements in the field of machine translation (mt). the current state-of-the-art involves llms for machine translation (aka large translation model, ltm) that leverage llms, categorized into prompting-based and tuning-based approaches. prompting-based ltms @xcite @xcite @xcite initiate translations by appending examples to a base llm, offering * corresponding author a training-free method that allows for swift customization to specific translation needs, such as domain-specific translations. however, this approach is constrained by static parameters, limiting its adaptability and translation depth. in contrast, tuning-based ltms @xcite @xcite @xcite enhance translation capabilities through supervised fine-tuning on mt-specific datasets, resulting in improved performance due to parameter updates. yet, this method tends to overlook the model's ability to leverage incontext learning, particularly in scenarios beyond zero-shot sentence-level translation. acknowledging the advantages and limitations of both ltm types, we propose the demonstrationaware ltm, which is developed by lightweight lora @xcite fine-tuning on the samples with demonstrations. to cover possible translation scenarios during training, we propose a mixture of demonstration types, which determines the demonstration type of a training sample by randomly choosing either the sentence pairs from the training set as sentence-level demonstrations or the contiguous contextual text as document-level demonstrations. we also design different strategies during the inference phase. for the domain translation, we use only sentence-level demonstrations and retrieve high-quality samples from the training data as demonstrations. when switching to the document-level translation, to ensure the specific information (e.g., style), we propose using hybrid demonstrations that concatenate the documentlevel and retrieval-based demonstrations. our experimental findings highlight that vanilla tuning-based ltms may not effectively learn to utilize demonstrations. conversely, our demonstration-aware ltm consistently outperforms a strong tuning-based ltm in demonstrationutilizing scenarios (e.g., domain-specific translation and document-level translation), underscoring the synergistic potential of combining prompting and tuning-based approaches. additionally, our analysis corroborates the utility of demonstrations in facilitating the translation of rare words. however, it also brings to light a critical challenge: mitigating the detrimental effects of noisy demonstrations, which can compromise the translation quality of medium-or high-frequency words. the main contributions are as follows: ## related work prompting-based ltm prompting-based ltm has demonstrated substantial significant icl capabilities, where carefully crafted prompts can yield remarkable outcomes in mt tasks @xcite . optimal selection of context examples is pivotal, as it can activate the intrinsic mechanisms of prompting-based ltms to produce the anticipated outputs, as evidenced by @xcite . consequently, there has been considerable research focused on optimizing prompting strategies for ltms in mt, encompassing the development and evaluation of prompt templates @xcite , the curation of demonstration sets @xcite , and the in-depth exploration of the models' capacity to learn from such demonstrations @xcite . further investigations have explored the method of using a pre-trained neural retriever to retrieve knowledge from databases and integrate external knowledge sources into ltms to elevate translation accuracy @xcite @xcite . despite the demonstrated efficiency in domain adaptation and ease of utilization, prompting-based ltms have not yet achieved the general translation efficacy of their tuning-based counterparts, as highlighted by comparative analyses @xcite . this limitation somewhat constrains their broader applicability in mt contexts. tuning-based ltm tuning-based ltms finetune foundational llm models on machine translation datasets, adjusting parameters to better perform on specific translation tasks. with the advent of open-source large models, research into tuning-based ltm @xcite increasingly gains attention. to rapidly adapt models to the translation domain, researchers have constructed numerous instruction-tuning datasets for mt @xcite @xcite . to enhance the translation capabilities of ltm, @xcite incorporate additional training data to improve tuning-based ltm performance. given that most current large models are primarily based on english as the basic language, there has been research aimed at optimizing translations from english to other languages @xcite . additionally, novel tuning methods have been designed. @xcite focus on models primarily on tuning from extensive multilingual non-parallel corpora. their method involves two stages: initial fine-tuning on monolingual data followed by fine-tuning on a small set of high-quality parallel data. @xcite propose a tuning-based ltm for low-resource languages and introduce contrastive alignment instructions to enhance cross-lingual supervision. @xcite try adapting ltm as automatic posteditors for document-level translation, showing that fine-tuning for automatic post-editors significantly improves translation metrics. @xcite propose a two-stage generation ltm through selfreflection, where the ltm first generates the initial translation, then conducts self-assessments, and refines the translation in the next stage based on the evaluation results. while these advancements have significantly elevated translation capabilities, they often overlook the potential of integrating prompting strategies during the inference stage. this paper endeavors to amalgamate the strengths of prompting-based ltms with those of tuning-based ltms, aiming to endow the result-ing model with superior demonstration learning capabilities while ensuring exemplary translation performance. ## conclusion to enhance the demonstration learning capability of tuning-based ltm, this paper introduces the demonstration-aware ltm. we crafted sentencelevel and document-level demonstrations to better enable the ltm to perceive and utilize demonstrations. following these developments, we also employed retrieval and self-generation techniques to create high-quality demonstrations that fully leverage the ltm's capacity for model inference. our experimental results in domain translation and document-level translation demonstrate that our proposed method enables tuning-based ltm to learn from demonstrations significantly and stably.