Large Language Model for Multi-Domain Translation: Benchmarking and Domain C o T Fine-tuning (2024)

Abstract: Achieving consistent high-quality machine translation (MT) across diverse domains remains a significant challenge, primarily due to the limited and imbalanced parallel training data available in various domains. While large language models (LLMs) have demonstrated impressive general understanding and generation abilities, their potential in multi-domain MT is under-explored. We establish a comprehensive benchmark for multi-domain translation, featuring 25 German⇔English and 22 Chinese⇔English test sets respectively covering 15 domains. Our evaluation of prominent LLMs reveals a discernible performance gap against traditional MT systems, highlighting domain overfitting and catastrophic forgetting issues after fine-tuning on domain-limited corpora. To mitigate this, we propose a domain Chain of Thought (CoT) fine-tuning technique that utilizes the intrinsic multi-domain intelligence of LLMs to improve translation performance. This method inspires the LLM to perceive domain information from the source text, which then serves as a helpful hint to guide the translation process. Despite being trained on a small dataset of four domains, our CoT fine-tune approach achieves notable enhancements in translation accuracy and domain robustness than traditional fine-tuning, as evidenced by an average 1.53 BLEU score increase in over 20 German→English distinct out-of-domain tests.

## introduction significant progress has been made in the field of machine translation (mt) with the integration of deep learning @xcite . nonetheless, mt systems often display uneven translation quality when faced with diverse scenarios in real-world applications @xcite . this inconsistency in translation quality is evident when comparing the competent performance in domains extensively seen during the training process against the inadequate performance in low-resource or unseen domains. this phenomenon highlights a critical multi-domain issue in mt @xcite @xcite , where the translation ability of a model is hindered by the limited domain coverage and imbalanced distribution in parallel training corpora, leading to poor generalization across diverse domains. despite this, acquiring a comprehensive and balanced training dataset that covers all domains is extremely challenging. generative large language models (llms) such as @xcite have shown impressive general understanding and generation abilities across a broad range of tasks @xcite . we believe llms pre-trained on massive amounts of corpora are inherently good at multi-domain understanding and generation, which could potentially address the dependence on parallel data for multi-domain mt. however, the potential of llms in multi-domain mt remains under-explored with most studies focusing on a narrow range of domains @xcite . to comprehensively evaluate the translation capabilities while avoiding the inaccuracies caused by test leakage, we establish the most extensive multi-domain mt benchmark, which includes 25 german⇔english and 22 chinese⇔english test sets covering 15 domains. by benchmarking prominent llms on four language directions, we find that llms, while promising, still exhibit obvious imbalanced performance across different domains (as illustrated in figure 1 ). recent efforts on llms such as prompt strategies @xcite @xcite and fine-tuning @xcite have shown effective performance improvement for downstream tasks. nevertheless, in line with prior research @xcite , our study uncovers that when llms are fine-tuned on domain-specific parallel corpora (in-domain), their ability to generalize to unseen domains (out-of-domain) remains constrained, often resulting in issues such as catastrophic forgetting and overfitting @xcite . while certain methods like retrieval-based approaches @xcite and multi-perspective thinking @xcite offer performance improvement, these techniques come at the cost of reliance on external knowledge resources and additional computational demands during decoding, aggravating the difficulty of practical deployment of llms. in this study, our objective is to leverage the inherent multi-domain understanding and generative capacity of llms to improve multi-domain translation performance and enhance out-of-domain robustness. our methodology is designed to inspire llms to elicit domain-specific insights from the source text, which are then used as a helpful hint prompt to guide the translation process. to accomplish this, we introduce a domain chain-of-thought (cot) fine-tuning technique, which jointly cultivates llm to recognize domain-specific information and output translation with given domainspecific hints during training. despite being trained on only tens of thousands of parallel examples from four domains, our cot fine-tuning method exhibits stronger translation quality and domain generalization than traditional fine-tuning. on the german⇒english translation task, it achieves an average 1.53 bleu score improvement on 21 distinct out-of-domain tests and 0.83 average improve-ment on 4 in-domain tests. notably, this performance gain is magnified as the dataset is expanded to 400,000 examples and the model size is scaled to 70 billion. our approach has outperformed industry systems such as google, gpt-4 and chatgpt by exceeding 1.8 bleu points averaged on the 25domain benchmark, demonstrating its scalability and robustness when subjected to vastly larger data volumes and model capacity. ## benchmarking llms for multi-domain machine translation the currently available multi-domain benchmarks suffer from deficient domain coverage, and the publicly available test sets have potential risks of data leakage in the mt training process, leading to skewed performance evaluations. moreover, as mt systems exhibit advancing performance and generalization capabilities, there lacks a finer-grained and more precise evaluation to keep pace with the rapid advancements in multi-domain mt. to provide a thorough and precise evaluation, we construct the most comprehensive multi-domain mt benchmark for two language pairs, primarily sourced from opus, wmt, tedtalks, in-house tests and prior works @xcite . there are 25 tests for german-english (de⇔en) and 22 for chinese-english (zh⇔en), each spanning 15 distinct domains such as news, global voices, covid, medical, industry, lecture, subtitle, chat, book, law, government, finance, it, religion, science, novel, energy. 1 settings we conducted experiments on bidirectional mt tasks for de⇔en and zh⇔en. to better evaluate translation performance, we adopt two widely-used metrics: sacrebleu @xcite , a n-gram matching-based metric, and comet foot_1 @xcite , a supervised model that based on pre-trained language models. we evaluate seven leading llms, using tailored few-shot inference strategies to suit the unique features of each llm. for llama-2-7b, which isn't trained to follow instructions, we use a 5-shot approach. for models with instruction following ability, such as llama-2-70b, gpt-4 foot_2 and chat-gpt, we employ a 1-shot strategy. for models specifically trained for mt, like alma-7b @xcite , and parrot-7b @xcite , we apply a 0-shot method. the details of prompt templates are shown in appendix b. we also present the commercial general mt google translate as a reference. ## analyzing llm's translation performance with fining-tuning to evaluate the efficacy of fine-tuning llms for translation tasks, we follow prior work @xcite to construct a translation instruction dataset in the stanford alpaca @xcite format for the fine-tuning process. ## conclusion in this paper, we present a comprehensive study of llms for multi-domain mt. our study begins by constructing a multi-domain mt testing benchmark and evaluating the performance of popular llms. our findings reveal that multi-domain mt remains a challenging task on llms. then, we conduct an analysis of previous approaches, noting that many of them have not fully addressed the challenges of multi-domain mt. finally, we propose a domain chain of thought fine-tuning strategy, which involves jointly training both domain hint generation and domain translation tasks. our experimental results demonstrate that our approach significantly outperforms direct fine-tuning in both in-domain and ood scenarios.