Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation (2025)

Abstract: Large Language Models (LLMs) have achieved impressive results across numerous NLP tasks, and fine-tuning them for Machine Translation (MT) has improved their performance. However, vanilla fine-tuning often leads to catastrophic forgetting, compromising the broad general abilities of LLMs and introducing potential security risks. These abilities, which are developed using proprietary and unavailable training data, make simple data replay methods ineffective. To overcome this issue, we propose a novel approach called **Ra**tionale **Dis**tillation. RaDis harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then “replayed” to prevent forgetting. These rationales connect prior knowledge with new tasks, acting as self-distillation targets to regulate the training process. By jointly training on reference translations and self-generated rationales, the model can learn new translation skills while preserving its general abilities across other tasks. Additionally, RaDis provides a fresh perspective on using rationales in the CL field and has the potential to serve as a general continual learning method for a variety of tasks.

## introduction large language models (llms) have demonstrated exceptional performance across diverse natural language processing (nlp) tasks but still fall short compared to conventional supervised encoder-decoder models in the realm of machine translation (mt). recent studies have sought to enhance the translation performance of llms through continual instruction-tuning with parallel corpora @xcite . while this approach effectively boosts translation performance, it often introduces catastrophic forgetting @xcite . as illustrated in figure 1 , fine-tuning instruction-tuned llms results in a significant decline in these models' performance on mt-bench @xcite , indicating a loss in general instruction following abilities. such degradation not only limits the broad application of llms but also hinders their translation performance, such as formality steering and contextualization capabilities @xcite . while various replay-based approaches have been proposed to mitigate forgetting @xcite @xcite , these methods do not apply to our target scenario. this limitation arises because the superior performance of llms largely depends on high-quality, proprietary data, to which downstream users have no access for replay. previous studies have explored the use of open-source monolingual data @xcite or instruction-following datasets @xcite @xcite as alternatives. however, their effectiveness remains limited as open-source datasets struggle to match the performance of proprietary ones. to address this problem, this paper explores leveraging the strong generative ability of llms to synthesize their own replay data. however, given the vast task space of llms and the limited data we could use, generating high-quality synthesis data that encapsulates diverse general knowledge remains a non-trivial question. we found that instruction-tuned llms are capable of generating detailed rationales when tasked with translation requests (see section 3.1 and appendix g for more details). further analysis suggests that these rationales encapsulate the internal knowledge llms utilize during translation. building upon these findings, we propose a novel training method named radis (rationale distillation). it prompts the llm to generate rationales for the reference translations in mt training data and then concatenates references and rationales, forming an enriched dataset for subsequent training. by incorporating both the rationales and the references into the training, radis ties llms' internal knowledge with new tasks to learn and introduces a selfdistillation loss on the rationale, thereby mitigating the forgetting issue. comprehensive experiments using three widely adopted llms, llama-2-7b-chat @xcite , mistral-7b-instruct-v0.2 @xcite , and qwen2.5-7b-instruct @xcite validates the effectiveness of radis. as depicted in figure 1 , radis improves translation performance to a level comparable to vanilla fine-tuning while maintaining the model's original proficiency on general ability benchmarks. further analysis reveals that distilling self-generated rationales not only outperforms distilling from external rationales generated by a stronger model but also avoids the conflict between learning new tasks and consolidating the original ability. together, these findings offer additional insights into radis' effectiveness and future study. in summary, this work makes the following contributions: radis overcomes this problem by replaying self-generated rationales, which provides a fresh angle in the field of cl. ## method we begin by presenting a key observation: when tasked with translation requests, instruction-tuned llms can generate detailed rationales that encapsulate the internal general knowledge leveraged during translation (section 3.1). building on this insight, we introduce rationale distillation (radis), which leverages these self-generated rationales as replay data to help the model retain its broad general capabilities (section 3.1). finally, we demonstrate that the radis training objective can be decomposed into a conventional mt loss and a selfdistillation loss on rationale tokens, which helps prevent excessive deviation of model parameters (section 3.3). instruction-tuned llms exhibit a strong ability to follow instructions and engage in conversational interactions, delivering helpful responses across a wide range of tasks. unlike conventional models that simply output the answer, instructiontuned llms are known to be able to generate rationales @xcite . as illustrated in figure 2 , when presented with a translation request, instruction-tuned llms not only generate the translation but also provide an accompanying rationale. ## experiments this section outlines the datasets, baselines, implementation details, and evaluation metrics employed in our experiments and the main results. ## conclusion in this paper, we demonstrate that llms suffer from severe degeneration of general abilities while fine-tuning for translation tasks. to address this issue, we propose a simple yet effective strategy, radis. radis prompts llms to generate rationales for the reference translation and utilizes these rationales to mitigate forgetting in a self-distillation manner. mirroring the human learning process, these rationales connect prior knowledge with new tasks and tie internal concepts together, thereby enhancing knowledge retention. extensive experiments show that radis greatly enhances the translation performance while preserving the models' general ability, which also benefits translation tasks. additionally, radis provides a fresh angle for utilizing rationales in the cl field and can help future research on building llms capable of excelling in specialized tasks without compromising their generality or safety.