MT - PATCHER : Selective and Extendable Knowledge Distillation from Large Language Models for Machine Translation (2024)

Abstract: Large Language Models (LLM) have demonstrated their strong ability in the field of machine translation, yet they suffer from high computational cost and latency. Therefore, transferring translation knowledge from giant LLMs to medium-sized machine translation models is a promising research direction. However, traditional knowledge distillation methods ignore the capability of student and teacher models, therefore repeatedly teaching student models on the knowledge they have learned, and failing to extend to novel contexts and knowledge. In this paper, we propose a framework called MT-Patcher, which transfers knowledge from LLMs to existing MT models in a selective, comprehensive and proactive manner. Considering the current translation ability of student MT models, we only identify and correct their translation errors, instead of distilling the whole translation from the teacher. Leveraging the strong language abilities of LLMs, we instruct LLM teachers to synthesize diverse contexts and anticipate more potential errors for the student. Experiment results on translating both specific language phenomena and general MT benchmarks demonstrate that finetuning the MT model on about 10% examples can achieve comparable results to the traditional knowledge distillation method, and synthesized potential errors and diverse contexts further improve MT performances on unseen contexts and words.

## introduction large language models (llm) have shown their impressive capabilities across almost all natural language tasks @xcite . however, their ability strongly correlates with the model size. in the field of machine translation, competitive results can only be evidenced on larger llms, while medium-sized llms like alpaca @xcite and parrot @xcite still lag behind supervised nmt systems by a large margin @xcite . how to efficiently transfer knowledge from larger llms to existing mt models that are affordable to deploy, is an important research direction. the most common method for knowledge transferring is knowledge distillation (kd) @xcite , where given an unlabeled corpus, a student model is trained to mimic the output of a teacher model on the corpus. although kd is a well-studied technique and has proven effective in many previous works @xcite @xcite , we argue that when transferring knowledge from giant llms to existing mt models, the traditional kd method does not take the capability of the student and teacher model into consideration, therefore leaving much room for improvement in terms of both efficiency and effectiveness. firstly, in contrast to student models in previous works @xcite @xcite that are randomly initialized, recent student mt models @xcite already exhibit a reasonable level of language proficiency, i.e., they can already accurately translate most examples in the unlabeled corpus. this renders the fine-tuning of student models on all teacher outputs both redundant and inefficient. secondly, the efficacy of kd is significantly constrained by the coverage of the monolingual corpus, which impedes their performance when translating words in novel contexts or words unseen in the monolingual corpus. however, modern llms grasp strong translation and language knowledge, as well as the ability to follow human instructions. this enables the development of more efficient and effective strategies for addressing these problems. in this paper, we introduce mt-patcher, a novel framework designed for the knowledge transfer from llms to existing mt models in a selective, comprehensive, and proactive manner. the design philosophy of mt-patcher is inspired by effective teaching strategies observed in real-world scenarios. rather than subjecting students to endless drills, an effective teacher would first assess the student's current abilities, then design practice to reinforce areas of weakness and extend learning to new situations @xcite . leveraging the strong language capabilities of llms, our method seeks to emulate these pedagogical strategies. specifically, we gather instructional data from gpt-4, which demonstrates how to identify and correct errors in student model translations, anticipate additional potential errors that the student models may commit, and synthesize diverse contexts for relevant translation knowledge that aids the student model in rectifying these errors. we subsequently finetune an existing proficient llm on these data to transform it into an mt-patcher model. we conduct experiments on translating specific language phenomena (chemistry materials and chinese idioms) and on general machine translation benchmarks (wmt22 @xmath0, @xmath1. experimental results show that finetuning the student model on only 10% examples selected by mt-patcher is equivalent to finetuning on all examples as in kd, and enlarging the finetuning corpus via the context synthesis and proactive error prediction technique further improves the translation performance. ## methodology in this section, we present mt-patcher , a framework that distills knowledge from llms to existing mt systems more efficiently and effectively. the process of mt-patcher undergoes two stages: figure 1 illustrates how mt-patcher works. ## experiments we evaluate our method on @xmath2. ## discussion we provide further analysis on how mt-patcher works and its applicability to real-world scenarios. all experiments are conducted on the wmt22 chinese-to-english translation datasets, and the student mt model is nllb 3.3b. ## conclusion we introduce mt-patcher, a framework designed to leverage capabilities of llms to en-hance the efficiency and effectiveness of translation knowledge transfer from llms to existing mt models. our approach involves a pipeline that initially generates feedback on translations produced by mt models, followed by the synthesis of potential errors and diverse contexts to systematically rectify these translation errors. through experimentation on both general and narrow domain mt benchmarks, we demonstrate that mt-patcher effectively improves student mt models' performances compared to seqkd baselines, and exhibits successful transferability across different models. in the future, we plan to refine our method from two angles. firstly, previous works @xcite have identified translationese as a significant issue, and training on pseudo data generated by llms can exacerbate this problem. a promising solution could involve retrieving target sentences containing correction words and back-translating them to the source side. secondly, the feedback's reason field contains a wealth of valuable information. we intend to explore more efficient strategies to harness this data.