LAND e RMT : Dectecting and Routing Language-Aware Neurons for Selectively Finetuning LLM s to Machine Translation (2024)

Abstract: Recent advancements in large language models (LLMs) have shown promising results in multilingual translation even with limited bilingual supervision. The major challenges are catastrophic forgetting and parameter interference for finetuning LLMs when provided parallel training data. To address these challenges, we propose LANDeRMT, a Language-Aware Neuron Detecting and Routing framework that selectively finetunes LLMs to Machine Translation with diverse translation training data. In LANDeRMT, we evaluate the awareness of neurons to MT tasks and categorize them into language-general and language-specific neurons. This categorization enables selective parameter updates during finetuning, mitigating parameter interference and catastrophic forgetting issues. For the detected neurons, we further propose a conditional awareness-based routing mechanism to dynamically adjust language-general and language-specific capacity within LLMs, guided by translation signals. Experimental results demonstrate that the proposed LANDeRMT is very effective in learning translation knowledge, significantly improving translation quality over various strong baselines for multiple language pairs.

## introduction conventional neural machine translation (nmt) usually requires a huge amount of parallel training data @xcite @xcite . in contrast, multilingual llms, e.g., @xcite , llama2 @xcite , in spite of being trained with mainly monolingual data, require only a few examples to demonstrate remarkable prowess in multilingual translation via in-context learning (icl) @xcite . however, such llm-based mt exhibits a major drawback that the quality of yielded translations is highly sensitive to the provided examples in icl @xcite and outputs might suffer from overgeneration @xcite . to address these issues, existing studies attempt to use various finetuning methods, such as adapterbased method @xcite , instructionbased tuning method @xcite . however, these approaches primarily focus on balancing between the original llms and new finetuning translation data they use only incremental data to acquire new knowledge without considering catastrophic forgetting of knowledge originally captured by llms @xcite @xcite . many studies have shown that catastrophic forgetting indeed exists across languages as llms are fine-tuned on one language pair and then used to translate another language on which llms are not fine-tuned @xcite . additionally, as llms are usually generally developed for multiple tasks (i.e., sharing parameters across different tasks), finetuning llms for mt task may cause parameter interference for other tasks @xcite . in section 4.3, we find that full-parameter finetuning of llms cannot always improve translation quality on all language pairs. therefore, is it possible to design a new finetuning method for llms, which can mitigate the issues of catastrophic forgetting and parameter interference during the finetuning process of llms to multilingual machine translation? in multilingual nmt, previous efforts evaluate the importance of model neurons to each language pair and only tune language-specific neurons for the current language pair during training @xcite . recent studies on llm unveils that many neurons in the feed-forward networks (ffn) are only activated for specific tasks and become "dead" for irrelevant tasks @xcite . inspired by these studies, we propose lan-dermt, a language-aware neuron detecting and routing framework for selectively finetuning llms to mt, which aims to mitigate the issues of catastrophic forgetting and parameter interference. first, we evaluate the mt awareness of each neuron in ffns. for neurons that are related to multilingual mt tasks, we further evaluate the relevance of each neuron to each language pair. according to their mt/language "awareness/relevance", we divide neurons into the unactivated neurons, language-general neurons and language-specific neurons. after that, we finetune llms on multilingual parallel training data. during finetuning, only the parameters of language-general neurons and language-specific neurons for the current language pair are tuned. this selective finetuning process can alleviate the parameter interference issue. as language-general and language-specific capacity matters for mt @xcite , we propose a conditional awareness routing mechanism to dynamically schedule language-general and languagespecific capacity across sub-layers in llms under the guidance of translation signals. in doing so, we can alleviate the catastrophic forgetting issue and facilitate llms to be adapted to mt. the main contributions of this work are summarized as follows: ## related work llms, with a few examples provided via in-context learning, have demonstrated impressive capabilities in machine translation without requiring explicit supervision from parallel training data @xcite @xcite . however, llms with icl for mt suffer from the sensitiveness to the provided examples @xcite and yielded translations might be overgenerated @xcite . another line of research on llms, known as domain-adaptive pretraining, focuses on finetuning llms to downstream tasks @xcite . although these approaches have demonstrated efficacy in adapting various llms and result in enhanced performance on downstream tasks @xcite @xcite , they rarely apply to multilingual generation tasks, e.g., multilingual mt. in order to efficiently adapt llms to mt, recent years have witnessed efforts on finetuning llms for mt @xcite . @xcite show that adapter-based finetuning with lora @xcite matches the performance of traditional finetuning while reducing the number of training parameters by a factor of 50. @xcite investigate the multilingual generalization when finetuning llms. however, they do not explicitly overcome catastrophic forgetting and parameter interference issues. to address these issues, our work starts with analyzing the neurons within the model, and finetunes llms by distinguishing neurons. research interests in understanding the inner workings of llms and nmt models have been growing recently @xcite @xcite . @xcite focus on neurons inside ffns and find that the network is sparse and represents many discrete features in llms. they find many of the alive neurons are reserved for discrete features and act as token and n-gram detectors for different languages. in addition, previous nmt efforts evaluate the importance of nmt neurons in each language pair and only finetune language-specific neurons for the current language pair participate in training for conventional multilingual nmt @xcite . partially motivated by these studies, we propose a language-aware neuron detecting and routing framework for selectively finetuning llms to mt. in our method, we use awarenessbased evaluation of neurons in llms and divide the neurons into language-general and languagespecific neurons. we only update the parameters of language-general neurons and the corresponding language-specific neurons for the current language pair during training to overcome catastrophic forgetting and parameter interference to enhance the multilingual translation ability of llms. ## methodology the proposed landermt is illustrated in figure 1 . we first propose a method to analyse which ffn layers of llms have strong relevance to sourcetarget language pair. this allows us to exclusively concentrate on layers that are related to the mt task, hence reducing the distraction from unrelated parameters. then, we employ taylor expansion (te) @xcite to evaluate the strength of awareness (relevance) of neurons at those layers to the given language pairs of the mt task. finally, we only route and finetune the detected languageaware neurons for the mt task. this can ensure that we only need to update a small number of relevant parameters of llms for mt. ## experiments we conducted extensive experiments with involving multiple models across various translation directions to evaluate the proposed framework against a set of strong baselines. ## conclusion in this paper, we have presented a novel approach that not only improves translation quality but also mitigates the risk of forgetting previous knowledge while adapting to new data. we propose a te to evaluate neuron awareness scores for mt tasks and categorize them into language-general neurons and language-specific neurons. the proposed routing mechanism ensures optimal allocation of resources across language-specific and languagegeneral capacities, further enhancing the adaptability of llms. our experimental results, conducted across ten language pairs, validate the effectiveness of our model, showcasing superior performance compared to existing baselines.