Neuron Specialization: Leveraging Intrinsic Task Modularity for Multilingual Machine Translation (2024)

Abstract: Training a unified multilingual model promotes knowledge transfer but inevitably introduces negative interference. Language-specific modeling methods show promise in reducing interference. However, they often rely on heuristics to distribute capacity and struggle to foster cross-lingual transfer via isolated modules. In this paper, we explore intrinsic task modularity within multilingual networks and leverage these observations to circumvent interference under multilingual translation. We show that neurons in the feed-forward layers tend to be activated in a language-specific manner. Meanwhile, these specialized neurons exhibit structural overlaps that reflect language proximity, which progress across layers. Based on these findings, we propose Neuron Specialization, an approach that identifies specialized neurons to modularize feed-forward layers and then continuously updates them through sparse networks. Extensive experiments show that our approach achieves consistent performance gains over strong baselines with additional analyses demonstrating reduced interference and increased knowledge transfer.

## introduction jointly training multilingual data in a unified model with a shared architecture for different languages has been a trend @xcite encouraging knowledge transfer across languages, especially for low-resource languages @xcite . however, such a training paradigm also leads to negative interference due to conflicting optimization demands @xcite . this interference often causes performance degradation for highresource languages @xcite and can be further exacerbated by limited model capacity @xcite . modular-based methods, such as languagespecific modeling @xcite and adapters @xcite , aim to mitigate interference by balancing full parameter sharing with isolated or partially shared modules @xcite . however, they heavily depend on heuristics for allocating task-specific capacity and face challenges in enabling knowledge transfer between modules @xcite . specifically, such methods rely on prior knowledge for managing parameter sharing such as language-family adapters @xcite or directly isolate parameters per language, which impedes transfer @xcite . research in vision and cognitive science has shown that unified multi-task models may spontaneously develop task-specific functional specializations for distinct tasks @xcite , a phenomenon also observed in mixture of experts transformer systems @xcite . these findings suggest that through multi-task training, networks naturally evolve towards specialized modularity to effectively manage diverse tasks, with the ablation of these specialized modules adversely affecting task performance @xcite . despite these insights, exploiting the inherent structural signals for multitask optimization remains largely unexplored. in this work, we explore the intrinsic taskspecific modularity within multi-task networks in multilingual machine translation (mmt), treating each language pair as a separate task. we focus on analyzing the intermediate activations in the feed-forward networks (ffn) where most model parameters reside. to our knowledge, our study is the first to show that neurons activate in a languagespecific way, yet they present structural overlaps that indicate language proximity in general. moreover, this pattern evolves across layers in the model, suggesting that neurons consistently transition from language-specific to language-agnostic. building on these observations, we introduce neuron specialization, a novel method that leverages intrinsic task modularity to reduce interference and enhance knowledge transfer. in general, our approach selectively updates the ffn parameters during back-propagation for different tasks to enhance task specificity. specifically, we first identify task-specific neurons from pre-trained unified translation models, using standard forward-pass validation processes without decoding. we then specifically modularize ffn layers using these specialized neurons and continuously update ffns via sparse networks. extensive experiments on small-(iwslt) and large-scale ec30 @xcite translation datasets show that our method consistently achieves performance gains over strong baselines with various configs. moreover, we conduct indepth analyses to show that our method effectively mitigates interference and enhances knowledge transfer in high and low-resource languages, respectively. our main contributions are summarized as follows: language-specific modeling. recent methods enhance the unified model by utilizing languagespecific (ls) modules such as adapters @xcite , ls layers @xcite and ls hidden states @xcite . although the unified model serves as a common foundation, these methods strictly isolate modules per language. such designs present no knowledge sharing among modules and thus offer fewer benefits to low-resource languages. alternatively, approaches like language family adapters @xcite seek to facilitate sharing among language-specific modules, however, they heavily depend on heuristics such as using priori linguistic knowledge to enable more flexible parameter sharing. additionally, these modular-based methods exhibit parameter inefficiency when handling numerous languages, resulting in increased memory requirements and extended inference times @xcite . similarly, techniques such as parameter differentiation (wang and zhang, 2022) and language clustering training @xcite alleviate interference by expanding the unified model with substantial extra parameters. sub-networks in multi-task models. the lottery ticket hypothesis @xcite states that within dense neural networks, sparse subnetworks can be found with iterative pruning to achieve the original network's performance. following this premise, recent studies attempt to isolate sub-networks of a pre-trained unified model that captures task-specific features @xcite @xcite . nonetheless, unlike our method that identifies intrinsic modularity within the model, these approaches depend on fine-tuning to extract the task-specific subnetworks. this process may not reflect the original model modularity and also can be particularly resource-consuming for multiple tasks. specifically, these methods extract the taskspecific sub-networks by fine-tuning the original unified multi-task model on specific tasks, followed by employing pruning to retain only the most changed parameters. we argue that this process faces several issues: 1) the sub-network might be an artifact of fine-tuning, suggesting the original model may not inherently possess such modularity. 2) this is further supported by the observation that different random seeds during fine-tuning lead to varied sub-networks and performance instability @xcite . 3) the process is highly inefficient for models covering multiple tasks, as it necessitates separate fine-tuning for each task. ## neuron structural analysis recent work aims to identify a subset of parameters within pre-trained multi-task networks that are sensitive to distinct tasks. this exploration is done by either 1) selecting hidden states that greatly influence task performance @xcite or possess high magnitude values @xcite ; or 2) fine-tuning the unified model on task-specific data to extract sub-networks @xcite @xcite . these approaches, however, raise a fundamental question, namely whether the modularity is inherent to the original model, or simply an artifact introduced by network modifications. in this paper, we perform a thorough identification of task-specific modularity through the lens of neuron behaviors, without altering the original parameters or architectures. we focus on the neurons -the intermediate activations inside the feed-forward networks (ffn) -to investigate if they indicate task-specific modularity features. as ffn neurons are active (>0) or inactive (=0) due to the relu activation function foot_1 , this binary activation state offers a clear view of their contributions to the network's output. intuitively, neurons that remain inactive for one task but show significant activation for another may be indicative of specialization for the latter. more importantly, this approach ensures that both parameters and hidden states remain unchanged, affirming the observed modularity is inherent to the original model. ## neuron specialization training our neuron structural analysis showed the presence of specialized neurons within the feed-forward network (ffn) layers of a multilingual network. we hypothesize that continuously training the model, while leveraging these specialized neurons' intrinsic modular features, can further enhance taskspecific performance. building on this hypothesis, we propose neuron specialization, an approach that leverages specialized neurons to modularize the ffn layers in a task-specific manner. ## experimental setup we describe the experimental setups in this section. note that we utilize the same training data for both pre-training and methods involving fine-tuning or continual training. more details of the datasets are in appendix a.1. ## conclusions in this paper, we have identified and leveraged intrinsic task-specific modularity within multilingual networks to mitigate interference. we showed that ffn neurons activate in a language-specific way, and they present structural overlaps that reflect language proximity, which progress across layers. we then introduced neuron specialization training to leverage these natural modularity signals to structure the network, enhancing task specificity and improving knowledge transfer. our experimental results, spanning various resource levels, show that our method consistently outperforms strong baseline systems, with additional analyses demonstrating reduced interference and increased knowledge transfer. our work deepens the understanding of multilingual models by revealing their intrinsic modularity, offering insights into how multi-task models can be optimized without extensive modifications.