Unraveling B abel: Exploring Multilingual Activation Patterns of LLM s and Their Applications (2024)

Abstract: Recently, large language models (LLMs) have achieved tremendous breakthroughs in the field of NLP, but still lack understanding of their internal neuron activities when processing different languages. We designed a method to convert dense LLMs into fine-grained MoE architectures, and then visually studied the multilingual activation patterns of LLMs through expert activation frequency heatmaps. Through comprehensive experiments on different model families, different model sizes, and different variants, we analyzed the similarities and differences in the internal neuron activation patterns of LLMs when processing different languages. Specifically, we investigated the distribution of high-frequency activated experts, multilingual shared experts, whether multilingual activation patterns are related to language families, and the impact of instruction tuning on activation patterns. We further explored leveraging the discovered differences in expert activation frequencies to guide sparse activation and pruning. Experimental results demonstrated that our method significantly outperformed random expert pruning and even exceeded the performance of unpruned models in some languages. Additionally, we found that configuring different pruning rates for different layers based on activation level differences could achieve better results. Our findings reveal the multilingual processing mechanisms within LLMs and utilize these insights to offer new perspectives for applications such as sparse activation and model pruning.

## introduction large language models (llms), proficient in utilizing a wide range of linguistic structures, have attained notable advancements in the field of natural language processing @xcite . however, how an llm uses multiple languages within * corresponding authors. we still lack an intuitive understanding of the internal neuron activity of llms when processing different languages-it remains like a black box. this is due to the difficulty of decomposing llms into recognizable components. this presents a significant obstacle for us to better utilize llms. therefore, we aim to investigate the differences and connections in internal neuron activations of llms when processing different languages. we refer to the different activation patterns exhibited by internal neurons of llms when processing different languages as multilingual activation patterns. to open the black box and intuitively understand and explain the internal multilingual activation patterns of llms, we devised a method that involves converting dense llms into fine-grained moe architectures and then calculating the activation frequencies of experts. subsequently, we visualize the multilingual activation patterns of llms by heatmaps. we conducted a comprehensive experimental study on the multilingual activation patterns of different llms, including various model families (llama 2, llama 3, mistral), different sizes (7b-70b), and different variants @xcite . we analyzed the distri-bution of high-frequency activated experts, multilingual shared experts, whether the activation patterns of different languages are related to language families, and the impact of instruction tuning on activation patterns. considering that dense llms can be transformed into moe architectures @xcite , based on obtaining multilingual activation patterns, we further propose to leverage the activation frequency differences of various experts to guide sparse activation and pruning, aiming to minimize the amount of computation and inference cost, while maintaining model performance as much as possible. we advocate using only high-frequency experts for inference, excluding the parameters of other experts to achieve sparse activation and pruning. to this end, we propose two specific pruning schemes: pruning based on frequency thresholds and pruning based on frequency sorting. in experiments across various models and metrics, our method significantly outperformed random expert pruning and even exceeded the performance of the original unpruned models in some languages. this further validates the effectiveness of the multilingual activation patterns discovered by our method, providing a new feasible path for model pruning. meanwhile, we found that equal proportional pruning at each layer is inferior to that of unequal pruning, which verifies the inter-layer differences in activation levels of llms. therefore, we suggest configuring different pruning rates for different layers based on differences in activation levels. overall, we have not only analyzed the multilingual activation patterns within various llms, but also explored their connections with language families and instruction tuning. furthermore, based on the identified multilingual activation patterns, we provide a new and effective approach to achieve sparse activation and pruning. ## related work initially, in the task of neural machine translation (nmt) using small language models, some works @xcite @xcite explored language-specific components. @xcite studied the exceptional performance of gpt-3 in catalan, despite the small proportion of catalan in the training corpus. @xcite evaluated the language specificity of the detector in the xglm model using a parallel corpus of czech and english, by conceptualizing the ffn as a system of detector, selector, and combiner. several recent works @xcite @xcite investigated the existence of language-specific neurons in llms and the modification of these neurons. @xcite identified a core region corresponding to linguistic competence in llms through fine-tuning with various languages. in terms of moe, the llama-moe model @xcite explored transforming llama 2 7b into an moe model and its continued training. however, exploring the multilingual activation patterns within llms and how to leverage these patterns remains an area that has not been sufficiently investigated. ## exploring multilingual activation zhang et al. (2022) proposed moefication, exploring the conversion of the feed-forward networks (ffns) in pre-trained transformers into moe structures without altering the original model parameters, and maintaining the performance on downstream tasks by conditionally selecting experts. inspired by this, we propose to study the internal neuron activation patterns of llms when processing different languages by visualizing the expert activation frequency and its variations after converting the model to an moe structure. expert construction our first step is to split the parameters of the ffns into different experts. the ffns in llama/mistral models comprise three layers: up-projection, gate-projection, and downprojection, as illustrated in the schematic diagram presented in appendix a. based on the intuition that the neurons with similar parameters exhibit similar activation patterns, we adopt the parameter clustering split method to cluster the parameters of each ffn layer, dividing them into different experts. specifically, we perform balanced k-means clustering (malinen and fränti, 2014) on the parameters of the up-projection layer, dividing it into 256 clusters. then, we divide the neurons and their parameters of down-projection and gate-projection layers into different clusters based on the clustering results of the up-projection layer. dividing neurons into different experts can significantly reduce the computational burden in subsequent experiments and enable us to directly observe the internal multilingual activation patterns of llms through heatmap visualization. to achieve fine-grained pa-rameter splits while maintaining the visualization effects of heatmaps, we divide each ffn layer into 256 experts. cross-layer expert selection next, we design the cross-layer expert selection method to identify experts with higher activation levels and frequencies as the language-specific experts for each language. llms contain multiple ffn layers, but the moefication method is limited to selecting experts within a single ffn layer, which fails to reflect the differences in activation levels across layers of varying depth within the llm. therefore, we need to extend the moefication method to cross-layer expert selection. for each input token, we use the sum of the activation values of all neurons for each expert as the score of this expert, representing the activation level. given the direct incomparability of activation magnitudes across different ffn layers, to facilitate a global comparison of activation levels across layers, we perform a z-score normalization on the scores of all experts within each layer. subsequently, we rank the scores of all experts across all layers. we select approximately the top 10% of experts (for the 7b/8b models, we select the top 800, and for the 70b model, we select the top 2000) as the activated experts for a given input token, increasing their activation counts by 1. by normalizing and then performing cross-layer comparison, we better identified the experts whose activation values stood out relative to other experts. expert activation patterns after extensive testing with a multitude of tokens, we calculate the activation frequency for each expert (activation count/total number of tokens). finally, the activation frequencies of all experts across n layers are compiled into an n × 256 activation matrix for heatmap visualization. data. we selected the nine most widely spoken languages from the 46 languages contained in the roots corpus @xcite for experiments. the data sources are in appendix b. the language families and genera of these languages are presented in ## conclusion in this study, we investigated the multilingual activation patterns in various llms from the perspective of moe models. we also explored their connections with language families and instruction tuning, as well as the potential for guiding sparse activation and model pruning. these findings can assist us in better developing and utilizing the multilingual capabilities of llms. we hope these findings inspire new research in related fields.