Language-Specific Neurons Do Not Facilitate Cross-Lingual Transfer (2025)

Abstract: Multilingual large language models (LLMs) aim towards robust natural language understanding across diverse languages, yet their performance significantly degrades on low-resource languages. This work explores whether existing techniques to identify language-specific neurons can be leveraged to enhance cross-lingual task performance of low-resource languages. We conduct detailed experiments covering existing language-specific neuron identification techniques (such as LanguageActivation Probability Entropy and activation probability-based thresholding) andneuron-specific LoRA fine-tuning with models like Llama 3.1 and Mistral Nemo. We find that such neuron-specific interventions are insufficient to yield cross-lingual improvements on downstream tasks (XNLI, XQuAD) in low-resource languages. This study highlights the challenges in achieving cross-lingual generalization and provides critical insights for multilingual LLMs.

## introduction acquiring multilingual capabilities in llms remains a challenge, particularly for low-resource languages @xcite @xcite . despite their remarkable success in tasks that require cross-lingual transfer, models such as llama 3.1 @xcite and mistral nemo @xcite do not perform consistently across languages, particularly underperforming on lowresource languages @xcite . this is largely due to the imbalance in highquality training data across languages, thus limiting the ability of multilingual models to effectively scale to low-resource languages @xcite . a tool that has recently emerged to better understand the nature of multilinguality in these llms is the use of language-specific neurons @xcite @xcite . these neurons are claimed to encode unique languagespecific features pertaining to each language, thus potentially enabling targeted language interventions. previous studies @xcite @xcite have demonstrated that these neurons play an important role in language generation tasks. however, the extent to which these neurons contribute to or affect crosslingual transfer to low-resource languages when evaluated on downstream tasks such as natural language inference (xnli) and question answering (xquad) remains unclear. in this study, we systematically probe the role of language-specific neurons in facilitating crosslingual transfer within multilingual llms. by utilizing existing techniques to identify languagespecific neurons such as language activation probability entropy (lape) @xcite and low rank adaptation (lora)-based finetuning @xcite , we aim to identify and analyze neurons that mainly contribute towards language-specific representations. our experiments span two popular cross-lingual benchmarks, xnli for nli @xcite and xquad for qa @xcite . after identifying language-specific neurons using existing techniques for a target language, we modify the activations of these language-specific neurons using different aggregation schemes in an attempt to amplify their role in cross-lingual transfer. our results show that such test-time (trainingfree) interventions via language-specific neurons are not very effective in enabling cross-lingual transfer, yielding very modest overall performance improvements of less than 1 absolute point in accuracy for low-resource languages. fine-tuning strategies like neuron freezing and activation sub-stitution were shown to significantly impact generation @xcite but do not show any consistent impact on cross-lingual task performance. a deeper analysis revealed that language-specific neurons often lack independence and we hypothesize that this polysemantic nature of neuron activations limits the effectiveness of targeted adjustments in multilingual llms @xcite . ## methodology the goal of this work is to explore whether targeting language-specific neurons in multilingual llms can be used to improve downstream performance on tasks such as xnli and xquad. previous studies @xcite @xcite have shown that distinct neuron subsets exist in multilingual models that encode language-specific features. prior work @xcite further indicates that language-specific representations are largely prevalent within feedforward networks. while prior work focused on how deactivating language-specific neurons degrades the quality of language generation, there has been little investigation into whether activating or fine-tuning these neurons can positively influence task performance @xcite . this forms the main motivation for our work. we aim to evaluate the role of language-specific neurons by aiming to enhance cross-lingual task performance through targeted neuron interventions. our results indicate that manipulating language-specific neurons, either by activating or fine-tuning them, does not lead to significant improvements in downstream task performance. ## conclusion in this work, we investigate if language-specific neurons in multilingual llms could be manipulated to improve cross-lingual task performance. our results show that test-time interventions and fine-tuning of language-specific neurons do not yield meaningful improvements. altering these activations often disrupt task-relevant information likely due to the polysemantic nature of llm neurons. we found the same behaviour across different methods for identifying language neurons, such as lape and activation probability. additionally, setting activations to zero did not significantly degrade performance, suggesting that zero is not a true indicator of deactivation. these findings indicate that language-specific neurons do not function independently but interact with broader model components, and need further investigation as tools of cross-lingual transfer.