Revealing the Parallel Multilingual Learning within Large Language Models (2024)

Abstract: Large language models (LLMs) can handle multilingual and cross-lingual text within a single input; however, previous works leveraging multilingualism in LLMs primarily focus on using English as the pivot language to enhance language understanding and reasoning. Given that multiple languages are a compensation for the losses caused by a single language’s limitations, it’s a natural next step to enrich the model’s learning context through the integration of the original input with its multiple translations. In this paper, we start by revealing that LLMs learn from parallel multilingual input (PMI). Our comprehensive evaluation shows that PMI enhances the model’s comprehension of the input, achieving superior performance than conventional in-context learning (ICL). Furthermore, to explore how multilingual processing affects prediction, we examine the activated neurons in LLMs. Surprisingly, involving more languages in the input activates fewer neurons, leading to more focused and effective neural activation patterns. Also, this neural reaction coincidently mirrors the neuroscience insight about synaptic pruning, highlighting a similarity between artificial and biological ‘brains’.

## introduction many of the recent large language models (llms) are multilingual. unlike language-specific nlp systems, such as machine translation systems specialized to a given language pair, these models are generally trained on large-scale multilingual datasets, using a unified vocabulary. because of this training approach, it is possible to learn a universal representation of texts across different lan- more specifically, we find that in addition to the performance improvements from incorporating more languages, llms can gain advantages from extensive languages even involving ones that do not surpass baseline performances. with the help of high-quality machine translation, we efficiently acquire abundant parallel input, enabling us to apply this method to various tasks. experimental results across 8 datasets, 7 languages, and 10 llms further demonstrate the effectiveness and applicability of pmi. since previous neuron activation statistics are primarily designed for the vanilla transformer model @xcite , we have extended these methods to analyze more advanced llm architectures. when llms receive pmi, we observe simultaneous performance improvements and neuron inhibition. in addition, pmi selectively activates only a small portion of the most commonly used neurons while inhibiting the rest. further analysis reveals that few-shot learning produces a similar effect on neuron activation, and integrating it with pmi enhances this neural reaction. these findings are consistently sustained across different models and tasks. we introduce our pmi and evaluate it with human translation in section 2.1. subsequently, we comprehensively analyze the performance gains brought by pmi in section 2.2 and explain its effectiveness from a view of neuron activation in section 3. moreover, we apply pmi to various tasks under real scenario setups in section 4. 21 . ## pmi can help: from a view of neuron activation although llms benefit from pmi, there is still no idea about how this mechanism works. considering that knowledge is memorized in different neurons in transformer models @xcite , hence a straightforward hypothesis is that giving the input in multiple languages may increase the number of activated neurons in the inference process. to quantify how many neurons in transformer models are activated during inference, some works propose to make statistics of the nonzero values in the intermediate output of multi-layer perceptrons (mlps) after a relu activation function @xcite . this is based on the idea that, in matrix multiplication, zero can be omitted; therefore, neurons that output zero are considered inhibited while others are activated. next, we will explain this statistical method. ## wide evaluation of pmi without human translations next, we focus on evaluating the pmi method on downstream tasks under real scenario setups. ## related work multi-way neural machine translation. multiway input is a successful method to enhance multilingual neural machine translation (mnmt) systems by providing the source language and its translations in different languages @xcite . in the inference stage, most works rely on highquality translations from human experts @xcite @xcite . however, this ground truth multilingual data is scarce in reality, limiting the application of multi-way input. different from multi-way mnmt, we find that llms benefit from pmi even when parallel multilingual input is derived from automatic mt systems, enabling us to apply pmi on a wide range of tasks. ## conclusions we reveal that llms can learn from parallel multilingual input. firstly, comprehensive experiments across 8 typical datasets, 10 commonly used multilingual llms, and 7 languages demonstrate the effectiveness and applicability of pmi. secondly, statistics of activated neurons indicate that pmi optimizes performance by inhibiting neurons and promoting more precise neuron activation, which performs like one-off synaptic pruning. in future work, we aim to explore applying pmi to multimodal tasks and observing neural activation behaviors in large multimodal models.