Pruning Multilingual Large Language Models for Multilingual Inference (2024)

Abstract: Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. This study introduces a promising direction for enhancing non-English performance through a specialized pruning approach. Specifically, we prune MLLMs using bilingual sentence pairs from English and other languages and empirically demonstrate that this pruning strategy can enhance the MLLMs’ performance in non-English language.

## introduction large language models (llms) have demonstrated remarkable language reasoning capabilities, particularly in english contexts. however, these llms limited their proficiency in non-english language @xcite . recognizing this limitation, recent research endeavors have given rise to multilingual large language models (mllms), such as xglm @xcite , mgpt @xcite , and bloom @xcite , designed to address the challenges figure 1 : the overlap ratios among the top-and bottom-30% features in the 27-th layer of xglm, ranked by their magnitude. the row and column labels correspond to languages and language pairs used in few-shot monolingual (en, fr, etc.) and translation (fr-en, es-en, etc.) demonstrations, respectively. each element represents the ratios of overlapping features between the top-and bottom-30% in magnitude within each demonstration. this figure shows that specific features are active only when inputting translation demonstrations. achieving performance parity in non-english languages with english, which represents the upper bound of mllms, necessitates the alignment of english and non-english texts. @xcite has demonstrated the effectiveness of the translatetest, a methodology translating non-english texts into english using an external machine translation (mt) system and running inference over the translated texts, which serves as an approach to superficially align english and non-english. however, the translate-test approach increases inference costs due to its reliance on the mt system. conversely, recent research indicated that mllms can perform translation by incorporating few-shot translation demonstrations into the contextual information @xcite @xcite . these findings suggest that mllms already have alignment capability between english and non-english, and it is manifested through the incorporation of few-shot translation demonstrations. we believe that if the alignment capability can be brought out in tasks other than translation, non-english performance elevates to the level equivalent to english. previous researches @xcite have identified two distinctive characteristics within hidden state features of llms: first, the presence of large magnitude features, a small set of hidden state features that emerge with significantly larger magnitudes than the remaining ones; and second, the essential nature of these features for the predictive capabilities of llms. referring to the researches, we analyzed magnitudes of features of mllms when inputting few-shot monolingual and translation demonstrations. fig. 1 shows the overlap ratios within features between the monolingual and translation demonstrations, and indicates that specific features are predominantly active only when inputting translation demonstrations. in addition, we will see later that the large magnitude features are relevant for the translation performance. motivated by the results, we hypothesized that mllms carry out zero-shot inferences while accentuating their alignment capability by forcing them to use large magnitude features that are active when inputting few-shot translation demonstrations. to achieve this, we retained and pruned weights of mllms following @xcite that involve operations for the large magnitude features and others, respectively. we observed that the pruned mllms improved zero-shot performance in non-english languages compared to pre-pruned mllms. our contributions in this study are threefold: ## task setting in this study, we conduct zero-shot learning with mllms under lin et al. ( 2022 )'s scenario. we consider a language @xmath0. to perform in-context learning, we convert x l to a cloze-style format that contains a [mask] symbol using a template t and map each candidate label @xmath1. an input prompt p(x l , y) is obtained by substituting the [mask] symbol in t (x l ) with v(y). in zero-shot in-context learning, a prediction ŷ is the label with the maximum likelihood: one of the problems with using llms is that it requires a lot of computational resources for inference. to overcome this problem, several studies have attempted quantization of the llms. @xcite proposed a novel quantization method called llm.int8() that performs matrix multiplications for large magnitude features and others in 16-bit and 8-bit, respectively. they empirically demonstrated that llm.int8() reduces performance degradation compared to performing all operations in 8-bit. this result suggests that the large magnitude features of llms are crucial for their prediction. motivated from the success of llm.int8(), @xcite proposed a weights pruning approach named pruning by weights and activations (wanda). wanda drops weights that do not involve operation for the large magnitude features. consider a linear layer's weight of k-th layer of a model t ×d in , where t denotes the number of tokens included in calibration data x. wanda calculates importance scores s @xmath2. specifically, a score of i, j-th element s i,j is calculated as: enhancing multilingual performance of multilingual pre-trained models as outlined in §1, several studies have attempted to train multilingual large language models (mllms) using datasets that exhibit a more balanced linguistic distribution compared to english-dominant data used in llms. although mllms have impressive multilingual capability, their performances in non-english languages do not achieve those in the english level, which is an upper bound level of mllms. this phenomenon has been observed in multilingual pre-trained masked language models, such as mbert @xcite and xlm-r @xcite . several studies have enhanced the alignment of hidden states between target languages and english using bilingual resources during either the pre-training or fine-tuning phases, demonstrating improved performance in the target languages @xcite @xcite @xcite . few-shot cross-lingual transfer, which involves fine-tuning the multilingual pre-trained models with a small amount of supervised data in the target language for a downstream task, is a promising approach, and several studies have validated its effectiveness @xcite . however, the efficacy of these methodologies when applied to mllms remains unverified, and such training approaches require substantial computational resources for adaptation to mllms. @xcite undertook research closely aligned with our objective, specifically aiming to enhance non-english performance of mllms in a zero-shot learning scenario without the necessity for fine-tuning. they introduced a novel methodology termed language representation projection (lrp2). lrp2 adjusts the hidden states at the a-th layer by subtracting and adding vectors that correspond to the target language and english, respectively. subsequently, the inverse operations are applied at the b-th layer. these vectors are derived from the mean-pooled vectors across all tokens in each respective language's dataset at the corresponding layers. in this research, we provide a promising direction for enhancing non-english performance through comprehensive experiments on various mllms. specifically, we identify large-magnitude features that are relevant for bringing out the inherent alignment capabilities of mllms @xcite @xcite . motivated by the results, we encourage mllms to leverage these prominent features by implementing wanda @xcite . subsequent sections will show that our approach improves the performance of non-english languages across multiple mllms, surpassing that of lrp2. ## detecting translation features our challenge is accentuating the alignment capability of mllms to enhance their non-english performance even when tasks other than translation. to explore how to emphasize the alignment capability, we first analyze the behavior of mllms during translation. while previous studies have suggested that large magnitude features are important for inference, they did not reveal whether large magnitude features are the same or different across tasks. if the large magnitude features that activate exclusively during translation are instrumental for mllms' translation performance, the features may be key to bringing out the alignment capability. therefore, in this section, we conduct analyses based on the following research questions to show step-by-step that there are large magnitude features that affect for translation performance. rq1: do few-shot translation demonstrations activate specific features? previous studies have suggested that mllms can achieve translation performance equivalent to supervised machine translation models by incorporating few-shot translation demonstrations into contextual information. we consider that salient features when inputting fewshot translation demonstrations play an important role in translation. therefore, in this study, we analyze magnitudes of the hidden state features x k src-@xmath3, those of few-shot translation demonstrations x src-@xmath4, ..., x n src-tgt } output from k-th layer of an mllm. here, t src-tgt is a total amount of tokens included in x src-tgt . to construct a translation demonstration x 1 src-tgt , we use n-shot bilingual sentence pairs between src and tgt language randomly sampled from bilingual data. in addition, we find the hidden state features x k @xmath5 t src ×d in and x k @xmath6, those of few-shot monolingual demonstrations x @xmath7, ..., x n src } and x @xmath8, ..., x n tgt }, respectively. please refer to appendix a for detailed descriptions on how to construct each demonstration (x src-tgt , x src , and x tgt ). to investigate the magnitude of each feature of x k src-tgt , x k src , and x k tgt , we find @xcite , which are vectors that each element is l2-norm of the feature aggregated across the tokens of the corresponding hidden states. specifically, j-th feature of we examine whether there are features that have extremely large magnitude compared to others. furthermore, we measure overlap ratio between the top and bottom β% of features, ranked according to their magnitudes. specifically, the overlap ratio between the features in the top β% of ∥x k src-tgt ∥ 2 and those in the bottom β% of ∥x k src ∥ 2 is calculated as follows: ## multilinguality of pruned mllms our main goal is to enhance the zero-shot performance of mllms in non-english contexts by leveraging the alignment capability between english and non-english languages. in the previous section, we demonstrated that the large magnitude features that are active only when processing few-shot translation demonstrations play an important role to maintain the translation performance of mllms. this implies that these prominent features are essential for bringing out the alignment capability. we hypothesize that zero-shot in-context learning is performed while accentuating the alignment capability by prioritizing the use of the large magnitude features from few-shot translation demonstrations. we can accomplish this by employing the pruned weights, denoted as θ src-tgt , based on fewshot translation demonstrations. the pruned model focuses on the large magnitude features relevant for performing translation, disregarding other features during the prediction process, thereby accentuating alignment capability regardless of the target task. we conduct zero-shot in-context learning based on the pruned weights θ src-tgt as follows: ## conclusion in this study, we showed that there are large magnitude features activated when inputting few-shot translation demonstrations and the pruned mllms (i.e. xglm and mgpt) based on the features enhances zero-shot performance on non-english languages by utilizing the english inference capabilities. additionally, we reformulated the scoring metric to eliminate weights associated with operations for large magnitude features in programming language generation and demonstrated that 5 see appendix e for the detailed descriptions of rankc. the observation from the result of pruning based on the reformulated metric paves the way for further inquiry into the selective pruning of model weights to optimize performance across diverse linguistic tasks. for future work, we would like to delve deeper into this aspect to identify weights that should be retained or pruned to enhance the performance of mllms.