Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency (2022)

Abstract: Structured pruning has been extensively studied on monolingual pre-trained language models and is yet to be fully evaluated on their multilingual counterparts. This work investigates three aspects of structured pruning on multilingual pre-trained language models: settings, algorithms, and efficiency. Experiments on nine downstream tasks show several counter-intuitive phenomena: for settings, individually pruning for each language does not induce a better result; for algorithms, the simplest method performs the best; for efficiency, a fast model does not imply that it is also small. To facilitate the comparison on all sparsity levels, we present Dynamic Sparsification, a simple approach that allows training the model once and adapting to different model sizes at inference. We hope this work fills the gap in the study of structured pruning on multilingual pre-trained models and sheds light on future research.

## introduction large-scale pre-trained monolingual language models like bert @xcite and roberta @xcite have shown promising results in various nlp tasks while suffering from their large model size and high latency. structured pruning has proven to be an effective approach to compressing and accelerating these large monolingual language models @xcite @xcite , making them practical for real-world applications. similarly, multilingual pre-trained models (conneau and lample, 2019; @xcite @xcite are also powerful and even have more parameters. however, little attention has been paid to evaluating the effectiveness of structured pruning on these multilingual models. applying pruning to multilingual pre-trained models is non-trivial, as it typically involves many languages and needs to carefully design the roles of modules within the network. for example, most attention heads have little impact on the performance of monolingual pre-trained models @xcite , while it is the opposite for multilingual pre-trained models (see section 5.3 and also @xcite ). this work intends to examine how structured pruning reacts to multilingual pre-trained models. we take the most representative multilingual pre-trained model family, xlm-r @xcite for our case study and evaluate the pruning performance on nine crosslingual understanding tasks in xtreme @xcite . we investigate three aspects of structured pruning: settings, algorithms, and efficiency. settings traditional pruning produces a single small model, which is shared across languages (shared setting). recent work on multilingual translation @xcite @xcite suggests that tailoring pruning to one language could achieve better results (non-shared setting). however, our comprehensive experiments show that neither of the two settings can consistently outperform the other one (see section 5.2). algorithms there exists a broad spectrum of pruning algorithms @xcite , and it is impossible to test all of them considering the cost of pre-training. we focus on two pruning algorithms that have been studied the most in monolingual pretrained models: the regularization-based pruning @xcite (and our improved version) and the gradient-based pruning @xcite @xcite (see section 4). we experimentally find that the simplest gradient-based pruning is more effective for xlm-r (see section 5.2). efficiency one meaningful way to measure pruning algorithms is to study how the performance and speed of the pruned model vary with the sparsity @xcite . however, most pruning algorithms, including those we study in this work, require training the model for each specific sparsity. this limitation makes comparisons against a range of sparsity levels infeasible due to the prohibitive training cost. to solve this issue, we propose the dynamic sparsification (ds for short), a simple method that parameterizes subnetworks at any sparsity level and shares their weights afterward (see section 6.1). ds only trains the model once but can obtain models at any sparsity level during inference. experiments on xnli @xcite show that ds does not degrade the performance much while dramatically reducing the training cost. interestingly, we observe that the model size and inference speed are not strongly correlated in xlm-r. this observation suggests that one could not obtain a fast model by simply making the model small by using vanilla pruning algorithms (see section 6.2). ## related work settings recent multilingual translation research suggests that adapting subnetworks for each language or language pair rather than for all of them gives better results. among them, @xcite train a shared multilingual model, then select layers for each language pair. @xcite also prune a shared multilingual model for each language pair, though on the level of entries in weight matrices. instead, @xcite prune attention heads and feedforward networks for each language. @xcite first identify general and languagespecific neurons in a shared multilingual network, then tune those neurons using the data of their corresponding language only. these findings inspire us to extend from multilingual translation to see how non-shared pruning settings work on multilingual pre-training. algorithms there are many structured pruning techniques proposed for monolingual pre-trained language models recently. @xcite propose a simple gradient-based importance score to prune attention heads. @xcite ; @xcite extend to prune other components like the feedforward network of the transformer @xcite . @xcite decompose the pre-trained model weights and apply l 0 regularization @xcite to regulate the ranks of decomposed weights. @xcite study layer pruning and show that directly dropping the top layers performs the best in fine-tuning. @xcite further show that by carefully choosing layers to drop, structured pruning can achieve a performance close to those trained by knowledge distillation @xcite . ## background in this section, we briefly review the structure of xlm-r @xcite , a transformer encoder @xcite pre-trained by masked language modeling task @xcite . we also revisit how conventional structured pruning algorithms are applied to transformers by introducing additional gating variables and setting appropriate values to them (see figure 1 and also @xcite ; @xcite ). the xlm-r model consists of n layers. each layer is made of the multihead attention and feedforward networks, followed by the residual connection and layer normalization. attention following michel et al. (2019)'s formula, the multihead attention is written as: ## extending pruning algorithms to pruning settings this section will first introduce pruning algorithms that we study and then describe how to adapt them to two pruning settings. the first is the shared setting that shares the pruned network across languages (default setting that all pruning algorithms could run on), and the second is the non-shared setting that prunes one subnetwork for each language @xcite . ## conclusion in this work, we study three aspects of structured pruning on multilingual pre-trained models: settings, algorithms and efficiency. experiments show interesting phenomena: the best pruning setting depends on the choice of algorithms; the simplest pruning algorithm performs the best; a fast model does not mean it should be small. we hope this work will give insight to future research. paul michel, omer levy, and graham neubig. 2019. are sixteen heads really better than one? in advances in neural information processing systems 32: annual conference on neural information processing systems 2019, neurips 2019, december 8-14, 2019, vancouver, bc, canada, pages 14014-14024. joakim nivre, rogier blokland, niko partanen, and michael rießler. 2018. universal dependencies 2.2. xiaoman pan, boliang zhang, jonathan may, joel nothman, kevin knight, and heng ji. 2017. crosslingual name tagging and linking for 282 languages. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1946-1958, vancouver, canada. association for computational linguistics.