Intriguing Properties of Compression on Multilingual Models (2022)

Abstract: Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages. Compression techniques are widely relied upon to reconcile the growth in model size with real world resource constraints, but compression can have a disparate effect on model performance for low-resource languages. It is thus crucial to understand the trade-offs between scale, multilingualism, and compression. In this work, we propose an experimental framework to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning.Applying this framework to mBERT named entity recognition models across 40 languages, we find that compression confers several intriguing and previously unknown generalization properties. In contrast to prior findings, we find that compression may improve model robustness over dense models. We additionally observe that under certain sparsification regimes compression may aid, rather than disproportionately impact the performance of low-resource languages.

## introduction scaling language models benefits multilingual settings, since it is difficult to maintain performance across a growing number of languages at a constant model size, a property also called the "curse of multilinguality" @xcite @xcite . however, the extent of growth in language model (lm) size @xcite @xcite has made deployment to resource-constrained environments much more challenging @xcite @xcite . to benefit from the performance gains conferred by scale, efficiency techniques that reduce model size * work was done while at google research. while maintaining comparable aggregate performance are widely used, such as quantization @xcite , compression @xcite and distillation @xcite @xcite . while most compression techniques have minimal impact on aggregate performance numbers @xcite @xcite @xcite , the impact on individual sub-populations in the data, such as low-resource languages, can be far more severe @xcite @xcite . disparities in resource availability become more apparent at larger scale, both in terms of data and deployment resource availability. this makes compression all the more necessary, but also motivates a thorough consideration of the subsequent impact of compression on generalization. in this work, we develop an experimental framework to investigate the impact of compression during fine-tuning of pre-trained multilingual models which we apply to named entity recognition (ner) across 40 languages of the wikiann benchmark @xcite . we study the impact of compression on groups of languages across multiple dimensions-resourcedness, script, and language family-and evaluate the sensitivity of models to input perturbations along these groupings. this leads us to discover the following intriguing properties: (1) lower-performing languages disproportionately suffer under extreme levels of sparsity, as pruning amplifies disparities. however, low-resource languages present an intriguing flipflop moment, where their performance may benefit from medium regimes of sparsity. (2) we find that dense models overfit to typical test cases, achieving a close-to-0 f1 score on slightly perturbed inputs, while compression can recover close to the original test performance. our results stand in contrast to previous work that find that sparsity erodes robust-ness, suggesting more work is needed to understand the dynamics between compression and robustness. ## related work the "curse-of-multilinguality" creates a tradeoff between number of languages and size of a model @xcite . however, training smaller models supporting fewer languages may not always be feasible @xcite . compressing large models has been shown to combat the curse, either by compressing the pretrained model @xcite , or by compressing during finetuning, as in our case. while many studies investigate the impact of pruning on aggregate metrics in monolingual pretrained lms @xcite @xcite @xcite @xcite @xcite , fewer works focus on multilingual settings (mukherjee and hassan awadallah, 2020; @xcite ). yet, prior analyses find a disparate effect of removing attention heads or model layers on languages and language families distant from the training data in ner @xcite , demonstrating the importance of looking into subpopulations as we do in this study. studies that compare the robustness of compressed and dense models further find that compression may lead to erosion of performance on "challenging" samples and poor generalization @xcite @xcite , a finding that we expand on and connect to language resourcedness. the technique we use to study robustness expands on studies that perturb training @xcite or evaluation data @xcite in ner by introducing perturbations specific to languages, language families, and scripts. ## methodology data we conduct our experiments on wikiann @xcite , a multilingual ner dataset. wikiann was sourced from wikipedia articles and automatically annotated with loc (location), per (person), and org (organisation) labels in the iob2 format @xcite . it is considered a "silver standard" due to its automatic entity labels and noise @xcite , but with its 176 languages it covers the most languages of any ner dataset. we focus our experiments on the 40 languages from the xtreme benchmark @xcite , with train-test splits defined by @xcite . these training sets were built with stratified sampling to create a balance across entity types @xcite , and are thus a subset of the total available data from the original wikiann. perturbations we test the robustness of compressed models by perturbing named entities in the test set. previous work @xcite show that sparse pretrained language models are less robust than their dense equivalents when evaluated on adversarial test sets, even when they perform similarly on in-distribution test sets. we adopt a data perturbation technique from dai and adel (2020) called entity mention replacement; an entity is randomly swapped with another entity of the same type (example sentences shown in app. d). we first perturb entities within same language for all the languages in our dataset (in-language). secondly, we propose a new benchmark appropriate for testing the cross-lingual robustness of multilingual models on our downstream task. we perturb entities across different languages that share common linguistic properties. in particular, we group languages by family and script and perturb entities across languages within the same group (in-script, in-family). model we use the cased multilingual bert (mbert) @xcite for all our experiments because it is one of the most widely used and studied multilingual lms (e.g., @xcite @xcite @xcite ). foot_0 mbert is trained on wikipedia data from 104 languages, has approximately 177 million parameters, and a vocabulary size of around 120,000. we finetune mbert by appending a linear classification layer to the model and updating all its parameters. full hyperparameters are listed in app. a.1. we evaluate the impact of sparsity in two settings: 1) in the monolingual setting we fine-tune on individual languages. for this setting, we select 10 languages with different available data size, language family and scripts (see pruning we induce sparsity by applying iterative magnitude pruning (imp) @xcite during fine-tuning. imp iteratively removes weights that are below a certain threshold until a desired target sparsity is reached. imp is widely used and competitive with far more compute intensive approaches @xcite @xcite , while allowing us to sparsify to an exact level. we compare two pruning strategies: 1) partial where we prune all dense layers except for embedding layers, 2) incl. embeddings where we prune all dense weights including embedding layers. embeddings make up more than half (91m) of the 177m parameters in mbert, while dense weights make up the rest. hence, pruning embeddings allows us to significantly reduce the number of mbert parameters. we consider five sparsity levels: 50%, 70%, 80%, 90%, 95% and 98%, corresponding to the percentage of weights pruned (hyperparameters in app. a.2. preliminary experiments were conducted with lower sparsity levels (10%-40%) and yielded similar findings to those at moderate sparsity levels (50%-70%), motivating the sparsity intervals chosen. the chosen sparsity levels also align with general best practice in sparsity evaluation as presented in previous works. moderate to high sparsity levels (50%+) are necessary for efficiency gains in the real-world and are usually studied in literature @xcite @xcite . ## conclusion this work investigates the effects of compression on multilingual pre-trained language models during fine-tuning. our analysis revealed several intriguing properties of pruning that should inform future work in this direction: (1) pruning dense layers up to ∼ 70% may improve quality for lowfrequency examples in the data and enhance model robustness. (2) the decision to prune embeddings may have critical impact on model robustness to out-of-distribution performance. (3) while lowperforming languages benefit from moderate pruning, they are disproportionately harmed when pruning more aggressively. based on these intriguing properties, we also make several recommendations to machine learning practitioners.