What Do Compressed Multilingual Machine Translation Models Forget? (2022)

Abstract: Recently, very large pre-trained models achieve state-of-the-art results in various natural language processing (NLP) tasks, but their size makes it more challenging to apply them in resource-constrained environments. Compression techniques allow to drastically reduce the size of the models and therefore their inference time with negligible impact on top-tier metrics. However, the general performance averaged across multiple tasks and/or languages may hide a drastic performance drop on under-represented features, which could result in the amplification of biases encoded by the models. In this work, we assess the impact of compression methods on Multilingual Neural Machine Translation models (MNMT) for various language groups, gender, and semantic biases by extensive analysis of compressed models on different machine translation benchmarks, i.e. FLORES-101, MT-Gender, and DiBiMT. We show that the performance of under-represented languages drops significantly, while the average BLEU metric only slightly decreases. Interestingly, the removal of noisy memorization with compression leads to a significant improvement for some medium-resource languages. Finally, we demonstrate that compression amplifies intrinsic gender and semantic biases, even in high-resource languages.

## introduction over the recent years, pre-trained transformer @xcite models have reached a substantial improvement in a variety of natural language processing (nlp) tasks. this improvement mostly comes from increasing their parameter size @xcite @xcite which escalates the cost of training @xcite @xcite , and hurts the memory footprint and latency at inference @xcite @xcite . specially in neural machine translation (nmt) task, massively mnmt models @xcite @xcite ) demonstrated promising results. they have been shown particularly interesting for low-resource languages which benefit a lot from knowledge transfer. on the other hand, it has also been observed that the curse of multilinguality may hurt the performance in high-resource languages. the strategy employed to overcome this problem @xcite @xcite is to scale up the number of parameters, thus attaining state-of-the-art performance in both high and low-resource languages. consequently, efficient inference with these very large models has become a crucial problem. this challenge can be overcome through model compression, e.g. knowledge distillation @xcite @xcite , pruning @xcite @xcite , and quantization @xcite @xcite @xcite @xcite . these methods can be applied with a little loss in top-line metrics, while reducing the memory-footprint, and enhancing inference time. however, recent work @xcite @xcite @xcite has demonstrated that under-represented features can suffer from a drastic decrease in performance which is not necessarily reflected by global (aggregated) metrics. in multilingual nmt, the overall metrics are often reported as an average across all the language pairs, where the performance between individual language pairs can vary a lot. therefore it is even more critical to understand what would be the exact impact of compression on multilingual nmt models, beyond the aggregated metrics. in this work, we illustrate the impacts of applying compression methods to massively multilingual nmt models, that are pre-trained in a great number of languages in several domains. to the best of our knowledge, this is the first attempt to analyze how compression impacts massively multilingual models. we hope it could be a starting point to bringing a comprehensive understanding between fairness and compression in multilingual nmt models. in this study, we concentrate on light compression techniques, specifically post-training quantization and magnitude pruning without any further finetuning. 2 we exploit the recent and largest mnmt model, @xcite ) that covers 100 languages and contains nearly 12b parameters and analyze the impact of compression on different language pairs evaluated on flores-101 benchmark @xcite (covering 101 languages). we also consider @xcite and @xcite benchmarks allowing us to assess different types of biases that could be present in the data and mnmt model. to sum up, our contributions are as follows: in section 2, we describe light compression methods we rely on, and mnmt model. section 3 presents our experimental setup and evaluation benchmarks. section 4 shows the analysis of the impact of the compression for nmt benchmarks. ## related work the first connection between compression and bias amplification has been made by @xcite in the case of image classification. the same authors proposed an approach to find a subset of the dataset which contains samples that have disproportionately high errors after the compression. there is also recent work that analyzes the effect of compression on pre-trained language models @xcite @xcite . notably, @xcite demonstrated a higher gender bias in compressed pre-trained language models. concerning nmt, 23 detailed results are provided in appendix k. while @xcite demonstrated that optimization of inference speed up may result in gender bias amplification. to the best of our knowledge, this work is the first in-depth study of the impact of compression on massively multilingual models. we hope our findings would encourage further research on this topic. ## conclusion we demonstrate the impacts of applying compression methods to the massively multilingual machine translation models by evaluating compressed models on flores-101 @xcite , gender bias benchmark @xcite , and word sense disambiguation benchmark @xcite . we show that while average bleu drops negligibly, the performance of under-represented language pairs drops drastically. interestingly, sparsity improves the performance of some medium-resource language pairs by removing the noisy memorization. by evaluating our compressed models on gender bias and word sense disambiguation benchmarks, we show that the compression amplifies the intrinsic gender and semantic biases, even in high-resource language pairs. we hope our findings could be a starting point to consider the fairness aspects when compressing multilingual models.