Too Brittle to Touch: Comparing the Stability of Quantization and Distillation towards Developing Low-Resource MT Models (2022)

Abstract: Leveraging shared learning through Massively Multilingual Models, state-of-the-art Machine translation (MT) models are often able to adapt to the paucity of data for low-resource languages. However, this performance comes at the cost of significantly bloated models which aren’t practically deployable. Knowledge Distillation is one popular technique to develop competitive lightweight models: In this work, we first evaluate its use in compressing MT models, focusing specifically on languages with extremely limited training data. Through our analysis across 8 languages, we find that the variance in the performance of the distilled models due to their dependence on priors including the amount of synthetic data used for distillation, the student architecture, training hyper-parameters and confidence of the teacher models, makes distillation a brittle compression mechanism. To mitigate this, we further explore the use of post-training quantization for the compression of these models. Here, we find that while Distillation provides gains across some low-resource languages, Quantization provides more consistent performance trends for the entire range of languages, especially the lowest-resource languages in our target set.

## introduction while nlp has made giant strides in producing more accurate models, these benefits are often not transferred representatively to end-users who would eventually use a language-technology (ethayarajh and jurafsky, 2020; @xcite . bloated sizes, cumbersome inference times @xcite and a limited set of languages that these models serve are a few reasons for this. more specifically, their usage is hindered by access bottlenecks such as (a) infrastructural obstacles: a large percentage of end-users do not have sustained access to internet or high-compute devices to enjoy a stable access to cloud-inferencing of current nlp models @xcite , (b) latency requirements: certain nlp services (chat-bots, real-time assistance interfaces, etc.) require very low-inference time which requisite lightweight-models (c) privacy constraints: the outflow of sensitive user data which is fed for inferencing to remotely hosted nlp models also has well documented issues @xcite @xcite . within the research that focuses on evaluating and mitigating these practical constraints, the focus on low-resource language setups has been fairly limited @xcite . for instance, while the compression of large language models has received consistent attention through analysis of pruning @xcite , distillation @xcite @xcite and even quantization @xcite -much of this work has focused on compressing language models for high-resource languages. in this paper, we report the results of a comparative analysis of the performance of distillation and quantization. by focusing on compressing seq2seq multilingual models across a range of languages with data ranging from 7000 to 3m samples -we especially demonstrate the different priors that need to be ascertained for the successful distillation of the model. we are unaware of any previous study that demonstrates the performance of these mechanisms on such low resource languages. the utility of this work is in commenting on the feasibility of these two compression techniques for rapid development and deployment of mt models for low resource languages @xcite . more specifically, we believe that distillation's reliance on several priors can be addressed naively through a resource-intensive exercise, where the optimal values of these priors are computed exhaustively. however, in the absence of such a budget, we expect this to be a major impediment in the development of lightweight models for such languages. since low resource language communities may also be marginalised in other ways, exhaustive investment of data and compute might not be feasible for such communities as well as the language technologists working on these languages @xcite @xcite . the main contributions of this work are: based on our analysis, we conclude that the suprising stability of naive post-training quantization, especially in the compression of extremely-low resource languages (training data between 5000 and 25000 samples) over distillation. we release a combination of lightweight, offline support mt models for these languages along with the scripts for generation and offline inference to further reproducible research in this domain 1 . ## approach -model and size adaptations in this section, we describe the languages (2.1), architectures under consideration (2.1), the adaptations that we make for training and fine-tuning these models (2.2) and the adaptations we make to compress their size. ## results in section 4.1, we present the performances of our base models in while answering these questions, we also analyze in parallel the performance of the quantized variants of these models implicitly indicating the reduced sensitivity of these variants from most of the previously discussed priors in spite of their competitive performances. ## related work owing to the known benefits of compressing language models due to their lower-memory footprint, improved inference speed and even improved performance in some cases, compression techniques have been explored widely in nlp. quantization while the work on quantizing encoder-models is replete @xcite @xcite the focus on quantizing decoder-only models @xcite , and specifically seq2seq models has been relatively much lower. recent work like, edgeformer, @xcite , llm.int8() @xcite have recently demonstrated the generation of seq2seq quantized models which provide a high-compression ratios and competitive performances though this work has also been done with much higher resource languages. distillation work within distillation is replete, even for the multilingual-type of models that we focus on. work like kaliamoorthi et al. (2021); jiao et al. (2021); yang et al. (2022) represent the major body of work in multi-lingual distillation -that is also centered across the encoder-only space. relatively lesser work has been done in the space of mutli-lingual distillation (soltan et al., 2021; mukherjee et al., 2021) of seq2seq models and even though work like zhang et al. (2020); he et al. ( 2019 ) extends this analysis to relatively low-resource languages, they rely on the use of monolingual data for the target language, a luxury that we cannot afford for half of the languages in our language set. note that since both processes are orthogonal, their conjunctive use has also been explored - @xcite for instance, get competitive results by applying token level contrastive distillation and module-wise dynamic scaling while quantizing generative models. note that we made the conscious decision of excluding pruning from our analysis because while it is known to demonstrate very effective parameter reduction, it is generally not as aggressive in it's memory footprint reduction as much as quantization and distillation @xcite . as we'll discuss further in section 7, size-reduction was an implicit focus of this work that is one of the most fundamental bottlenecks of community deployment a.4. ## discussion while this work explicitly focuses on only the performance comparison between distillation and post-training quantization, it's efficacy can also be viewed in demonstrating the development of lightweight, machine translation models for extremely low-resource languages. this is a very critical outcome as performance-oriented machine translation (mt) models for low-resource languages are often not suited for the immediate consumption of the community. the access bottleneck introduced by these bloated models, can especially affect those communities which haven't traditionally enjoyed access to a digital ecosystem, often widening the gap between those who can and cannot access these tools. towards this direction, the exploration of compression strategies for these models -especially when tied to end-user centric nlp services such as translation is imperative. in this work, the size of all models being evaluated after compression was less than 400mbthe quantized models are at least 3x lighter the size of the native hm and the distilled models give even more impressive gains of upto 8x smaller than their uncompressed counterparts. this size reduction, coupled with the increased speed of inference associated with this reduction in most cases can enable a suite of accessible translation models for these languages foot_3 . this establishes a very promising potential in achieving deployment-constraint aware models: for instance, in areas where users do not enjoy a sustained access to the internet -these lightweight models may be adapted to operate on edge in an offline fashion. ## conclusion and future work in this work we established that hard-distillation is sensitive to several priors which makes it a brittle mechanism of compression, especially for languages with extremely low-resources. in relative comparison, post-training quantizaton provides a competitive, stable and cost-effective compression mechanism that works effectively for extremely low-resource languages as well. moving forward, we wish to explore the effect of using additional data (augmented or natively available) on the compressed variants of these models and extend distillation's analysis to utilizing logit distributions of the teacher (soft-distillation). having observed the poor confidence measures of the transformer -and it's relatively random distributions we expect to get more interpretable evidence towards the suitability of these models for soft distillation through such an analysis. the use of continual pretraining with monolingual data has been shown to be very useful in improving the transfer for low-resource languages. in our cases, our lowest resource languages, i.e, bribri, wixarica, gondi and mundari, did not have any monolingual data available natively so we explored the augmentation of the same using lexicons @xcite . we also generated forward translated data using the hm that we developed to fuse with the lexicon-adapted data. for continued pretraining we use a fixed learning rate of 0.001. results of our experiments are logged in we did observe a small gain in performance upon the addition of la data during pretraining -though the post-quantization performance and the distilled model' significant performance degradation called for a deeper investigation on the effects of continued pretraining for this language.