Are Compressed Language Models Less Subgroup Robust? (2023)

Abstract: To reduce the inference cost of large language models, model compression is increasingly used to create smaller scalable models. However, little is known about their robustness to minority subgroups defined by the labels and attributes of a dataset. In this paper, we investigate the effects of 18 different compression methods and settings on the subgroup robustness of BERT language models. We show that worst-group performance does not depend on model size alone, but also on the compression method used. Additionally, we find that model compression does not always worsen the performance on minority subgroups. Altogether, our analysis serves to further research into the subgroup robustness of model compression.

## introduction in recent years, the field of natural language processing (nlp) has seen a surge in interest in the application of large language models (llms) @xcite @xcite . these applications range from simple document classification to complex conversational chatbots. however, the uptake of llms has not been evenly distributed across society. due to their large inference cost, only a few well-funded companies may afford to run llms at scale. to address this, many have turned to model compression to create smaller language models (lms) with near comparable performance to their larger counterparts. the goal of model compression is to reduce a model's size and latency while retaining overall performance. existing approaches such as knowledge distillation @xcite have produced scalable task-agnostic models @xcite @xcite . meanwhile, other approaches have shown that not all transformer * corresponding author: jg717@sussex.ac.uk. ## related works most compression methods belong to one of the following categories: knowledge distillation @xcite , pruning @xcite , or quantization @xcite . additionally, there exists orthogonal approaches specific to lms such as vocabulary transfer @xcite . previous works looking at the effects of model compression have focused on the classes or attributes in images. @xcite analyzed the performance of compressed models on the imbalanced classes of cifar-10, imagenet, and celeba. magnitude pruning and post-training quantization were considered with varying levels of sparsity and precision respectively. model compression is found to cannibalize the performance on a small subset of classes 1 https://github.com/wearepal/ compression-subgroup to maintain overall performance. hooker et al. (2020) followed up by analyzing how model compression affects the performance on sensitive attributes of celeba. unitary attributes of gender and age as well as their intersections (e.g. young male) were considered. the authors found that overall performance was preserved by sacrificing the performance on low-frequency attributes. stoychev and gunes (2022) expanded the previous analysis on attributes to the fairness of facial expression recognition. the authors found that compression does not always impact fairness in terms of gender, race, or age negatively. the impact of compression was also shown to be nonuniform across the different compression methods considered. to the best of our knowledge, we are the first to investigate the effects of model compression on subgroups in a nlp setting. additionally, our analysis encompasses a much wider range of compression methods than were considered in the aforementioned works. ## experiments the goal of learning is to find a function f that maps inputs @xmath0. additionally, there exists attributes @xmath1. the subgroups can then be defined as @xmath2. ## results model size and subgroup robustness. we plot the overall results in figure 1 and note a few interesting findings. first, in multinli and sco-tus, we observe a trend of decreasing average and worst-group accuracies as model size decreases. in particular, tinybert 6 appears to be an outlier in multinli by outperforming every model including bert base . however, this trend does not hold in civilcomments. instead, most compressed models show an improvement in wga despite slight drops in average accuracy. even extremely compressed models like bert t iny are shown to achieve a higher wga than bert base . we hypothesize that this is due to civilcomments being a dataset that bert base easily overfits on. as such, a reduction in model size serves as a form of regularization for generalizing better across subgroups. additionally, we note that a minimum model size is required for fitting the minority subgroups. specifically, the wga of distilled models with layers fewer than 6 (bert m ini , bert t iny , and tinybert 4 ) is shown to collapse to 0 in scotus. second, we further analyze compressed models with similar sizes by pairing distilbert with tinybert 6 as well as post-training quantization (bert dq and bert sq ) with bert qat according to their number of parameters in ## conclusion in this work, we presented an analysis of existing compression methods on the subgroup robustness of lms. we found that compression does not always harm the performance on minority subgroups. instead, on datasets that a model easily overfits on, compression can aid in the learning of features that generalize better across subgroups. lastly, compressed lms with the same number of parameters can have varying performance due to differences in weight initialization after compression.