A Comparative Study on the Impact of Model Compression Techniques on Fairness in Language Models (2023)

Abstract: Compression techniques for deep learning have become increasingly popular, particularly in settings where latency and memory constraints are imposed. Several methods, such as pruning, distillation, and quantization, have been adopted for compressing models, each providing distinct advantages. However, existing literature demonstrates that compressing deep learning models could affect their fairness. Our analysis involves a comprehensive evaluation of pruned, distilled, and quantized language models, which we benchmark across a range of intrinsic and extrinsic metrics for measuring bias in text classification. We also investigate the impact of using multilingual models and evaluation measures. Our findings highlight the significance of considering both the pre-trained model and the chosen compression strategy in developing equitable language technologies. The results also indicate that compression strategies can have an adverse effect on fairness measures.

## introduction despite their increasing popularity, machine learning models have been known to exhibit biases in their outputs, present privacy risks, and have potentially negative environmental consequences from their training and deployment. @xcite . language models suffer from biases that result in unequal resource distributions (allocational harms), in addition to the undesired tendency to reproduce biases and stereotypes in content that is reflective of hegemonic worldviews (representational harms). although measures have been proposed in tasks such as text classification @xcite to investigate the disparate allocational treatment of different classes, much of the research on fairness in language models centers on addressing representational harms * equal contribution @xcite . the potential of these models to further stigmatize marginalized communities is demonstrated in @xcite , which illustrates how recidivism prediction systems are biased against black defendants, who have a higher baseline risk for repeat offences. biases are also prevalent in computer vision applications such as facial recognition technologies. within nlp, @xcite , one of the first forays that studied this phenomenon in language, noted that word embeddings contained stereotypical associations with respect to gender. language models can exhibit biases toward different dialects for tasks like toxicity and hate speech detection @xcite , generate stereotypical representations and narratives @xcite , and are capable of the outright erasure of underrepresented identities @xcite . compressed models that are biased may have detrimental consequences in the real world, as they are typically deployed on edge devices, which can further disadvantage communities without access to other forms of technology. consequently, these issues have compelled a shift towards developing more inclusive systems. @xcite demonstrates how compression techniques, when applied to models that deal with tabular data, lead to the disparate treatment of less-represented classes. however, equivalent studies in nlp @xcite @xcite do not provide a conclusive observation as to whether compression methods are effective for reducing bias in nlp, and are centered mainly solely around model distillation being the compression technique of choice. this paper aims to resolve the following questions by benchmarking a wide range of metrics and datasets to study bias in text classification systems. talat et al. (2022); orgad and belinkov (2022); field et al. (2021); blodgett et al. (2020) provide critical insights into the current state of fairness in nlp and delve into the details of what research studies must consider when conducting work in this area. the discussion thus far concerning fair-ness, in general, has mainly been anglo-centric, but recent forays @xcite @xcite have explored bias in multilingual spaces and languages beyond english. in the context of model compression, @xcite show that while larger models produce fewer gendered errors, they produce a greater proportion of gendered errors in coreference resolution whilst @xcite suggest that distillation and pruning have a regularizing effect that mitigates bias in text classification. on the other hand silva et al. (2021); ahn et al. (2022); hessenthaler et al. (2022) all demonstrate how distillation can have an adverse impact on model fairness. @xcite strongly casts doubt on the results from @xcite by showing that knowledge distillation decreases model fairness. additionally, the findings from @xcite point toward the fact that pruning can amplify bias in multilingual machine translation models. it must also be noted that with the exception of @xcite ; @xcite ; @xcite ; @xcite , many of these studies do not validate the fairness of these models over downstream tasks. this is essential as bias measurements over a model's pretrained representations cannot be used as a proxy to assess bias in its downstream outputs @xcite . @xcite ; @xcite explore the efficient debiasing of models via the use of adapters and an adapted form of distillation, respectively. to our knowledge, our work is the first comprehensive study on fairness in nlp with respect to pruning, distillation and quantization, in addition to which it addresses both monolingual and multilingual models. ## extrinsic measures for extrinsic measurement over downstream tasks, we have used multiple datasets with different fairness definitions (details in the other monolingual extrinsic measure includes the african american vernacular english-standard american english (aave-sae) dataset @xcite , which consists of intent-equivalent sae and aave sentence pairs. @xcite has shown that aave language is more likely to be identified as hate speech compared to the standardized form of american english. a fair, unbiased model on this data would produce similar sentiment scores for both aave and sae. we have also included the results for the equity evaluation corpus (eec), a templatebased dataset that evaluates the emotional intensity of sentiment classification systems over four categories of data-anger, fear, sadness, and joy, in the appendix (section c.1). ## multilingual datasets to investigate whether the observed trends in a monolingual setting extend to a multilingual scenario, we conducted experiments using a separate set of models, with information about their size ## conclusion in this work, we conduct a comprehensive evaluation of fairness in compressed language models, covering multiple base models, compression techniques, and various fairness metrics. while prior studies have evaluated the fairness of compressed models, the results have not always been conclusive. in contrast, our extensive benchmarking provides evidence that challenges recent research suggesting that model compression can effectively reduce bias through regularization, and we demonstrate that this is the case for both multilingual and monolingual models across different datasets. the compression of language models through distillation, quantization, and pruning is crucial for the practical use of language technologies in various real-world applications. while it is essential to preserve performance during compression, it is equally imperative that the compressed versions of language models maintain or even enhance fairness measures to avoid potential harm.