Towards Fair Dataset Distillation for Text Classification (2022)

Abstract: With the growing prevalence of large-scale language models, their energy footprint and potential to learn and amplify historical biases are two pressing challenges. Dataset distillation (DD) — a method for reducing the dataset size by learning a small number of synthetic samples which encode the information in the original dataset — is a method for reducing the cost of model training, however its impact on fairness has not been studied. We investigate how DD impacts on group bias, with experiments over two language classification tasks, concluding that vanilla DD preserves the bias of the dataset. We then show how existing debiasing methods can be combined with DD to produce models that are fair and accurate, at reduced training cost.

## introduction training and inference with deep neural networks is generally costly in terms of storage and computing resources. model compression methods such as knowledge distillation @xcite and pruning @xcite are popular ways of reducing model size to make inference more efficient. an alternative approach is dataset distillation ("dd": @xcite al. (2018) ), which compresses the training set into a small synthetic dataset. although there is significant pre-cost associated with dd in learning synthetic instances, dd has been shown to achieve almost identical performance to training over the original training set @xcite , at reduced computational cost. for example, sucholutsky and schonlau show that, on the imdb binary sentiment classification dataset @xcite , a randomly initialized model trained over the distilled dataset with 10 50 100 150 200 250 size of the distilled dataset 0.75 0.80 0.85 accuracy fairness figure 1 : performance and fairness over an occupation classification task, using models trained over the original (dotted lines) and distilled datasets (dashed lines) with different data sizes. for both metrics, larger is better. see section 3.2 for full results. it is well known that naively-trained models learn and amplify dataset biases, potentially leading to discrimination such as opportunity inequality @xcite . with the potential for compression methods to reduce training and storage cost, it is important to study their impact on model fairness. here, we take dd as a case study and ask: (a) how does dd impact fairness; and (b) can we ensure fairness within this paradigm? specifically, we investigate how dd impacts on group fairness, and present experiments over two fairness benchmark datasets for text classification. our contributions are: (1) we show that while dd preserves model performance, it also retains the dataset bias (e.g., figure 1 ); and (2) we combine bias mitigation approaches with dd, and show that they improve fairness substantially. ## conclusion this paper evaluated the effect of dataset distillation on fairness in the context of two text classification tasks. empirically, we showed that distilled datasets retain unwanted biases. in order to learn fairer synthetic datasets, we employ adversarial learning and balanced training for bias mitigation, which results in substantial fairness improvements. we conclude that dd can be effective for fair and efficient text classification, specifically for simpler tasks where a small distilled data set is sufficient.