Deep Model Compression Also Helps Models Capture Ambiguity (2023)

Abstract: Natural language understanding (NLU) tasks face a non-trivial amount of ambiguous samples where veracity of their labels is debatable among annotators. NLU models should thus account for such ambiguity, but they approximate the human opinion distributions quite poorly and tend to produce over-confident predictions. To address this problem, we must consider how to exactly capture the degree of relationship between each sample and its candidate classes. In this work, we propose a novel method with deep model compression and show how such relationship can be accounted for. We see that more reasonably represented relationships can be discovered in the lower layers and that validation accuracies are converging at these layers, which naturally leads to layer pruning. We also see that distilling the relationship knowledge from a lower layer helps models produce better distribution. Experimental results demonstrate that our method makes substantial improvement on quantifying ambiguity without gold distribution labels. As positive side-effects, our method is found to reduce the model size significantly and improve latency, both attractive aspects of NLU products.

## introduction datasets constructed for natural language understanding (nlu) tasks, such as natural language inference (nli) and text emotion analysis, contain a large amount of ambiguous samples @xcite . as exemplified in since transformer-based @xcite ) pre-trained language models (plms) @xcite have become popular for nlu tasks, the accuracies of various nlu models have been substantially improved. nevertheless, they are still not good at approximating the human opinion distributions @xcite , or label distributions drawn from a larger number of annotators, and their predictions tend to be over-confident @xcite . if nlu products frequently produce overconfident predictions for ambiguous samples, it is not likely that they would be reliable for users who have different opinions. as an attempt to address this problem, previous work @xcite has demonstrated that label smoothing @xcite helps make the prediction distributions close to human opinion distributions, simply addressing the issue of over-confidence. however, this does not explicitly address how to exactly capture the degree of relationship between each sample and its candidate classes (i.e., how to estimate p(@xmath0. some researchers @xcite @xcite have tried to use empiricallygold label distributions for directly learning the relationship, but these approaches require significant additional annotation costs. in this paper, we propose a novel method that employs compression techniques for deep learning models, namely layer pruning @xcite and knowledge distillation (kd) @xcite , and show how these compression techniques help models capture such a degree of relationship. we first observe that hidden states in lower layers more accurately encode the information about the sample-classes relationship, and that validation accuracies from internal classifiers inserted between adjacent layers are converging. this indicates that pruning a part of higher layers can make the models well represent the relationship information with their prediction distribution, while retaining the accuracy. we also observe that transferring the distribution knowledge that represents more accurate information about the relationship from a lower layer into the final classifier at the top of the pruned network can help the models produce better distribution. experimental results demonstrate that our method significantly outperforms existing ones that do not use additional distribution datasets. without using such additional resources, our method also outperforms, or is comparable with, those that do use these resources over nli benchmarks. moreover, since our method uses compression techniques for deep learning models, this also reduces the model size significantly and improves latency as well. both are attractive aspects of nlu products because they lead to consequent reduction in the cloud cost or to deployment on cheaper ondevices. deep model compression aims at eliminating redundant components of pre-trained deep learning models (via pruning or low-rank factorization @xcite ) to improve latency and reduce the model size. at the same time, maintaining the performance of the original model (via kd) is essential. while the goal of compression itself is not directly relevant to capturing ambiguity, we demon-strate that compression methods can also be used for accurately capturing ambiguity and suggest that such an approach presents another novel research direction for this task. ## related work recent work has revealed that the state-of-the-art plms fine-tuned to predict single gold labels with cross-entropy loss function fail to properly estimate human opinion distributions @xcite and tend to produce over-confident predictions @xcite . this issue of over-confidence is well-known in modern complex deep neural networks, because they can easily overfit one-hot labels of a training dataset. moreover, this issue arises regardless of the correctness of predictions @xcite . in a situation where there exist a large number of ambiguous samples in an nlu dataset, it does not make sense to tolerate over-confident predictions. naturally, in order to obtain better human opinion distributions, the use of label smoothing @xcite has been proposed @xcite . label smoothing softens target training label distributions (i.e., one-hot labels) by shifting α probability mass from the target labels equally to all the labels. as a result, it prevents models from overfitting one-hot distribution. @xcite and @xcite have shown that label smoothing is effective at better estimating human opinion distributions. however, it makes all predictions less-confident, compared with using one-hot labels, not considering how to capture the degree of relationship between each sample and its candidate classes, which is an essential aspect to address ambiguity. monte carlo dropout (mc dropout) @xcite addresses the drawback of label smoothing. for a given sample, this method makes k stochastic forward passes from a pre-trained neural network with dropout, where k different prediction distributions are then averaged to form a final distribution for the sample. since different forward passes could produce different plausible predictions for ambiguous samples, mc dropout also captures the aforementioned degree of relationship. using mc dropout also improves the quality of output distributions @xcite , but this suffers from several drawbacks, such as its non-deterministic nature and higher latency for inference. directly learning from human opinion distributions has also been studied. zhang et al. (2018) and meissner et al. (2021) trained models with the empirically-gold label distributions to match predictions and human opinion distributions. as postediting, zhang et al. (2021), wang et al. (2022), and zhou et al. (2022) used temperature scaling @xcite , with which output logits from a fine-tuned model are rescaled with hyperparameter t , and the softmax distribution becomes accordingly smoother and closer to target distributions. t is tuned on the distribution labels from a validation set by minimizing the kl-divergence between the predicted distributions and human opinion distributions. these additional resources significantly improve the ability to quantify ambiguity, but are accompanied with enormous annotation costs. we propose to address all these limitations, considering how to exactly capture the degree of relationship between each sample and its candidate classes without the need for extra resources. in the next section, we explain how deep model compression can be made to account for the relationship without additional human opinion distribution information. ## experiments in this section, we investigate how exactly our method of using compression techniques can capture the ambiguity of each sample without empirically-gold label distributions. we re-ran all experiments three times with different random seeds to identify variance. the standard deviation value of accuracy is smaller than 0.0155 on all methods and datasets and of jenson-shannon distance (jsd) @xcite is smaller than 0.0081, both of which are negligible. ## discussion which compression method is more effective? as described in can our observations be reproduced over different datasets? the design of our proposed methods is based on our three observations in section 3.1. if these do not manifest in other datasets, our method may not work in general. therefore, we conducted the same procedure chaossnli chaosmnli emotion diff.↓ jsd↓ diff.↓ jsd↓ diff.↓ jsd↓ std 0.6092 0.3299 0.5749 0.4219 0.3987 0.4203 mc 0.5476 0.2984 0.5187 0.3718 0.3711 0.4044 ls 0.5753 0.2723 0.5469 0.3540 0.3850 0.4057 ts 0.5342 0.2626 0.4957 0.3095 0.3663 0.3859 ldl 0.4819 0.2185 0.4435 0.2991 0.2866 0.3338 pruning 0.5997 0.3197 0.5614 0.4091 0.3791 0.4069 +kd 0.5265 0.2635 0.4686 0.2799 0.3610 0.3935 5 : the degree of improvement when our method is applied to the fine-tuned models with label smoothing described in section 3.1 on the datasets used in our experiments. as described in figure 3 , in these datasets, we observed that the validation accuracy is starting to converge in the lower layers (observation (1)). we also found that feature representations from a lower layer contain richer information about the degree of relationship (see figure 5 in appendix a). we also observed again that after the most rapid drop of entropy value occurs, models started to converge as described in figure 3 . these suggest that our method can be applied to various other nlu datasets as well. is maximizing the probabilities for ground truth labels when predictions are incorrect a valid solution for tuning λ of our kd loss? in order to validate the tuning approach, we measure the average difference between the ground truth probabilities and predicted probabilities for the ground truth labels as described in can models trained with label smoothing be improved with our proposed method? since our method is applied to a fine-tuned model, we looked into whether the proposed method can further improve the estimation ability for human opinion distributions on models trained with label smoothing. in this case, we used smoothed labels instead of one-hot labels y during knowledge distillation. as shown in what are additional benefits of our method? in our experiments, 1 layer is pruned for the emotion analysis model and 3 layers are pruned for the nli models. these result in significant reduction in the number of model parameters (from 125m (roberta-base) to 117m and to 103m, respectively). we also measured the average latency per 300 token input on a low-end mobile device (i.e., samsung galaxy tab s6 lite). the pruned network is also found to significantly reduce the latency on the mobile device (from 2.42 sec. to 2.22 sec. and to 1.86 sec., respectively). ## conclusion in this work, we proposed a novel method for capturing ambiguity with deep model compression techniques, namely layer pruning and knowledge distillation. experimental results demonstrate that our method substantially improves the ability of quantifying ambiguity and provides efficient compressed models for nlu products. as future work, we would further investigate the availability of different compression methods such as pruning self-attention heads and fnn because redundant components in modern complex deep learning may lead to over-confidence @xcite . in another direction, we may also address limitations that are revealed in our work, such as multiple training procedures or hyperparameter tuning for each method (e.g., how much we allow accuracy drop during layer pruning).