On the Effects of Transformer Size on In- and Out-of-Domain Calibration (2021)

Abstract: Large, pre-trained transformer language models, which are pervasive in natural language processing tasks, are notoriously expensive to train. To reduce the cost of training such large models, prior work has developed smaller, more compact models which achieves a significant speedup in training time while maintaining competitive accuracy to the original model on downstream tasks. Though these smaller pre-trained models have been widely adopted by the community, it is not known how well are they calibrated compared to their larger counterparts. In this paper, focusing on a wide range of tasks, we thoroughly investigate the calibration properties of pre-trained transformers, as a function of their size. We demonstrate that when evaluated in-domain, smaller models are able to achieve competitive, and often better, calibration compared to larger models, while achieving significant speedup in training time. Post-hoc calibration techniques further reduce calibration error for all models in-domain. However, when evaluated out-of-domain, larger models tend to be better calibrated, and label-smoothing instead is an effective strategy to calibrate models in this setting.

## introduction large pre-trained transformer language models like bert @xcite have revolutionized natural language processing, achieving state-of-the-art results in several tasks. the process of applying these models on a downstream task consists of two components: (1) self-supervised pre-training on a large amount of text corpora and (2) supervised fine-tuning on the downstream task. due to the very large number of parameters of such transformer based architectures, the high downstream accuracies comes at a large computational cost @xcite during http://cogcomp.org/page/publication_view/953 the pre-training stage and also to a lesser extent, while fine-tuning. to alleviate this computational cost, several models with fewer parameters have been proposed that significantly speed-up both the pre-training and the fine-tuning stages @xcite @xcite . for example, the smallest model in @xcite consists of only 4 million parameters compared to bert-base which has 110 million parameters; this leads to a 65x speedup for pretraining time. it has been widely observed @xcite ) that smaller models achieve comparable downstream task performance with a very significant speedup in training time. a second issue with pre-trained models with a massive number of parameters, is their lack of calibration, which measures how well the model confidences (posterior probabilities) are aligned with the empirical likelihoods. in other words, for a calibrated model the probability associated with the predicted class label should reflect its ground truth correctness likelihood. importantly, in the seminal work of @xcite , the authors demonstrate that for deep neural architectures increasing model size negatively affects its calibration, even though classification accuracy increases. in this paper, we extend this to investigate the dependence of calibration on model size for pre-trained transformer models. since miscalibrated models can make very confident predictions even when they make errors, especially on out-of-distribution data @xcite , it is crucial to carefully study model calibration. recently, there has been some progress on studying the calibration of deep neural networks and specifically, pre-trained transformers @xcite @xcite . however, a careful study of how the size of the pre-trained model influences calibration is lacking. with the computational constraints of training large transformers like bert and the increasingly wide adoption of smaller mod-els, it becomes essential to study the calibration of these variants. in this work, we make a thorough empirical study of the calibration properties of smaller transformer architectures of the bert family, for a wide set of classification tasks. the set of models have rich variations over number of layers, number of hidden neurons and embedding representation. additionally, we analyze the effects of techniques designed to help calibrate models: during training (eg: label smoothing) and post-hoc (eg: temperature scaling), on the smaller models, for both in-and out-of-domain datasets. we establish the following results in this paper: 1. when evaluated in-domain, smaller models are as well calibrated as bert-base, both with and without temperature scaling. 2. when evaluated out-of-domain, smaller models are worse calibrated than bert-base. this persists, to a lesser extent, even after temperature scaling. 3. label smoothing, on the other hand, is not effective in-domain, but helps smaller models attain better calibration than bert-base out-of-domain. it also helps improve accuracy as compared to the non-smoothed models, on out-of-domain data. ## background in this section, we describe how we measure calibration and two techniques that help calibrate models: temperature scaling and label smoothing. calibration metric: let us define the following notation: k is the number of classes, z i denotes the raw logits from the model for the i th example and σ (k) denotes the k th value of the softmax layer σ, corresponding to the probability for the k th class (for @xmath0, ..., k]). then, the confidence on the a model is well calibrated if the confidence on a prediction is aligned with the accuracy on that prediction, in expectation. the widely adopted expected calibration error (ece) metric @xcite measures exactly this: difference in expectation between confidence and accuracy. empirically this is approximated by dividing the data into m confidence based bins, i.e., b m (where @xmath1, 2, ..., m }) contains all datapoints i for which if acc(b m ) and conf (b m ) denotes the average accuracy and prediction confidence for the points in b m , ece is defined as: k , ∀i, which is the uniform distribution with maximum uncertainty. as @xmath2, the probability collapses to a point mass (q @xmath3, p @xmath4. the optimal temperature t is tuned on the dev-set by a line search algorithm. label smoothing @xcite leads to a modified fine-tuning procedure to address overconfident predictions. while maximum likelihood estimation (mle), sharpens the model's posterior distribution around the target labels, label smoothing introduces uncertainty to smoothen the posterior over the labels. label smoothing constructs a new target vector from the one-hot target vector, with a probability of 1 -α on the target label and α k-1 on all the other labels. then, in the standard manner, the cross entropy loss is minimized between the model predictions and the modified target vectors. label smoothing has been shown to implicitly calibrate neural networks @xcite and @xcite ) have shown it is effective for calibrating models on out-of-distribution data. ## conclusion we presented a thorough empirical study of the effects of model size (number of parameters) on calibration. through experiments on a number of tasks, we demonstrated that smaller transformer models are as well, and sometimes better, calibrated compared to bert-base, when evaluated in-domain. on out-of-domain evaluation, larger models are better calibrated, out-of-box. label-smoothed models are better calibrated and more accurate, on out-of-domain data, with smaller models benefiting more from label smoothing. alex wang, amanpreet singh, julian michael, felix hill, omer levy, and samuel bowman. 2018. glue: a multi-task benchmark and analysis platform for natural language understanding. in proceedings of the 2018 emnlp workshop blackboxnlp: analyzing and interpreting neural networks for nlp, pages 353-355. alex warstadt, amanpreet singh, and samuel r bowman. 2018. neural network acceptability judgments. arxiv preprint arxiv:1805.12471. adina williams, nikita nangia, and samuel r bowman. 2017. a broad-coverage challenge corpus for sentence understanding through inference. arxiv preprint arxiv:1704.05426. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, rémi louf, morgan funtowicz, et al. 2019. huggingface's transformers: state-of-the-art natural language processing. arxiv preprint arxiv:1910.03771.