On the Calibration of Massively Multilingual Language Models (2022)

Abstract: Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer. While there has been much work in evaluating these models for their performance on a variety of tasks and languages, little attention has been paid on how well calibrated these models are with respect to the confidence in their predictions. We first investigate the calibration of MMLMs in the zero-shot setting and observe a clear case of miscalibration in low-resource languages or those which are typologically diverse from English. Next, we empirically show that calibration methods like temperature scaling and label smoothing do reasonably well in improving calibration in the zero-shot scenario. We also find that few-shot examples in the language can further help reduce calibration errors, often substantially. Overall, our work contributes towards building more reliable multilingual models by highlighting the issue of their miscalibration, understanding what language and model-specific factors influence it, and pointing out the strategies to improve the same.

## introduction massively multilingual language models (mmlms) like mbert @xcite , xlmr @xcite , mt5 @xcite and mbart @xcite have been surprisingly effective at zero-shot cross-lingual transfer i.e. when fine-tuned on an nlp task in one language, they often tend to generalize reasonably well in languages unseen during fine-tuning. these models have been evaluated for their performance across a range of multilingual tasks @xcite @xcite and numerous methods like adapters @xcite , sparse fine-tuning @xcite and few-shot learning @xcite have been proposed to further improve performance of crosslingual transfer. despite these developments, there has been little to no attention paid to the calibration of these models across languages i.e. how reliable the confidence predictions of these models are. as these models find their way more and more into the real word applications with safety implications, like hate speech detection @xcite it becomes important to only take extreme actions for high confidence predictions by the model @xcite . hence, calibrated confidences are desirable to have when deploying such systems in practice. @xcite showed that modern neural networks used for image recognition @xcite though perform much better than the ones introduced decades ago @xcite , but are significantly worse calibrated and often over-estimate their confidence on incorrect predictions. for nlp tasks specifically, @xcite showed that classifiers trained using pre-trained transformer-based models @xcite are well calibrated both in-domain and out-of-domain settings compared to non-pre-trained model baselines @xcite . @xcite highlights, since zero-shot cross-lingual transfer represents shifts in the data distribution the point estimates are likely to be miscalibrated, which forms the core setting of this work. in light of this, our work has three main contributions. first, we investigate the calibration of two commonly used mmlms: mbert and xlm-r on four nlu tasks under zero-shot setting where the models are fine-tuned in english and calibration errors are computed on unseen languages. we find a clear increase in calibration errors compared to english as can be seen in figures 1a and 1b , with calibration being significantly worse for swahili compared to english. second, we look for factors that might affect the zero-shot calibration of mmlms and find in most cases that the calibration error is strongly correlated with pre-training data size, syntactic similarity, and sub-word overlap between the unseen language and english. this reveals that mmlms are miscalibrated in the zero-shot setting for low-resource languages and languages that are typologically distant from english. finally, we show that model calibration across different languages can be substantially improved by utilizing standard calibration techniques like temperature scaling @xcite and label smoothing @xcite without collecting any data in the language (see figure 1c ). using a few examples in a language (the few-shot setting), we see even more significant drops in the calibration errors as can be seen in figure 1d . to the best of our knowledge, ours is the first work to investigate and improve the calibration of mmlms. we expect this study to be a significant contribution towards building reliable and linguistically fair multilingual models. to encourage future research in the area we make our code publicly available foot_0 . ## calibration of pre-trained mmlms consider a classifier h : @xmath0, where [k] denotes the set of labels {1, 2, • • • , k}. we assume h can predict confidence of each of the [k] labels and is given by h k (x) ∈ [0, 1] for the k th label. h is said to be calibrated if the following equality holds: in other words, for a perfectly calibrated classifier, if the predicted confidence for a label k on an input x is q, then with a probability q the input should actually be labelled k. naturally, in practical settings the equality does not hold, and neural network based classifiers are often miscalibrated @xcite . one way of quantifying this notion of miscalibration is through the expected calibration error (ece) which is defined as the difference in expectation between the confidence in classifier's predictions and their accuracies (refer appendix a.1 for details). in our experiments we compute ece on each language l's test data and denote their corresponding calibration errors as ece(l). ## experiments we seek to answer the following research questions: a) how well calibrated are fine-tuned mmlms in the zero-shot cross lingual setting? b) what linguistic and model-specific factors influence calibration errors across languages? c) can we improve the calibration of fine-tuned models across languages? ## conclusion in this work we showed that mmlms like mbert and xlmr are miscalibrated in a zero-shot cross lingual setting, with the calibration errors being even worse on low resource languages and languages that are typologically distant from the pivot language (often english). we then demonstrated the effectiveness of standard calibration techniques for improving calibration across languages both with and without collecting any new languagespecific labelled data. we recommend that researchers and practitioners consider, measure and report the calibration of multilingual models while using them for scientific studies and building systems. in future work, we aim to bridge the gap between zero-shot and few-shot calibration methods by exploring unsupervised calibration methods under domain shift @xcite that utilizes unlabelled data in new domains to improve calibration. investigating the cross lingual calibration of mmlms for tasks other than sentence classification like sequence labelling @xcite and question answering @xcite ) is another natural extension of our work.