Calibrating Zero-shot Cross-lingual (Un-)structured Predictions (2022)

Abstract: We investigate model calibration in the setting of zero-shot cross-lingual transfer with large-scale pre-trained language models. The level of model calibration is an important metric for evaluating the trustworthiness of predictive models. There exists an essential need for model calibration when natural language models are deployed in critical tasks. We study different post-training calibration methods in structured and unstructured prediction tasks. We find that models trained with data from the source language become less calibrated when applied to the target language and that calibration errors increase with intrinsic task difficulty and relative sparsity of training data. Moreover, we observe a potential connection between the level of calibration error and an earlier proposed measure of the distance from English to other languages. Finally, our comparison demonstrates that among other methods Temperature Scaling (TS) generalizes well to distant languages, but TS fails to calibrate more complex confidence estimation in structured predictions compared to more expressive alternatives like Gaussian Process Calibration.

## introduction while deep neural networks, especially large pretrained language models, have driven striking improvements on various standard benchmarks @xcite , it is never a good practice to assume their predictions are accurate and should be taken blindly. in many cases, it is important to understand "what a model does not know" through its estimation of its uncertainty. for example, reliable model confidence is important in high-stakes domains @xcite , or when downstream tasks leverage confidence scores to mitigate error propagation @xcite . moreover, accurate confidence can serve as a measure of the value of information in iterative data collection or human-in-the-loop learning @xcite . whether the model confidence is accurate is usually measured by how well it matches the observational data -through confidence calibration @xcite . yet modern neural networks are criticized for being overconfident with their predictions, given their increased capacity to fit the training dataset @xcite . this problem is exacerbated by domain-shift @xcite or zero-/few-shot transfer @xcite ). an important task that is often concerned with such data shift is zero-shot cross-lingual transfer, which has been viewed as a natural extension to domain adaptation @xcite . existing studies in natural language processing have mainly focused on zero-shot transfer accuracy alone @xcite @xcite , without concern for the uncertainty measures of massive cross-lingual pre-training models @xcite @xcite . on the other hand, large-scale uncertainty estimation and calibration work have mostly been conducted in the vision domain @xcite . large-scaled calibration studies put predominant importance on computer vision. in natural language processing, while model calibration has wide applications w.r.t tasks such as text classification @xcite , seq2seq generation @xcite @xcite , question answering @xcite and zeroshot learning @xcite , benchmarking results are not as comprehensive as in vision. in this work, we evaluate how the calibration of large-scale multilingual models is affected by the zero-shot cross-lingual transfer, and whether we might mitigate calibration error with standard techniques relying solely on the source language foot_0 . we conduct our experiments on six standard crosslingual transfer tasks across seven typologically diverse target languages, using english as the annotated source language. our key findings include: calibration of large scale models noticeably, @xcite ; @xcite have produced large-scale benchmarks over a variety of tasks and existing calibration methods with mixed results. while empirically @xcite shows that the traditional post-training calibration methods such as temperature scaling do not always transfer under domain shift, results from @xcite indicates that there is a correlation between in-domain and out-of-domain calibration error for models with large capacities like vit @xcite , and that model calibration decreases more slowly than accuracy. in @xcite shows that pretrained transformer models achieve better calibration and that temperature scaling further reduces calibration error in-domain. @xcite demonstrates that the benefit of the pretrained model diminishes as the domain shift increases. our work extends these analyses to model calibration under zero-shot cross-lingual transfer. calibration of structured prediction calibration of structured prediction models is relatively under-explored, due to the difficulty in defining the calibration setting @xcite . jagannatha and yu (2020) proposed a general calibration scheme where the calibration is measured on the sequence level. yet under challenging transfer conditions for difficult tasks, the top-k sequences do not contain enough positive events, and letting the event set of interest depends on model prediction making cross-method comparison difficult. in this paper, we investigate model calibration of structured prediction tasks as well as classification, given the high interest in tasks with a sequence tagging nature where one has to model inter-label dependencies in the multilingual community. we employ a slightly different setting with (jagannatha and yu, 2020) whence either tag-wise calibration is measured @xcite , or a balanced set of a positive or negative set of spans is used to construct the event set of interest. in section 3.3 we discuss our formulation in detail, and show that it is compatible with the framework proposed by @xcite . ## conclusions we explore the model calibration of large language models under the zero-shot cross-lingual transfer scenario. our results show that the extent of miscalibration varies according to a number of aspects second, transferring from english to non-english intensifies miscalibration as the target language is farther from english. also, larger models are likely to be less miscalibrated when the model is transferred to a different target language zero-shot. moreover, our result shows that temperature scaling and gaussian process calibration methods are among the top-performing methods. while temperature scaling is easy to implement and generalizes well to distant languages, it's less effective when applied to complex structured probabilities. finally, models are least calibrated on "high levels" tasks like xnli and span extraction. models are most calibrated on simple "low-level" tasks like pos. our results demonstrate that model confidence scores are useful metrics to understand the model behavior and language transfer pairs in crosslingual tasks. we encourage users to calibrate their model before zero-shot deployment to produce more reliable confidence estimation and prevent over-confidence for downstream tasks.