Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration (2023)

Abstract: Pretrained multilingual encoder models can directly perform zero-shot multilingual tasks or linguistic probing by reformulating the input examples into cloze-style prompts. This is accomplished by predicting the probabilities of the label words at the masked token position, without requiring any updates to the model parameters. However, the performance of this method is limited by the model’s bias toward predicting label words which frequently occurred during the pretraining. These words typically receive high probabilities. To address this issue, we combine the models with calibration techniques which modify the probabilities of label words predicted by the models. We first validate the effectiveness of a proposed simple calibration method together with other existing techniques on monolingual encoders in both zero- and few-shot scenarios. We subsequently employ these calibration techniques on multilingual encoders, resulting in substantial performance improvements across a wide range of tasks.

## introduction prompt-based learning @xcite has emerged as an important research area. recent research demonstrates that multilingual encoder models are capable of accomplishing zero-shot cross-lingual learning (zhao and schütze, 2021; @xcite and linguistic probing @xcite by using cloze-style prompts. these prompts consist of an input sample, a task-specific context and a mask token. the encoder model applies masked language modeling (mlm) @xcite to generate predictions for the mask token using a selection of prescribed candidate tokens from the vocabulary. these predictions can be subsequently utilized for label classification or probing purposes. for example, the sentiment analysis of assigning the product review "worked as stated!" to class pos can be reformulated as: "worked as stated! all in all, it was @xcite ." the model is requested to fill in the word "good" at the mask token position, which is mapped to the pos label. however, earlier studies indicate that the output of masked token prediction is biased towards certain label words in the candidate token list @xcite . this bias not only affects the predicted class probabilities @xcite , but also deteriorates the model's overall performance @xcite . according to @xcite and @xcite , label words with higher frequency in the pretraining corpus tend to be predicted with higher probabilities. besides, the prompt context can also influence the degree of bias present in the masked token prediction. figure 1 illustrates the impact of the above mentioned biases on the model predictions. it shows the results of a binary sentiment analysis task with the bert base @xcite be found in in this paper, we contribute by (1) proposing a simple yet effective calibration method that involves adding trainable penalties to the probabilities of the label words, (2) demonstrating its effectiveness in achieving performance enhancements comparable to other existing calibration techniques, (3) refining the calibration parameters with only a few training examples for further improvement, and (4) boosting the zero-shot performance of multilingual encoders by introducing calibration methods. ## experimental setup dataset we first validate the effectiveness of the different calibration methods on several monolingual tasks. we study sentiment analysis using two datasets: binary amazon polarity @xcite and the english subset of 5label multilingual amazon reviews @xcite , topic categorization using two datasets: the ag news and yahoo answers topics @xcite , sentence pair classification using two datasets: english subsets of mnli @xcite and paws-x (yang et al., 2019), and 5 datasets from the glue benchmark @xcite : cola (warstadt et al., 2019), mrpc @xcite , qqp, rte @xcite , and wnli @xcite . for the evaluation of multilingual encoders, we use multilingual amazon reviews, xnli and paws-x. besides, following nie et al. setup in our monolingual experiments, we use the pretrained models bert-base-cased (devlin et al., 2019) and roberta-base @xcite . in the multilingual experiments, we use their multilingual counterparts bert-base-multilingual-cased and xlm-roberta-base @xcite . we use pytorch @xcite and the huggingface framework @xcite . we repeat each experiment 5 times with different random seeds and report the mean and variance. details of the experimental setting can be found in appendix a. ## conclusion in conclusion, our work focuses on boosting the zero-shot learning performance of multilingual encoders in language understanding tasks through probability calibration techniques. we address the bias issue in the mask token prediction of label words by introducing various calibration techniques that modify the probabilities of these words. we first test the efficacy of different calibration methods in monolingual encoders. we also prove that with a minimal number of training examples, the calibrated probabilities yield further enhancements compared to the zero-shot calibration method. our experiments on multilingual encoders demonstrate that all calibration methods bring a performance improvement across various tasks.