Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models (2022)

Abstract: Massively Multilingual Transformer based Language Models have been observed to be surprisingly effective on zero-shot transfer across languages, though the performance varies from language to language depending on the pivot language(s) used for fine-tuning. In this work, we build upon some of the existing techniques for predicting the zero-shot performance on a task, by modeling it as a multi-task learning problem. We jointly train predictive models for different tasks which helps us build more accurate predictors for tasks where we have test data in very few languages to measure the actual performance of the model. Our approach also lends us the ability to perform a much more robust feature selection, and identify a common set of features that influence zero-shot performance across a variety of tasks.

## introduction multilingual models like mbert @xcite and xlm-r @xcite have been recently shown to be surprisingly effective for zero-shot transfer @xcite , where on fine-tuning for a task on one or a few languages, called pivots, they can perform well on languages unseen during training. the zero-shot performance however, is often not uniform across the languages and the multilingual models turn out to be much less effective for low resource languages @xcite and the languages that are typologically distant from the pivots @xcite . what affects the zero-shot transfer across different languages is a subject of considerable interest and importance @xcite @xcite , however there is little conclusive evidence and a few papers even show contradictory findings. @xcite recently, showed that it is possible to predict the zero shot performance of mbert and xlm-r on different languages by formulating it as a regression problem, with pretraining data size and typological similarities between the pivot and target languages as the input features, and the performance on downstream task as the prediction target. along similar lines @xcite and @xcite explore zero-shot performance prediction with a larger set of features and different regression techniques. however, the efficacy of these solutions are severely limited by the lack of training data, that is, the number of languages for which performance metrics are available for a given task. for instance, for most tasks in the popular xtreme-r @xcite benchmark, there are data points for 7-11 languages. this not only makes zero-shot performance prediction a challenging problem, but also a very important one because for practical deployment of such multilingual models, one would ideally like to know its performance for all the languages the model is supposed to handle. as @xcite shows, accurate performance predictors can also help us build better and fairer multilingual models by suggesting data labeling strategies. in this work, we propose multi-task learning @xcite as an approach to mitigate training-data constraints and consequent over-fitting of the performance predictors to tasks and/or datasets. the contributions of our work are fourfold. first, we experiment with different multitask learning approaches, such as group lasso @xcite , collective matrix factorization @xcite , multi-task deep gaussian process regression @xcite and meta agnostic meta learning @xcite for 11 tasks. we observe an overall 10% reduction in performance prediction errors compared to the best performing single-task models. the gains are even stronger when we just consider the tasks with very few data points (≤ 10), where we see a 20% drop in the mean absolute errors. second, an interesting consequence of modelling this problem via multi-task learning is that we are able to predict performance on low resource languages much more accurately, where in some cases single-task approaches may perform even worse than the simple averaging baselines. third, apart from the features used for zero-shot performance prediction in the previous work @xcite @xcite , we also utilize metrics quantifying the quality of multilingual tokenizers as proposed in @xcite as features in our predictive models, which turn out to have strong predictive power for certain tasks. to the best of our knowledge, our work is the first to explore the impact of tokenizer quality specifically on zero-shot transfer. and fourth, our multi-task framework in general lends us with a much more robust selection of features affecting the zero-shot performance. this, in turn, lets us investigate the critical open question on what influences the zero-shot performances across languages more rigorously. as we shall see, our findings corroborate some of the previous conclusions, while others are extended or annulled. performance prediction. prior work has explored predicting the performance of machine learning models from unlabelled data by either measuring (dis)agreements between multiple classifiers @xcite or by utilizing underlying information about data distribution @xcite . in the context of @xcite explored predicting the performance of a machine translation system by utilizing different explanatory variables for the language pairs. @xcite proposed a learning to rank approach to choose transfer languages for cross lingual learning using several linguistic and dataset specific features. recently, there has been an interest in predicting the performance of nlp models without actually training or testing them, by formulating it as a regression problem. @xcite showed that using experimental settings for an nlp experiment as inputs it is possible to accurately predict the performance on different languages and model architectures. @xcite extended this work by proposing methods to do a fine-grained estimation of the performance as well as predicting well-callibrated confidence intervals. specifically predicting the zero-shot performance of mmlms was first explored in @xcite , where they used a linear regression model to estimate the cross-lingual transfer performance based on pretraining data size and linguistic relatedness features. @xcite tackled this problem by utilizing xgboost regressor for the prediction along with a larger set of features. @xcite explored individual syntactic features for zero-shot performance prediction instead of working with aggregate similarity values, and showed about 2 to 4 times gain in performance. we extend all of these works by considering a multi-task learning approach, where performance prediction in a task utilizes not only the data available for that task, but also the patterns observed for other tasks. ## problem setup we begin by defining the multi-task performance prediction problem and then describe the different linguistic and mmlm specific features used. ## approaches we extensively experiment with a wide-array of multi-task as well as single-task regression models to provide a fair comparison between different approaches to zero-shot performance prediction. ## experimental setup in this section, we discuss our test conditions, datasets and training parameters for the different experiments. ## conclusion and future work in this paper, we showed that the zero-shot performance prediction problem can be much more effectively and robustly solved by using multi-task learning approaches. we see significant reduction in errors compared to the baselines and single-task models, specifically for the tasks which have test sets available in a very few languages or when trying to predict the performance for low resource languages. additionally, this approach allows us to robustly identify factors that influence zero-shot performance. our findings in this context can be summarized as follows. this last finding is especially interesting. as described earlier, geographical proximity is a more clear, noise-free and complete feature compared to the other relatedness metrics. however, one could also argue that since neighboring languages tend to have high vocabulary and typological feature overlap due to contact processes and shared areal features, geographical distance is an extremely informative feature for zero-shot transfer. two direct implications of these findings are: (1) for effective use of mmlms, one should develop resources in at least one pivot language per geographic regions, and (2) one should work towards multilingual tokenizers that are effective for most languages. there are a number of directions that can be explored in future related to our work. the prediction models can be extended to a multi-pivot and few-shot settings, as described in @xcite . further probing experiments could be designed to understand the role of sub-word overlap on zero-shot transfer of retrieval tasks. the gp prior will be defined by replacing the task specific kernel k t in the equation 4 with the multi-task kernel k m . we use the optimization steps similar to dgp and the inference is done by using the standard gp formulae. relating mdgpr to equation 1, the global parameters θ are the parameters of the deep network g, and the task specific parameter φ is the positive semi-definite matrix k task . model agnostic meta learning (maml): maml @xcite is a popular meta learning algorithm that can be used to quickly adapt deep neural networks on new tasks in a few-shot setting. in maml, the set of initialization parameters for the neural network are explicitly learned such that the network can generalize well on a new task with a small number of gradient steps and training samples. relating to equation 1, the global parameters θ can be considered as the initial set of parameters for the neural network that are learned and shared across all the tasks. task specific parameters φ are adapted from θ by taking k gradient steps using the task's performance data. for evaluating a task t, we consider rest of the tasks in our dataset as helpers (t ′ ∈ t-{t}) and use them to train the initial set of parameters θ. the initial parameters are then updated by fine-tuning the network on the training set for t using gradient descent.