Performance Prediction via B ayesian Matrix Factorisation for Multilingual Natural Language Processing Tasks (2023)

Abstract: Performance prediction for Natural Language Processing (NLP) seeks to reduce the experimental burden resulting from the myriad of different evaluation scenarios, e.g., the combination of languages used in multilingual transfer. In this work, we explore the framework ofBayesian matrix factorisation for performance prediction, as many experimental settings in NLP can be naturally represented in matrix format. Our approach outperforms the state-of-the-art in several NLP benchmarks, including machine translation and cross-lingual entity linking. Furthermore, it also avoids hyperparameter tuning and is able to provide uncertainty estimates over predictions.

## introduction natural language processing (nlp) is an empirical discipline, with progress driven by a myriad of tasks, domains, computational models and datasets @xcite . models are often developed to be applicable to a large number of languages. given that there are more than 7000 languages spoken in the world @xcite , evaluating the model performance of each possible nlp scenario leads to a combinatorial explosion and would require excessive time for training and testing, as well as the significant costs of evaluation resource creation. this is very time consuming and computationally expensive, especially for large models and massively multilingual applications @xcite . to illustrate, for neural machine translation (mt) models, retraining and testing for each new language pair is required @xcite , which quickly increases the number of experiments (as shown in figure 1 ). in those scenarios, whether the model is appropriate for the considered test languages is not clear upfront. similarly, for crosslingual transfer (tsf) tasks like parsing @xcite or part-of-speech (pos) tagging jpn kor lit mar mkd nld nob ara bos bul ces cmn dan deu ell eng epo est eus (de @xcite , the performance on all languages of interest is only known after training and testing has been performed. this is particularly interesting for the development of nlp technologies for low-resource languages @xcite . in this paper we propose a method for performance prediction which can solve the above problems through automatically estimating performance scores of machine learning models, thus avoiding the need for excessive training and test cycles, and data creation costs. the goal is to predict the performance of a model solely based on past experimental records -a task of great interest for decision making in aforementioned high-cost applications. we propose to use bayesian probabilistic matrix factorization methods and incorporate context features describing the language pairs of interest. this is realisable because many performance prediction problems in nlp can be modelled as a matrix, as illustrated in figure 1 . the result will inevitable be an incomplete matrix because some data are missing, as they were either to costly to obtain or no datasets were available @xcite . our key contribution is a bayesian approach for performance prediction, contrasting with prior work which is exclusively frequentist. this confers the following advantages: 1) it enables us to quantify the uncertainty of our predictions by providing credible intervals (cis); and 2) predictions of performance scores can be provided in a more principled way via a posterior distribution after integrating over the model's parameters and hyperparameters. we provide an extensive experimental study of matrix factorization methods, including our proposed bayesian technique, applied to several important benchmarks, and show several improvements over the state-of-the-art. ## problem formulation and methods as demonstrated in figure 1 , the performance scores for a suitable nlp task can be arranged in a matrix with the rows displaying the source languages and the columns showing the target languages foot_0 . the matrix is the score matrix @xmath0, t) contains either the performance score for a source-to-target language relationship or is empty. the number of considered source and target languages is denoted by n s and n t , respectively. the performance of a model, measured by an evaluation metric (e.g. bleu, @xcite ), depends on properties like the model architecture m, training dataset d, languages l, the training procedure p and the test dataset d ′ . the score matrix r is then given by with f θ (•) being a function which denotes a linear or non-linear relationship of its arguments and φ x represents the feature set of each characteristic x . the dependency on test set features is omitted. we assume that the distributions of training and test datasets are the same, ignoring a possible domain shift (following @xcite ). as the completion of an arbitrary matrix is illposed, we reformulate f θ (•) as a low-rank matrix approximation problem. in its simplest, noiseless form, the matrix r can be reconstructed as @xcite r ≈ w t h, using two lower rank matrices @xmath1, with w and h being the latent matrices for the source and target languages, respectively. due to the low-rank assumption, for the latent dimensions it holds that d ≪ n s , n t . hence, each individual score r s,t of a source-target pair can be calculated by r s,t ≈ w t s h t , where each latent source and target language vector is given by a low-dimensional vector, w t @xmath2, h @xmath3, respectively @xcite . the matrix factorisation (mf) approach described above is inspired by research on recommender systems, which use latent factor models for preference predictions @xcite . in those models, the low rank behavior is a widely used assumption. we assume in our model that the performance of a bilingual task is determined by a small number of unobserved factors. using a linear factor model the performance for translating from a source language is modeled by linearly combining target language factor vectors using source language specific coefficients. ## experiments and analysis following the evaluation setting of nlperf @xcite , we consider a set of tasks described in the following, all of which allow a reformulation into a matrix representation. the performance scores are predicted for bilingual lexicon induction (bli); machine translation on aligned wikipedia data (wiki mt ), and with cross-lingual transfer for translation into english (mt tsf ); cross lingual dependency parsing (parsing tsf ); cross-lingual pos tagging (pos tsf ) and cross-lingual entity linking (el tsf ). despite also considering cross-lingual transfer tasks, for reasons of simplicity and without any loss of generality, we refer to each cell in the matrix as a source-target language pair unless otherwise stated. all tasks and their matrix properties are stated in to allow a fair comparison to the closest related work nlperf @xcite , we consider an identical set of features for our predictions: dataset size, word/subword vocabulary size, average sentence length, word/subword overlap, type-token ratio, type-token ratio distance, single tag type, fused tag type, average tag length per word, dependency arcs matching wals features and six distance features from the uriel typological database (littell @xcite . further information regarding each feature can be found in @xcite . all features were normalized using their corresponding z-score. for each task, (except bli), we predict scores for a single model. this means that there is exactly one underlying statistical or neural model considered for each task. for bli however, our dataset consists of two metrics vecmap and muse. the performance is obtained as in the equivalent multi-model scenario in @xcite , by averaging over both metrics. the nlperf framework is based on gradient boosting trees @xcite and implemented with xgboost @xcite . the parameterisation of this predictor was kept as in @xcite , assuming a squared error as the objective function for the regression, a fixed learning rate of 0.1, a maximum tree depth of 10 and the number of trees being 100. the default setting was used for the regularization terms. we evaluate the performance of our methods by using the root mean squared error (rmse). predictions for mf are obtained according to equation ( 8 ) but through omitting the context information, whereas mf ctx makes use of the latter. the conditional distribution over observed ratings as given in equation ( 2 ) is used to obtain predictions for bpmf, and adapted accordingly for pmf. all non-bayesian models are trained using nested 5-fold cross validation and stochastic gradient descent (sgd) over 2000 training iterations. referring to figure 1 , note that during training of the latent vectors w s and h t , all available scores in the corresponding column s and row t are used. the regularization parameters are chosen using a vali-dation set. all non-probabilistic experiments are averaged over 10 runs. training of the probabilistic models was performed using mcmc sampling and the results are averaged over 2 runs, due to mcmc time complexity. 3 more information about the execution times of each experiment is given in appendix b. ## related work matrix completion (mc) is an important technique used to recover a matrix from a subset of its entries, widely applied and studied in many areas of research like machine learning, data science, signal processing and communications @xcite . in nlp, it is found in keyword searches or recommender systems, among others @xcite . mc in the context of recommender systems can be interpreted as e.g. the task of predicting a product to recommend to costumers. in the famous netflix price competition, targeting recommender systems for movies, it has been shown that an approach based on matrix factorization (mf) via low-rank latent factor models is the basis for the best algorithm to predict user ratings @xcite . there are two main directions of performance prediction in machine learning. the performance can either be predicted as a function of certain training dataset properties or as a function of its training time or number of iterations @xcite . following the former, current state of the art approaches like nlperf @xcite or litmus @xcite leverage the advantages of regression models like gradient boosting trees. we use nlperf as a competitive baseline in this work. note that nlperf was shown to outperform several simple mean-value baselines, namely the average over the performances of all available test instances; the average over shared source languages; and the average over models if multiple models are used. this approach can easily incorporate additional features of the experiments, but they do not provide a direct measure of uncertainty. in @xcite methods used for performance prediction are based on tensor-regression-based ap-proaches being robust pca and cp decomposition. those are developed to provide a more fine-grained performance measure. while confidence intervals in the frequentist sense are provided, no bayesian analysis was conducted. in @xcite collective mf was used for performance prediction of massive multilingual language models, where it underperformed xgboost. however, no experiments were shown for bilingual tasks. we differ in our work because we explore the framework of mf extensively and are able to provide a range of experiments contrasting probabilistic and non-probabilistic approaches. additionally our approach can give cis as a measure of uncertainty of our predictions. closer to our approach is the work of @xcite , where instead of evaluating domain similarity or diversity, a linear classifier is used to predict the performance drop under domain shift using metrics based on h-divergence, reverse classification accuracy and confidence measures. in contrast to this body of literature, our methods provide the additional benefit of bayesian uncertainty quantification. an area of application for performance prediction, to be additionally highlighted here, is data selection, during which the method can be used to identify a suitable training dataset under domain shift, which will lead to the best model performance. earlier work in this field in e.g. @xcite uses the unsupervised a-distance measure of divergence between domains, while existing literature on phrase based machine translation in @xcite and @xcite show that perplexity-or cross-entropy based scoring methods are beneficial to select suitable sentences for training, increasing the overall model performance. connected to this stream of literature is also the work of @xcite . they present a bayesian optimisation approach, which is model-independent and is used to learn data selection measures for transfer learning. additionally, incorporating not only information about the data set, but also about the model, @xcite shows that the h-discrepancy, which is a generalisation of the source guided discrepancy @xcite , can be used to identify the best generalization performance of discourse models. ## conclusion in this work, we presented an extensive study of various mf methods applied to the problem of per-formance prediction. using a bayesian approach we can give a measure of uncertainty in terms of cis in addition to point estimates. additionally, we show that leveraging the obtained bounds of the cis can guide decision-making regarding whether it is lucrative to deploy a model for certain languagepairs and whether the corresponding datasets for the language-pairs should be acquired. our results confirm that bilinear mf methods can be used to predict the performance scores of various nlp tasks, which is computationally less expensive than using a non-linear model like xgboost in nlperf. furthermore, we show that the mf framework is suited to predict performance scores in cold-start scenarios. the mf framework was not able to outperform the sota for the two-model scenario bli, which might be due to the small matrix size and unsuitable features. however, bpmf ctx clearly outperformed lr. this suggests that further studies are required regarding non-linear mf methods. while we chose to present a specific set of tasks within this work, the proposed mf methods can be used for other nlp scenarios where the considered problem can be modelled as a matrix.