Predicting Performance for Natural Language Processing Tasks (2020)

Abstract: Given the complexity of combinations of tasks, languages, and domains in natural language processing (NLP) research, it is computationally prohibitive to exhaustively test newly proposed models on each possible experimental setting. In this work, we attempt to explore the possibility of gaining plausible judgments of how well an NLP model can perform under an experimental setting, without actually training or testing the model . To do so, we build regression models to predict the evaluation score of an NLP experiment given the experimental settings as input. Experimenting on~9 different NLP tasks, we find that our predictors can produce meaningful predictions over unseen languages and different modeling architectures, outperforming reasonable baselines as well as human experts. %we represent experimental settings using an array of features. Going further, we outline how our predictor can be used to find a small subset of representative experiments that should be run in order to obtain plausible predictions for all other experimental settings.

## introduction natural language processing (nlp) is an extraordinarily vast field, with a wide variety of models being applied to a multitude of tasks across a plenitude of domains and languages. in order to measure progress in all these scenarios, it is necessary to compare performance on test datasets representing each scenario. however, the cross-product of tasks, languages, and domains creates an explosion of potential application scenarios, and it is infeasible to collect high-quality test sets for each. in addition, even for tasks where we do have a wide variety of test data, e.g. for well-resourced tasks such as machine translation (mt), it is still computationally prohibitive as well as not environmentally friendly @xcite to build and test on systems for all languages or domains we are interested in. because of this, the common practice is to test new methods on a small number of languages or domains, often semi-arbitrarily chosen based on previous work or the experimenters' intuition. as a result, this practice impedes the nlp community from gaining a comprehensive understanding of newly-proposed models. toward that end, through drawing on the idea of characterizing an experiment from @xcite , we propose a framework, which we call nlperf, to provide an exploratory solution. we build regression models, to predict the performance on a particular experimental setting given past experimental records of the same task, with each record consisting of a characterization of its training dataset and a performance score of the corresponding metric. concretely, in §2, we start with a partly populated table (such as the one from bli method evaluation set ## problem formulation in this section we formalize the problem of predicting performance on supervised nlp tasks. given an nlp model of architecture m trained over dataset(s) d of a specific task involving language(s) l with a training procedure (optimization algorithms, learning rate scheduling etc.) p, we can test the model on a test dataset d and get a score s of a specific evaluation metric. the resulting score will surely vary depending on all the above mentioned factors, and we denote this relation as g: s m,p,l,d,@xmath0, p, l, d, d ). in the ideal scenario, for each test dataset d of a specific task, one could enumerate all different settings and find the one that leads to the best performance. as mentioned in section §1, however, such a brute-force method is computationally infeasible. thus, we turn to modeling the process and formulating our problem as a regression task by using a parametric function f θ to approximate the true function g as follows: in the next section we describe concrete instantiations of this function for several nlp tasks. ## nlp task instantiations to build a predictor for nlp task performance, we must 1) select a task, 2) describe its featurization, and 3) train a predictor. we describe details of these three steps in this section. task dataset citation source target transfer # models # exs task langs langs langs metric wiki-mt schwenk et al. (2019) 39 39 -single 995 bleu ted-mt qi et al. (2018) 54 1 -single 54 bleu tsf-mt qi et al. (2018) 54 1 54 single 2862 bleu tsf-parsing nivre et al. (2018) -30 30 single 870 accuracy tsf-pos nivre et al. (2018) -26 60 single 1531 accuracy tsf-el rijhwani et al. (2019) -9 54 single 477 accuracy bli lample et al. (2018) 44 44 -3 88×3 accuracy ma mccarthy et al. (2019) -66 -6 107×6 f1 ud zeman et al. (2018a) -53 -25 72×25 f1 tasks we test on tasks including bilingual lexicon induction (bli); machine translation trained on aligned wikipedia data (wiki-mt), on ted talks (ted-mt), and with cross-lingual transfer for translation into english (tsf-mt); crosslingual dependency parsing (tsf-parsing); crosslingual pos tagging (tsf-pos); cross-lingual entity linking (tsf-el); morphological analysis (ma) and universal dependency parsing (ud). basic statistics on the datasets are outlined in for wiki-mt tasks, we collect experimental records directly from the paper describing the corresponding datasets @xcite . for ted-mt and all the transfer tasks, we use the results of @xcite . for bli, we conduct experiments using published results from three papers, namely @xcite , @xcite and @xcite . for ma, we use the results of the sigmorphon 2019 shared task 2 @xcite . last, the ud results are taken from the conll 2018 shared task on universal dependency parsing @xcite . featurization for language features, we utilize six distance features from the uriel typological database @xcite , namely geographic, genetic, inventory, syntactic, phonological, and featural distance. the complete set of dataset features includes the following: dataset size: the number of data entries used for training. 2. word/subword vocabulary size: the number of word/subword types. 3. average sentence length: the average length of sentences from all experimental. 4. word/subword overlap: predictor our prediction model is based on gradient boosting trees @xcite , implemented with xgboost @xcite . this method is widely known as an effective means for solving problems including ranking, classification and regression. we also experimented with gaussian processes @xcite , but settled on gradient boosted trees because performance was similar and xgboost's implementation is very efficient through the use of parallelism. we use squared error as the objective function for the regression and adopted a fixed learning rate 0.1. to allow the model to fully fit the data we set the maximum tree depth to be 10 and the number of trees to be 100, and use the default regularization terms to prevent the model from overfitting. in this section we investigate the effectiveness of nlperf across different tasks on various metrics. following @xcite , we conduct kfold cross validation for evaluation. to be specific, we randomly partition the experimental records of l, d, c, s tuples into k folds, and use k-1 folds to train a prediction model and evaluate on the remaining fold. note that this scenario is similar to "filling in the blanks" in for evaluation, we calculate the average root mean square error (rmse) between the predicted scores and the true scores. baselines we compare against a simple mean value baseline, as well as against language-wise mean value and model-wise mean value baselines. the simple mean value baseline outputs an average of scores s from the training folds for all test entries in the left-out fold (i) as follows: note that for tasks involving multiple models, we calculate the rmse score separately on each model and use the mean rmse of all models as the final rmse score. the language-wise baselines make more informed predictions, taking into account only training instances with the same transfer, source, or target language (depending on the task setting). for example, the source-language mean value baseline ŝ(i,j) in a similar manner, we also compare against a model-wise mean value baseline for tasks that include experimental records from multiple models. now, the prediction for the j th test instance in the left-out fold i is an average of the scores on the same dataset (as characterized by the language φ l and dataset φ d features) from all other models: ## related work as discusssed in @xcite , there are two main threads of work focusing on predicting performance of machine learning algorithms. the first thread is to predict the performance of a method as a function of its training time, while the second thread is to predict a method's performance as a function of the training dataset size. our work belongs in the second thread, but could easily be extended to encompass training time/procedure. in the first thread, @xcite attempt to infer learning curves based on training data features and extrapolate the initial learning curves based on bleu measurements for statistical machine translation (smt). by extrapolating the performance of initial learning curves, the predictions on the remainder allows for early termination of a bad run @xcite . in the second thread, @xcite adopt linear regression to capture the relationship between data features and smt performance and find that the amount of reordering, the morphological complexity of the target language and the relatedness of the two languages explains the majority of performance variability. more recently, elsahar and gallé ( 2019 (2020) explore the functional form of the dependency of the generalization error of neural models on model and data size. we view our work as a generalization of such approaches, appropriate for application on any nlp task. ## conclusion and future work in this work, we investigate whether the experiment setting itself is informative for predicting the evaluation scores of nlp tasks. our findings promisingly show that given a sufficient number of past training experimental records, our predictor can 1) outperform human experts; 2) make plausible predictions even over new-coming models and languages; 3) extrapolate well on features like dataset size; 4) provide a guide on how we should choose representative datasets for fast iteration. while this discovery is a promising start, there are still several avenues on improvement in future work. first, the dataset and language settings covered in our study are still limited. experimental records we use are from relatively homogeneous settings, e.g. all datasets in wiki-mt task are sentencepieced to have 5000 subwords, indicating that our predictor may fail for other subword settings. our model also failed to generalize to cases where feature values are out of the range of the training experimental records. we attempted to apply the predictor of wiki-mt to evaluate on a low-resource mt dataset, translating from mapudungun (arn) to spanish (spa) with the dataset from @xcite , but ended up with a poor rmse score. it turned out that the average sentence length of the arn-spa data set is much lower than that of the training data sets and our predictors fail to gener-alize to this different setting. second, using a categorical feature to denote model types constrains its expressive power for modeling performance. in reality, a slight change in model hyperparameters @xcite , optimization algorithms (kingma and ba, 2014), or even random seeds @xcite may give rise to a significant variation in performance, which our predictor is not able to capture. while investigating the systematic implications of model structures or hyperparameters is practically infeasible in this study, we may use additional information such as textual model descriptions for modeling nlp models and training procedures more elaborately in the future. lastly, we assume that the distribution of training and testing data is the same, which does not consider domain shift. on top of this, there might also be a domain shift between data sets of training and testing experimental records. we believe that modeling domain shift is a promising future direction to improve performance prediction.