Towards More Fine-grained and Reliable NLP Performance Prediction (2021)

Abstract: Performance prediction, the task of estimating a system’s performance without performing experiments, allows us to reduce the experimental burden caused by the combinatorial explosion of different datasets, languages, tasks, and models. In this paper, we make two contributions to improving performance prediction for NLP tasks. First, we examine performance predictors not only for holistic measures of accuracy like F1 or BLEU, but also fine-grained performance measures such as accuracy over individual classes of examples. Second, we propose methods to understand the reliability of a performance prediction model from two angles: confidence intervals and calibration. We perform an analysis of four types of NLP tasks, and both demonstrate the feasibility of fine-grained performance prediction and the necessity to perform reliability analysis for performance prediction methods in the future.

## introduction performance prediction (p 2 ) aims to predict a machine learning system's performance based on features of the underlying problem, dataset, or learning algorithm. while this topic is still relatively unexplored in the nlp context, there are a few examples of predicting performance as: (i) a function of training or model parameters for determining the number of training iterations @xcite or value of hyperparameters @xcite and identifying and terminating bad training runs @xcite . (ii) a function of dataset characteristics to illustrate which factors are significant predictors of system performance @xcite , or find a subset of representative experiments to run in order to obtain plausible predictions @xcite . in this paper, we ask two @xcite (1, 2] (2, 3] (3, 6] entity length interval 0.6 0.7 0.8 0.9 1.0 f1 research questions with respect to performance prediction: can we predict performance on a more fine-grained level, and can we quantify the reliability of performance predictions? with respect to the first contribution, previous p 2 methods have almost entirely focused on predicting holistic measures of accuracy such as entity f1 @xcite or bleu score @xcite over the entire dataset ( §2.2). however, from a perspective of understanding the workings of our models, work on model analysis has demonstrated the need for more fine-grained analysis over a wide variety of tasks @xcite @xcite @xcite . these methods calculate separate accuracy scores for different types of examples (e.g. accuracies for entity recognition by entity length). our first contribution is to examine experimental settings where we predict these fine-grained evaluation scores ( §2.3), and also propose performance prediction methods particularly suited to this fine-grained evaluation setting ( §3). our second contribution is the development of methods for estimating the reliability of performance predictions. while allowing estimation of experimental results without actually having to run the corresponding experiments may improve efficiency, if the performance predictor is wrong it may lead to missing results of a potentially important experiment. this particularly becomes an issue when developing methods for fine-grained performance prediction, as the number of data points which can be used to predict each performance number decreases as we subdivide datasets into finer-grained categories. thus, we make methodological steps towards answering two specific questions: (i) how can we define and calculate a confidence interval over performance predictions? (ii) how well does the confidence interval of prediction performance calibrate with the true probability of an experimental result? fig. 1 is an example of performance prediction and reliability analysis over fine-grained performance estimates (f1 scores over different entity length buckets) of an ner system are obtained in two ways: (i) calculated based on results from the ner system itself (in gray); (ii) estimated based on a performance prediction model, without running an actual experiment (in red). we can observe that: (1) with fewer test samples (e.g. 49), confidence intervals of both actual and predicted f1 become much wider, suggesting larger uncertainty. (2) in the last bucket, the predicted f1 (red point) is far from the actual f1 (gray point), but with a confidence interval of predicted performance (red bar), the actual f1 still falls within it, indicating the importance of knowing the level of confidence. in experiments, we investigate the efficacy of different performance prediction models on four typical nlp tasks under both holistic and finegrained settings, then explore methods for the reliability analysis of these performance prediction models. major experimental results show: 1) there is no one-size-fits-all model: best-scoring performance prediction systems in different scenarios are diverse. in particular, one of our proposed models achieved the best results on the partof-speech task ( §6.1). 2) a better performance prediction model doesn't imply better calibration ( §6.2). 3) all four performance prediction models (including previous top-scoring ones) produce confidence intervals over-confidently ( §6.2). ## performance prediction: formulation and applicable scenarios in this section, we will mathematically define performance prediction and its application in the holistic and fine-grained evaluation. ## parameterized regression functions the performance prediction model takes in a set of features that characterize an experiment's peculiarities and predict performances based on different parameterized regressors g(•) in eq. 2. we first describe methods explored by previous works and then present a tensor regression-based approach that is particularly well-suited for fine-grained performance prediction. ## statistical preliminaries before going into our second contribution to establishing reliability of performance prediction, we describe two relevant concepts from statistics. ## experiments in this section, we break down our experimental results into answering two research questions sections: (1) how well do our underlying performance predictors work, particularly the newly proposed tensor-based predictors and on the newly proposed task of fine-grained performance prediction? (2) how well can we estimate the reliability of our performance predictions? models besides the four performance prediction models (cp, pca, xgboost, lgbm) that we have introduced in §3, following @xcite , we additionally use a simple mean value baseline model which predicts an average of scores s from the training folds for all test entries in the left-out evaluation fold: hyper-parameters detailed information about the hyper-parameters used in training the performance prediction models in various tasks is provided in the appendix. tasks we explore performance prediction on four tasks: (1) machine translation (mt) @xcite , (2) part-of-speech tagging (pos), (3) named entity recognition (ner), (4) chinese word segmentation (cws). to compare the performance of tensor-based models and gradient boosting models on the same dataset, we convert the datasets used in different prediction tasks to tensors. statistics of the tensor data are shown in the appendix.