Revisiting Sample Size Determination in Natural Language Understanding (2023)

Abstract: Knowing exactly how many data points need to be labeled to achieve a certain model performance is a hugely beneficial step towards reducing the overall budgets for annotation. It pertains to both active learning and traditional data annotation, and is particularly beneficial for low resource scenarios. Nevertheless, it remains a largely under-explored area of research in NLP. We therefore explored various techniques for estimating the training sample size necessary to achieve a targeted performance value. We derived a simple yet effective approach to predict the maximum achievable model performance based on small amount of training samples – which serves as an early indicator during data annotation for data quality and sample size determination. We performed ablation studies on four language understanding tasks, and showed that the proposed approach allows us to forecast model performance within a small margin of mean absolute error (~0.9%) with only 10% data.

## introduction labeled data play an important role in creating performant machine learning models, which makes data annotation a fundamental process for any natural language application pipeline @xcite . recent work has sought to reduce the annotation costs through the use of active learning @xcite and data sampling @xcite @xcite . indeed, these approaches are shown to be effective in identifying or constructing data subsets needed to achieve a competitive model performance. for instance, the active learning paradigm adds new data iteratively to the existing set before model retraining @xcite , improving upon the traditional human annotation pipeline that obtains the entire labeled set all at once. nevertheless, the data labeling process typically annotates as much data as the annotation budget permits, or by clearly defined stopping criteria to terminate the labeling process. unfortunately, this is usually challenging as annotators do not have the knowledge of the effect of added labels to model performance nor how much more data is needed to arrive at the desired model generalizability @xcite . the stopping condition is in fact tied to the quality of data samples w.r.t. model parameters @xcite , which influences the effective sample size foot_0 , and it is then beneficial to obtain an approximation of the expected performance @xcite @xcite . therefore, knowing the approximate amount of training data needed for this particular performance would serve as an useful knowledge not only for deciding when to stop adding labeled data, but also as an early indication for the data quality. for instance, by having early label quality signals, we can decide between two different types of annotation, or even between two pools of annotators with different expertise. to this end, we explored the relationship between data sample size and model performance in the context of language understanding via learning curve modeling, which defines model performance as a function of dataset sizes. by modeling this relationship in low resource settings, we obtain useful early signals with approximated accuracies for any given the labeled set, which can provide an idea for the sample size and data quality @xcite . previous studies have shown that nonlinear weighted curve fitting methods such as inverse power laws or exponential functions can provide decent approximations of the empirical predictive performances @xcite . we thus put forward an ensemble of these functions which we showed to display a consistently highly correlated behavior across four language understanding benchmarks and with as little as 10% of the entire training set. this work makes the following contributions: ## background our method is a sample size determination technique that helps to design annotation projects by determining the necessary sample size. previous methods have focused on identifying the sample size required to reach a specific target performance, such as a high correlation coefficient @xcite @xcite , which often involves predicting the sample size necessary for a classifier to attain a specific accuracy level @xcite . there are two main approaches for predicting the sample size needed to achieve a particular classifier performance: (1) @xcite present a model-based method for predicting the number of samples required for classifying microarray data. (2) a more general approach involves fitting a classifier's learning curve to inverse power law models @xcite . examples of this approach include algorithms proposed by @xcite ; @xcite ; @xcite . ## the approach learning curve modeling. a learning curve is a graphical representation of how a classifier's performance changes as the size of the training set increases. the curve typically has three sections: an initial section where performance improves rapidly with increasing training set size, a middle section where the rate of improvement slows down, and a final section where the classifier reaches its maximum performance and further increases in training set size do not lead to significant improvements. this relationship can be quantified using a set of data points, each of which represents the expected performance of the classifier e acc on a particular training set size d k . these data points can be plotted to create the learning curve, which can help to understand the behavior of the classifier and inform decision-making about how much training data is needed to achieve a desired performance level. task description. given a downstream classification task with n total data points, a learning curve model f predicts the expected performance e acc when a classifier trained on the an observed range of training set size (d k ; @xmath0. the empirical learning curve is assessed by the parametric models for the learning algorithm performance extrapolation. in our settings, we set @xmath1, where few data points consisting of (e acc , d k ) are to be obtained. ## results and analysis we present results of ensemble method for learning curve modeling on the nlu benchmarks. ## conclusions and future works in this work, we investigated techniques for estimating the amount of training data needed to achieve a target performance in four natural language understanding benchmarks. we demonstrated that our approach allows for accurate prediction of model performance using only a small portion of the data, which can be useful in scenarios with limited resources. nevertheless, we also recognize the limitation in our current study. for instance, we did not explore sampling techniques other than random sampling; while recent works @xcite @xcite have shown promising directions in data sampling that outperforms random sampling. another interesting direction is to explore the model architecture's influence on generalizability, and thus the learning curve, which we left for future works.