Hard and Soft Evaluation of NLP models with BOO t ST rap SA mpling - B oo S t S a (2022)

Abstract: Natural Language Processing (NLP) ‘s applied nature makes it necessary to select the most effective and robust models. Producing slightly higher performance is insufficient; we want to know whether this advantage will carry over to other data sets. Bootstrapped significance tests can indicate that ability. So while necessary, computing the significance of models’ performance differences has many levels of complexity. It can be tedious, especially when the experimental design has many conditions to compare and several runs of experiments. We present BooStSa, a tool that makes it easy to compute significance levels with the BOOtSTrap SAmpling procedure to evaluate models that predict not only standard hard labels but soft-labels (i.e., probability distributions over different classes) as well.

## introduction text classification is one of the main applications of nlp, with hundreds of papers published every year at nlp conferences. while these publications cover various domains, they essentially follow the same steps: pick a classification problem. identify baseline models, standard models from previous literature, and state-of-the-art (sota) models. propose a novel approach to the problem. show that it outperforms the previous ones on benchmark data sets. developing better methods for a task is a common feature of the computational linguistics literature, and selecting the best model from a range of options is crucial for the whole experimental procedure. however, showing absolute improvements on several data sets (let alone one) is not sufficient. variations in model initialization, batch sampling and other factors might result in an improvement that does not generalize. we can use significance tests to assess whether the observed improvements are likely to hold on future data sets. however, identifying the best method(s) depends on two different but correlated aspects of the evaluation process. the metrics adopted for the performance measurement and the significance test carried out for the models' comparison. to compute the significance of the models' improvements over comparison sets, boostsa relies on bootstrap sampling @xcite . as @xcite discussed, the effect size, that is, the performance gap between different methods, can be modelled as a random variable. when this random variable follows a normal distribution, it is possible to use student's t-test to estimate the significance in the performance difference. however, the assumption of normal distribution does not hold in most nlp applications. therefore, randomized, sample-based, non-parametric tests such as bootstrap sampling are better suited for nlp. however, the correct implementation of this method can be non-trivial (especially on top of other experiments). to allow for a safe, flexible application, we release boostsa to the community. since the results of bootstrap methods are "very sensitive to sample size, [...] as well as to the existence of multiple metrics" @xcite , boostsa incorporates some constraints in the hyper-parameter choice (section 2.3) to prevent the accidental misuse of the methods. concerning the metrics, the scenario differs when the prediction uses hard labels or soft labels. hard labels are standard one-hot encoded labels, where one class is correct and is assigned the value of 1, and the others are wrong and have a value of 0. in the case of soft labels, the actual label value is ultimately uncertain. this uncertainty is expressed as a probability distribution over classes, each receiving a value from 0 to 1, all summing up to 1. in the first case, the use and the interpretation of metrics such as accuracy (acc), precision (prec), recall (rec), and f-measure (f1) are well-understood in the literature @xcite . boostsa computes these metrics for hard labels. in the second case, no metric is generally accepted to evaluate the divergence between probability distributions. boostsa follows the approach of @xcite and provides cross entropy (ce), jensen-shannon divergence (jsd), entropy similarity (e-sim) and entropy correlation (e-corr) for soft labels. contributions. we release boostsa, an opensource application that computes: the tests can be run efficiently, even for complex experimental designs comparing many different models trained in several runs of experiments. the package can be installed with pip and is released on github/fornaciari/boostsa (documentation at boostsa.readthedocs.io). ## experiment figure 2 shows the boostsa's output for a real use case. we use the barely significant results of the part-of-speech (pos) tagging classification task carried out by @xcite , ## instances. we consider two metrics, accuracy and fmeasure, and six different sub-sample sizes, from 5% to 50% of test set rate. the h0's accuracy and f-measure were 85.84 and 74.56, the h1's accuracy and f-measure were 86.27 and 75.13, with a delta of 0.43 and 0.57, respectively. similar to the results shown by @xcite , significant p-values appear only for sample sizes greater than 30% of the whole test set. ## limitations besides the constraints discussed in section 2.3, aimed to prevent the significance test misuse, boostsa assumes that h0 predictions, h1 predictions, and targets are available. when comparing with results from previous literature, this assumption might not hold: in these cases, to apply boostsa it is necessary to find or reproduce the h0 outcomes. ## conclusion we present boostsa, a python package to allow nlp practitioners to efficiently compute significance values for hard and soft labels using a safe set of hyper-parameters. we discuss how the test hyper-parameters can affect the outcome, introducing safety constraints for the test use and suggesting, as a good practice, to report the hyper-parameters with the experiments' results. while the metrics for hard labels consolidated in literature, those for soft labels that measure the divergence between probability distributions are not yet widely used, and the consensus about their interpretation is still on the way to be reached. we follow the extensive survey of @xcite , which takes into consideration the most recent trends in nlp. we also agree with @xcite , who point out that incorporating into the models the information about the intrinsic entities' ambiguity, expressed as inter-coders disagreement and represented as a probability distribution over different classes, is a necessary step to create models that carry out nlp task with human-like performance. psychology has seen a growing scandal around p-hacking, i.e., the generation of enough experimental variations to produce a significant outcome in one of them. this risk is low in nlp, as significance alone is not sufficient for publication: typically, proof of predictive performance on ideally several test sets is necessary instead. reporting significance in nlp is therefore an additional robustness measure, indicating the model's generalizability. while users might abuse the capabilities of boostsa to make significant results more likely, this would require deliberate tampering (and even then would not guarantee significance). if used as intended, however, boostsa should reduce unintended variance via researcher degrees of freedom and make results more comparable and reproducible.