Is the Best Better? B ayesian Statistical Model Comparison for Natural Language Processing (2020)

Abstract: Recent work raises concerns about the use of standard splits to compare natural language processing models. We propose a Bayesian statistical model comparison technique which uses k-fold cross-validation across multiple data sets to estimate the likelihood that one model will outperform the other, or that the two will produce practically equivalent results. We use this technique to rank six English part-of-speech taggers across two data sets and three evaluation metrics.

## bayesian model comparison imagine a scenario where one wishes to compare the performance of two classifiers across q data sets. by performing m k-fold evaluations, the experimenter obtains a vector of @xmath0, i.e., differences in scores, between the two models: the values in these vectors are a positive cross-correlation ρ because cross-validation introduces overlap in training data. let δ i denote the mean difference score on the ith data set, and let δ 0 denote the average population-level difference. @xcite propose a hierarchical probabilistic model after the model learns the parameter distributions from experimental data, we obtain a posterior probability distribution p(δ 0 , σ 0 , μ | d). to infer whether one classifier is more likely to outperform another-or whether they are practically equivalent-we draw n s samples from the posterior distribution. we use decision counters n left , n rope , and n right to keep track of how many times the left model was more likely to outperform the right model, to be practically equivalent to the right model, and to be outperformed by the right model, respectively. for each sample of the parameters, we define the posterior of the mean difference accuracy on a new unseen data set δ next as t(δ 0 , σ 0 , μ). we obtain the outcome probabilities by integrating the distribution over the three intervals-e.g., we obtain the probability that the left model is better than the right by integrating from the left end of the distribution to the left edge of the rope interval, and so on-and then incrementing the decision counter for the region with the highest outcome probability. finally, we compute likelihoods for the three scenarios by dividing the decision counts by the number of samples drawn: p(left) = instead of significance, we thus estimate the likelihoods that one method is better than the other (or are practically equivalent). these estimates follow from observing the beliefs of a bayesian model that models the probability of the methods' mean difference on unseen data sets, after sampling parameters from a meta-distribution which estimates the difference and variance over the population of data sets with μ degrees of freedom. ## materials and methods to compare the results of our study with the ones obtained by @xcite , we use the same models, data sets, and evaluation metrics. 1 that is, we compare implementations of the tnt @xcite , collins @xcite , lapos @xcite we perform 20 randomized 10-fold cross validations, obtaining 200 measurements of each model's performance on each data set. in each run, 80% of the data is used for training, 10% for validation, and 10% for evaluation. we fit bayesian models using the baycomp library foot_0 and draw 50,000 samples from the posterior. following gorman and bedrick, we use three evaluation metrics. token accuracy is simply the number of test data tokens correctly tagged divided by the total number of tokens, and is the standard intrinsic evaluation metric used for this task. oov accuracy is similar to token accuracy but is restricted to out-of-vocabulary tokens, i.e., those found in the test data but not in the training data. finally, sentence accuracy is the number of test data sentences that contain no tagging errors, divided by the number of test sentences. groundtruth data is provided by human annotators. foot_1 ## results posterior distributions of the hierarchical models are visualized in figures 1 2 3 and summarized in the hierarchical model estimates, for example, that tnt, the simplest tagger, would be outperformed in token accuracy by any of the other five taggers 80-90% of the time. however, there is a surprisingly high chance of practical equivalence in token accuracy between the collins tagger, lapos, and the stanford tagger; for instance, the probability of practical equivalence of the latter two is 84%. this result is contrary to gorman and bedrick's replication of a standard split evaluation-they report that lapos is significantly better than the collins tagger, and that the stanford tagger is significantly better than la-pos, according to two-tailed mcnemar tests at α = .05-but it is consistent with their subsequent failure to consistently reproduce this ranking using randomly-generated splits and bonferronicorrected mcnemar tests. in contrast, nlp4j and flair are quite likely to outperform the other taggers, and flair has an 80% chance of outperforming nlp4j. similar results are obtained with sentence accuracy, a less-commonly used metric. tnt is once again quite likely to be outperformed by other models. whereas lapos is quite likely to outperform the collins tagger, there is an 82% probability that lapos and stanford taggers will yield practically equivalent results. both nlp4j and flair are both quite likely to outperform earlier models, and flair is most likely to outperform nlp4j. there is a 55% chance of practical equivalence between tnt and the collins tagger for oov accuracy. this is somewhat surprising because the two models use rather different strategies for oov inference: tnt estimates hidden markov model emission probabilities for oovs using a simple suffix-based heuristic @xcite , whereas the collins tagger, a discriminativelytrained model, uses sub-word features developed by @xcite to handle rare or unseen words. similarly, whereas nlp4j and flair also use distinct oov modeling strategies, we estimate that they have a 62% likelihood to achieve practical equivalence on this metric. ## discussion using the methods above, we obtain the following likelihood-based performance rankings: we also find some divergences from the results reported by gorman and bedrick. for instance, they find that the stanford tagger has significantly higher token accuracy than lapos on the penn treebank standard split. according to our model, the two taggers are most likely practically equivalent, a result which is consistent with their later finding that stanford outperforms lapos on only 1 out of 20 penn treebank random splits. we also find out that while both taggers were practically equivalent in both token and sentence accuracy, stanford is likely to outperform lapos in oov words, which could have impacted the statistical significance in the original experiment, as the repetition of the k-fold procedure causes strong variation -of the vocabulary available at training and oov token sets -between experimental runs. we note that bayesian comparison and the precise quantities it estimate may give insights into the particular strengths and weaknesses of the various models and evaluation metrics. for instance, we infer that whereas the collins tagger improves upon tnt, and flair improves upon nlp4j, in both token and sentence accuracy, these improvements are not likely to be due to differences in the models' handling of out-of-vocabulary words. this is because tnt and the collins tagger, and nlp4j and flair, are most likely practically equivalent in their tagging accuracy for oov words. in bayesian approaches as we are thinking about probabilities of a method outperforming another method. as a result we can do what was not possible in the nhst approach taken by gorman and bedrick. we can order methods into at a partial ordering to gain an insight into which methods are more likely to perform better than others. we can do this based on the modeled likelihoods, but it would not be in a nhst framework, because there are currently no multiple comparison correction procedures that take into account the variance of repeated runs of a method on the same data set. gorman and bedrick reported that lapos would be sure to outperform collins on ptb (20 out of 20 times), but not on ontonotes (7 out of 20 times) in token accuracy. we found out that that the most likely scenario, when the performance is modeled using a hierarchical model on evidence from both data sets jointly, that these difference are likely within practical equivalence. we set the interval of practical equivalence of observed accuracies to match the 95% confidence intervals reported by bedrick and gorman, to maintain a capacity for comparing the two experimental approaches. however, we believe it is much more useful to have an interpretable and intuitively understandable definition of what practical equivalence means in the experiment. instead of setting it based on statistical confidence intervals, we recommend selecting the rope to represent the scale of human annotator differences, or the error level that does not negatively impact a downstream task that depends on the prediction quality of evaluated methods. ## conclusions we compare the performance of six part-of-speech taggers on two data sets using twenty repetitions of a ten-fold cross-validation procedure and statistical system comparison performed using hierarchical bayesian models. by sampling from the posterior distribution of these models, we estimate the likelihood that a given tagger will be better than, worse than, or practically equivalent to other taggers on three different evaluation metrics. these estimates are valid insofar as the data sets used to estimate the bayesian models comprise a representative sample of a coherent population of data sets. this method provides a principled way to perform statistical model comparison using k-fold crossvalidation, a data-efficient evaluation technique. it also allows us to incorporate results obtained across multiple data sets and to make populationlevel inferences. we finally compare the results obtained with the proposed method to those computed using randomly generated splits and traditional nhst-based model comparison. the results provide new insights into the strengths and weaknesses of english part-of-speech tagging models, complementing other approaches to model comparison and interpretation.