Verification, Reproduction and Replication of NLP Experiments: a Case Study on Parsing U niversal D ependencies (2020)

Abstract: As in any field of inquiry that depends on experiments, the verifiability of experimental studies is important in computational linguistics. Despite increased attention to verification of empirical results, the practices in the field are unclear. Furthermore, we argue, certain traditions and practices that are seemingly useful for verification may in fact be counterproductive. We demonstrate this through a set of multi-lingual experiments on parsing Universal Dependencies treebanks. In particular, we show that emphasis on exact replication leads to practices (some of which are now well established) that hide the variation in experimental results, effectively hindering verifiability with a false sense of certainty. The purpose of the present paper is to highlight the magnitude of the issues resulting from these common practices with the hope of instigating further discussion. Once we, as a community, are convinced about the importance of the problems, the solutions are rather obvious, although not necessarily easy to implement.

## introduction the practice of independent verification of empirical findings has been at the core of the modern science. there have been, however, recent failures of reproduction in many fields @xcite , which also instilled public interest with the popular name reproducibility crisis @xcite . in computational linguistics and natural language processing (nlp), worries about reproducibility have been voiced for over a decade with a notable increase in recent years @xcite @xcite @xcite @xcite , just to name a few). as some of these studies point out, it is often unclear what is meant by the terms replication and reproduction. the definitions of these terms are often blurred, and they may even be used in opposite meanings in different studies @xcite . in this paper, we use the term (exact) replication to refer to the activity of running the same code on the same data set with the aim of producing the same measurements reported in the original publication. we use the term reproduction to refer to the activity of verifying the claims or findings by varying the experimental settings in meaningful ways. as argued earlier @xcite , the scientifically interesting and useful activity is reproduction, while replication has rather little or no use for the purpose of scientific verification. the confusion, however, is not only limited to the usage of these terms. in many studies, the aim of the activity is also unclear, or often understood as obtaining the same numbers reported in an original study (in our terms exact replication). the emphasis on state-of-the-art scores (even with small increase over the previous state of the art) also motivates the exact, 'fair', comparisons based on a single-best score. on the other hand, many of the statistical data-driven methods involve a number of inevitable sources of variation. for example, any machine learning method will produce varied results when trained and tested on different parts of the data, and many others are also sensitive to other stochastic processes, such as random initialization of their parameters or the order of training instances. even though this variation is important for evaluating and comparing statistical models, there has been an increasing emphasis on exact replication of the published scores. for example, the semeval 2020 reviewer form includes the following scoring instructions for reviewers (emphasis ours): hence, although lightly, not being exactly replicable due to sample variation is penalized, encouraging participants to eliminate, or hide, all sources of variation. the same or similar reviewing criteria have recently become standard for major publication venues for computational linguistics. in this paper, we show that the emphasis on exact replication, in fact, hinders the idea of verification of the results. we support this claim through parsing experiments on universal dependencies treebanks @xcite , investigating the effects of two common or well-established practices that aim to facilitate replication. namely, fixing random seed of the pseudo random number generator used during experiments, and standard training, development and test set splits. ## experimental setting our experiments consist of replicating and reproducing parsing scores reported with version 1.2 of ud-pipe @xcite on ud version 2.5 treebanks @xcite . udpipe is an open-source pipeline for tokenization, tagging, and dependency parsing designed particularly for the ud annotation scheme. the system obtained near-top results in the recent conll shared tasks on dependency parsing from raw text @xcite . besides its performance and the ease of use, another interesting aspect of udpipe for the current study is the fact that maintainers provide pre-trained models for the majority of the ud treebanks, where each model is trained and tested on the standard ud splits. the authors also publish common performance metrics, and provide scripts for replication. 1 our experiments consist of (1) exact replication of the reported results using the same code and the standard splits, (2) reproducing the results by varying the random seed, and (3) reproducing the results on random treebank splits. to facilitate exact replication of the published scores, the udpipe model distribution fixes the random seed used for initializing the model parameters and other sources of variation (e.g., shuffling of training instances). for our main experiments, the udpipe is modified to set the random seed based on the time stamp (a reasonably random quantity for our purposes). we use the same embeddings distributed with the udpipe models, and the same hyperparameter settings as the original models. the experiments are run using the scripts provided in the udpipe 2.5 models package, with slight modifications for parametrizing the input, and parallelizing it with gnu parallel @xcite . we report results for all treebanks for which a udpipe 2.5 model was published, except for the largest five treebanks. 2 for the experiments with different data splits, we simply combine all the sentences from the standard splits, and create 10 versions of each treebank by randomly splitting the sentences into train, dev, and test sets of the same size as the original split. in all cases, since the parser does not use any extra-sentential information, we sample the sentences randomly, without paying attention to document boundaries even if they are marked. in the main text, we only present and compare labeled attachment scores (las). other metrics, including tokenization and tagging scores, are presented in appendix a. ## results we first verify the exact replication of the published scores. besides the fixed treebank splits, since original study eliminates all sources of random variation, our results exactly match theirs. to establish the expected variation of the parser on the same treebank splits, we perform multiple experiments with standard splits after allowing random variation in model initialization. the black bars af afribooms ar padt be hse bg btb ca ancora cop scriptorium cs cac cs cltt cs fictree cu proiel da ddt de gsd el gdt en ewt en gum en lines en partut es ancora es gsd et ewt eu bdt fa seraji fi ftb fi tdt fr gsd fr partut fr sequoia fr spoken fro srcmf ga idt gd arcosg gl ctg gl treegal got proiel grc perseus grc proiel he htb hi hdtb hr set hu szeged hy armtdp id gsd it isdt it partut it postwita it twittiro it vit ja gsd ko gsd ko kaist la ittb la perseus la proiel lt alksnis lt hse lv lvtb lzh kyoto mr ufal mt mudt nl alpino nl lassysmall no bokmaal no nynorsk no nynorsklia orv torot pl lfg pl pdb pt bosque pt gsd ro nonstandard ro rrt ru gsd ru taiga sk snk sl ssj sl sst sme giella sr set sv lines sv talbanken ta ttb te mtg tr imst ug udt uk iu ur udtb vi vtb wo wtb zh gsd zh gsdsimp in figure 1 present the range of las obtained in 10 experiments on the standard splits (all las figures presented in this paper are percentages). 3 all scores in figure 1 are relative to the scores published by the original study, a positive value indicating a score higher than the published score on a particular treebank. the scores do not diverge from the original study substantially. nevertheless, there is considerable variation. the range of scores are between 0.40 and 7.89 las points with an average of 1.30 las. the range of las scores from 10 experiments with different random seeds do not contain the published result in 21 of the 90 experiments. it is also worth noting that in cases the reported value is within the range, it is not necessarily at the center of the range. as expected, the range of scores obtained on random treebank splits (shown as light blue bars in figure 1 ) indicate an even wider spread. the ranges vary between 0.65 and 9.68 las points, with an average of 2.25. the scores of random-split experiments also differ from the published results considerably. the las on random splits and the published results differ 2.72 points on average, with a maximum of 12.30. in the majority of the cases, the published results are not within the range of scores obtained on randomsplit experiments, only 33 out of 90 differences falling within this range. most differences however, are positive. minimum las obtained in random-split experiments are higher than the published results for 53 treebanks, indicating that standard splits underestimate the success of the parser. this is expected, since most treebanks do not sample test sets randomly, conflicting with the i.i.d. assumption of many machine learning models. the published results are above the random-split range only for 4 treebanks, probably due to the official splits with easier to parse test sets. ## discussion the experiments presented above demonstrate the effects of two sources of variation on parsing scores. the variation displayed in figure 1 is rarely reported in the literature. furthermore, common practices promoted for enabling exact replication leads researchers to eliminate this variation from their experiments, reporting single-figure scores without an indication of the expected variation. such single-figure scores, as demonstrated above, are often biased in an arbitrary direction. the first type of variation we discuss is due to model training and evaluation, e.g., because of random initialization of the model parameters, or random ordering of the data before online or batch training. any non-trivial statistical model produces different results when trained and tested on the same data. this is typically negligible for convex optimization procedures. 4 the variation is naturally higher for models trained with non-convex optimization, such as neural networks which are the dominant approach to parsing as well as many other nlp tasks. the variation stemming from these two sources is rarely reported in the literature. on the contrary, due to heavy emphasis on replicability in the field @xcite , for example), when researchers release their software, they often fix the variation in an arbitrary way, e.g., by fixing the initial random seed used by pseudo random number generators as in our case study above. the point here is not to promote systems with large variation. we want to reduce the variation when possible. for example, ensembling or model averaging are useful for reducing the variation. however, there will still be some variation based on initialization differences and sampling, and reporting this variation is useful for understanding the success of the model, as well as for meaningful model comparison. our experiments indicate the range of las obtained by udpipe on ud treebanks over 10 random initializations is 1.30 on average, and goes up to 7.89 on individual treebanks. to facilitate a rough comparison, the las (averaged over all treebanks) differences between udpipe and the systems above and below udpipe in conll 2018 shared task are less than 0.20, indicating that the arbitrary choice of random seed could have made the difference in this ranking. the solution to this problem is clear. rather than 'hiding' the variation by settling on a fixed model instance, we should report the variation caused by the inevitable sources of variability. as the quote from the review form in section 1 demonstrate, reporting results with sampling variation is perceived as a negative property of a paper submitted to many computational linguistics conferences. hence, to be able to increase chances of acceptance of a paper, the authors are encouraged to fix all sources of (natural) variation of models in an arbitrary manner. the present paper, hopefully, makes it clear that this is wrong. however, since the incorrect behavior is currently being promoted by the community at large. the real solution lies in further discussion of what should really be expected from individual experimental studies. the second issue we discuss is the adverse effects of standard training, development and test set splits, which has also been brought up by others before @xcite . the standard splits are common among all nlp data sets, but tradition is probably older for treebanks, going back to early parsing research. 5 the recent treebanking efforts, e.g., ud treebanks, continued this tradition with official splits of training, development, and test sets. the standard test sets are often justified by replicability of the results obtained on these data sets, and fair comparisons between different systems. another motivation for test split is to avoid 'peeking into test data' @xcite . however, the latter goal is difficult to entertain for a standard test that remains fixed for decades. independent of the motivation of the standard splits, the experiments reported above show that the use of standard treebank splits hides a large amount of variation, and it often biases the parsing results arbitrarily. when the scores are calculated over multiple random splits, the variation can be as large as 9.68 las and the average scores obtained can differ from the reported results up to 12.30 las. furthermore, for the majority of the treebanks, scores reported in the original study are not in the range of scores obtained in random-split experiments. clearly, these differences blur model comparisons. more importantly, in any practical use of these tools, the user should be aware of the expected variation. fortunately, in our experiments, most of the results would come as a pleasant surprise, since we obtained ranges of scores better than the original report. this also means the standard test set splits are biased in certain ways, and, hence, model comparison on these sets may arbitrarily favor certain types of parsers. the solution is, again, obvious: reporting results (with their variation) on different data splits. in other words, abandoning standard splits. however, this goes against the well-established traditions. again, true solution require more discussion of the issues in the research community. many data sets include a standard training, development and test set split. as a result, for fair comparison, researchers are encouraged, or sometimes required to report results on a single standard test set. for example, it is unlikely for a paper on parsing to be accepted if it reports results using non-standard splits. hence, similar to reporting sampling variation, individual authors are forced to follow current practices. the real solution requires community-wide changes to acceptable practices. the issues discussed above are orthogonal to the issue addressed by statistical significance testing. although statistical assurances for significant differences have their merits, it is important to note that they indicate statistically significant differences on a single test set. hence, in their typical application, they may even contribute to a false sense of significant difference that disappears when the systems are tested on different test sets, or even the same model trained with different initializations. interestingly enough, the differences between the minimum and the maximum las scores in our experiments conducted on the standard test sets with varying random seeds are statistically significant for all treebanks. 6 hence, a statistically significant finding reported, e.g., for two alternative training settings, may very well be due to the random initialization at least with a 10 % chance according to our experiments. the use of above-chance statistical differences are only helpful when the systems are tested on multiple data sets @xcite . however, in most tasks, multiple data sets are not a luxury we enjoy, especially when the research or application at hand is in a particular domain. using multiple random splits of available data or use of cross-validation is often the closest we can get to a general solution. hidden in the issues related to variation with different data splits is the size and quality of the data sets. for many languages or tasks, the data sets are small and sometimes low quality, which eventually affect the models trained and tested on them. creating, annotating and curating data sets is a labor-intensive, expensive, yet error-prone task. on the other hand, creating more data sets for which there are already existing resources, and improving the existing data sets are not rewarding tasks. again, to be able to test our models on multiple and better data sets, we, as a community, need ways to encourage and reward creation and maintenance of more and high-quality data sets. although our case study is based on parsing results with udpipe, the problems outlined are neither specific to udpipe nor parsing. in any statistical, data-driven nlp experiment, some amount of variation is inevitable in experimental results, and the standard test set splits and avoiding variation due to other factors in model training hides this variation. ## concluding remarks this paper presented a demonstration of adverse effects of avoiding variation in the results of nlp experiments. we demonstrate the effects of two sources of variation in nlp experiments: the variation due to sampling and initialization. for sampling variation, a well-established practice in the literature is to use a fixed standard training-test split across different studies. the only clear advantage of using a single standard test set is allowing fair comparisons between different studies. a fair comparison is definitely desirable in a competition, but for research we should perhaps emphasize meaningful comparisons more. the results presented above demonstrate that comparisons on a single (standard) tests set are not necessarily meaningful. another, less-established but still common practice is to fix the random seed of the pseudo random number generator used during the experiments. we show that both practices hide the inherent variation in parsing scores, and bias the published results in arbitrary ways. as a result, these practices also defeat their very own purpose, making it difficult to interpret the comparison between different models outside this fixed setting. the purpose of this paper is to point out the problem and promote further discussion of the topic. our outcomes indicate that for insightful evaluations, we should acknowledge and report the variation in experimental results, rather than fixing the variation in arbitrary ways. this includes, (1) letting the model initializations vary, and (2) running the experiments on multiple data splits rather than a single training/development/test split. for both cases, reporting the variation based on multiple experiments. however, the implementation of the solutions is not easy. it requires community-wide changes beyond the power of individual researchers.