Exploring Variation of Results from Different Experimental Conditions (2023)

Abstract: It might reasonably be expected that running multiple experiments for the same task using the same data and model would yield very similar results. Recent research has, however, shown this not to be the case for many NLP experiments. In this paper, we report extensive coordinated work by two NLP groups to run the training and testing pipeline for three neural text simplification models under varying experimental conditions, including different random seeds, run-time environments, and dependency versions, yielding a large number of results for each of the three models using the same data and train/dev/test set splits. From one perspective, these results can be interpreted as shedding light on the reproducibility of evaluation results for the three NTS models, and we present an in-depth analysis of the variation observed for different combinations of experimental conditions. From another perspective, the results raise the question of whether the averaged score should be considered the ‘true’ result for each model.

## introduction recently there has been a promising surge of interest in reproducibility of nlp models, supported by challenges @xcite , shared tasks @xcite , conference tracks @xcite , and even the reality check theme at this conference. the outcome of this surge in interest has been a flurry of reproducibility studies and related investigations @xcite @xcite . however, the collective findings from these efforts have been alarming. with interest in reproducibility growing, the evidence is mounting that scores are substantially affected by changes not only to arbitrary factors like random seed and different data splits, but also by incidental factors such as the type of gpu on which an experiment is run, and the run-time environment. in many cases, near-identical scores can be guaranteed only when an experiment is re-run in fully containerised form. in effect, this means that even perfect sharing of information (once regarded as the answer to all our reproducibility problems foot_0 @xcite ) cannot guarantee identical results in all cases. all this raises questions about reporting, experimental design and the informativeness of scores regarding the relative merits of different methods. underlying these is the question of where the boundary lies -seemingly between the two extremes. on the one hand, exploration of methodological variations and reporting of separate scores is part and parcel of method development. on the other hand, arbitrary and incidental factors such as random seed are not part of method development, because they do not generalise to future applications of the same method. for the former, clearly, comparing and reporting different scores is important; for the latter, how to interpret, address or report variation in scores is an open question. in this paper, we tackle this question by conducting a systematic and comprehensive investigation coordinated across two nlp groups to study the variation of the results across three neural text simplification (nts) models under many different experimental conditions. we experiment with different random seeds, run-time environments, and dependency versions to ensure broad coverage of our study. we observe that reporting average score and its coefficient of variation is a more reliable standard than reporting the maximum value, and we urge researchers to record all methodological conditions, control incidental ones, and abstract away arbitrary factors to promote the reproducibility of their scientific contributions. in the following subsections we describe the four different systems ( §2.2), the single data set/split and four text processing variants ( §2.3), and the two evaluation methods ( §2.4) which were included in our exploration, either because they were part of the original study or because we added them. §2.5 provides an overview of the incidental and arbitrary variation arising in our different runs which we also analysed. ## results we report the results from both team a and team b, for each of the studied conditions. while both teams struggled to get the original repository to a working state, team a failed to install all the required dependencies as many are deprecated. team b reported similar concerns about reproducing and reusing the original source code; however, ultimately, they managed to get the repository to a running state. for random seed search, we included two worst and best-performing models in this table, while full results of this search can be found in appendix. averaged scores for each of the three models together with the standard deviations and coefficients of variation @xcite are presented in according to the averaged sari score, the transformer model performs best; however, the newest version performs worse than the old one. according to the averaged bleu score, 6 lstm-v2w and transformer have very similar performance, but the newest version of the transformer is the best of all while the first version is the worst. we used the r package cvequality (version 0.2.0; @xcite ) to test for significant differences of coefficients of variation (cv). this package implements two of the most widely used statistical significance tests, proposed by @xcite and @xcite . the null hypothesis for each of the two automatic metrics is that there is no difference in cv between the three models. we use the results reported in the besides the mentioned analysis, we found it hard to provide distinct and unique observations from the results. this is likely due to the fact that the results are not conclusive and the variance is high. we do not believe this is a flaw in our experimental design but rather a good representation of the complexities of comparing different models across varying conditions. the number of experiments conducted in this study is more than 60, a number that exceeds the number of experiments conducted in most other studies by a large margin. one of the concerning issues we encountered is the issue of software deprecation. while this is not a new problem, and it is as old as software itself, it is becoming more and more prevalent. this is due to extreme reliance on empirical results and the complexity of publications that utilise neural networks. often source codes use several external libraries and dependencies, any of which may become deprecated at any time. increased availability of source code and the abundance of tools are signs of a healthy research community. seeing new tools and libraries developed and improved daily is encouraging. at the same time, we believe researchers should practice caution when introducing new tools and libraries into their experiments, as doing so may shorten the usability of their source code. ## conclusions in this work, we conducted a series of experiments for a single task using the same data under different experimental conditions. we categorized these conditions into three different categories: methodological, arbitrary, and incidental. we report the results of our experiments to demonstrate the wide results variation that can occur due to these factors. we propose that researchers should record all methodological conditions, control incidental ones, and abstract away arbitrary factors. lastly, we observed that using average score and its coefficient of variation (cv) instead of the maximum value provides far more reliable results. we recommend that researchers adopt this practice when documenting the findings from their own studies. we are aware that this is easier said than done. we are, however, optimistic that the field can move closer to this ideal over time. in the meantime, it is our hope that this recommendation highlights the contrast between what is currently a common practice (unfortunately, inadequate recording and reporting that do not address necessary factors for reproducibility) and what is needed to support successful, reproducible research in our field.