Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization (2022)

Abstract: Replication of research results has become more and more important in Natural Language Processing. Nevertheless, we still rely on results reported in the literature for comparison. Additionally, elements of an experimental setup are not always completely reported. This includes, but is not limited to reporting specific parameters used or omitting an implementational detail. In our experiment based on two frequently used data sets from the domain of automatic summarization and the seemingly full disclosure of research artefacts, we examine how well results reported are replicable and what elements influence the success or failure of replication. Our results indicate that publishing research artifacts is far from sufficient, that that publishing all relevant parameters in all possible detail is cruicial.

## introduction replicability is gaining more and more attention in the nlp world with dedicated workshops foot_0 , replication checklists foot_1 etc. while this improves the situation considerably, and the availability of research artifacts is improving, there is still the question if replicability is possible if all artifacts necessary are available. additionally, often results from the literature are cited, but it is far from clear whether the reported results are obtained experimentally (by re-implementing or re-running a particular method) or also cited. one domain where the availability of research artifacts is almost perfect, is automatic summarization. standard benchmark data sets published in the course of various shared tasks are available, the evaluation method is well known, its implementation is available and resulting data submitted to shared tasks have also been made available by the organizers. therefore, it should be straightforward to replicate results reported by the organizers of the shared task, as well as results reported in the literature. this would hardly be a submission to a workshop on insights from negative results if things were that easy. normally, successfully replicating previous results would just appear as one or more number in a table used for comparison. but our results indicate that despite this near-perfect conditions, reporting and replicating results is far from straightforward. based on a literature review and experiments in replicating results we show the discrepancies that occur both in cited results, as well as when experiments are replicated. our contributions are therefore a closer look and comparison of reported results from the domain of automatic summarization and results from replicated experiments and factor benefitting or hindering complete replication. ## replication in nlp experiments in reproducing results in the nlp domain such as those presented by @xcite are still quite rare. one reason is, that when undertaking such projects, "sometimes conflicting results are obtained by repeating a study" foot_2 . @xcite report, that their experiments on two tasks in nlp are difficult to carry out and to obtain meaningful results. preprocessing, experimental setup, versioning, system output, and system variation cause experimental variation according to the authors. the 4real workshop 4 focuses on the "the topic of the reproducibility of research results and the citation of resources, and its impact on research integrity". their call for papers asks for submissions of "actual replication exercises of previous published results" @xcite . results from this workshop suggest that reproducing experiments gives additional insights, and is therefore beneficial for the researchers as well as for the community @xcite . @xcite present a study on the replication of results in the context of part-of-speech tagging and whether lstms really work as well as the literature suggests. the results are mixed and show that the replicability depends on parameters such as tagset complexity. crane (2018) looks into the area of question answering and finds that "source code without a reproducible environment does not mean anything". the author presents a set of experiments to show, that different parameters can lead to different results, similar in magnitude to those reported in the literature. @xcite give a more general overview on this issue, as they perform a replicability study on various nlp tasks. they find that the increasing amount of evaluation data sets is a two-edged sword and only beneficial if the data reflects a variety of linguistic phenomena and are heterogeneous at least with respect to language or domain. otherwise, showing that results are valid on one data set is probably sufficient. other authors look into the availability of research artifacts (i.e. @xcite who found that a large proportion of research artifacts are not available. a recent study by @xcite systematically looked into the replicability of various publications from the nlp domain, finding, that only approx. 14 % of the examined publications were replicable. for both data sets the organizers of the shared task published reference summaries as well as submitted summaries. furthermore, the evaluation results are available as well. @xcite introduced an automatic evaluation metric, which became the standard both for subsequent shared tasks, as well as for automatic summarization in general. rouge has a range of parameters, which have to be set prior to running the evaluation. several of these parameters are not binary, which results in a extensive parameter space. @xcite gives details on these parameters and the resulting issues. both data sets that have been widely used in the past 15 to 20 years and therefore provide a reasonable basis for our analysis, which contains three parts: first, we will look into results reported in the literature and we aim to replicate those reported results. second, we use available summarization methods out of the box or retrain them and evaluate the results. third, we use a data set published by @xcite to replicate their results. in our experiments, we stick as close as possible to the description offered in the cited publications and cite the results given. ## discussion we looked into the question of whether the fact that all necessary research artifacts are available for specific benchmark data sets in automatic summarization allow for a straightforward evaluation and replication. we also looked into results reported in the literature, as often results are cited in subsequent works as baselines or for comparison. we observed quite severe differences not only in the exact values obtained by running the evaluation, but also in the conclusions drawn from these with respect to the ranking of the system outputs. we also observed that the results highly depend on the parameters used for evaluation. if evaluation parameters and system output results are given, results are reproducible, as we were able to show with the data and results presented by @xcite . using their data and the evaluation parameters, our results were almost identical to those reported in the original publication. as only some results differed, it remains open if the observed differences are due to changes on the hardware and/or software level. also, not all three evaluation metrics differed. as most values were in the range @xmath0.1 one assumption is, that this is due to differences in rounding. in order to evaluate this, a more detailed analysis of individual results is required. if the method used to produce the summaries has been described in enough detail, it is possible to achieve similar results as we did with work by @xcite . despite the seemingly ideal circumstances, we failed to reproduce the results for system 65 in duc 2004. for the duc 2002 task we were only partially able to replicate or reproduce results reported in the literature, despite similar circumstances. we could not reproduce results reported in the literature. also our experiments with mead were not conclusive. they showed that depending on the parameters used for evaluation, the results can vary considerably, sometimes even significantly, even though the system implementation is available and the evaluation metric is known. a closer look at the publications analyzed for this study, we found that only about 40% report the full set of evaluation parameters. almost 50% of the publications did not mention the evaluation parameters at all. 12 replicating or even reproducing results for these publications is therefore unnecessarily complicated and involves testing all possible combinations of parameters. as the correct parameter set is unknown in these cases, comparisons are as not as valuable as they could be. additionally, re-implementations such as py-rouge foot_12 do not offer all the parameters rouge originally offered, making comparisons even harder. therefore, one of our next steps is to re-evaluate the presented experiments using py-rouge. more analysis, also in other areas of nlp would be beneficial to strengthen the results of this study. while rouge has quite an extensive parameter range, it is negligible compared to modern machine learning approaches and as has been pointed out by crane (2018) they "often go unreported". nevertheless, our results highlight a problem that will become more severe the more complicated the methods developed in nlp become: disclosing all parameters used for creating and evaluating a specific system is crucial. publishing the algorithms and the resulting data is not enough to ensure replicable results. and even having details about the evaluation procedure (including relevant parameters) does not ensure that results can be replicated and conclusions in line with previous work can be drawn. while this might sound trivial, our results indicate that this is not being done in enough detail to ensure replicability and reproducibility of results.