R epro H um #0729-04: Partial reproduction of the human evaluation of the M em S um and N eu S um summarisation systems (2025)

Abstract: In this paper, we present our reproduction of part of the human evaluation originally carried out by Gu et al. (2022), as part of Track B of ReproNLP 2025. Four human annotators were asked to rank two candidate summaries according to their overall quality, given a reference summary shown alongside the two candidate summaries at evaluation time. We describe the original experiment and provide details about the steps we followed to carry out the reproduction experiment, including the implementation of some missing pieces of code. Our results, in particular the high coefficients of variation and low inter-annotator agreement, suggest a low level of reproducibility in the original experiment despite identical pairwise ranks. However, given the very small sample size (two systems, one rating), we remain cautious about drawing definitive conclusions.

## introduction in recent years, several editions of the reprogen and repronlp shared tasks have been carried out -see, e.g., @xcite )-, which contributed to making the nlp community more aware of the importance of reproducibility when running and reporting on experiments. this year, the re-pronlp organisers proposed two tracks @xcite : track a (open) was for reproductions of any evaluation result, while for track b (repro-hum), a set of 20 papers was preselected based on their suitability for being reproduced (availability of code, of instructions to evaluators, of detailed evaluation results, etc.). the present paper reports on one of the two reproductions for paper #0729-04 from @xcite : memsum: extractive ## original experiment this section contains a summary of the original experiment and a detailed description of the human evaluation procedure. ## our reproduction in this section, we describe which experiment we reproduced and how we carried it out. all our code and documentation can be found on github, 1 and details of our evaluation can be found in the human evaluation data sheet (heds) @xcite ). 2 ## results and discussion ## conclusions we conducted a reproduction study of @xcite overall quality human evaluation of two summarisation systems, memsum and neusum. even though the outcome of our study is at first sight in line with the original study's results, mem-sum achieving a slightly higher overall score that neusum with no statistically significant differ-ence, both our quantified reproducibility assessment results (high coefficients of variation) and our detailed analysis of the global and per-annotator scores (marginal overall system scores difference and a very low inter-annotator agreement) suggest a low level of reproducibility of the original study. thus, our interpretation of the evaluation results differs slightly from that of the original study: based on our analysis, the two systems appear to be very similar in terms of quality. this similarity may be attributed to both memsum and neusum being extractive summarisers, with a significant proportion of the sentences selected by each system overlapping, which could make judgments difficult for annotators (i.e. because it is a relative evaluation, ranking two similar things is hard). however, considering the very small sample size (two systems, one criterion), we remain cautious in our interpretation. more reproductions would be needed to draw more solid conclusions. finally, although the reproduction process was not entirely straightforward and required some effort (see section 3), we found that the majority of the necessary materials were available, and the reproduction in general was feasible and relatively smooth.