R epro H um #0729-04: Human Evaluation Reproduction Report for “ M em S um: Extractive Summarization of Long Documents Using Multi-Step Episodic M arkov Decision Processes” (2025)

Abstract: Human evaluation is indispensable in natural language processing (NLP), as automatic metrics are known to not always align well with human judgments.However, the reproducibility of human evaluations can be problematic since results are susceptible to many factors, the details of which are often missing from the respective works.As part of the ReproHum project, this work aims to reproduce the human evaluation of a single criterion in the paper “MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processes” (Gu et al, 2022).The results of our reproduction differ noticeably from those of the original study. To explain this discrepancy, we discuss differences in the experimental setup, as well as more general characteristics of the selected domain and the generated summaries.

## introduction human evaluation is generally considered the gold standard in nlp research @xcite . while automatic metrics are usually easier and cheaper to use, they have been shown as problematic in different ways: for example, standard metrics are often used in inappropriate settings and without reporting important details such as version information, and they do not always correlate well with human judgments @xcite @xcite @xcite . human evaluations can solve some of those issues, but come with their own challenges. apart from higher costs and time expenditures, it has been shown that human evaluations in existing research do not always rely on the same terminology @xcite and that the evaluation outcomes can be affected by a multitude of parameters, the details of which are often missing from reports @xcite . as a consequence, reproducibility is a core issue for human evaluation, potentially casting doubt on the validity of reported results and conclusions @xcite . against this background, the reprohum project and associated repronlp shared task @xcite aim to systematically test the reproducibility of human evaluations and strengthen transparency and reliability in nlp research. as part of this project, we attempt to reproduce the human evaluation in the paper "memsum: extractive summarization of long documents using multi-step episodic markov decision processes" @xcite . in the following, we first outline the content of @xcite 's work. after that, we describe the details of the human evaluation carried out in this study and the differences from the original work. finally, we compare the results of our reproduction study with the original findings. ## original study in their work, @xcite look at extractive summarization, i.e., selecting a subset of sentences from a source document which adequately summarize the content of the full text. for this, the authors propose memsum, an extractive summarizer based on reinforcement learning that is designed for long documents. memsum utilizes a multi-step episodic markov decision process to iteratively select sentences from the source document. for each decision, the system considers a broad set of information, i.e., the content of the current sentence, the global context of the rest of the document, and the sentences selected in previous steps. the system is tested with rouge @xcite ) as an automatic metric, showing state-of-the-art performance on long document datasets such as pubmed, arxiv @xcite , and govreport @xcite . in addition to this, @xcite human evaluation where memsum is compared to a strong baseline, i.e., the existing neusum @xcite summarizer. the evaluation is divided into two parts, where memsum generates summaries with adaptive length (experiment i) or is fixed to a number of 7 sentences (experiment ii). in both experiments, evaluators rate summaries for texts from the pubmed dataset which are generated by memsum and neusum, respectively. the generated summaries are compared to ground-truth abstracts written by humans with respect to coverage, non-redundancy and overall quality. the results show that neusum achieves slightly better coverage, but memsum summaries are rated significantly higher for non-redundancy. memsum also exceeds the baseline for overall quality, although this difference is not statistically significant. in this work, we aim to reproduce experiment ii in @xcite , focusing on the overall quality criterion and disregarding coverage and nonredundancy. with regard to this scope, the main finding of the original study can be summarized as follows: memsum generates summaries of higher overall quality than the neusum baseline. ## method in our evaluation setup, we tried to follow the procedure of @xcite as closely as possible. our code is available on github 1 . more details can be found in our human evaluation data sheet (heds, shimorina and belz 2022; belz and thomson 2024) 2 . 1 github.com/clause-bielefeld/reprohum0729-04 2 github.com/nlp-heds/repronlp2025 ## 563 sum and the other by the baseline neusum system. using html radio buttons, the evaluators should indicate which of the two generated summaries has a higher overall quality or is more consistent with the reference summary. for additional assistance, the interface included a highlight function that marks text spans in color that correspond to the content of an input query. sent2vec sentence embeddings @xcite were used to determine the relevance of text passages. for each item, the system rated as better is ranked #1, while the other is ranked #2. as in the original paper, we tested the item pairs for instances where both systems gave the exact same response and replaced the evaluator ratings with rank #1 for both systems in those cases. in our results section we report the mean ranks per system, averaged over all items and evaluators. ## reproduction results the results of the original study and our reproduction can be seen in ## discussion as discussed in the previous section, we were unable to reproduce the main findings from @xcite with regard to the overall quality criterion in experiment ii. while the proposed memsum model surpassed the neusum baseline in the original study, our results show the opposite trend, i.e., neusum is rated as better than memsum on average. the high cv* scores corroborate these differences. however, it is important to note that the difference between the two systems is fairly small and not statistically significant, and the interannotator agreement reveals substantial differences in the judgments of individual evaluators. reasons for the deviations from the original results and the measured uncertainty in our evaluators can be seen both in properties of the stimuli and in the experimental setup of this reproduction. properties of stimuli as described in section 3.1, @xcite used samples from the pubmed dataset for their evaluation. importantly, this dataset consists of domain-specific texts from the medical field written in highly technical language, making it possible that evaluators, who are not domain experts, struggle to evaluate the content and textual quality of the summaries. in addition, both systems often select similar sentences: as shown in the histogram in figure 2 , in more than half of the evaluation samples the outputs of the two systems overlap by at least 5 sentences, with a total length of 7 (see section 3.1). as a result, the summaries of both systems often only vary in detail, making it difficult to rank the methods. both of these aspects -the technical jargon and the high similarity between generated summarieswere named by participants as complicating factors subsequent to the evaluation. ## 565 differences in experimental setups another reason for the discrepancies between the original results and our reproduction may lie in the definition of the quality criterion. the notion of overall quality is relatively underspecified, which could lead to uncertainty regarding the exact properties of the texts against which they should be evaluated, although the evaluation interface provides somewhat more precise instructions, see section 3.3. importantly, as noted in section 3.4, @xcite also included ratings for coverage and non-redundancy, which could affect the evaluation of overall quality -for example, if the ratings for these more specific criteria are included in the evaluation of overall quality. finally, as described in section 3.4, the evaluation interface in the original study included an option to skip items if the summaries were too similar or if a decision could not be made for other reasons. our interface lacks this function, forcing evaluators to decide on a ranking in all cases. given the high similarity between the generated summaries for many items, this could be a reason for a higher rate of arbitrary decisions compared to @xcite 's evaluation, although it is unknown how many items were actually skipped in the original study. ## conclusion in this paper, we attempted to reproduce the human evaluation from the work of @xcite . our evaluation produced clear differences from the original results, in particular we could not demonstrate that the proposed memsum system produces summaries with higher overall quality than the baseline neusum system. at the same time, the narrow margin between the systems and the low inter-annotator agreement suggest fundamental uncertainties among our evaluators. to explain these discrepancies, we discussed differences between the original study and our reproduction, as well as general characteristics of the chosen domain and the generated summaries. the mixed results in our study underline the problem that it is often difficult to reproduce the results of human evaluations in published papers, and stress the importance of projects like reprohum.