R epro H um: #0744-02: Investigating the Reproducibility of Semantic Preservation Human Evaluations (2025)

Abstract: Reproducibility remains a fundamental challenge for human evaluation in Natural Language Processing (NLP), particularly due to the inherent subjectivity and variability of human judgments. This paper presents a reproduction study of the human evaluation protocol introduced by Hosking and Lapata (2021), which assesses semantic preservation in paraphrase generation models. By faithfully reproducing the original experiment with careful adaptation and applying the Quantified Reproducibility Assessment framework (Belz and Thomson, 2024a; Belz, 2022), we demonstrate strong agreement with the original findings, confirming the semantic preservation ranking among four paraphrase models. Our analyses reveal moderate inter-annotator agreement and low variability in key results, underscoring a good degree of reproducibility despite practical deviations in participant recruitment and platform. These findings highlight the feasibility and challenges of reproducing human evaluation studies in NLP. We discuss implications for improving methodological rigor, transparent reporting, and standardized protocols to bolster reproducibility in future human evaluations. The data and analysis scripts are publicly available to support ongoing community efforts toward reproducible evaluation in NLP and beyond.

## introduction reproducibility is a cornerstone of scientific progress, ensuring that research findings are reliable, verifiable, and can form a solid foundation for subsequent studies. in the field of natural language processing (nlp), where models and systems evolve rapidly, reproducibility is especially critical to validate claims and foster cumulative knowledge. central to this effort are evaluation strategies that assess model performance, broadly categorized into automatic metrics and human judgments. automatic evaluation offers efficiency and consistency but often fails to capture nuanced language understanding, whereas human evaluation provides richer, more context-sensitive insights, at the expense of scalability, objectivity, and cost @xcite @xcite . these trade-offs highlight the complementary roles of both approaches in nlp research. as nlp models increasingly approach or surpass the limits of traditional automatic metrics and benchmarks, the role of human evaluation has become more pronounced. however, the inherent subjectivity in human judgments introduces challenges for reproducibility. variability in annotator expertise, demographic factors, evaluation environments, and subtle differences in protocol implementation can all introduce variability into human evaluation results @xcite . addressing these challenges requires clear, consistent, and transparent protocols for human evaluation to preserve the integrity and comparability of nlp research. the reprohum project @xcite represents a concerted effort to address these issues by developing systematic approaches to enhance the reproducibility of human evaluations in nlp. by formalizing methodological frameworks and providing practical guidelines, re-prohum seeks to mitigate sources of inconsistency and enable more reliable comparisons across studies. this initiative complements a growing body of meta-analytical work aimed at improving reproducibility and rigor across scientific disciplines @xcite . motivated by these considerations, we undertake a focused reproduction study of the human evaluation protocol introduced by @xcite to what extent can human evaluation results be faithfully reproduced when following the original experimental setup? the remainder of this paper is organized as follows: section 2 reviews related work and foundational concepts; section 3 describes the original protocol and details our reproduction methodology; section 4 presents results and analyzes observed differences; section 5 discusses broader implications and proposes guidelines informed by our findings; finally, section 6 concludes and outlines directions for future research. the input, preference data, and analysis are made available in github @xcite . ## background human evaluation remains the most trusted method for assessing nlp system outputs, yet reproducibility in these evaluations continues to pose major challenges. these difficulties largely arise from inconsistent methodologies, ambiguous reporting, and incomplete experimental details. the field suffers from considerable heterogeneity in quality criteria and terminology, as demonstrated by @xcite , who surveyed 165 nlg papers using human evaluation and found over 200 distinct terms describing quality. crucially, about two-thirds of those papers failed to define what quality aspects they measured, and many omitted essential information such as system inputs, outputs, or participant demographics. this fragmentation hampers comparability and aggregation of results, underscoring the need for consistent evaluation designs and terminology. building on this foundation as part of the re-prohum initiative, @xcite examined 177 nlp papers with human assessments, identifying that only around 13% contained sufficiently accessible information to allow confident reproduction. frequent issues included missing participant details, unclear instructions, and methodological flaws, all exacerbated by incomplete documentation and limited author involvement. they recommended adopting structured recording protocols such as the human evaluation data schema (heds) @xcite and called for stronger standardization in experimental design to improve reproducibility. efforts to assess reproducibility directly have been made through organized shared tasks like those in the humeval and repronlp initiative @xcite @xcite . in 2023 and 2024, reprohum partners attempted to reproduce existing human evaluation studies, revealing common obstacles such as inconsistent bug fixes, procedural deviations, and variability in evaluator numbers. these factors often led to divergent results and illustrated the persistent lack of uniformity in quality criteria and evaluation protocols. such challenges emphasize the necessity for multiple reproductions and diverse quantitative reproducibility measures to ensure reliable conclusions. recent critical analyses echo and extend these concerns by highlighting fundamental limitations in human evaluation methodologies. for example, @xcite demonstrate that human preference feedback used in training and evaluating large language models tends to systematically underrepresent important aspects such as factuality, and is biased by factors like assertiveness and output complexity. similarly, @xcite survey widespread flaws affecting human evaluation, automatic metrics, and datasets, arguing that current protocols are unsustainable for distinguishing advanced models and advocating for causal frameworks in evaluation reporting. complementing these perspectives, @xcite document pervasive experimental flaws in repeated human evaluations, ranging from coding errors to deviations from scientific best practices and inaccurate result reporting. together, these studies caution that human evaluation, while indispensable, is imperfect and vulnerable to methodological weaknesses. they recommend comprehensive reforms including multiple annotators, improved experimental rigor, better annotation aggregation, transparent reporting, and stronger oversight to increase reliability and scientific rigor. taken together, these findings-from foundational surveys to reproducibility analyses, shared task experiences, and broad methodological critiques-paint a consistent picture: human evaluation is essential for nlp system assessment but is hindered by inconsistent reporting, incomplete details, ambiguous quality definitions, and practical execution flaws. there is a collective imperative to adopt comprehensive standardization of evaluation methodologies, rigorous documentation protocols, better annotator training and incentivization, and more reliable aggregation and analysis techniques. addressing these challenges is crucial for advancing scientific rigor and trustworthiness in nlp evaluation. ## experimental setup and reproduction this section details the human evaluation methodology reproduced from @xcite . we first describe the original evaluation protocol used to assess paraphrase generation models on semantic preservation, dissimilarity, and fluency. next, we outline our reproduction setup focusing exclusively on the semantic preservation criterion, including participant recruitment and survey implementation. then, we summarize key deviations from the original experiment and justify the necessary adaptations, highlighting how these changes were managed to maintain the integrity and validity of the reproduction. finally, we describe the quantified reproducibility assessment (qra) framework @xcite used to evaluate the reproducibility of our results. this framework provides a structured approach to assess reproducibility across various dimensions, including statistical consistency, inter-annotator agreement, and qualitative findings. ## results this section presents the outcomes of our reproduction study evaluating semantic preservation in paraphrase generation models. we closely followed the original human evaluation protocol @xcite , comparing four systems: the vae baseline, separator, latent bow, and dips. ## discussion our quantitative reproducibility analysis results demonstrate successful reproduction of the original human evaluation protocol for assessing semantic preservation in paraphrase generation models. this conclusion is supported by low coefficient of variation (cv*) values, strong correlation coefficients, moderate inter-annotator agreement (as measured by fleiss' κ), and confirmation of all original findings, collectively indicating a high degree of reproducibility. nonetheless, several aspects warrant further exploration. first, although the power analysis was conducted using analysis of variance (anova), linear mixed-effects models @xcite may be more appropriate for this type of data since the requirements for using analyses of variances are often not met @xcite . second, the interpretation and acceptable ranges of fleiss' κ values are context-dependent; different tasks and domains often yield varying κ distributions @xcite . currently, clear guidelines for interpreting fleiss' κ within the context of human evaluation across diverse tasks and settings are lacking. the automated evaluation of nlg systems, particularly for open-ended and creative tasks, remains an open challenge. reference-based automated metrics and emerging llm-based evaluation methods are the two primary approaches in this domain. consequently, developing automated evaluation frameworks that are both more reliable than existing metrics and more cost-effective than human evaluations would represent a significant advancement @xcite . such frameworks could enable researchers to conduct broader and more systematic evaluations, facilitate robust model comparisons across diverse tasks, and assist practitioners in selecting models best suited to specific applications. moreover, these frameworks hold promise for supporting continuous model evaluation and monitoring systems that adapt dynamically to evolving requirements and user needs. although automated evaluation techniques have advanced, traditional reference-based metrics for open-ended text generation continue to exhibit significant shortcomings. metrics such as bleu @xcite , rouge @xcite , and meteor @xcite primarily measure n-gram overlap between generated and human-written texts. while widely used, these metrics frequently fail to capture semantic equivalence and correlate poorly with human judgments @xcite @xcite @xcite @xcite @xcite @xcite . to address these limitations, recent approaches leverage contextual embeddings to better assess semantic similarity. for instance, bertscore @xcite and alignscore @xcite use pretrained language model embeddings to evaluate the closeness of generated outputs to references in embedding space. building on this concept, the llm-as-a-judge framework employs large language models directly as evaluators by utilizing their capacity to assess generated texts from multiple perspectives @xcite @xcite @xcite . this framework shows promise in aligning well with human judgments; however, challenges remain, including sensitivity to prompts, potential brittleness, and inherent biases within the models @xcite . given the increasing use of llms or foundation models as evaluators, and the growing availability of high-quality human judgment data, including over thirty studies involving human evaluation @xcite , it is essential to further investigate the reliability of such models as judgment agents. the availability of this data, combined with detailed annotation of experimental protocols, will facilitate the development of improved recommendations and practical guidelines for evaluation metrics in diverse contexts. ## conclusion this study contributes to advancing reproducibility in nlp human evaluation by successfully reproducing the semantic preservation assessment protocol introduced by @xcite . our reproduction closely matched the original results, confirming the relative semantic fidelity of diverse paraphrase generation models with strong statistical validation and moderate inter-annotator agreement. the application of the quantified reproducibility assessment framework @xcite provided a multifaceted and quantitative perspective on reproducibility, highlighting areas of strong consistency as well as aspects sensitive to experimental conditions. despite this success, challenges remain, including contextual interpretation of agreement metrics, the effects of platform and participant differences, and the necessity for more robust statistical modeling approaches. these results reinforce the need for comprehensive standardization of human evaluation methodologies, detailed and transparent documentation such as the human evaluation datasheet @xcite , and wider adoption of reproducibility-focused frameworks. looking forward, integrating improved automated evaluation methods, and particularly those leveraging llms @xcite , offers promising avenues to complement human judgments and reduce reliance on costly and variable human annotations. finally, we encourage the nlp community to embrace collaborative reproducibility initiatives such as the reprohum project @xcite and to make evaluation data, protocols, and analyses openly accessible. such collective efforts are crucial to strengthening the scientific rigor and trustworthiness of human evaluation in nlp, thereby accelerating reliable and cumulative progress in the field. our own data and analysis scripts supporting this reproduction are available in a public repository @xcite .