R epro H um #0744-02: A Reproduction of the Human Evaluation of Meaning Preservation in “Factorising Meaning and Form for Intent-Preserving Paraphrasing” (2025)

Abstract: Assessing and improving the reproducibility of human evaluation studies is an ongoing concern in the area of natural language processing. As a contribution to this effort and a part of the ReproHum reproducibility project, we describe the reproduction of a human evaluation study (Hosking and Lapata, 2021) that evaluates meaning preservation in question paraphrasing systems.Our results indicate that the original study is highly reproducible given additional material and information provided by the authors. However, we also identify some aspects of the study that may make the annotation task potentially much easier than those in comparable studies. This might limit the representativeness of these results for best-practices in study design.

## introduction reproducibility is a central requirement for human evaluation studies. given the same data and setup, other researchers should be able to independently arrive at similar conclusion as the original works. however, in practice, reproducibility of human evaluation studies remains problematic in the field of natural language processing @xcite . in this context, systematic reproductions of human evaluation studies play an important role in assessing the state of reproducibility in the field and in establishing best practices. as part of the reprohum project @xcite , this report describes our effort to reproduce a human evaluation study of paraphrasing systems, originally conducted by @xcite . based on information submitted by hosking and lapata to the reprohum organizers, we attempt an otherwise independent reproduction that closely mirrors the original study. we also provide an heds (shimorina @xcite form for our reproduction study, which is accessible in the shared repronlp repository. 1 our results indicate that the original study is highly reproducible, even in the face of a change in annotator recruitment and study scope. 2 however, we also find that this is in part due to the large quality differences in the systems in the study and not an exclusive consequence of the original design decisions. ## original study the basis of our reproduction study is an evaluation of paraphrasing systems conducted by @xcite . they propose a neural paraphrasing system that, given an input question, outputs a distinct paraphrase that conserves the original meaning. the system combines two encoder representations to generate the paraphrases: a continuous variational representation derived from the input to represent question semantics and a discrete syntactic representation to indicate the desired surface form of the paraphrase. in keeping with the original work, we refer to this system as separator. the study in question is part of the evaluation in @xcite and focuses on comparing the newly introduced system separator against competitors in three dimensions, which the authors describe as follows: fluency "which system output is the most fluent and grammatical?" meaning "to what extent is the meaning expressed in the original question preserved in the rewritten version, with no additional information added? which of the questions generated by a system is likely to have the same answer as the original?" dissimilarity "does the rewritten version use different words or phrasing to the original? you should choose the system that uses the most different words or word order." all dimensions were evaluated jointly in the same form by human annotators. the goal of the original study was to demonstrate that the newly proposed approach preserves meaning and fluency while maintaining adequate dissimilarity to the input. the following, more detailed description of the study is based both on the original paper, as well as on additional materials and resources that were obtained by the reprohum organizers from the authors. at no point was there any direct interaction between the authors of the original study and the authors of this reproduction study. ## reproduction study following the guidelines of the reprohum project, we reproduce the study as closely as possible following the setup described in section 2.1. repro-hum organizers were able to obtain the original batches used in the study, as well as the original interface template. we employ both in our reproduction study. however, we introduce two major deviations from the original study setup: both changes can potentially impact the results of the reproduction. another threat to reproducibility is the relatively large time difference between the original study and the reproduction. while the original work was published in august 2021, with experiments likely concluded at least a few months before this date, all annotations in the reproduction study were elicited on may 3rd, 2024. this is particularly relevant, since there is some evidence that llms increasingly penetrate crowd-working platforms @xcite , which might alter annotator behavior. ## study statistics due to a bug in annotator assignment 6 , we elicited a total of 121 batch annotations. since we only require a total of 120 (= 3 repeat annotations × 40 batches), we randomly discard one repeat annotation from the over-annotated batch. we find only a single missed attention check in the entire annotation set. due to the low prevalence of missed attention checks and the cost associated with resubmission, we opt to not discard the related submission in a slight deviation from the original protocol. the median completion time, as measured from the time a study was accepted on prolific to the time the annotator submitted the completion code to prolific, is 7:16 minutes. this shorter than estimated completion time results in an average actual hourly pay of £16,51, well above our nominal target rate of £12,00. ## discussion while the high degree of reproducibility of the original study is encouraging, we note that this result was achieved with access to information and resources that are not directly available from either the original publication or the associated repository. specifically, neither the original batch assignment we use to reproduce results nor the original annotation interface are publicly available. both were made accessible to the reprohum project upon request. we also highlight that, as an exact reproduction, we can only make statements about how well results reproduce under the same selection of documents for annotation. however, the claims of the original study should be replicable for any subset of the original dataset that is sampled according to the study description. evaluating this goes beyond the scope of this study and it is reasonable to expect a higher variation if a different subset was sampled from the input. finally, if this study is interpreted to assess best practices for reproducible study design, we should be mindful that reproducibility is greatly aided by the clear-cut differences between systems. specifically, we identify two properties of the dataset that likely make annotation much easier than for other, similar studies: to illustrate how these dataset properties make the task comparatively easy, figure 5 shows the hypothetical relative preferences that systems would receive under the following deterministic annotation rules: the resulting scores already closely resemble the original ranking. in particular, we can easily reproduce the very strong performance of vae and the very weak score of dips. lbow and separator are close, just like in the original study, although the ranking is inverted. however, both manual inspection of the data and the original study scores for the fluency score suggest that separator is much more grammatical than lbow, which sometimes outputs paraphrases that are difficult to parse. this is likely to skew results in favor of separator as observed in the original study, but is not captured by our simple setup. ## conclusions in this report, we have given an account of our reproduction attempts of a study of meaning preservation annotation in paraphrasing systems. our results show an encouragingly high degree of reproducibility with the resources provided by the authors. in particular, the availability of original batches and interfaces makes it easy to design a highly similar setup to the original study. however, our analysis also shows that care needs to be taken not to over-interpret the outcomes of this study when it comes to making recommendations about best-practices in general. in particular, we find that the dataset contains many decisions which are likely to be very easy for annotators due to exact correspondence between inputs and outputs and the presence of obvious defects in some paraphrases, which make them unreadable. it thus remains unclear to which extent the results of this reproduction study are representative of more challenging annotation studies. this is particularly relevant, since current generations of nlg systems are well known to be much less prone to the kind of obvious mistakes present in the original study.