R epro H um #0669-08: Reproducing Sentiment Transfer Evaluation (2025)

Abstract: We describe a reproduction of a human annotation experiment that was performed to evaluate the effectiveness of text style transfer systems (Reif et al., 2021). Despite our efforts to closely imitate the conditions of the original study, the results obtained differ significantly from those in the original study. We performed a statistical analysis of the results obtained, discussed the sources of these discrepancies in the study design, and quantified reproducibility. The reproduction followed the common approach to reproduction adopted by the ReproHum project.

## introduction human evaluation is considered to be the gold standard for assessing natural language processing (nlp) systems, although many factors can affect its reliability. subjectivity in human ratings can make experiments difficult to reproduce @xcite ; the definitions of the evaluated criteria are often inconsistent @xcite and may confuse annotators @xcite . furthermore, external factors such as interface design can bias annotator behavior in unexpected ways @xcite . in some cases, issues such as unclear instructions, inappropriate dropping of outliers, or overlooked implementation bugs are only revealed during reproduction @xcite . therefore, efforts such as the reprohum project @xcite help us identify these challenges and develop more robust and transparent evaluation practices. in this report, we describe our reproduction study of human evaluation of sentiment transfer, originally performed by @xcite . we focus on a single quality evaluated in the original experiment: semantic preservation, i.e., how much of the original meaning was preserved after performing the sentiment transfer. we also limit our evaluation to a single style: more positive (see section 2). the original experiment is described in section 2. we reproduce the setting of the original study as closely as possible and describe this process in section 3. the results of our human annotation are shown in section 4. section 5 describes how we compared key numerical results to assess reproducibility and compares the findings of our reproduction against the original study. finally, in section 6 we discuss reasons for differences between the original and reproduced results. ## original experiment the original study @xcite presents a zero shot prompting method with large language models (llms) for text style transfer. the text style transfer task transforms or adds stylistic attributes to a text while preserving the global structure, e.g. converting "it is a nice day." to a more positive "it is a truly magnificent day!" @xcite . @xcite 's llm prompting method can perform any arbitrary text transformation (e.g. "more melodramatic") without fine-tuning or presenting specific exemplars in the prompt. the style is transferred for 50 randomly chosen sentences from the reddit writing prompts validation set @xcite . the sentences are transformed into three standard styles (more positive, more negative, more formal) and six nonstandard styles (more melodramatic, more comical, include the word "baloon", include the word "park", include a metaphor, more descriptive). the researchers compared the following six systems: the prompts were executed with the lamda and lamda-dialog language models @xcite , as well as gpt-3 @xcite . as text style classifiers @xcite are not available for all target styles, the researches relied on human evaluation with six professional annotators. the annotators evaluated three aspects on 1-100 scale: (1) transfer strength -to what extent the output matches the target style; (2) semantic preservation -how well the output preserves the meaning of the input, excluding the style change; (3) fluency. to achieve good inter-annotator agreement, the researchers run an initial calibration session where annotators rated a small subset of data (excluded from the main results) and asked clarifying questions about the instructions. each triple of input-transformationoutput was rated by three annotators. target styles commonly used in research for style transfer (positive and negative sentiment and formality), where data are available, are also evaluated on the yelp polarity dataset @xcite and gramarly's yahoo answers formality corpus (gyafc) @xcite . those are also evaluated with automatic metrics: the hug-gingface transformers sentiment classifier @xcite for transfer strength, semantic similarity to human examples from @xcite through the bleu score, and fluency as measured by gpt-2's @xcite perplexity. ## reproduction study we reproduced the human annotation of a single style transfer transformation -more positive -and one evaluation aspect -semantic preservation. we followed the original experiment as closely as possible. however, instead of using internal annotators which are not available to us, we recruited annotators from the prolific crowdsourcing platform. 1 in effect, we could not perform the initial calibration session. the setup of the reproduction is based on the original study's design and the reprohum guidelines @xcite : datasets we use the same 50 sentences from the reddit writing prompts as the original study, the more positive transformation, and the outputs of all the six systems that were compared. ## main results the results of our reproduction are presented in the observed differences between the original and reproduced results are significant. our annotators seem to be more strict when assessing semantic preservation, as the overall average across all the methods is more than 20 percentage points lower 3 standard deviation of a mean is σ √ n . the relative increase in deviation caused by a smaller sample is σ/ √ 140 σ/ √ 150 = 150 140 = 1, 0351. the observed differences from the original study are at least three times higher. there are also substantial differences in the ranking of the evaluated methods. in the original study, the paraphrase method was ranked the highest, while human-written texts were outperformed -or scored the same -by four out of the five automatic methods. in the reproduction, the outputs of paraphrase method received the lowest score and humans outperformed all automatic methods. the rest of the systems receive similar ranks in both studies. inter-annotator agreement we measured the inter-annotator agreement of obtained annotations with krippendorff's α @xcite . to identify potential outliers, we also conducted ablation analyses by recalculating agreement scores after excluding each annotator's ratings in turn. the results are presented in according to @xcite , the interannotator agreement obtained should be interpreted as poor. the original study did not report interannotator agreement, leaving it unclear whether our result is due to the lack of the initial annotator calibration session (conducted in the original experiment but omitted in the reproduction) or from the inherent difficulty of the annotation task. our ablation analysis in ## quantifying reproducibility the reproduction targets were determined based on the categories outlined in the reprohum shared task guidelines @xcite , sect. a5) and qra++ @xcite . the targets in the following categories were identified: type i following the quantified reproducibility assessment by @xcite , we computed the small sample coefficient of variation (cv*) as a measure of the degree of reproducibility for numerical scores. the results are given in the values of cv* computed for the original study and the reproduction are in the range of 13-33, except for the substantially higher value for style transfer performed by paraphrasing. type ii results are evaluated with pearson and spearman correlation @xcite , as well as with the root-mean-square deviations from the original results. the results are presented in type @xcite summarises the findings from the original study as follows: "the outputs from our method were rated comparably to both human-generated responses and the two prior methods". however, these conclusions are not confirmed by our reproduction. as previously mentioned, human-written responses obtained the highest scores, with a difference of 9 percentage points to the approach proposed in @xcite . in our study, this approach was outperformed by both baseline methods, but the difference to one of them was relatively small. ## discussion one major difference between the original experiment @xcite and the reproduction study is that the original experiment performed an annotation calibration procedure on 10 examples. these 10 examples were excluded from the evaluated data and allowed the authors to align their expectations with the annotators, who were free to ask questions during this process. we hypothesize that the absence of this calibration step affected the reproduction, especially since measuring meaning preservation in sentiment transfer is counterintuitive and requires clear guidance for consistent annotation. given that the original experiment was conducted in 2021 (i.e., before the introduction of llms to the general public), we also cannot rule out the possibility that people have increased their expectations of ai, leading to the lower scores we observed.