Same Trends, Different Answers: Insights from a Replication Study of Human Plausibility Judgments on Narrative Continuations (2023)

Abstract: We reproduced the human-based evaluation of the continuation of narratives task presented by Chakrabarty et al. (2022). This experiment is performed as part of the ReproNLP Shared Task on Reproducibility of Evaluations in NLP (Track C). Our main goal is to reproduce the original study under conditions as similar as possible. Specifically, we follow the original experimental design and perform human evaluations of the data from the original study, while describing the differences between the two studies. We then present the results of these two studies together with an analysis of similarities between them. Inter-annotator agreement (Krippendorff’s alpha) in the reproduction study is lower than in the original study, while the human evaluation results of both studies have the same trends, that is, our results support the findings in the original study.

## introduction reproduction studies of human evaluations in the field of natural language processing (nlp) are attracting increasing attention @xcite . due to the inherent limitations of automatic evaluation, especially in natural language generation tasks which often imply high variability in the output, human evaluation is often considered to provide more reliable assessments (van der lee et al., 2019). however, initial results observed in the context of reprohum 1 , a coordinated, multilab reproducibility project which the present work is also part of, suggest that the majority of human evaluations in nlp face the challenge of being unreproducible due to various reasons @xcite . this clashes with the importance of ensuring high levels of experimental reproducibility, which has been gaining increasing recognition in 1 https://reprohum.github.io/ the nlp community @xcite @xcite . in the context of our participation in the re-pronlp shared task on reproducibility of evaluations in nlp (track c -reprohum project) foot_0 , this paper reports on our experience when trying to reproduce as closely as possible a previously run human evaluation. specifically, we aimed to reproduce human evaluations conducted by @xcite on the continuations of narratives generated with various systems or written by humans. in order to harmonise and coordinate all replication efforts within reprohum, the project leaders have created a spreadsheet that each lab in charge of a reporduction experiment was asked to fill in and submit, acting as pre-registration for the replication. this human evaluation datasheet (heds) is included in appendix b. following the shared reproduction approach provided by the reprohum's coordinators, we first summarize the original study explaining the task addressed, and the human evaluation setting (section 2), followed by our replicated experiment (section 3). although we did try to perform our new experiments under conditions as similar as possible to those of the original study, we still ended up with some differences between our setup and the original paper (e.g. we raise the payment to give the annotator a fairer reward); we discuss these in detail. finally, we report and analyze the results obtained in our reproduction study by comparing them to the original experiments (section 4), and draw some conclusions on the feasibility of a full experimental reproduction (section 5). ## original study we aim to repeat the experiment conducted in "it's not rocket science: interpreting figurative language in narratives" by @xcite . ## reproduction study as mentioned, we only replicated the absolute evaluation from the original paper. three differences between the reproduced experiment setting and the original one exist. first of all, we recruited a total of 75 workers for each task with no additional requirements. this was done after thorough consideration: the total number and requirements of workers employed for absolute evaluation are not mentioned in the original paper. still from the file containing the result data, we could infer, via anonymised ids, that the total number of annotators is much smaller than the number of hits. however, since all hits were published in one row, the selecting criteria for workers were unclear and we received no further clarification from the original authors; hence, we chose to also publish the hits with no additional restrictions on workers in one row for each task, which ultimately led to recruiting one worker for each hit. no control on whether one worker can work on both the idiom and simile tasks was put in place: in other words, one worker can potentially work on at most two hits in total, one hit of each task. due to invalid results received, we re-published some of the hits to obtain new assessments so that evaluations from a total of 127 workers were collected. in the original study, no rejecting or re-publishing of hits was observed, but one invalid result is included in the original outputs for the simile task. secondly, we raised the monetary compensation from @xmath02.21 per hit, following the general recommendation of the reprohum project to meet the minimum hourly salary in the uk, assuming it takes 10 minutes to finish one hit properly. besides setting differences, we received some invalid responses due to the original survey layout and thus had to re-publish some hits. thirdly, we changed the examples given in the idiom tasks. only the specific simile examples were made available by the authors, therefore we chose a narrative including an idiom and its corresponding continuations from the development set and then used them as examples for the crowdworkers. ## results in this section, we report our results compared to the original experiment, and the reproducibility assessment. for each narrative-continuations pair we collect three responses, and whether a continuation is plausible is determined by majority voting, following what was described in the original paper. as a first check, we assess inter-annotator agreement (iaa) using krippendorff's α, which is appropriate for categorical labels attached to text spans, and which was used in the original experiment. the original experiment reports krippendorff's α = 0.68, while our replicated experiment shows krippendorff's α = 0.11 and the krippendorff's α of the original experiment is 0.33 using our calculation method. this discrepancy might have to do with the fact that in our replication there are many more annotators, and with the way the score was calculated (accounting or not for the same annotator possibly doing more hits in the original study). the original paper reports quantitative results of each model for each task, and describes the reported data as the "percent of times that the generation from each of the models and human-written references was chosen as plausible by the majority of workers." since in each task we only collect assessments from each worker for one single continuation per model, there is no confusion on how to calculate the quantitative results. however, responses of multiple human-written continuations from each worker are collected, and the original paper did not detail how they came to the reported results for the human-produced continuations. after several attempts, 7 out of 8 the original results of the plausibility of human-written continuations were successfully reproduced using the following procedure: the calculated results are shown in surprisingly, our replicated experiments show that knowledge-enhanced models outperform humans, which was not the case in the original study. one plausible assumption is that the workers recruited in the original study have all been given additional training on evaluating continuations. another possible reason is that the workers in the original study might unconsciously think that the second half of continuations is more plausible. each of them works on multiple hits, and in each hit the first three continuations are always model-generated continuations and the rest are human-written continuations. the second problem is avoided in our replicated study as we avoided letting one worker evaluate several hits. nevertheless, both the performance difference and low iaa suggest that normal workers cannot fully understand, or reach an agreement on determining whether a continuation is plausible, with only the example and instructions given on the hit page. overall, the original paper concludes from the results that "a knowledge-enhanced model outperformed the baseline gpt-2 model...the context model was favored for idioms while the literal model was favored for similes," and the general trend of our results, albeit at times largely different in scores, also supports this conclusion. ## conclusions although the results of our replicated experiment support the general findings of the original paper, the human evaluation process of the original paper could not be fully reproduced properly. two aspects need particular attention. first, the reproducing process is intrinsically difficult. even though we tried our best, and we gained substantial help from the original authors, several questions still emerged during the replication stage which could not be answered. the detail of the worker recruitment process for example was not available and might be not fully known to the original authors either, due to platform specifications that can be 3 secondly, as shown in reprohum: pre-experiment record 1.1.2 which experiment within the paper is this form being completed for? absolute evaluation of plausibility (idiom and simile) in section 5. post tasks in the crowdsourcing platform (mturk). 3.2.4 what training and/or practice are evaluators given before starting on the evaluation itself? we can provide evaluators with detailed guidelines and examples of generated sentences along with plausible assessments. however, it is not known if guidelines and examples were provided in the original paper.