Reproduction of Human Evaluations in: “It’s not Rocket Science: Interpreting Figurative Language in Narratives” (2023)

Abstract: We describe in this paper an attempt to reproduce some of the human of evaluation results from the paper “It’s not Rocket Science: Interpreting Figurative Language in Narratives”. In particular, we describe the methodology used to reproduce the chosen human evaluation, the challenges faced, and the results that were gathered. We will also make some recommendations on the learnings obtained from this reproduction attempt and what improvements are needed to enable more robust reproductions of future NLP human evaluations.

## introduction reproducible and repeatable evaluations lay at the heart of good science. however, there has been increasing concern with natural language processing (nlp) on whether human evaluations are in fact reproducible and repeatable. this is particularly important within the field of nlp as human evaluations are seen as the "gold standard" as compared to automatic metric based evaluations. this has lead to an interest in trying to understand and quantify the degree to which evaluations are reproducible. one such effort is the reprohum project 1 , which attempts to investigate human evaluations within nlp by systematically uncovering the extent of problems of reproducibility. as part of this project multiple partner labs, consisting of both academic and industry institutions, were invited to participate in a multi-lab study reproducing human evaluations from a chosen set of research papers. these papers were vetted by the organising committee of the reprohum project to ensure that sufficient details in terms of materials @xcite and evaluation procedures were present for a successful attempt at reproduction by a given partner lab. in addition to the original paper author(s) consent and 1 reprohum - https://reprohum.github.io co-operation was sought to enable the reproduction of human evaluations in their paper. in this paper, we describe the current challenges facing human evaluations in nlp and reproducibility (section 2). afterwards we give details on the attempt to reproduce a specific human evaluation within the paper "it's not rocket science: interpreting figurative language in narratives" by @xcite (section 3) and how the reproduction of the paper was conducted with details on the challenges involved (section 4). finally, we detail the results obtained from the reproduction (section 5) and the recommendations (section 6) we would make based on the experiences of this experiment that would enable more robust reproductions of future nlp human evaluations. ## background within recent years there has been a great interest in understanding and quantifying the reproducibility of experiments across several areas of scientific research. this also includes experiments in the field of natural language understanding (nlu), where researchers have questioned the degree to which experiments and results can reliably be reproduced. recent work exploring the reproducibility of past nlu work has found significant issues such as only a minority of systems reproducing previously reported scores and systems not working due to non-functional code or resource limits @xcite . in fact some estimates place the percentage of papers being repeatable without any significant barriers as low as 5% and at 20% if the original author(s) help is sought @xcite . additionally, there has been growing awareness of systematic issues with regard to how human evaluations are being conducted. in particular, the lack of standardisation and significant underreporting of key human evaluation details @xcite . there has been an attempt to make human evaluation reporting more standardised and comparable between different papers through an introduction of a classification system that defines quality criterion properties @xcite . however, as noted by @xcite , whilst the problems of evaluations are known and proposals have been made to improve the situation, the adoption of best practices remains lacking. the reprohum project is a subsequent followup of the reprogen shared tasks foot_0 @xcite in 2021 and 2022. in these shared tasks participants either selected a paper proposed by the organisers or self-selected a paper for human evaluation reproduction. the results from these shared tasks showed some indications that human evaluations that have different evaluation cohorts can disadvantage the reproducibility of a given experiment @xcite . however, lowering the cognitive loads on individual evaluators could potentially lead to be better reproducibility of results @xcite . ## reproduction experiment for this reproduction experiment we were tasked with reproducing a specific human evaluation in the paper "it's not rocket science: interpreting figurative language in narratives" by @xcite . the paper explores the interpretation of figurative languages (idioms and similes) in english by exploring plausible and implausible continuations from a given fictional narrative. the authors of the paper used models to generate plausible idioms and similes from a given narrative. these generated texts were compared to human written ones in both automatic and human evaluations. the focus for this experiment is to reproduce the human evaluation conducted by authors. in particular, reproducing the absolute human evaluation, which asked human amazon mechanical turk workers on whether the computer generated and/or the human references are plausible or not for the given narrative. this task is illustrated in figure 1 , which is taken from @xcite . in the original experiment crowd workers were shown a narrative, the meaning of the idiom or the property of the simile and a list of three automatically generated continuations. one from a baseline supervised gpt-2 model, one from a context model, and the third from the literal model. in addition to the automatic continuations, participants were shown three human alternatives for idioms or five for similes. for each continuation (automatic or human) participants were asked to rate whether the text is plausible or not. each example was rated by three workers and the result aggregated using majority voting. both evaluations were done on 25 randomly sampled narrative texts for both the absolute idiom and simile scenarios. this equates to 50 narrative texts in total. the original paper incorrectly states "50 narratives for each task", however prior to the reproduction experiment this was clarified by the authors to be a mistake. in the original evaluation the authors of the paper reported the following results for the absolute evaluation: in addition to the above reported results the authors also make a mention of the fact that "the context model was favoured for idioms, the literal model was favoured for similes". this result will also be checked in this reproduction attempt. ## methodology & challenges the original evaluation collected human evaluations using amazon mechanical turk (mturk) crowd workers. like the original, the reproduction also used amazon mechanical turk as well. however, the paper by @xcite does not detail whether any controls were applied or not for the selection of crowd workers. nor were any details provided about the cohort of participants in terms of demographic data and the total number of participants recruited. for the reproduction experiment a total of 80 workers were recruited across both tasks (35 for idiom and 45 for simile). in agreement with the re-prohum organisers each worker was paid the uk living wage 3 of £10.90, in a us dollar equivalent amount, to give fair compensation for the workers time and effort across both tasks. the experimental data and user interface was taken from the original published source code repository 4 . however several challenges were encountered in attempting to reuse the original experimental data and user interface: to re-create the csv files needed for the plausibility idiom and simile tasks on mturk the output json files from the original experiment were used. in particular, the narrative, the automatic and human continuation texts for each of the scenarios were extracted from these json files using a python script. original results are from @xcite . out of the 45 workers who participated on the simile plausibility evaluation only 5 workers had answered all 25 texts. in the idiom plausibility evaluation out of the 35 workers that had participated only 2 had completed all 25 texts, with the next highest participant completing 17 in total. whilst this is lower than the original experiment, we believe this should not affect the results reported significantly as the analyses for idioms were constrained to 17 instead of 25 texts. additionally, as shown later, a similar percentage of the idiom human texts were rated as plausible as the original study. for the analysis, the code was written independently from scratch as no analysis code is present within the authors code repository. when analysing inter-annotator agreement, the difference between the original study and the reproduction is a drop in the absolute krippendorf's α score from 0.68 to 0.39. more granular analysis showed that the krippendorf's α inter-annotator agreement was 0.3761 for the idiom task and 0.3971 for the simile task between the three respective annotators. it is possible that a difference in the type of annotators used in reproduction as compared the original study resulted in a difference in the inter-annotator results seen between two the studies. when evaluating majority worker preference between the context model and the literal model for idioms and similes, we observed that for idioms preference was greater with the context model (83.33%) than the literal model (66.66%). for similes we were able to see a larger preference for the literal model (64%) over the context model (48%). this confirms the preferences that @xcite observed with human annotators in their original experiment. whilst we could not replicate the moderate interannotator agreement found in the original study nor the preference for the context model for idioms, we were able to successfully replicate the results for the percentage of idioms and similes considered plausible through majority worker voting for human and the baseline model generated texts. additionally, we were able to replicate the preference for the context model for idioms and the literal model for similes. the fact that the results were either the same or very close to the original study shows in some aspects shows that some results were successfully replicated in this reproduction study. ## conclusion & recommendations in this paper we have conducted a partially successful reproduction of the results obtained in the absolute idiom and simile human evaluations. whilst we were able to reconfirm the results for the judgement on whether human and baseline model generated texts idioms and similes are plausible, the same could not be said for the literal and context model texts. additionally, inter-annotator agreement scores show that there is a significant difference between the results obtained as compared to the original study. one possible reason for this could be due to the difference in the cohorts of annotators recruited between the two studies. a similar challenge was found in the reproduction experiment by @xcite where the difference in recruited participant cohorts was speculated as a possible probable cause for the inability to reproduce results from the original study. nev-ertheless, it has been noted that recruiting mturk crowd workers that have high inter-annotator agreement with each other can be challenging even with a structured process in place to filter out unsuitable workers @xcite and therefore in itself may not guarantee reproduction success. based on the experiences of this reproduction study there are several key recommendations to reduce uncertainty for reproduction attempts: the first recommendation is very straightforward. without the information on the type of participants that were used for the evaluation it is very likely that any reproduction attempt may not succeed as the differences between the two recruited groups might be too far significant to enable a comparable evaluation. therefore, data, such as participant demographics, would enable any reproduction attempt to focus on recruiting the right participants for a given study. when combined with the second recommendation, this would help to give confidence to ensure that participants who do not qualify for the experiment are rightfully excluded. once having recruited the right participants, it is important the exact same datasets and user interfaces are provided to ensure comparability with the original experiment and armed with the same analysis code to reduce any possibilities of discrepancies occurring. with these recommendations and the learnings from others in this area, it is hoped that future attempts at performing reproduction experiments will be more successful than at present.