Some lessons learned reproducing human evaluation of a data-to-text system (2023)

Abstract: This paper presents a human evaluation reproduction study regarding the data-to-text generation task. The evaluation focuses in counting the supported and contradicting facts generated by a neural data-to-text model with a macro planning stage. The model is tested generating sport summaries for the ROTOWIRE dataset. We first describe the approach to reproduction that is agreed in the context of the ReproHum project. Then, we detail the entire configuration of the original human evaluation and the adaptations that had to be made to reproduce such an evaluation. Finally, we compare the reproduction results with those reported in the paper that was taken as reference.

## introduction an experiment or study is reproducible when independent researchers can replicate it by following the documentation shared in the original report and draw the same conclusions, which is also a clear synonym of reliability. in natural language processing (nlp), reproducibility is not limited to specifying the parameters chosen to train a model, but it goes beyond that and requires the specification of all the details of the evaluation process by which the reported results are obtained. in nlp, until recently, not too much attention has been paid to the reproducibility of neither automatic nor human evaluations. in the case of automatic metrics, there is a reproducibility checklist @xcite , but in the case of human evaluations not so much progress has been made. in addition, some papers have been published about reproducibility in nlp, regarding reproducibility tests based on the fulfillment of certain properties in human evaluations @xcite but also proposing a template for recording the details of human evaluations in nlp experiments, with the aim of improving the replicability of these processes @xcite . the work presented in this paper is part of the reprohum foot_0 project, that investigates the factors that make a human evaluation more reproducible in nlp by launching multi-lab sets of reproductions of human evaluations. as members of one of the 21 partner labs in this project, we performed a reproduction of an nlp study in which a data-totext system is assessed and compare the results obtained in the reproduction with the original ones. the rest of the manuscript is organised as follows. in section 2 we introduce related work and the common approach defined as a global requirement for all the reproducibility experiments within reprohum project. section 3 describes the reproduction of the nlp evaluation, first, explaining the content of the paper chosen for reproduction and then, explaining all the details of the evaluation that is going to be reproduced. in section 4, the results of the reproduced evaluation compared to the original paper are reported and discussed. finally, section 5 concludes with final remarks and future work. ## background in the context of the shared task reprolang @xcite a replication of a human evaluation of a neural text simplification system by @xcite was performed @xcite , obtaining worse results in the reproduction study, in terms of grammaticality and meaning preservation. with the aim of developing theory and practice of reproducibility assessment, the reprogen shared task arose and in its two editions @xcite several studies involving the reproduction of different experiments were carried out. @xcite replicated an evaluation of machine translation outputs where errors related to comprehensibility and meaning correctness were annotated in texts by marking up word involved in an error @xcite . they found that 4 out of 6 system rankings were the same in both studies, but error rates for minor error types have lower reproducibility than those classified as major error types. @xcite reproduced human evaluations of data-to-text systems. despite differences in the number and type of raters, authors found poor reproducibility when assessing the effect of hedges on preference judgments between native and fluent english speakers. @xcite faced the evaluation reproduction of a stance-expressing football report generator @xcite , finding good reproducibility for stance identification accuracy, but lower reproducibility for clarity and fluency. in addition, it is worth noting that in the context of the reprohum project, adhering to the following guidelines is mandatory when reproducing experiments: it must be noted that during all the reproduction process described above is not allowed to contact the authors of the original paper or communicate with other project labs carrying out this or any other reproduction experiment to avoid affecting the reported outcomes. thus, all the information and resources provided should be in the common resources folder provided by the project team and in case of any question we were asked to only contact the reprohum project managers who act as a proxy with the authors of the work to be reproduced. ## reproduction of an nlp evaluation in this section we describe how we applied the reprohum guidelines previously introduced. for the purpose of human evaluation reproduction, we were assigned the paper published by @xcite . based on the evaluation details described in the paper, the appendices, the resources available in the associated public repository, and the resources provided by the reprohum managers after contacting the authors, we reproduced the evaluation as close as possible to the original one. ## results the evaluation process was organized in the abovementioned 4 mini-batches to complete the total 300 tasks. following the procedure explained in section 3.2, we had to repeat 59 hits in order to obtain 300 valid responses. at the end, a total of 144 different crowd-workers participated in the evaluation. notice that, in the original paper it is reported a total of 131 crowd-workers participating in the study, but for 600 tasks instead of 300, i.e., for both the ro-towire and mlb datasets. since we reproduced the experiment with the rotowire dataset, the number of unique participants is quite high, considering that we requested half of the hits. furthermore, in the original paper it is reported an inter annotator agreement of 0.44 for supported and 0.42 for contradicting facts, using krippendorff's α. of course, these values are calculated for the 600 tasks performed for both datasets, and the total of 131 crowd-workers. we calculated the same agreement measure, by adding the number of supported/contradicitng facts of each task, i.e., by adding the count of each of the four sentences in the hit to have two scores per hit: contradicting and supported facts scores. the results gave us an agreement of 0.188 for supported and 0.219 for contradicting facts. therefore, the agreement in our evaluation is lower than in the original one, although we must take into account that the number of tasks and unique evaluators is different in our reproduction experiment, so a direct comparison is not really fair. looking at the original scores shown in contrasting the original vs. reproduction results, we can see that for the macro supported facts the score only differs in 0.08, but in the rest of the compared systems it differs more from the original study, reaching the largest difference in the templ system (1.3, i.e., a 17% less supported facts in average). comparing the results of the reproduction for the macro supported facts with respect to the rest of the systems, macro obtains significantly smaller values only with respect to the templ system, while in the original study it is also significantly smaller compared with the rbf-2020 system. regarding the contradicting facts, for all the systems the reproduction scores are higher than in the original experiment, being the ed+cc system which yields the largest difference, with a 0.91 of counted contradicting facts in the original paper against a 1.95 in the reproduction experiment (i.e., an increase of the 114%). surprisingly, the macro system achieves the lowest score in terms of contradicting facts in our reproduction experiment, while in the original experiment the templ system had the best performance. looking at macro results against the other systems, we can say that macro achieves only significantly lower scores respect to the ed+cc system, whilst in the original experiment also significantly differences with the rbf-2020 were concluded. it must be noted that if we pay attention to the unbiased coefficient of variation (cv*) in after analyzing the results shown in ## concluding remarks and future work in this work we performed a reproduction experiment of a human evaluation in nlp. following the work by @xcite , in the reproduced evaluation, a data-to-text system with a macro planning stage (macro) is assessed in terms of contradicting/supported facts generated in the sports domain, i.e., rotowire dataset. when counting the supported facts of the different systems, there is not a clear change pattern in the reproduction scores respect to the original ones. all of the scores are slightly different from the original ones, whether higher or lower. but, considering that having more supported facts is better, the macro system shows a mildly improvement in the reproduction study in terms of score and also obtaining less statistically significant smaller scores with respect to the rest of systems in the comparison. despite of that, the templ system @xcite is still the best in terms of supported facts, mainly because, as it is also pointed in the original paper for reproduction, the system essentially parrot facts. regarding the count of contradicting facts, there is a clear increase in general for all the systems and, surprisingly the system with the smallest number of contradicting facts is the macro system, instead of the templ system which was the best system in the original study. however, the macro system produces statistically significant smaller scores only with respect to the ed+cc system. the reproduction results show a similar tendency regarding supported facts, where the templ system still produces the bigger number of supported facts. however, the tendency changes regarding contradicting facts. in addition to the general increase of contradicting facts, templ and gold summaries, which were the baselines with the less contradicting facts, are outperformed by macro, being the model with the less contradicting facts generated. there are certain factors in human evaluation that cause the results of a reproduction study, despite replicating all the settings, not to be exactly the same as those reported in the original study. one of the most distinguishing factors are the evaluators. in this case, the same amt crowd-worker requirements were applied to select a profile of workers in the crowd-sourcing platform equal to that of the original study, but they will never be the same evaluators. moreover, a different number of evaluators have participated than in the original study, since they could choose how many tasks to perform freely. this is obviously one of the reasons why a reproduction of a human evaluation can lead to a difference in the results. in connection to the amt crowd-worker requirements, the following experience with a worker from the platform is worth to be mentioned here. as mentioned in the section 3.2, following the original evaluation settings we indicated as a requirement to perform our task to have a minimum of 1,000 hits completed. a few hours after launching the first mini-batch of the experiment, we received a message in which an experienced amt crowd-worker welcomed us to the platform and very kindly told us the following: "if this is your first batch posted here, welcome to mturk! just a heads up, posting work with insufficient qualifications tends to yield some terrible results. i'd suggest making the qualifications 10,000 approved hits". since this was a reproducibility experiment, we had to stick to the conditions specified in the original paper and kept the minimum number of hits at 1,000. anyways, this advice is worth to be mentioned here for consideration in future experiments. having seen that some of the results of the replicated experiment differed from the original, and that the agreement between the raters was poor, we believe that the minimum number of hits required may have had an influence. when the original evaluation was launched (in 2021) this requirement was probably enough to achieve good results in the platform, but currently, as the worker recommended, probably we should increase the minimum number of required hits to 10,000 in order to get equivalent results to those reported by @xcite . taking advantage of the fact that this worker had contacted us, we asked him/her about a special qualification granted in amt that we were curious about, despite not being used in our experiment: the so-called "masters qualification". on the official amt website there is no clear information about the requirements that crowd-workers must meet to obtain this qualification and what it exactly means, so we asked the worker what he/she knew about it and the worker told us that amt is notoriously tight-lipped about the masters qualification and even the workers of the platform do not know what is the criteria for granting this type of qualification. this fact made us think about the platform's lack of transparency even with workers and why sometimes amt has bad reputation, despite being a powerful tool that so many people use. in the light of these findings, this reproduction study emphasizes the critical importance of providing comprehensive details about human evaluations in nlp. the standarization of reporting practices for human evaluations by tools such as the human evaluation datasheet (heds) in the framework of a common approach for reproduction, increases the reproducibility and, therefore reliability of any work. thus, we encourage researchers to further document their nlp evaluations using these standards, with the aim of enhancing the quality of the works in the field. as future work, we plan to repeat the evaluation for the mlb dataset with the aim of checking if reported results differ in a similar way as observed in the rotowire dataset. also, we would like to reproduce the other human evaluation reported in the original paper, i.e., the quality of the generated texts in terms of grammar, conciseness and coherence, by comparing pairs of summaries.