Another PASS : A Reproduction Study of the Human Evaluation of a Football Report Generation System (2021)

Abstract: This paper reports results from a reproduction study in which we repeated the human evaluation of the PASS Dutch-language football report generation system (van der Lee et al., 2017). The work was carried out as part of the ReproGen Shared Task on Reproducibility of Human Evaluations in NLG, in Track A (Paper 1). We aimed to repeat the original study exactly, with the main difference that a different set of evaluators was used. We describe the study design, present the results from the original and the reproduction study, and then compare and analyse the differences between the two sets of results. For the two ‘headline’ results of average Fluency and Clarity, we find that in both studies, the system was rated more highly for Clarity than for Fluency, and Clarity had higher standard deviation. Clarity and Fluency ratings were higher, and their standard deviations lower, in the reproduction study than in the original study by substantial margins. Clarity had a higher degree of reproducibility than Fluency, as measured by the coefficient of variation. Data and code are publicly available.

## introduction recent years have seen growing interest in, and concern about, reproducibility across the natural language processing (nlp) field. the reprogen shared task on reproducibility of human evaluations in natural language generation @xcite was the first shared task to focus on reproducibility of human evaluations (rather than metrics). we report on our participation in repro-gen, where our contribution was in track a, the 1 https://github.com/thiagocf05/ reprogen2021-vanderlee main reproducibility track. more specifically, we repeated the human evaluation study reported by van der @xcite . in this paper, we describe how we approached this task, present the results obtained, and compare our results with those reported in the original paper, using different methods of analysis. data and language sources: automatically scraped football match data from goal.com, foot_0 subsequently stored in xml-format, is used as input data, and the memo fc corpus as reference data. system architecture: the pass 3 architecture is a data-to-text pipeline consisting of the following modules: (1) the governing module (used in slightly different versions for different report parts) processes topics one by one, and interacts with the other modules as necessary; (2) the topic collection module extracts topics from the match data and orders them; (3) the lookup module retrieves all matching template categories for a given match event and their corresponding templates from a database; (4) the between-text variety module removes templates that were used in the last match report to ensure variety; (5) the ruleset module checks whether constraints associated with a given template category are met; (6) the template selection module selects templates from the remaining categories in a weighted random fashion; (7) the template filler module fills empty template slots with the relevant information from the match data; ( 8 ) the text collection module combines the text produced for the different report parts in the right order; (9) the information variety module removes repeated information; and (10) the reference variety module replaces repeated referring expressions. ## study design we aimed to keep all aspects of study design the same to the extent that was possible. in the sections below, we consider different aspects of study design and describe common features and differences, before summarising same/different properties in section 3.5. ## conclusion in this paper, we reported work which aimed to repeat the human evaluation experiment reported by van der @xcite as closely as possible. we characterised the properties which we know to be either the same or different in our reproduction study (compared to the original study), and presented scores from the reproduction study side by side with scores reported in the original paper ( note that we have not speculated about the likely reasons for the differences between the two sets of results. we know that those properties marked as different in section 3.5 are all possible reasons. out of these, it would seem likely that a sizeable part of the difference is down to the different cohorts of evaluators: older, known to us, mostly from computer science backgrounds in the reproduction study, vs. younger, random passers-by recruited in the science building of a university in the original study. the human evaluation studied here is about as simple as such evaluations get: just one system was evaluated, on three quality criteria and 10 output pairs, each evaluated by the same 20 raters. the coefficient of variation gives a measure of degree of reproducibility that is comparable across measures and across studies, so we can e.g. make the (relative) assessment that clarity was found to have a higher degree of reproducibility than fluency. however, the measure does not enable us to make an (absolute) assessment whether either one of them had good reproducibility. in order to do this, we would have to know what normally counts as good reproducibility in similar circumstances in nlp. since nlp currently has very few reproduction studies, and none that report coefficients of variation for human evaluations, such assessments are not possible at this point in time. they will become possible over time if more studies start to report cv (or other measures of precision) for reproduction studies.