TUDA -Reproducibility @ R epro G en: Replicability of Human Evaluation of Text-to-Text and Concept-to-Text Generation (2021)

Abstract: This paper describes our contribution to the Shared Task ReproGen by Belz et al. (2021), which investigates the reproducibility of human evaluations in the context of Natural Language Generation. We selected the paper “Generation of Company descriptions using concept-to-text and text-to-text deep models: data set collection and systems evaluation” (Qader et al., 2018) and aimed to replicate, as closely to the original as possible, the human evaluation and the subsequent comparison between the human judgements and the automatic evaluation metrics. Here, we first outline the text generation task of the paper of Qader et al. (2018). Then, we document how we approached our replication of the paper’s human evaluation. We also discuss the difficulties we encountered and which information was missing. Our replication has medium to strong correlation (0.66 Spearman overall) with the original results of Qader et al. (2018), but due to the missing information about how Qader et al. (2018) compared the human judgements with the metric scores, we have refrained from reproducing this comparison.

## introduction reproducibility is an utmost priority in research to ensure reliability of scientific findings. informally, it describes the ability to repeat a study, beginning with the same starting point, using the same resources (if possible) and achieving the same results and conclusions @xcite . reproducibility requires that approaches in publications be recorded in such a way that previously uninvolved parties can comprehend and recreate them @xcite . however, reproducibility is a complex requirement which often fails because of missing details (like not described data sets or missing key parameters)-such aspects, even though they may appear minor at first sight, either prevent reproducibility altogether or at least distort the results @xcite . one reason for such failures of reproducibility may be lack of widely accepted definitions and practical conceptualization of reproducibility, as there is currently no consensus on how and to what level of detail research should be documented @xcite . the shared task reprogen @xcite ) deals with the reproducibility problem. in particular, it aims to investigate reproducibility of human evaluation. the findings of reprogen should yield general insights into how reproducibility can be improved. the task in reprogen is to replicate either one of the pre-selected studies or a self-selected study from the field of natural language generation (nlg) and to document the findings. in this paper, we report on our reproducibility of the work "generation of company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evaluation" (compdesc for short) by @xcite . this work analyzes multiple sequence-to-sequence models that were used to generate short company descriptions from wikipedia articles. this includes both automatic and human evaluation which are then compared with each other. our replication focuses on the human evaluation, in accordance with the general outline of reprogen. ## compdesc and our replication we first describe the paper compdesc, then outline how we replicated its human evaluation. finally, we compare both experiments. ## results in addition, we calculated spearman's ρ and pearson's r correlations between the values in the two tables on each axis. from after that, despite not discovering the original setup, there is one noteworthy case (see since @xcite were not able to provide us with the original code or the corresponding information, it was impossible to determine the reason for the difference. for this reason and the consequent non-comparability of the results, we have refrained from reproducing the correlation matrix using the human evaluation results obtained in this replication study. ## conclusion in this replication, we could not reproduce all results of the original study "generation of company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evalu- in contrast to the first one, our secondary objective, namely to investigate whether we could obtain comparable inferences with the reproduced correlation matrix based on our human evaluation results, was not successful. we had to make several assumptions of missing information and even with that, we were not even able to recalculate the original results by using the human evaluation results from @xcite . therefore, we have refrained from a comparison with our data.