R epro H um #0031-01: Reproducing the Human Evaluation of Readability from “It is AI ’s Turn to Ask Humans a Question” (2025)

Abstract: The reproducibility of results is the foundation on which scientific credibility is built. In Natural Language Processing (NLP) research, human evaluation is often seen as the gold standard of evaluation. This paper presents the reproduction of a human evaluation of a Natural Language Generation (NLG) system that generates pairs of questions and answers based on children’s stories that was originally conducted by Yao et al. (2022). Specifically, it replicates the evaluation of readability, one of the most commonly evaluated criteria for NLG systems. The results of the reproduction are aligned with the original findings and all major claims of the original paper are confirmed.

## introduction reproducibility is one of the main measures for good science. by reproducing studies from other researchers and confirming their results, scientific findings can be independently verified. in recent years, surveys about reproducibility revealed widespread problems across disciplines @xcite . natural language processing (nlp) is no exception and also suffers from a variety of problems with regard to the reproducibility of scientific results @xcite . evaluation metrics, like precision, recall, and accuracy, but also more sophisticated metrics, like bleu @xcite , rouge @xcite , or the bert-score @xcite , are widely used in the evaluation of nlp systems. they are not only cheap and fast to calculate but also promise a high degree of reproducibility: most metrics guarantee that for the same input, they will always produce the same score. 1 however, they do not always correlate well with human judgments @xcite and, particularly for natural language generation (nlg), 1 yet, reproduction of metric-based results can still be difficult as pointed out by @xcite . human evaluation, therefore, in the field of nlp plays an important role and is often seen as the best available option for evaluation @xcite . however, designing good, reproducible human evaluations is much more difficult than the use of automated metrics and many existing human evaluations suffer from problems that limit their reproducibility @xcite . the reprohum project and the associated repronlp shared taks @xcite @xcite aim to address this problem by analysing the reproducibility of human evaluations in nlp and developing a methodological framework to assess the reproducibility of human evaluations. 2 as part of the project, multiple partner labs attempt to reproduce results from selected papers that report on human evaluations in nlp. this paper presents the results of such a reproduction study for the paper "it's ai's turn to ask humans a question" by @xcite . the original paper introduces a new approach for the generation of question-answer pairs and compares that approach against two baselines in a human evaluation. a previous reproduction of the study was conducted by @xcite who concluded that "all in all, we managed to replicate the original study". we believe that the design of the reproduction presented in this paper, which comes to a similar conclusion, is closer to the original design, particularly, because we recruited participants with the same background as the original participants (namely nlp experts; see also section 3.2), while the participants recruited by @xcite have a different background (namely undergraduate students). while the original paper and the reproduction by @xcite investigate three quality criteria in the human evaluation, namely readability, question relevancy, and answer relevancy, we only reproduced the evaluation of one quality criterion, namely readability. as shown by @xcite , readability is one of the most frequently evaluated quality criteria in human evaluation of nlg systems, although it is not always evaluated under this specific term and the definitions of it vary. 3 based on the information provided in the original paper and additional information obtained from the original authors by the reprohum project team, we were able to reproduce the main findings of the original paper: while the scores obtained in the reproduction of the human evaluation slightly differ from the original scores, the ranking of the compared systems and all major claims made in the original paper with regard to the readability evaluation could be verified. the results of our reproduction, the code used to calculate the reported metrics, and a human evaluation datasheet (heds, @xcite ) for the reproduction experiment are available on github. 4 the heds file is also available in the central repronlp 2025 heds repository. 5 2 original study @xcite introduce a question-answer pair generator that is designed for educational purposes: based on story books for readers from kindergarten to eighth-grade, the system automatically generates question-answer pairs that are designed to test different dimensions of comprehension skills. the architecture of the system consists of three main components: the modules have been fine-tuned on the fairy-taleqa dataset @xcite . the dataset consists of over 10,000 qa pairs from almost 300 children's books, which have been specifically designed to test reading comprehension. ## reproduction the paper and its human evaluation have been chosen by the reprohum team to be reproduced by a partner lab. we conducted the reproduction according to the reprohum guidelines and instructions. ## results the results of the reproduction are shown in in the reproduction the best readability scores were achieved by the "groundtruth" qa pairs (mean 4.38 on a scale from 1 (worst) to 5 (best)), followed by the pairs generated by the system introduced by @xcite (mean 3.85), and the paq baseline (mean 3.14). ## conclusion the results of our reproduction confirm all major claims and results of the original experiment with regard to the readability of the generated questionanswer pairs. while the order in which the systems have been evaluated is the same, the absolute scores received by each system vary from the original results by up to 20%. given how vaguely defined readability and the scale it was judged on (from 1 to 5) were in the experiment, this does not seem surprising. however, other factors could have also influenced that the results of the reproduction were overall more critical, e.g. the composition of the participants (the reproduction consisted only of nlp researchers) or temporal effects (while the original study was conducted before the public availability of large language models, like chatgpt, the reproduction was conducted in 2024, when expectations towards the quality of ai-generated texts might already have been higher). lastly, it is worth pointing out that our reproduction heavily relied on information that was not available in the original paper, but was kindly provided by its authors to the reprohum project. if we would have based our reproduction attempt solely on the information provided in the paper, our experimental setup would have, in all likelihood, looked significantly different and might well have yielded different results.