R epro H um #0033-05: Human Evaluation of Factuality from A Multidisciplinary Perspective (2025)

Abstract: The following paper is a joint contribution for the 2025 ReproNLP shared task, part of the ReproHum project. We focused on reproducing the human evaluation based on one criterion, namely, factuality of Scientific Automated Generated Systems from August et al. (2022). In accordance to the ReproHum guidelines, we followed the original study as closely as possible, with two human raters who coded 300 ratings each. Moreover, we had an additional study on two subsets of the dataset based on domain (medicine and physics) in which we employed expert annotators. Our reproduction of the factuality assessment found similar overall rates of factual inaccuracies across models. However, variability and weak agreement with the original model rankings suggest challenges in reliably reproducing results, especially in such cases when results are close.

## introduction although natural language processing (nlp) represents a field that strongly focuses on computational approaches and the use of automatic evaluation, human evaluation remains an important practice for assessing nlp systems. automated metrics may be scalable and robust, however, they often fail to capture nuances of natural language, such as emotional tone, cohesion, and coherence, in the same manner as humans, as stated by @xcite and first noticed by @xcite . since humans are also prone to errors, there still is a need for proper guidelines of human evaluation @xcite . @xcite reviewed several evaluation methods in nlp and showcased the role of human evaluators in assessing aspects that computational approaches struggle to take into account. however, reproducibility of human evaluation and proper guidelines remain a * equal contribution. complex and difficult-to-achieve task @xcite . this paper is a contribution to the repronlp 2025 shared task @xcite , which is part of the reprohum project foot_0 , a multi-lab cooperative project aiming to test the reproducibility of human evaluations through large-scale reproduction. our paper focuses on evaluating the factuality of artificially generated scientific definitions from @xcite . the original work was reproduced before by @xcite , and @xcite , who concentrated on evaluating the fluency of the generated definitions. the paper begins with introducing the research and then discussing about related studies. next up we present our reproduction steps followed by our additional experiment, concluding with results, participant feedback and final remarks. according to standard scientific procedures, we provide all data and code used to help with further research in this area. we also follow the project's coordination team's guidelines by completing the human evaluation datasheet (heds) @xcite . 2 ## original study in the original study @xcite , human evaluation is employed on a dataset of 300 generated scientific definitions by three models considered to be the best performing: dexperts, gedi, and an svm reranker. the authors define two types of generated definitions that are in scope: "highcomplexity" (which use more academic and technical language) and "low-complexity" (which use terms more approachable for the general public). in this sense, they compile two separate sets of data: scientific news articles designed for training of lower-complexity definitions and scientific journal abstracts, which are later used for highcomplexity definitions training. the first model uses the dexperts architecture introduced by @xcite , which consists of an ensemble of three different language models: an "expert" (trained on text with desired features), an "anti-expert" (trained on text with unwanted qualities), and a base model. the difference between the logits from the first two is merged with the logits of the latter. for their study, the authors opt to continue the training of pretrained bart-large models for the expert and anti-expert models, utilizing the abstracts dataset and the news dataset. for the generation of more complex definitions, the abstracts dataset is used in the training of the expert model, while the news dataset is employed for the anti-expert model training, with the setup being reversed to generating less complex definitions. the original work also includes the usage of the gedi (generative discriminators) method @xcite , which utilizes a class-conditioned language model trained on text with required attributes, more specifically in this context referring to the two sets of data. another well-performing approach is the one of reranking, which is introduced in the original paper. this method consists of producing 100 candidate definitions for each term during test time using a bart model, which are then reranked by a discriminator model trained to discern between text from scientific journals and from science news. for the human evaluation, the authors select 50 terms from the test data at random, generating for each both low-and high-complexity definitions with all three best-performing models described earlier, which results in a set of 300 texts to be manually annotated. they use two annotators to rate the generated definitions based on three criteria: fluency, relevancy, and factuality. specifically for factuality, the annotators assign a binary label to each definition, indicating whether or not it contains any factually wrong information. in case of errors, they use a 1-4 likert scale to score the extent of the inaccuracies. the paper reports a krippendorff's alpha of 0.59 when assessing whether a definition contained factually incorrect information, showcasing a modest agreement between the human evaluators, as per the official interpretation guidelines @xcite . the inter-rater agreement is maintained to almost the same extent in the case of evaluating the severity of errors (krippendorff's alpha of 0.55). ## previous work in the reproduction by @xcite , they closely followed the original paper @xcite with minor changes due to missing details, looking specifically at the fluency ratings. their results showed similar patterns, with fluency rating being significantly different among the svm model and both gedi and dexperts. however, inter-annotator agreement was lower (krippendorff's alpha decreased by 0.11). additionally, they conducted a second study where they gathered evaluations from eight additional annotators and analyzed the variability of the ratings. also focused on fluency, @xcite reproduced and found out that even if overall performance was lower, the relative performance of the three systems matched the original findings. the authors stated that lower agreement among annotators and their feedback suggests that ambiguity significantly affects human judgment. ## our reproduction our reproduction concentrates on the factuality criterion. following the standard procedures for human evaluation reproducibility, by using quantified reproducibility assessment @xcite , we try to track the original study as closely as possible. this entails the setup of two annotators that rate 300 definitions, first with a binary label of yes/no as an answer to the question "does this definition contain factually incorrect information?" and, in case of factual inaccuracies, to further provide a score on a set scale from 1 to 4 for the extent of the error, with the specification that 1 represents the lowest severity of error, while 4 is the highest. moreover, an additional experiment is carried out on domain-specific questions. in this sense, we define separate question sets depending on their domain, focusing on medical and physics-related questions. these term sets are then each assessed by a pair of participants with expert knowledge in the respective question set domain. the category of each term is established automatically, with subsequent human validation. this pipeline utilizes a llama 3.2 llm with the setup shown in box 1. all terms are assessed in this manner, after which a manual validation by one of the authors is performed. for the additional experiment, only the medicine (174 questions) and physics (42 questions) categories are considered in order to align with the participants' areas of expertise. box 1: llama 3.2 prompt system prompt: you are a helpful assistant. user prompt: what exact science is this term from? examples: medicine, geography, physics, chemistry, computer science. respond with the name for each term, no explanations terms: [ {"id":<term_id>, "term_text":"<@xmath0, ... ] after the annotation process is performed, the raters receive a feedback form regarding their participation. ## results when looking at the results from the original setup with all 300 generated definitions, the following can be stated: annotators agree poorly on whether a definition was factually incorrect (krippendorff's α = 0.466), but display even more reduced agreement on how severe those errors are (krippendorff's α = 0.132). over half of the definitions (54.0 %) were flagged by both annotators as incorrect, and nearly four-fifths (78.0 %) by at least one of them. we have computed the same values for the additional experiment, separately for each domain. focusing on medicine, the agreement on the yes/no decision rose to a substantial level (krippendorff's α = 0.682), and consistency around severity improved moderately (krippendorff's α = 0.507). still, more than half of the medical definitions (58.6 %) were marked wrong by both annotators, and almost three-quarters (73.0 %) by at least one. in physics, experts showed a more robust consensus on whether a definition was wrong (krippendorff's α = 0.790), yet their views on the degree of error remained split (krippendorff's α = 0.369). almost all physics definitions were labeled as containing factual inaccuracies (92.9 % by both annotators, 95.2 % by at least one of them). these patterns reveal the following observations: first, having domain experts makes it easier to agree on the presence of factual mistakes; however, domain expertise is less effective at harmonizing severity ratings-even when evaluators share deep subject knowledge. if we want reliable severity scores in future studies, we may need simpler scales or clearer examples of what each level means. this intuition is supported by the participants' feedback, which will be presented in the next section. when we compare our reproduction with the original evaluation, as seen in these differences can attest that prevalence estimates can shift substantially across studies and that agreement on how bad an error is appears especially fragile. future work might include more detailed anchor examples or simplified severity scales to boost reproducibility. following the original experiment and under the framework of quantified reproducibility assessment @xcite , we have also computed percentages for flagged factual inaccuracies for each generation system (svm reranker, gedi, dexperts); to compare the obtained values, we have utilized the coefficient of variation for small samples (cv * ), introduced by @xcite , with all results available in when we consider the rate at which at least one annotator rated a definition as factually inaccurate, the svm reranker again exhibits high variability, rising from 38% to 78% (@xmath4.0%, cv * ≈ 69), while gedi shifts from 52% to 78% (@xmath5.0%, cv * ≈ 40). dexperts shows the smallest proportional change, going from 86% down to 78% (@xmath6.0%, cv * ≈ 10), showing its relative reproducibility also in this setup. the cv * results provide a useful ranking: dexperts' percentages remain within roughly one-fifth and one-tenth of their means, respectively, while the svm reranker's rates vary by more than half. gedi consistently falls between these extremes. this suggests that dexperts seems to be the most reproducibly labeled for factual inaccuracy, and the svm reranker seems to be the least, with gedi occupying a middle position. metric value p significance (α = 0.05) pearson's r -0.327 0.78 n.s. spearman's ρ -0.500 0.67 n.s. we have also investigated the correlations between the original study's percentages and our own, using both pearson's r and spearman's ρ; the results are visible in ## participant feedback after completing the feedback form, it seems that our main evaluators reported that on average, they spent about 3 hours completing the annotations. they both needed to use the internet on some occasions for their ratings regarding definitions related to biology and chemistry. one noteworthy aspect that counted in their annotation process was their academic background. this was also seen in the additional experiment participants, with one rater stating that "as i am studying medicine, i know the importance of details, so information should be complete, very clear and precise when it comes to medical terms". they all had difficulties grading incomplete definitions, as well as those with slight inaccuracies. when they had doubts and rated in the middle of the scale, they justified their ratings for incomplete or vague information, incorrect or imprecise terminology, oversimplification, or definitions that were unclear or failed to fully define the term. all annotators stated that their capacity to rate factuality would have been enhanced if they had a better understanding of specific terminology and concepts, clearer grading examples for different levels of inaccuracy, and more detailed instructions on completeness and coherence. ## conclusion as numerous studies have shown @xcite , human evaluation still remains important for properly evaluating technological development. our findings suggest that employing domainspecific experts and providing proper annotation guidelines represent crucial factors for accurate automated systems. however, both automated and human evaluations in the nlp field have drawbacks, hence the need for hybrid automated-human evaluation systems. especially when it comes to human evaluation of generated scientific definitions, only experts in such domains should be employed. while human evaluation of factuality may come off as an objective task, it actually relies heavily on subjective interpretation, human judgment comprising a certain degree of creativity and divergent thinking (a thought process used for generating creative ideas through exploring multiple possible solutions) as it was stated by @xcite , particularly when evaluators draw on multidisciplinary expertise, like in our case, from psychology, medicine, and physics. while absolute agreement values differed from those originally reported, the general trends regarding which models yield more factual inaccuracies were broadly maintained. however, statistical analysis revealed low and non-significant correlations for the case where both annotators labeled definitions as factually inaccurate. these results outline both the challenges and the value of reproducibility in human evaluation setups.