Using LLM Judgements for Sanity Checking Results and Reproducibility of Human Evaluations in NLP (2025)

Abstract: Human-like evaluation by LLMs of NLP systems is currently attracting a lot of interest, and correlations with human reference evaluations are often remarkably strong. However, this is not always the case, for unclear reasons which means that without also meta-evaluating against human evaluations (incurring the very cost automatic evaluation is intended to avoid), we don’t know if an LLM-as-judge evaluation is reliable or not. In this paper, we explore a type of evaluation scenario where this may not matter, because it comes with a built-in reliability check. We apply different LLM-as-judge methods to sets of three comparable human evaluations: (i) an original human evaluation, and (ii) two reproductions of it which produce contradicting reproducibility results. We find that in each case, the different LLM-as-judge methods (i) strongly agree with each other, and (ii) strongly agree with the results of one reproduction, while strongly disagreeing with the other. In combination, we take this to mean that a set of LLMs can be used to sanity check contradictory reproducibility results if the LLMs agree with each other, and the agreement of the LLMs with one set of results, and the disagreement with the other, are both strong.

## introduction while considered a particularly reliable form of evaluation @xcite , the cost and expertise required for human evaluation experiments prevent them from being used as standard in nlp. large language models now exhibit astonishing performance across a wide range of different tasks including problem-solving and reasoning tasks @xcite . in combination with their ability to interpret and follow provided instructions, this makes them tempting, more cost-efficient alternatives to human evaluation, and they are beginning to be used in place of human evaluators in approaches commonly referred to as 'llm-as-judge.' however, llm judgments sometimes do, and sometimes do not, agree with comparable human judgements, for reasons that are not entirely clear. this means their reliability needs to be demonstrated anew for each new domain and/or task via metaevaluation against human judgments, incurring the very cost their use is meant to obviate. there are nevertheless situations where we may not have to worry about this, namely where we wish to arbitrate between multiple comparable human evaluations whose results contradict each other. here it may be possible to use results from comparable llm judgments to decide which of the contradictory human evaluation results are more likely to reflect the true picture. in this paper, we explore this question in the context of contradictory reproducibility results for human evaluation experiments, using reproductions and reproducibility results from the repronlp shared tasks @xcite as our data. we start with a look at related research (section 2), followed by an overview of our study (section 3). we present the three sets of original studies and reproductions for them that constitute our data (section 4), and the llm-as-judge methods we use (section 5). for each of the three scenarios we then present side-by-side evaluation results, and correlation matrices between the different evaluations (section 6). we discuss the results (section 7) and finish with concluding remarks (section 8). code and resources can be found on github. foot_0 2 related work llm-as-judge evaluation methods have been shown to correlate remarkably strongly with human evaluations across a range of task contexts @xcite , including text summarisation assess-ment (e.g. g-eval @xcite , machine translation evaluation (e.g. gptscore @xcite , and code generation assessment @xcite , to name but a few. a number of studies have investigated ways to improve the reliability of llm-as-judge evaluations, including pairwise ranking, and using bigger, instruction-tuned models (e.g. gpt-4) @xcite ; varying evaluation item order, and using majority voting @xcite ; targeted prompt tuning @xcite ; deterministic settings for hyperparameters like temperature, top-k, fixed random seed @xcite ; and conducting systematic sweeps over prompt templates and decoding settings to identify the most stable configuration @xcite . overall, while techniques like the above have improved alignment with human judgments, and correlations are therefore often high, it remains unclear why this is not always the case, so that strictly speaking meta-evaluation tests against human judgments must be carried out every time llm-as-judge methods are to be used with a new llm, task or domain. to the best of our knowledge, applying llm-asjudge evaluation for sanity-checking human evaluations has not so far been explored. ## background and study overview consider the following scenario. the repronlp shared tasks @xcite produced sets of two or more highly comparable human evaluations, one of which was the original study, and one or more were reproductions carried out by shared task participants with precisely aligned experimental details controlled by the organisers. when conducting quantified reproducibility assessment with qra++ @xcite , the organisers found that in some cases, one of the (typically) two reproductions strongly agreed with the results from the original evaluation, while the other strongly disagreed. in such cases, the repronlp shared task organisers had no basis for deciding which of the two reproductions reflected the true picture: either the agreeing reproduction was right and the original study had excellent reproducibility, or the disagreeing reproduction was right and it had terrible reproducibility. the overarching aim of the study we report in this paper is to examine how llm-as-judge results behave in such scenarios, and whether they can provide a basis that was missing in the repronlp shared task for deciding between the two possibilities above. our starting point is three sets of comparable human evaluations from repronlp 2024, each consisting of (i) a set of human-produced system-level scores from the original study (o); and (ii) two sets of human-produced system-level scores from reproduction studies conducted by repronlp participants (r1 and r2). for each set of comparable human evaluations o, r1, r2 we produce directly comparable llmas-judge results using different llm ensembles j * . we then compute pearson's correlations between all pairs of sets of results and analyse them. we start below with an overview of the three original studies and two reproductions each that form the basis of our investigation, in terms of the common data and evaluation criteria used in them, and the experiment-level qra++ type ii and iv @xcite reproducibility results reported in the repronlp results reports for them (section 4). next we describe the llm-as-judge methods we used to compute the sanity checks, detailing the models and model combinations they comprise (section 5). finally, we present and discuss the side-by-side results and correlations between them (section 6). ## original studies and reproductions note that while the raw responses from the original experiment are available, the script to calculate system-level scores is not, and the two teams above arrived at different scores for the original results when reimplementing it @xcite . we also found slight differences when we reimplemented it. in order to be able to compare the reproduction results to the original results on an equal footing, we used the scores produced by our reim-plementation for all evaluations in the atanasova et al. scenario. evaluation criterion: atanasova et al. (2020) used coverage, non-redundancy and noncontradiction as the evaluation criteria, of which the reproduction studies use coverage only, where good coverage is defined as follows: the explanation includes important, salient information and does not omit any key points that contribute to the factcheck. repronlp type ii and iv results: the table below from the repronlp 2024 results report shows pearson's and spearman's correlations (type ii reproducibility) in the third and fourth columns, with proportion of matching rankings (type iv reproducibility) shown in the last column. as can be seen from the table, between o (the original study) and r1, strong correlations were found, and all findings were confirmed, but both measures were very poor between o and r2, and between r1 and r2. feng et al. (2021) used this dataset in the original study, the human evaluation of which was reproduced in repronlp'24 (belz and thomson, 2024) by two teams (fresen et al., 2024; lango et al., 2024). the human evaluation experiment involved summaries (abstracts) generated for 10 randomly selected dialogues. evaluation criterion: feng et al. (2021) evaluate informativeness, conciseness and coverage, of which the reproduction studies address informativeness, defined as follows: informativeness measures whether the abstract contains the key information from the original conversation. repronlp type ii and iv results: the table below from the repronlp 2024 results report shows that strong correlations are seen between the original study (o) and r2. however, correlations between o and r1, and between r1 and r2, are close to 0 (no correlation). at the same time, nearly all findings from o were confirmed by r2, but only about half of the findings were confirmed between r2 on the one hand, and o and r2 on the other. evaluation criteria: puduppully and lapata (2021) evaluated grammaticality, coherence and conciseness/repetition. the reproduction studies address all three evaluation criteria, defined as follows: grammaticality: is the summary written in well-formed english? coherence: is the summary well structured and well organized and does it have a natural ordering of the facts? conciseness/repetition: does the summary avoid unnecessary repetition including whole sentences, facts or phrases? repronlp type ii and iv results: as the table from the repronlp 2023 results report below shows, strong correlations were found, and all findings were confirmed, between o and r1. however, correlations were negative and only 1/3 of findings were confirmed both for o and r2, and for r1 and r2. ## results in this section, we present two types of results for each of our three sets of evaluations above: (i) sideby-side system-level scores, and (ii) correlation matrices between the scores obtained in each set. ## discussion we have looked at three scenarios where we had one original human evaluation and two contradicting reproductions of the original evaluation, one strongly agreeing with it, the other strongly disagreeing. in this situation, we would not normally have a way of telling whether (i) the reproduction that agrees with the original evaluation is right and the original evaluation has terrible reproducibility, or (ii) the reproduction that disagrees with the original evaluation is right and the latter has excellent reproducibility. for each of these three scenarios, we tested multiple llm ensembles as stand-in replacements for the human evaluators, and found that in all three scenarios, they not only all strongly agreed with each other, but also with the original evaluation and one of the reproductions. that the llms agree with each other may not come as a surprise, but that they also strongly agree with one set of human evaluation while strongly disagreeing with the other, is more so. this pattern held true for all twelve different llm ensembles we tested, whether they consisted of all small llms, all medium-sized llms, or a combination of both. in one scenario @xcite , the small-llms ensemble j c s agreed slightly less well with two of the other evaluations (r1, j c v ) than the other agreeing evaluations, but in the other scenario we tested it in @xcite , j c s and j c v had a correlation of @xmath0.99. interestingly, we saw different kinds of disagreement. in the feng et al. scenario, correlation coefficients were all very close to 0 indicating an entirely random relationship between the disagreeing evaluation and the others. in contrast, in the case of coherence and repetition in puduppully & lapata, we saw pronounced negative correlation scores throughout, indicating an inverse relationship between the disagreeing evaluation and the rest. finally, for the atanasova et al. evaluations, and grammaticality in puduppully & lapata, we see a mix of random and inverse relationships. all of which begs the question what this can tell us about the disagreeing evaluations? is there necessarily something wrong with them? in directly comparable human evaluations, the main difference will tend to be the sample of evaluators performing the assessments. clearly, different samples (from the population of all evaluators) will results that differ to different degrees, with a small proportion deviating substantially from true population-level result. the greater the deviation, the smaller the likelihood of it occurring, but it is possible that the disagreeing evaluations we have seen in this paper are due to rare sampling effects, whereas the llms are able to produce assessments closer to the population level, because trained on very large (in effect population-level) samples of text. the nature of the disagreement discussed above can provide more information. if we are dealing with a rare sampling effect, we would not expect to see near perfect random correlations (as in r1 in the feng et al. scenario) with multiple other evaluations. in this scenario therefore, it may be supposed that something has gone wrong, perhaps a coding error at some point in the pipeline from collecting the evaluator assessments to aggregating the results at system-level which resulted in the association between evaluation items and scores being lost. in the case of the negative correlations seen consistently with other evaluations in the coherence and repetition evaluations in the puduppully & lapata scenario, another explanation is needed. here, the relationship is not random; there is a pronounced association, but it is in the wrong direction. here it is possible that at some point in the analysis carried out in the r2 evaluation, the signs of the evaluation scores inadvertently became inverted, perhaps as a result of a bug in the best-worst scaling. this leaves just the mixed random and negative correlations seen in the grammaticality evaluation in the puduppully & lapata scenario. given the negative correlations seen consistently for the other two evaluation criteria (coherence and repetition), we would expect to see the same for grammaticality given that the evaluator sample was the same. the fact that we see a mix of random and mild to medium positive associations makes this picture very hard to interpret. note however that correlations between the other evaluations (both human and llm-based) are also considerably weaker and more mixed than in any of our other scenarios, perhaps indicating that the grammaticality evaluation task itself was somehow harder to perform consistently. ## conclusion in this paper, we have examined the behaviour of llm-as-judge methods in situations where they are used to obtain additional evaluation results to add to a set of comparable human evaluation studies of which at least two strongly disagree with each other. we have seen that in such scenarios, all twelve llm ensembles we tested invariably strongly agreed with one of the disagreeing human evaluations, and strongly disagreed with the other, providing evidence that the one they all agree with is the more reliable. drawing out the commonalities from the three different scenarios we examined (corresponding to five different evaluation experiments, each with one evaluation criterion), we conclude that llms can be used as sanity checkers to validate human evaluations in scenarios where: even in the case of single human evaluations, running multiple llm-as-judge methods in parallel could provide additional confirmation of results, provided the methods involve a variety of different types of llms, and they all agree with each other and with the (single) human evaluation. all in all, using llms as sanity checkers for human evaluations would seem to be one application of the 'llm-a-judge' paradigm where the built-in reliability check against human evaluations means results means they can be relied on without the need for independent validation by meta-evaluation for every new domain and/or dataset.