Ask Me Like I ’m Human: LLM -based Evaluation with For-Human Instructions Correlates Better with Human Evaluations than Human Judges (2025)

Abstract: Human evaluation in NLP has high cost and expertise requirements, and instruction-tuned LLMs are increasingly seen as a viable alternative. Reported correlations with human judgements vary across evaluation contexts and prompt types, and it is hard currently to predict if an LLM-as-judge metric will work equally well for new evaluation contexts and prompts, unless human evaluations are also carried out for comparison. Addressing two main factors contributing to this uncertainty, model suitability and prompt engineering, in the work reported in this focused contribution, we test four LLMs and different ways of combining them, in conjunction with a standard approach to prompt formulation, namely using written-for-human instructions verbatim. We meta-evaluate performance against human evaluations on two data-to-text tasks, and eight evaluation measures, also comparing against more conventional LLM prompt formulations. We find that the best LLM (combination)s are excellent predictors of mean human judgements, and are particularly good at content-related evaluation (in contrast to form-related criteria such as Fluency). Moreover, the best LLMs correlate far more strongly with human evaluations than individual human judges across all scenarios.

## introduction human evaluation remains the most reliable method for system evaluation in nlp @xcite , but its high cost, required expertise, and methodological inconsistencies limit its scalability and reliability @xcite . the emergence of large language models @xcite @xcite has caused a paradigm shift in text generation and understanding across many domains @xcite . llms are exhibiting state-of-the-art performance in problem-solving and reasoning tasks @xcite . llms also hold out the appealing vision of cheaper human-like evaluation, demonstrating adaptability and generalisation capabilities @xcite . while individual human judges are subject to inter-rater variability and require multiple annotators for reliability, llms may provide more consistent judgements when resources are constrained. 'llm-as-judge' approaches do address some of the issues with human evaluation, such as cost and evaluator inconsistency, but their reliability when applied to new tasks needs to be demonstrated via correlation tests with human judgements. in the experiments presented in this paper, we investigate the alignment between human and llm judgements across a range of criteria for two nlp data-to-text tasks. to standardise prompt formulation, we use the same instructions as those provided in human evaluations, and compare them with more conventional llm prompts, in conjunction with single models and model combinations of both varying and comparable sizes. in contrast to previous work, we conduct our llm-as-judge experiments using verbatim human evaluation instructions as a way of standardising prompt formulation. furthermore, we investigate llm-as-judge performance in this setting, comparing with more standard llm prompt formulations, in meta-evaluation against human judgements on data-to-text tasks. ## discussion and conclusion we have presented results for experiments with llm-as-judge approaches for two types of datato-text tasks and eight evaluation methods, using as a way of standardised prompt formulations the verbatim human instructions from previous evaluations. these were shown to work better than more conventional prompt formulations in all scenarios, irrespective of task or the length of input/output. an unexpected discovery was that llms can serve as sanity checkers for human evaluations. the repronlp shared task organisers had no basis for deciding which of two reproductions of puduppully and lapata (2021) they reported was right: either repro 1 (h 2 in this paper) was right and the work had excellent reproducibility, or repro 2 (h 1 ) was right and it had terrible reproducibility. because both of our llm majority votes strongly agreed with repro 1 and strongly disagreed with repro 2, the indication is that repro 1 (h 2 ) gave the better results out of the two reproductions. overall, we have found our best llms to be highly reliable predictors of human evaluations, and to benefit from human-type detailed evaluation instructions. the result that individual human judges correlate far less well with overall human judgements than llms do, implies that if the choice is between a small number of human judges and an llm you are better off using the llm.