LLM -as-a-Judge Failures at Automating the Identification of Poor Quality Outputs in Free-Form Texts (2025)

Abstract: Large language models (LLMs) such as GPT-4, Claude and LLaMA are routinely used to evaluate long-form text generated by language models. We study the ability of these models to identify low quality texts, an increasingly rare subset of output which is of great interest to pinpoint during development. We present experiments with a panel of LLM judges, and crowd-sourced approximations of reference judgments. Pinpointing sub-par outputs is a difficult task for both models and crowdworkers, with models doing overall better. Moreover, unlike findings in prior work on factoid question answering, panels of cheaper models do not agree as well with high quality developer judgments of low quality as panels of frontier models. We present qualitative and quantitative analysis of the relative strengths of models in the panel, gleaning insights why they yield better results over a single model.

## introduction the outputs of llms such as e.g. gpt-4 @xcite , claude @xcite are used by millions but the evaluation of their long-form text responses pose an outstanding research challenge @xcite . as llms produce texts with impressive overall fluency, accuracy, and usefulness, developers of user experiences powered by the models are interested in detecting the rare bad quality generations @xcite . aiming to identify poor outputs invites a rethinking of the criteria for automated evaluation. human agreement rates like accuracy can be misleading since they reflect the abundant good answers rather than the ability to detect poor ones (section 3.3). we use f 1 measure for the class of bad outputs (i.e. texts that experts would rate as 1, 2, or 3 on a five-point quality scale). we take developer judgments as the gold standard for evaluation and study how successfully these can be replaced by llms or crowd-sourcing. we expand on the growing body of research on llms as evaluators @xcite @xcite @xcite and their accompanying challenges @xcite , and are the first to focus on the detection of bad outputs. we carry out our experiments on summeval @xcite summaries paired with developer and crowd-sourced quality ratings. these fine-grained scores are important in llm development to exclude unacceptable outputs and guide the model to ideal responses @xcite . we build upon prior work from factoid question answering showing that a panel of multiple cheaper models better aligns with all human judgments than a single frontier model does @xcite . our results on long-text evaluation in contrast show that combining frontier models improves the ability to detect poor outputs, but not necessarily for weaker models (section 4). further, we seek to understand how a panel of judges can resolve some of the challenging cases that a single model cannot handle correctly. our quantitative and qualitative analysis show that (i) different models have distinct weaknesses in evaluating different response aspects but combining multiple strong models helps mitigate individual limitations; (ii) a panel of llms enhances the detection of challenging unacceptable responses while maintaining a stable and accurate evaluation of common acceptable cases. (iii) no evaluation is perfect, even 'high quality' human ratings have errors. using llms for automatic evaluations. llms align better with human evaluations @xcite @xcite , and a number of studies have identified gpt-4 as the single best automatic evaluator @xcite . llm evaluators are more flexible. they accommodate various evaluation tasks via prompt instructions, including reference-free scoring @xcite , chain-of-thought scoring @xcite , and pairwise preference assessments @xcite . although llm-asa-judge is gaining popularity @xcite , @xcite show that most existing benchmarks and evaluations still rely on classical evaluations, such as exact-match, binary yes/no responses, or multiple-choice formats, with only one benchmark incorporating long, open-ended answers that use llms to evaluate. further analysis of best practices and concerns are needed to trust reliability for automated llm evaluations @xcite . ## llms complement each other we now study the effectiveness of a panel of judges compared to single-model evaluations, and the optimal number of models in the panel. we then conduct a qualitative analysis of common failure modes of the best three llm and crowd evaluations. combining multiple models has higher alignment with developer judgments. we compute the macro f 1 score between automatic and developer evaluations. figure 2 shows the median and maximum scores for panels of different compositions. for the median score, when model selection is unknown, the shift from a single model to a three-model panel shows notable improvement, while adding more models provides minimal benefit. adding more models increases cost but yields little improvement in evaluation. for the maximum score, while using a single strong model might be adequate for specific evaluation tasks, figure 2 shows that different models excel in different aspects. different llms show strengths in different evaluation criteria, suggesting that relying on a single llm for all aspects does not yield optimal results. for all size panels, gpt-4, llama3-70b, and claude were part of the panel achieving the maximum score. combining these individually strong models yield the better results (coherence: 0.76; consistency: 0.84; fluency: 0.73; relevance: 0.76) than any of its constituents (individual best in ## ensemble combination analysis section 4 shows that ensemble of size three has the best alignment with developer judgments. we pick out the best model combination for sizes 1, 3, 5, and 7 for each aspect and show that for all aspects, gpt-4, llama3-70b, or claude always appear in one of the best combinations, showing that a strong model in the mix is necessary to improve reliable evaluations ( ## conclusion automating evaluation is easier for the common class than the rare class. however, identifying the rare class is crucial for detecting low quality responses. a failure to identify low quality responses can result in extra human annotations to accurately find the low-quality responses. while combining multiple strong models improves detection, it does not help weak models and crowdworkers. the percentage of examples crowdworkers identify correctly but llms cannot is rare, suggesting llm evaluations are more reliable than low-quality crowdsourced evaluations. even high-quality human annotations (developers, while original work refers to experts) can make mistakes. llms excel in aspects requiring detailed text analysis, such as consistency, but no evaluation is perfect. future work can explore semi-automated evaluations that combine human expertise with multiple llms, leveraging each of their strengths while minimizing human effort and cost to make evaluations reliable.