Human-Centered Design Recommendations for LLM -as-a-judge (2024)

Abstract: Traditional reference-based metrics, such as BLEU and ROUGE, are less effective for assessing outputs from Large Language Models (LLMs) that produce highly creative or superior-quality text, or in situations where reference outputs are unavailable. While human evaluation remains an option, it is costly and difficult to scale. Recent work using LLMs as evaluators (LLM-as-a-judge) is promising, but trust and reliability remain a significant concern. Integrating human input is crucial to ensure criteria used to evaluate are aligned with the human’s intent, and evaluations are robust and consistent. This paper presents a user study of a design exploration called EvaluLLM, that enables users to leverage LLMs as customizable judges, promoting human involvement to balance trust and cost-saving potential with caution. Through interviews with eight domain experts, we identified the need for assistance in developing effective evaluation criteria aligning the LLM-as-a-judge with practitioners’ preferences and expectations. We offer findings and design recommendations to optimize human-assisted LLM-as-judge systems.

## introduction recent advancements in large language models (llms) challenge traditional methods of assessing natural language generation (nlg) quality, as known metrics, such as bleu @xcite and rouge @xcite , fall short for creative tasks. the diverse and expanding ca-pabilities of llms @xcite present a selection challenge for practitioners, requiring evaluations of extensive outputs across contexts like summarization and retrieval-augmented generation @xcite . the subjective and use case-specific nature of emerging nlg tasks often demands human review, making the evaluation process hard to scale without suitable automatic metrics. while experts can perform evaluations, this is costly and impractical for rapid iteration in early development stages. @xcite . one potential solution to these challenges is to leverage the capabilities of llms to aid in the evaluation process. despite not always being accurate, llms have the potential to significantly reduce the workload by identifying outputs where they are not confident, thus indicating where human input may be required. additionally, llms can assist practitioners in identifying and customizing criteria specific to their use case-such as, for example, faithfulness to contextual information, naturalness of the conversation, and succinctness-with which they wish to conduct their evaluations. this customization enables a more targeted and effective assessment of model outputs, tailored to the specific requirements of their tasks. in this paper, we present results from a user study of evalullm @xcite , a tool designed to facilitate the evaluation of model outputs. evalullm simplifies how practitioners choose llms by offering a quick way to assess and compare their performance across various tasks. this method accelerates the development of evaluation criteria and helps manage the growing variety and capabilities of llms. to understand the challenges and user needs in model evaluation that leverage llm-as-a-judge to automate the process, we conducted formative, semi-structured interviews with 8 practitioners (data scientists, software engineers, and ai engineers) who have been involved in model performance evaluation projects over the past year. our interviews revealed various challenges and needs. for instance, practitioners highlighted the necessity for rapid performance comparison across different setups, the importance of defining evaluation criteria (e.g., structured and customizable templates aligned with specific use cases), and strategies for effectively integrating llm-as-a-judge into their workflow (e.g., starting with a small subset of data before scaling up). in this paper, we present the following contributions: ## related work llms trained to follow instructions can generate results that surpass the quality of data produced by humans. this makes it increasingly challenging to assess the quality of natural language generation (nlg) outputs @xcite ) @xcite ) @xcite . traditional referencebased metrics, such as rouge @xcite and bleu @xcite , might not effectively capture the essence of llm outputs, especially in scenarios where the output space is broad and varied. this means multiple different outcomes can all be valid, making it nearly impossible to create sufficiently comprehensive reference sets. consequently, these metrics may not be reliable indicators of nlg output quality, as they often demonstrate a low correlation with human judgments @xcite . recent advances highlight llms' potential as customizable judges, @xcite @xcite capable of adapting to various tasks beyond traditional evaluation methods. techniques like g-eval @xcite use chain-of-thought prompting and form-filling to assess nlg quality, while gptscore @xcite evaluates using conditional token probabilities, enhancing scoring granularity. alpacaeval @xcite ) compares model win rates, and prometheus @xcite ) is a fine-tuned llm specifically designed for evaluation tasks. these methods align closely with human preferences, especially in creative tasks, emphasizing llms' ability to mimic human judgment. their effectiveness relies on tailored prompt design and user-defined criteria for precise evaluations. while not part of this paper, in our own work, we have also done comprehensive benchmarking of human agreement of different llm-as-a-judge approaches for different use cases and we found that depending on use case, llms as judges, and judging approach, we were able to achieve good results. note that this is often a hard problem for humans too and interrater reliability can be a good reference. previous research has investigated using expertlabeled data to develop custom evaluation metrics like autocalibrate @xcite , but this method is limited by the availability of such data. for reference-free evaluations, interactive human involvement is preferable, allowing users to refine criteria effectively by reviewing outputs. con-stitutionmaker @xcite enables feedback on model outputs to iteratively refine prompts, focusing more on ai prototyping than evaluation. other tools like zeno @xcite , the what-if tool @xcite , and errudite @xcite help identify model vulnerabilities by analyzing specific data segments. evallm @xcite allows users to define criteria interactively, using llm-as-a-judges for output ratings, although this can be limited by llm reasoning capabilities @xcite . our study builds on these insights, proposing a system where practitioners define criteria in natural language for llms to perform pairwise comparisons, enhancing trust through a "human-in-the-loop" blind review process that eliminates the need for expert data. ## evalullm to explore how to support users in developing their own custom evaluation criteria for accurate and reliable evaluations that align with human preferences in a trustworthy manner, we designed and deployed evalullm @xcite . this tool enables users to generate evaluation outputs by providing a prompt, selecting multiple models, and defining llm-as-a-judge with custom metrics using natural language. users can then review the results and provide feedback, inspecting the agreement between human and ai evaluations through a blind review process. in this paper, we use eval-ullm as a conceptual design probe with users to explore the design space of how to support development of custom evaluation criteria for accurate and reliable evaluations that align with human preferences in a trustworthy manner. the overall user flow of evalullm comprises of three stages (see figure 1 ). the build experience focuses on defining the llm-assisted evaluation experience to initiate the auto-evaluation process, the review experience, providing a high-level summary of the evaluation results, and the inspect experience allows users to manually examine the generated outputs through a blind review process. the data generated from this process can be used to calculate the agreement rate, assisting practitioners in better assessing the agreement between human and llm-as-a-judges. this assessment is crucial for calibrating trust and aids in making informed decisions about whether to change configurations and rerun the evaluation. in the absence of reference data, related studies suggest that llms may not be entirely suitable for use as numerical judges @xcite . this is because grading based on single answers may fail to detect minor distinctions between specific pairs. furthermore, the outcomes could become unreliable, as absolute scores tend to vary more than relative pairwise results when there are changes in the judging model @xcite . to mitigate these challenges, evalullm uses a pairwise comparison approach, as it can reduce the complexity of the evaluation task by breaking down the comparison of multiple outputs into smaller decisions between pairs of data which might yield to more accurate evaluation results at the cost of additional inference operations. the evaluation method involves making pairwise comparisons between the outputs of different models, similar to the alpacaeval approach @xcite . however, instead of comparing outputs to a single reference, they are compared against one another. ## methodology our goal was to explore the challenges users encounter during llm-assisted model evaluations and, based on our observations, to design a framework that meets their needs and supports effective collaboration between humans and llm-as-ajudges. we used evalullm to facilitate the creation of evaluation tasks and conducted our research through semi-structured interviews using webex. participants accessed a prototype of eval-ullm, shared their screens, and used think-aloud methods to create evaluation tasks. each participant worked on the same task: using an llm-asa-judge to identify the best model for generating headlines from the cnn/daily mail dataset. ## results our data analysis identified nine themes, categorized into use case challenges, evaluation criteria, and evaluation workflow (for a full list with example quotes see ## discussion and design recommendations our findings highlight user needs across different use cases when using llm-as-a-judge. users require guidance to evaluate model outputs effectively. we discuss the implications of our findings and propose design recommendations for llm-asa-judge tools and user experiences. ## conclusion we studied evalullm, an ai-assisted tool utilizing llms alongside humans as judges for llmgenerated content. our findings highlight the potential of llms as customizable judges and underscore the importance of interactive, transparent, and user-centered evaluation processes. based on our findings, we offer design suggestions for practitioners that can help them build more effective , nuanced, adaptable, and user-friendly evaluation tools that meet diverse needs as compared to automated benchmarks. inspired by our user research, we are currently in the process of rolling out an evolved ai-assisted evaluation tool to a larger user population to observe "usage in the wild." "it was like eighty-twenty, eighty percent of the time they don't have it." -p5 support comparison with different setup "say we had five different models and for each model we had 20 different configurations or something like that. now that's 100 different combinations. um, we'd like the limited judge to be to run on like all hundred. give us an overview. which are the three that are actually worth looking at?" -p2 "gpt 4 as a baseline and we're just trying to see how close are we getting with these other models in order to replicate the performance." -p7 shifting evaluation priority "i know that's like a terrible metric [confusion matrix] to be used as the first one, but we have actually done this with a client because they asked us to do so. they're looking for just accuracy." -p5 evaluation criteria desire structured and customizable templates "a freeform text box is too simple. i would love there to be templates that i can utilize. and at the very least, be able to just edit so that i can get into my use case." -p7 "more examples might be nice." -p2 need for multiple rounds of iterations "it can be really hard to figure out how to express the evaluation criteria in a way that makes sense to the model. but it can also just be hard in your own mind to figure out what it means for a title to be good." -p2 "if i think, without having a clearer sense of what the evaluation is, sort of what a baseline evaluation is, it might be nice to have a couple of features of an evaluation that we could just select in like a checkbox. " -p3 display performance for each criteria individually "there might be times where you have to trade off on certain kinds of things and win rate is not necessarily the best metric because there are multiple categories to define what it means to win." -p7 "so i'm covering a lot of ground there, and i know that's hard for the model to deal with because now the model has to have a whole lot of different criteria, and it's all drawn up by the ones, but that's kind of what a good title headline is about." -p7 evaluation workflow run evaluation on subset of data first "we don't have a problem here because the data set is small. but, like, if there's like, a 1000. then it would it make sense to go through the entire batch and we find out your volume criteria needs to be tweaked." -p2 "i'd want to iterate on my judge enough for it to get a decent annotator agreement and then let it go wild." -p2 instant feedback on human-ai agreement "tell me when to quit." -p1 ensuring transparency for trustworthy evaluation "so i definitely want, as we discussed earlier, a lot of transparency and exactly what is being sent to the models to generate the responses and then what is then being sent to the llm as a judge." -p2 "maybe a small note on, like, you know what the prompt is, like, what the data set is and what the tool is doing." -p8