LLM -Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts (2024)

Abstract: This paper introduces a framework for the automated evaluation of natural language texts. A manually constructed rubric describes how to assess multiple dimensions of interest. To evaluate a text, a large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses. The LLM predictions often fail to agree well with human judges—indeed, the humans do not fully agree with one another. However, the multiple LLM distributions can be _combined_ to _predict_ each human judge’s annotations on all questions, including a summary question that assesses overall quality or relevance. LLM-Rubric accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters. When evaluating dialogue systems in a human-AI information-seeking task, we find that LLM-Rubric with 9 questions (assessing dimensions such as naturalness, conciseness, and citation quality) predicts human judges’ assessment of overall user satisfaction, on a scale of 1–4, with RMS error < 0.5, a 2× improvement over the uncalibrated baseline.

## introduction many fields that must assess large numbers of short documents have turned to nlp-assisted workflows. for example, lawyers conducting legal discovery must identify all relevant documents @xcite -a task also faced by journalists and historians. social scientists and market researchers must code survey responses @xcite enumerate.ai; atlas.ti). teachers or examiners must evaluate student writing @xcite and provide feedback @xcite . doctors, social workers, or public health agencies or researchers may assess an individual's mental health or safety from their social media posts (chancellor and de choudhury, 2020; @xcite or from clinical interviews and assessments @xcite . the above settings evaluate human-authored texts. in addition, nlp developers must assess the quality of their machine-generated texts-texts that are consumed by end users, but also hidden intermediate steps in agentic workflows (such as chains of thought, tool calls, and revisions). with the recent commercialization of conversational ai, for example, it is crucial to evaluate dialogue systems during development and monitor them after deployment. special care is needed in high-stakes settings like medical dialogue @xcite . manual evaluation has long been the gold standard for assessing text, including generated text @xcite . humans are often asked to consider multiple criteria and then provide a final assessment @xcite . humans may also be asked to produce reference answers to which other humans can compare the target text. yet manual evaluation is expensive, time-consuming, and not without its own quality and reliability issues @xcite @xcite . because of these challenges, and the increasing abilities of large language models (llms) @xcite , experimenters have recently been eliciting ratings directly from an llm @xcite @xcite @xcite . but can llm evaluation be trusted? it solves the time, scaling, and possibly cost issues, but leaves open the problem of aligning these llm ratings with human judgments. we present a general approach to this alignment problem. we demonstrate its value for the evaluation and comparison of llm-powered dialogue systems, in an information-seeking dialogue task @xcite similar to @xcite . evaluation in this setting is complex owing to competing factors that might affect a human judge's assessment of the dialogue. these may include correctness of responses, accuracy and helpfulness of citations, length and complexity of responses, and more @xcite . our llm-rubric approach begins with a manually authored evaluation rubric. the rubric's multiple-choice questions cover various evaluation dimensions, and it may also include a question that assesses overall quality or relevance. evaluating a text, such as a dialogue, then consists of two main steps: @xcite for each rubric question we elicit the llm's probability distribution over possible responses, by prompting it with the text and the rubric question, and (2) we aggregate and calibrate these distributions with a small feed-forward network that has been trained to match the individual preferences of human judges. a high-level overview of llm-rubric is shown in figure 1 . for research in generative nlp, once the rubric and llm are fixed, llm-rubric can be used like other metrics (bleu, rouge, etc.) to drive system development, monitor quality, demonstrate the value of a new technique, and conduct competitions. in our dialogue evaluation experiments, each user-ai dialogue is evaluated by 3 trained annotators (randomly drawn from a larger pool) who each answered the same 9 rubric questions. our method uses these data to train an automatic llm-based evaluator, without treating the 24 human annotators as interchangeable. overall, we find foot_1 that • personalized calibration of an llm evaluator of overall satisfaction on ă 750 synthetic dialogues significantly improves its prediction of human judgments and correlations with human judgments, but still works poorly. accurate automated text assessment could replace human assessment in many other settings, such as those reviewed at the start of this paper. it could also be used in new settings where human assessment was never feasible. in ai-powered user interfaces, instantaneous scoring of user-written text can feed into downstream decisions such as providing writing feedback or deciding how to proceed with a dialogue. an ai reasoning engine may internally apply a rubric to assess the validity of a proposed natural-language reasoning step @xcite . when processing a large document collection, an llm can be used to assess the compatibility of two text passages @xcite @xcite , potentially in a more nuanced way than vector similarity; this problem arises in workflows for matching, routing, clustering, and fact-checking @xcite ; and the papers just mentioned). finally, automated assessments could provide signals for training text generation @xcite @xcite . to allow llm-rubric to support all of these use cases, we release general code along with the datasets we created for this paper (see url on page 1). we discuss limitations at the end of the paper. ## the llm-rubric method it is challenging to model human preferences in a combinatorial space such as text. reasonable human judges may differ @xcite on @xcite what textual properties they happen to prefer (e.g., concise vs. detailed, formal vs. informal, novice vs. expert audience), (2) how they combine multiple preferences into an overall assessment, and (3) how they convey that assessment through a numerical score. figure 2 shows that in our dataset ( §3), different human judges indeed have very different marginal distributions of overall score. clearly these cannot all be matched by a judge-independent system (e.g., the llm shown at the lower right of figure 2 ). to expose the different properties and preferences at play, we ask the human judges a series of finer-grained questions about different evaluation criteria. it is already common in practical settings ( §1) to at least mention such criteria in instructions to human judges. we use the same questions to query an llm, 2 and train a calibration network to jointly adjust the llm's scores to match the scores of any given human judge. we refer to this methodology as llm-rubric. the gray curves in figure 2 show that on held-out dialogues, the calibrated overall score is now distributed like that of the given judge. we will see later that these scores are also more accurate on the individual dialogues. in this section, we present llm-rubric in a general way, but for concreteness, we also introduce details of our specific experimental setup. ## data conversational ai systems are now being widely deployed. to test our methods on dialogue evaluation, we invest in developing both synthetic and real datasets of human-ai conversations. we focus on english information-seeking dialogues in the "it help" (enterprise information technology) domain @xcite . as in many real world domains, dialogue data here is often proprietary to the system owner and/or private to the user. acquiring experimental access to live systems for evaluation is even more difficult. thus, we build and evaluate several llm-powered dialogue systems, which differ in their ability to search a corpus of websites related to microsoft azure 9 help topics. for training data, we generate a corpus of synthetic dialogues with simulated users, and have human judges rate them. collecting these artificial dialogues is efficient, since judges only have to annotate conversations and not interact with the systems first. for our final test data, we have our judges actually interact with the live systems as users and then annotate their own dialogues. all of our judges are professional annotators. to mine the topics for both synthetic and live evaluation, we use real user queries and click data from a large commercial web search engine, which further increases the realism of our experiments. below, §3.1 explains how we compile a corpus of background documents and how we select topics to ensure that the generated and collected conversations are diverse and are indeed informationseeking, rather than navigational or transactional. ## experiments we will evaluate how well llm-rubric can predict individual judges' assessments y a 0 of our q 0 (overall user satisfaction). we evaluate predictions ŷa 0 both in absolute terms (whether they achieve low root-mean-squared error, or rmse) and in relative terms (how well ŷa 0 correlates with y a 0 , i.e., whether ŷa 0 can be used to rank pt, aq pairs). we train our calibration networks on synthetic dialogues. we then evaluate them not only on heldout synthetic dialogues but also on real dialogues, to demonstrate that the llm scoring and its calibration can generalize from synthetic to real data. hyperparameter selection. to train a system on a given training set, we evaluate hyperparameter settings from a grid by 5-fold cross-validation on the training set, and then use the selected hyperparameters to train on the entire training set. we select the hyperparameters that maximize the main task objective, namely the log-likelihood of (held-out) annotations y a 0 . the hidden layer sizes h 1 , h 2 each range over t10, 25, 50, 100u, the batch size ranges over t32, 64, 128, 256u, the learning rate of the adam optimizer ranges over t0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01u, and the numbers of epochs for pretraining and fine-tuning each range over t5, 10, 20, 30, 40, 50u. foot_10 synthetic data evaluation. we test our calibration network on all 741 synthetic dialogues, using 5-fold cross-validation; the dataset is split at the dialogue level so that each dialogue appears in only one fold. different folds may select different evaluation hyperparameters, resulting in different architectures for the calibration network. 12 real data evaluation. we test our calibration network on all 223 real dialogues, after training on all of the synthetic dialogues (again selecting hyperparameters by 5-fold cross-validation). baseline methods. as we ablate different components of the oracle model by withholding the llm response vector from the model input and by depersonalizing the calibration network (oracle w/o personalized calibration) by dropping w a k . to restore a judge a's idiosyncratic distribution of q 0 scores (figure 2 ), without restoring their idiosyncratic computation of q 0 from other dimensions, we try correcting the output of the depersonalized calibration network using an a-specific isotonic regression model. our depersonalized oracle is similar to the oracle, but instead of using the responses of the actual target judge a, it uses the distribution of responses of all other judges (averaging their onehot vectors), holding out the target judge. 15 it also drops the personalized weights w a k . thus, the oracle provides a rough upper bound on llm-rubric. the depersonalized oracle provides a rough upper bound on a version of llm-rubric that produces a-independent results. ## results a trivial baseline of predicting a constant q 0 (the overall mean from training data) achieves an rmse of 0.82 on both synthetic and real conversations. llm-rubric roughly halves this (row 6 of ta-ble 1 ), so it explains « 3 4 of the variance in human judgments of q 0 across judges a and texts t . its predictions of q 0 have tolerably low error and correlate reasonably well with those of human judges. in sharp contrast, the llm's direct response to q 0 (row 2 or 3) does worse than the constant baseline. even calibrating its response distribution for each judge (row 4) barely improves on the baseline, explaining only 5-10% of the variance in human judgments and achieving only « 0.2 correlation with them. this suggests that the llm cannot help assess q 0 (user satisfaction) until we ask it about the finer-grained dimensions q 1 -q 8 . the results obtained by factscore (row 5) do not correlate any better with overall satisfaction, so percentage of factually correct sentences is also not a good indicator of overall user satisfaction. moreover, @xcite showed that dialogue systems were poorly evaluated by simple metrics of lexical overlap with human responses. ## analysis calibration. does our trained llm-rubric produce well-calibrated probability distributions for q 0 (as one would expect from maximumlikelihood training)? we checked on synthetic data. it obtained excellent smece values of ă 0.05 for each y 0 p y 0 " t1, 2, 3, 4u, where smece is the smoothed expected calibration error (błasiok and nakkiran, 2023). informally, this means that for each y 0 p y 0 , when we examine the held-out examples pt, 0, a, y a 0 q with pa py 0 | t, q 0 q « p, the fraction where y a 0 " y 0 was in fact « p. §k shows calibration plots and discusses how to use calibrated probabilities for downstream decisions. ## conclusions this work proposed llm-rubric-a rubricbased framework for automatic evaluation of text. we trained and tested it on novel datasets of information-seeking dialogues. llm-rubric performs multidimensional evaluation using a blackbox llm, then aggregates and calibrates these multidimensional responses for each human judge. although the llm's raw responses do not highly correlate with human judgments in such a complex task, we found that combining its response distributions on all questions can predict each human judge's responses, including overall satisfaction. we obtained substantial improvements on rmse and on both linear and rank-based correlation metrics, on held-out synthetic conversations (development data) and real ones (test data). below, we discuss limitations, ethics, uses, and extensions.