H uman R ank E val: Automatic Evaluation of LM s as Conversational Assistants (2024)

Abstract: Language models (LMs) as conversational assistants recently became popular tools that help people accomplish a variety of tasks. These typically result from adapting LMs pretrained on general domain text sequences through further instruction-tuning and possibly preference optimisation methods. The evaluation of such LMs would ideally be performed using human judgement, however, this is not scalable. On the other hand, automatic evaluation featuring auxiliary LMs as judges and/or knowledge-based tasks is scalable but struggles with assessing conversational ability and adherence to instructions. To help accelerate the development of LMs as conversational assistants, we propose a novel automatic evaluation task: HumanRankEval (HRE). It consists of a large-scale, diverse and high-quality set of questions, each with several answers authored and scored by humans. To perform evaluation, HRE ranks these answers based on their log-likelihood under the LM’s distribution, and subsequently calculates their correlation with the corresponding human rankings. We support HRE’s efficacy by investigating how efficiently it separates pretrained and instruction-tuned LMs of various sizes. We show that HRE correlates well with human judgements and is particularly responsive to model changes following instruction-tuning.

## introduction the evaluation of language models (lms) is a challenging problem and a prolific research subject. many benchmarks have recently been proposed aiming to evaluate the general capabilities of lms, covering both automatic and human evaluation @xcite . evaluating lms' capabilities as conversational assistants, i.e. its adherence to human instructions, is particularly challenging as model inputs and outputs are more unstructured and open-ended. ideally, human judgement should be employed to evaluate such open-ended figure 1 : overview of humanrankeval: given a question with multiple answers, we correlate human scores of each answer with the log-likelihoods of the lm. the unabridged answers can be found in figure 2 . outputs, typically either through interactive conversation with lms @xcite or by presenting participants with outputs from different lms and collecting their preferences (van der @xcite . as this approach is time-consuming and does not scale well, previous work proposed to substitute human judgement with auxiliary large lms. however, these efforts have so far only been applied with proprietary models, e.g. gpt-4 @xcite , and with mixed results @xcite . on the other hand, conventional automatic evaluation of lms on knowledge-based tasks such as multiplechoice question-answering (qa) @xcite @xcite , can measure specific task performance in a scalable manner, but is not necessarily indicative of how an lm would perform these tasks in an open-ended conversational setting @xcite . to this end, we introduce humanrankeval (hre), an automatic evaluation task for lms as conversational assistants that comprises a novel dataset and metric. the core idea behind hre is to measure an lm's alignment with human preferences (hp). intuitively, given a question (q) with multiple available responses (a1, . . . , a4), hre measures how well an lm's "preference" ranking over those answers aligns with those of humans (see figure 1 ). we approximate hp by collecting a set of questions and rated answers from stackoverflow and stackexchange. the hre dataset covers a diverse collection of 14 topics, each containing 500 information-seeking questions paired with the top-4 answers rated (on average) by 100+ domain experts. to estimate the "preferences" of an lm, we obtain the log-likelihood of each answer under the model's distribution. the hre metric is calculated as the correlation of the lm's rankings against the corresponding human rankings. we should note that we do not consider hre as a replacement for human judgement, but rather propose its usage for fast iterations during development. we support hre's efficacy by investigating how effectively it separates pretrained and instructiontuned lms of various sizes. we then compare our results against those of other evaluation frameworks, showing that hre correlates well with human evaluation of lms and provides unique insights. specifically, relative to openllm, a highly popular automatic evaluation leaderboard @xcite , hre is able to more effectively differentiate pretrained and instruction-tuned lms. our contributions are threefold: 1) we create a large-scale, high-quality, diverse qa dataset to capture/approximate human preferences, 2) introduce an efficient automatic method to evaluate lms as conversational assistants by measuring the correlation of lm and human preferences, and 3) perform analysis that shows hre correlates well with human judgement and provides unique insights. foot_0 ## related work the evaluation of lms is a highly active research topic, exemplified by a recent survey @xcite that tracks over 250 papers, with over 100 of those published in just the last 12 months. there are additional surveys focused on alignment @xcite , trustworthiness @xcite , morals @xcite and fairness @xcite as well as multiple benchmarks with leaderboards covering a wide variety of lm behaviours @xcite @xcite @xcite @xcite @xcite @xcite , to list just a few. therefore, we focus on methods relevant to evaluating lms as conversational assistants to differentiate from prior work. ## humanrankeval we now introduce humanrankeval, an automatic evaluation task (comprising a novel dataset and metric) for lms as conversational assistants. as mentioned earlier, the core idea behind hre is to evaluate lms by observing how an lm's "preference" ranking (derived from the model's logprobabilities over several answers) aligns with human-obtained rankings. to achieve this, we gather open-ended, information-seeking questions from popular online communities to capture hp. each question comes with several answers ranked by domain enthusiasts (see figure 2 for an example), indicating the order of responses (most to least preferable). our data sources consist of stack-exchange and stackoverflow. as both contain a plethora of topics, some of which may be considered subjective, we endeavoured to select the more objective/quantitative topics that we would expect to have a high degree of consensus among users, i.e. most people would agree on "good" answers. ## conclusions multiple benchmarks have been proposed for evaluation of lms as conversational assistants. however, these are either not specifically designed for this purpose, rely on large (usually proprietary) lms as the ground truth, or are difficult to scale in terms of sourcing reliable human judges. we have therefore introduced humanrankeval, a novel automatic evaluation task that comprises a dataset of human-authored questions and answers coupled with a metric. the votes for each question were obtained from over 100 participating domain experts (on average), resulting in high-quality human preferences. hre performs evaluation by measuring how well the lm's "preferences", estimated as log-likelihoods of answers, correlate with human ratings. to validate hre, we demonstrated that it cleanly separates pretrained and instructiontuned lms of various sizes, and showed that its scores correlate well with human ratings. relative to knowledge-based evaluation, hre is particularly adept at detecting changes to lms' behaviour introduced by instruction-tuning and/or preference optimization. while knowledge-based automatic evaluation can test for specific skills, undesirable biases and essential world knowledge, we expect hre to accelerate the development of lms as conversational assistants by providing unique insights.