Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation (2024)

Abstract: As large language models (LLMs) evolve, evaluating their output reliably becomes increasingly difficult due to the high cost of human evaluation. To address this, we introduce FLAMe, a family of Foundational Large Autorater Models. FLAMe is trained on a diverse set of over 100 quality assessment tasks, incorporating 5M+ human judgments curated from publicly released human evaluations. FLAMe outperforms models like GPT-4 and Claude-3 on various held-out tasks, and serves as a powerful starting point for fine-tuning, as shown in our reward model evaluation case study (FLAMe-RM). On Reward-Bench, FLAMe-RM-24B achieves 87.8% accuracy, surpassing GPT-4-0125 (85.9%) and GPT-4o (84.7%). Additionally, we introduce FLAMe-Opt-RM, an efficient tail-patch fine-tuning approach that offers competitive RewardBench performance using 25×fewer training datapoints. Our FLAMe variants outperform popular proprietary LLM-as-a-Judge models on 8 of 12 autorater benchmarks, covering 53 quality assessment tasks, including RewardBench and LLM-AggreFact. Finally, our analysis shows that FLAMe is significantly less biased than other LLM-as-a-Judge models on the CoBBLEr autorater bias benchmark.

## introduction the growing capabilities of large language models (llms) present a key challenge: how can we reliably evaluate their long-form responses? a promising approach is to use the models themselves as autoraters. after large-scale multitask instruction tuning, llms can generalize to follow new human instructions @xcite @xcite , making them suitable for this task. this is appealing because human evaluation, while essential, is limited by subjectivity @xcite , inconsistency among raters @xcite , and the high costs of extensive evaluations @xcite @xcite . training llm autoraters on human judgments is essential for aligning them with human preferences @xcite . however, gathering these judgments is both costly and time-consuming. reusing human evaluations from prior research is a promising approach, yet it faces challenges such as inconsistent standards, diverse criteria, inadequate documentation, and privacy or proprietary concerns. on the other hand, training autoraters on model outputs offers consistency @xcite but risks reinforcing biases and hallucinations @xcite and may also breach proprietary llm service terms. 2 to address these limitations, we curated and standardized human evaluations from prior research to create flame, a collection of 102 quality assessment tasks comprising more than 5.3m total human judgments ( §3). flame spans a wide variety of task types, from assessing summarization quality to evaluating how well ai assistants follow user instructions. we hypothesized that training on this large and diverse data collection would enable llm autoraters to learn robust, generalized patterns of human judgment, minimizing the impact of noisy or low-quality human judgments. for transparency and reproducibility, we use only publicly available human evaluation data with permissive licenses from previous studies ( §3.2). to address challenges due to the lack of standardization and documentation, we thoroughly exam- ## the flame collection we curated 5.3m human judgments across 102 training tasks, with an additional 53 tasks reserved for evaluation ( §5.1). appendix b lists our datasets. our data covers various task types and llm capabilities ( §3.2-3.3). we manually crafted task definitions and evaluation instructions, converting all tasks into a unified format ( §3.4). ## model we fine-tune the instruction-tuned palm-2-24b on the flame collection to create general-purpose llm autoraters that can be prompted to perform various tasks. we train three flame variants: 1) flame-trained with examples-proportional mixture weights @xcite ; 2) flame-rminitialized with flame and fine-tuned on a balanced mixture of four pairwise evaluation datasets covering chat, reasoning, and safety ( §4.2); and 3) flame-opt-rm-trained with rewardbenchoptimized mixture weights ( §4.3). ## experiments we compare flame to several popular llm-as-a-judge autoraters ( §5.2) using a suite of 12 autorater benchmarks (1 held-in and 11 held-out), covering a total of 53 quality assessment tasks ( §5.1). overall, flame variants outperform all llm-as-a-judge autoraters on 8 out of 12 benchmarks ( §5.3). ## further analysis of flame in this section, we depart from the typical focus on analyzing the effect of factors like model size, data size, and data quality in multitask learning, which have been extensively studied @xcite . instead, we examine potential biases in our llm autoraters. we find that our models are significantly less biased than popular llm-as-a-judge autoraters. in appendix f, we further demonstrate flame's potential utility for ai development, particularly in identifying highquality responses for code generation. ## conclusion we curated and standardized human evaluations from permissively licensed datasets, compiling a data collection of over 100 quality assessment tasks with 5m+ human judgments. we demonstrate that this collection can be used for training generalpurpose llm autoraters and optimizing them for specific downstream applications. our models outperform popular proprietary llm autoraters on 8 out of 12 autorater benchmarks, covering 53 tasks.