Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024)

Abstract: Proprietary LMs such as GPT-4 are often employed to assess the quality of responses from various LMs. However, concerns including transparency, controllability, and affordability strongly motivate the development of open-source LMs specialized in evaluations. On the other hand, existing open evaluator LMs exhibit critical shortcomings: 1) they issue scores that significantly diverge from those assigned by humans, and 2) they lack the flexibility to perform both direct assessment and pairwise ranking, the two most prevalent forms of assessment. Additionally, they do not possess the ability to evaluate based on custom evaluation criteria, focusing instead on general attributes like helpfulness and harmlessness. To address these issues, we introduce Prometheus 2, a more powerful evaluator LM than its predecessor that closely mirrors human and GPT-4 judgements. Moreover, it is capable of processing both direct assessment and pair-wise ranking formats grouped with a user-defined evaluation criteria. On four direct assessment benchmarks and four pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and agreement with humans and proprietary LM judges among all tested open evaluator LMs. Our models, code, and data are all publicly available.

## introduction evaluating the quality of outputs produced by language models (lms) is progressively becoming difficult, as the outputs cover an extremely diverse distribution of text and complex tasks. to address this issue, language model-based evaluation has emerged as a scalable and cheap paradigm for assessing lm-generated text @xcite figure 1 : weak evaluators (e.g., llama-2-chat-70b, prometheus, and gpt-3.5-turbo) achieve low scoring correlation with strong evaluators (e.g., humans, gpt-4, and claude-3-opus). on the other hand, scores provided by strong evaluators highly correlate with each other. @xcite . in this paradigm, lms are either prompted to output a scalar indicator of quality (denoted as direct assessment) @xcite @xcite or to determine which of two outputs are preferred (denoted as pairwise ranking) @xcite @xcite . prior works employing proprietary lms as evaluators have demonstrated not only high correlations with human evaluations but also increased speed and cost-effectiveness @xcite @xcite . however, relying on proprietary lms for evaluation poses significant challenges. the lack of transparency about their training data compromises both fairness and reproducibility, making it problematic to use them in evaluation pipelines. additionally, concerns regarding controllability and affordability also persist @xcite . to address these issues, recent works have focused on developing evaluator lms that are open-access, transparent, and controllable @xcite @xcite @xcite . yet, these models often yield scoring decisions that do not correlate well enough with human judgments or those made by proprietary lms, failing to effectively simulate them. moreover, open evaluator lms are not flexible since they are typically trained only to perform either direct assessment or pairwise ranking and assess based on general public preferences like helpfulness and harmlessness, limiting their ability to handle diverse real-life scenarios. to close the gap with proprietary lms, we investigate unifying the two model-based evaluation paradigms -direct assessment and pairwise ranking -to train a robust unified evaluator lm. we propose a recipe based on merging the weights of two evaluator lms trained separately on direct assessment and pairwise ranking formats. our key empirical observation is that weight merging can yield an evaluator lm that not only works in both formats, but also outperforms evaluator lms that are jointly trained or only trained on a single format. to demonstrate our approach, we develop the preference collection, a new fine-grained pairwise ranking feedback dataset that builds on the feedback collection @xcite , which is a direct assessment feedback dataset. we choose mistral-7b @xcite and mixtral-8x7b @xcite as our base models, and merge the weights of evaluator lms separately trained on the feedback collection and the preference collection to obtain our resulting models, prometheus 2 (7b & 8x7b). on four direct assessment benchmarks (vicuna bench, mt bench, flask, feedback bench), the prometheus 2 models demonstrate the highest correlation with both human evaluators and proprietary lm-based judges compared to existing open evaluator lms, with the pearson correlation surpassing other baselines by 0.2 units across all datasets. similarly, on four pairwise ranking benchmarks (hhh alignment, mt bench human judgment, auto-j eval, preference bench), the prometheus 2 models show the highest agreement with human evaluators among all the open evaluator lms we tested, reducing the performance gap with gpt-4 in half. our contributions are summarized as follows: ## methodology we propose a new recipe for training a unified evaluator lm based on merging the weights of models trained for direct assessment and pairwise ranking. we begin with background on direct assessment and pairwise ranking for evaluator lms (section 3.1, 3.2), followed by the construction process of our training data (section 3.3). finally, we present our methods to train state-of-the-art evaluator lms, prometheus 2 models (section 3.4). ## experimental setup the statistics of all the benchmarks are in the four direct assessment benchmarks are: the four pairwise ranking benchmarks are: in direct assessment, we conduct referencebased evaluations by appending the reference answer as the input. we use pearson, spearman, and kendall-tau as performance metrics to measure scoring correlations against reference evaluators. moreover, we include the results of the referencefree direct assessment evaluation in appendix f. in pairwise ranking, we conduct reference-free evaluations. based on judgments assigned by humans, we use accuracy as our metric to measure agreement between evaluator lms and humans. also, the mt bench human judgment and auto-j test set includes a 'tie' option assessed by human evaluators. we evaluate in two ways: by excluding all 'tie' options for pairwise ranking (denoted as 'w/o tie'), or by using direct assessment where responses scored as 'ties' are grouped, and pairwise rankings are applied to the remaining responses with differing scores (denoted as 'w/ tie'). ## experimental results in this section, we compare the evaluation capabilities of prometheus-2 models with other baselines using a direct assessment format (section 5.1) and a pairwise ranking format (section 5.2). additionally, we measure the consistency of the scores from evaluator lms in appendix e. ## analyses of weight merging to understand the effectiveness of our proposed weight merging method in the context of evaluations, we address the following research questions: