S edar E val: Automated Evaluation using Self-Adaptive Rubrics (2024)

Abstract: The evaluation paradigm of LLM-as-judge gains popularity due to its significant reduction in human labor and time costs. This approach utilizes one or more large language models (LLMs) to assess the quality of outputs from other LLMs. However, existing methods rely on generic scoring rubrics that fail to consider the specificities of each question and its problem-solving process, compromising precision and stability in assessments. Inspired by human examination scoring processes, we propose a new evaluation paradigm based on self-adaptive rubrics. Specifically, we create detailed scoring rubrics for each question, capturing the primary and secondary criteria in a structured format of scoring and deduction points that mimic a human evaluator’s analytical process. Building on this paradigm, we further develop a novel benchmark called SedarEval, which covers a range of domains including long-tail knowledge, mathematics, coding, and logical reasoning. SedarEval consists of 1,000 meticulously crafted questions, each with its own self-adaptive rubric. To further streamline the evaluation, we train a specialized evaluator language model (evaluator LM) to supplant human graders. Using the same training data, our evaluator LM achieves a higher concordance rate with human grading results than other paradigms, including GPT-4, highlighting the superiority and efficiency of our approach.

## introduction the rapid advancements in large language models (llms) have led to their widespread use @xcite @xcite . however, assessing these models in open-ended question-answering scenarios poses a significant challenge. automated metric-based evaluations offer speed and convenience but often fall short due to the diversity of ground truth * corresponding author. @xcite @xcite . in contrast, humanbased evaluations provide reliable assessments but require substantial resources. to bridge the gap, the llm-as-a-judge paradigm attempts to strike a balance between automated and human evaluation. prominent examples of this approach include mt-bench @xcite and arena @xcite , which leverage proprietary models to evaluate individual or comparative model responses. these benchmarks use pre-defined principles, such as the 3h principle (human-like, helpful, harmonious), to determine responses that align best with realistic human preferences. the widespread use of @xcite as an evaluator in these studies presents challenges, including high costs for research institutions and potential data leaks. some studies @xcite @xcite propose using open-source pretrained models @xcite @xcite to train specialized evaluator lms, offering a more costeffective and secure solution. however, these methods typically use a uniform, question-agnostic rubric to guide the scoring process, overlooking the unique characteristics of each question. each question has different emphases, with primary and secondary scoring points. a general rubric applies uniform criteria, failing to accurately reflect human preferences. to adaptively align the scoring process with human judgment, we propose a novel evaluation paradigm based on self-adaptive rubrics. unlike coarse-grained general rubrics, we provide finegrained rubrics for each task, detailing specific scoring and penalty points with primary and secondary information. by analyzing focus points, we assign different values to each point. additionally, we introduce penalty points to penalize models for generating rejected responses. the scoring pro-the vietnamese poet rabindranath tagore, who won the nobel prize in literature for "leaves of grass," was which number nobel laureate from asia? firstly, two errors need to be corrected. therefore, the score is 1 point + 2 points + 0 points -0 @xmath0. final score: 3 points. cess considers both preferred and rejected perspectives. the inconsistent coverage of positive and penalty points ensures a more refined constraint on the scoring process. these detailed scoring trajectories simplify the evaluation process to an instruction-following task, reducing dependency on a judge model's internal knowledge and skills, leading to more accurate and stable assessments. building on this paradigm, we construct a new benchmark called sedareval, presenting a taxonomy comprising 5 major categories, with a dataset of 1000 queries, that fully aligns with realistic scenarios. we further conduct ablation experiments on each component of the llm-as-a-judge paradigm to investigate training a specialized llm for scoring, revealing their respective importance. we analyze whether llms can correctly evaluate questions they can correctly answer and find that insufficient diversity in existing sft data and a lack of evaluation-format data limit model performance. we also propose human-ai consistency to ensure evaluator llms maintain alignment with human preferences while leveraging their chain of thought capability to improve evaluation performance. based on these findings, we develop a specialized evaluator llm tailored to the benchmark for automated scoring. this model surpasses gpt-4 in model-level and question-level pearson correlation, gsb, and acc metrics, demonstrating higher consistency with human judgment. ex-perimental results validate the effectiveness and efficiency of our proposed paradigm. our contributions are summarized as follows: ## sedareval benchmark in this section, we introduce sedareval, a benchmark constructed upon the self-adaptive rubrics paradigm. we begin by delving into the intricacies of the self-adaptive rubric paradigm, followed by a detailed explanation of the benchmark's core components -questions and their corresponding rubrics -along with the methodology for model evaluation using this benchmark. to ensure the quality of sedareval, we incorporate comprehensive human assessment into the construction process, meticulously filtering out samples that fail to meet the established quality standards. ## evaluator language model in this section, we introduce an evaluator lm aligned with the self-adaptive rubrics paradigm to substitute human evaluators. we begin by delineating the evaluation format. subsequently, we propose a novel data filtering strategy to align the chain-of-thought evaluation process with human judgments. finally, we discuss the automation of rubric generation. ## analysis in this section, we conduct a comprehensive experimental analysis of the robustness of the proposed benchmark evaluations, examining the data distribution, training phases, and training paradigms of evaluator lms. our findings reveal limitations in current training methodologies for evaluator lms. building on these insights, we develop an evaluator lm aligned with the self-adaptive rubrics paradigm. ## conclusion in this paper, we introduce a novel evaluation paradigm called self-adaptive rubrics, aligning the scoring process with human judgment and reducing bias by tailoring rubrics to specific questions. based on this paradigm, we develop a new benchmark, insda. to automate scoring, we analyze existing open-source evaluator language models and identify training phase data diversity issues. we then introduce human-ai consistency to align the chain-of-thought evaluation with human judgment and propose an evaluator lm that follows the self-adaptive rubrics paradigm. experimental results show our model achieves higher consistency with human evaluation compared to gpt-4. we hope our work inspires researchers to apply this paradigm to more tasks, aligning automated scoring with human judgment.