Constructing Domain-Specific Evaluation Sets for LLM -as-a-judge (2024)

Abstract: Large Language Models (LLMs) have revolutionized the landscape of machine learning, yet current benchmarks often fall short in capturing the diverse behavior of these models in real-world applications. A benchmark’s usefulness is determined by its ability to clearly differentiate between models of varying capabilities (separability) and closely align with human preferences. Existing frameworks like Alpaca-Eval 2.0 LC (CITATION) and Arena-Hard v0.1 (CITATION) are limited by their focus on general-purpose queries and lack of diversity across domains such as law, medicine, and multilingual contexts. In this paper, we address these limitations by introducing a novel data pipeline that curates diverse, domain-specific evaluation sets tailored for LLM-as-a-Judge frameworks. Our approach leverages a combination of manual curation, semi-supervised learning to generate clusters, and stratified sampling to ensure balanced representation across a wide range of domains and languages. The resulting evaluation set, which includes 1573 samples across 14 categories, demonstrates high separability (84%) across ten top-ranked models, and agreement (84%) with Chatbot Arena and (0.915) Spearman correlation. The agreement values are 9% better than Arena Hard and 20% better than AlpacaEval 2.0 LC, while the Spearman coefficient is 0.7 more than the next best benchmark, showcasing a significant improvement in the usefulness of the benchmark. We further provide an open-source evaluation tool that enables fine-grained analysis of model performance across user-defined categories, offering valuable insights for practitioners. This work contributes to the ongoing effort to enhance the transparency, diversity, and effectiveness of LLM evaluation methodologies.

## introduction large language models (llms) have dramatically changed the landscape of machine learning research and have been incorporated in products for the past few years. along with their rise, a multitude of benchmarks and frameworks @xcite have been proposed to assess the capabilities of llms which include knowledge tasks such as mmlu @xcite , reasoning tasks like @xcite and more standard nlp tasks @xcite . however, these benchmarks fail to capture the behavior that a user experiences in a chat/generative applications. typically, human evaluations are seen as a gold standard to determining which llm responses are preferable over others in a chat setting but is time-consuming and expensive to conduct @xcite . to address this shortcoming, zheng et al. introduced the concept of llm as a judge as an automatic evaluator alternative, which uses another llm the judging of model completions to another llm such as gpt-4 or gpt-4o @xcite . alpaca-eval is another benchmark designed under the paradigm of llm as an evaluator where a target llm's completions are compared against a reference llm's output (the default being gpt-4 turbo) and assigned a winrate against the reference @xcite . it has seen widespread adoption since it is cheap, fast, and mitigates length bias @xcite . similarly, arena-hard v0.1 is recent benchmark which focuses on distilling the hard category of chatbot arena into a smaller evaluation set @xcite . they use a topic clustering pipeline to cluster prompts with openai's embedding model (text-embedding-3-small) (ope-nai, 2024b) and score each cluster based on difficulty, creativity, and reasoning ability with gpt-3.5 turbo. they also introduce a notion of separability (how well can a benchmark differentiate between models) and agreement with human preferences (i.e. chatbot arena) as a measure of benchmark quality. unfortunately, there are still some limitations with the current open-source llm-as-a-judge framework. alpaca-eval 2.0 lc is dominated by general chat queries/instructions and has few prompts in domains such as coding, medical, finance, law and mathematics as shown in figure 2 . arena-hard v0.1 addresses some of these deficiencies by upweighting coding and mathematics prompts and restricting the general chat queries to 30% if the evaluation set. however, both evaluation sets are strictly in english therefore not accessing the model's multilingual capability and have a smaller number of prompts in more niche categories like law and medicine. as models are acquiring more capabilities across various data types such as charts/tables, domains and languages, it becomes crucial to determine how to evaluate each model's ability in a scalable manner. in this paper, we attempt to address challenges from alpaca-eval 2.0 lc and arena-hard v0.1 by introducing more diversity across domain knowledge and languages. to accomplish this, we introduce a simple data pipeline methodology to create a new evaluation set designed for these specific contexts. first, we source prompts from various open source datasets (shown in there are several advantages to our approach as shown in figure 1 . similar to arena-hard v0.1, our approach is robust to contamination as we can periodically run our data pipeline on the same data to get new samples or potentially even a new data mixture. as mentioned earlier, our methodology allows introduction of new datasets which enables diversity rather than offered by arena-hard v0.1 and alpaca-eval. in addition, our evaluation set more closely mirrors chatbot arena rankings; figure 4 shows a visual comparison of model rankings. in particular, our evaluation set places gemini-1.5-flash (deepmind, 2024) over gemma2 27b instruct (team, 2024) which aligns with chatbot arena rankings whereas the others rank gemma2 27b over gemini-1.5-flash. moreover, since we use open source models for the entire pipeline, practitioners can mold the pipeline and generate evaluation sets to test domains and capabilities they care about. after we have obtained the evaluation set, we execute the same procedure as llm-as-a-judge by generating the outputs completions from gpt-4o and using them as reference to construct a leaderboard from ten various open and closed-sourced models. with this labeling approach, we are able to breakdown the composition of prompts into various categories and report category win rates. we plan to release an evaluation tool which displays the category winrate for all models on the leaderboard and an explorer which displays both the target model as well as the reference model's completions for a prompt and the reasoning given by the llm judge. this analysis tool allows users to obtain fine-grained insights on where different models succeed and fail for their particular use-case. our main contributions can be summarized below: at their core, benchmarks are tool to estimate llm capabilities. there are many different flavors of benchmarks, spanning either across domains or various tasks. some popular benchmarks include: boolq @xcite , mmlu @xcite , @xcite , math @xcite , xsum @xcite , hellaswag @xcite , and mgsm @xcite ). an expanded framework of static benchmark is autobencher which automatically creates new benchmarks which finds holes in knowledge of current sota llms @xcite . these types of benchmarks have ground-truth references and compare how closely the llm's completion aligns with those references. an inherent limitation with static benchmarks is that they are hosted on the internet and thus are susceptible test leakage contamination @xcite . the other style of benchmarking relies on constructing a human evaluation trials on a set of evaluation prompts. due to the expensive nature of human evaluation, a recent, cheaper alternative is to use sota llms to evaluate model completions either through single score or pairwise comparison with a reference answer, popularly referred to as llm-as-a-judge @xcite @xcite @xcite . this motivates the need for "live, refreshable" benchmarks so that the integrity of the benchmark can be maintained. livebench is a framework which sources data from arxiv papers, news articles, and datasets to periodically replace the stale prompts @xcite . chatbot arena is an open platform that allows online users to send prompts to two different models and compare/contrast the models' response @xcite . users can then vote on which completion was superior. other live benchmarks include dyn-abench @xcite , livecodebench @xcite , and r2e @xcite . our work lies in the intersection between llm-as-a-judge and live benchmarks as our data pipeline enables periodic refreshing of the evaluation set from existing clusters. furthermore, our data pipeline is fairly general as it can consume a variety of diverse datasets (relative to arena-hard v0.1 and alpaca-eval), consists of using open-source models, and is flexible enough to work on the user's desired data. ## methodology in this section, we describe our approach to creating novel evaluation set using llm-as-a judge. we enumerate the datasets that we source from to create our unlabeled corpus and subsequently describe our data pipeline for generating the evaluation set. ## experimental setup in this section, we discuss finer details about the data pipeline we mentioned in the prior section. experimental setup on a set of ten highly rated models foot_0 as well as defining the metrics which determine the quality of the benchmark. ## limitations/future work there are certain limitations to our work. currently, the categories we enumerate in our data pipeline is manually specified by humans and significant curation is done to ensure high quality prompts; for future work, we want to expand to using llms as category generators as well as quality checkers to automate the human effort out of this pipeline. moreover, the diversity of prompts in the multilingual categories could be limited as we consider bucket all subdomains of a language into the same category. sub-categorization of domains in non-english languages is left for future work. for improving our leaderboard, we wish to add more models to be more representative of the entire spectrum of other leaderboards and futhur increasing the quality of the bradley-terry models we use to obtain the model's confidence intervals. in order to improve category separability, we look to creating a methodology on figuring out the minimum number of samples required to improve separability. the other aspect of future work relies to details regarding llm-as-a-judge evaluation. typically, the judge models are ablated but less explored is the quality of the reference answer and whether one can use a weaker model instead of a stronger one to see if metrics are maintained. current metrics define how separable a benchmark is and how much it aligns with human preferences but fails to account for the composition and diversity of the underlying data. for future work, we seek to quantify the diversity of each benchmark to understand how many capabilities/domains it spans. ## conclusion we introduce a data pipeline that leverages via semi-supervised learning with a k-nn to enable practitioners to create benchmarks on their own data for targeted domains. through evaluations of ten various closed and open-sourced models, we demonstrated that our benchmark achieves higher separability and agreement with ci with respect to chatbot arena, nearly 5 and 10 percentage points higher than the next best baseline, respectively. our benchmark covers a wide variety of topics such as finance, medicine, legal and different languages absent in other llm as a judge benchmarks. we hope that llm developers can use our data pipeline to create their own benchmarks to evaluate their models for their particular use-case.