_intro.md · ai2lumos/lumos_data

1. What is WildBench? Why should I use it?

🦁 WildBench is a benchmark for evaluating large language models (LLMs) on challenging tasks that are more representative of real-world applications. The examples are collected from real users by the AI2 WildChat project.
🆕 Motivation: We aim to provide a more realistic and challenging benchmark for evaluating LLMs, as opposed to existing benchmarks that do not capture the diversity and complexity of real-world tasks.

🌠 Key Features:

🌟 Challenging & Real: We carefully curate a collection of 1024 hard tasks from real users, which cover common use cases such as code debugging, creative writing, and data analysis.
🌟 Reliable AutoEval w/ Checklists: Instead of merely asking GPT-4 to choose between A and B, we provide an instance-specific Checklist (i.e., a list of evaluation questions) for it to reason before making a judgment. It’s similar to CoT. Thus, our eval is highly interpretable and easy-to-verify.
🌟 Length Penalty: GPT-4 judges tend to prefer longer outputs (although humans do too); to avoid this, we devise a simple method to add length penalty on Elo. You can even slide it on our leaderboard UI!
🌟 Task Categorization: We tag each example with 12 task types, so we can analyze task-specific performance of LLMs, in addition to their overall ranking.
🌟 Fair Comparisons: WildBench tests all examples on all LLMs. This is different from arena-style evaluation, where one example is only tested on a single pair of models and never seen again.
🌟 Easy & Fast: WildBench (v1.0) contains 1024 examples now, and it is extremely easy to add your own LLMs to our leaderboard! We will do the work for you!
🌟 Dynamic: WildBench will not be a static dataset. We will continue adding new examples and updating evaluation methods based on community feedback.
🌟 Human Evaluation (ongoing): We are collecting human preferences via our Leaderboard UI (check the 🔍 🆚 tab). Please help us vote! (We’re planning to recruit domain experts too.)
🌟 Community driven: We welcome everyone to contribute to human evaluation and create challenging examples. We also value your feedback and suggestions, and will continue enhancing our benchmark leaderboard accordingly.

2. Where are the examples of WildBench from?

WildBench was designed with a focus on capturing the real-world complexity and diversity of tasks that large language models (LLMs) encounter. The design process involved several key steps:

2.1. Task Collection from WildChat

WildChat, a dataset akin to ShareGPT but larger and with user consent, was utilized to gather human-GPT conversations. We filtered the data for English, non-toxic responses and used various popular LLMs to generate responses, which were then scored using reward models such as StarlingRM and PairRM. The examples with the highest score variance were shortlisted, from which 1024 were chosen for curating the WildBench v1.0, ensuring a mix of diversity and quality.

2.2. Task Categories

The tasks are classified into 12 categories to cover a broad spectrum of real-user scenarios. This categorization helps in maintaining a balanced task distribution, mirroring the task variety in WildChat and differing significantly from traditional benchmarks.

2.3. Additional Annotations

WildBench includes further annotations like secondary task types, conversation turn counts, user intents, moderation tags, and evaluation checklists, providing deeper insights into the tasks and enhancing response assessments. These annotations are generated by GPT-4.

3. How do you evaluate the performance of LLMs on WildBench?

3.1. Elo Rating

We show two Elo rating for each model in our Main table. The "Overall" Elo rating is the standard method of using bootstrap method to compute the Elo scores for each model. The "Task-Avg" Elo is computed by first computing standard Elo on subsets of our data for each task type and then perform the average of them.

3.2. Length Penalty

We know that GPT-based evaluation tends to prefer longer responses, which is also the case for human evaluation. To mitigate this, we use a length penalty to normalize the Elo rating of the responses. Specifically, we compute two versions of Elo ratings for each model: one is based on win rates, and the other is based on "longer rates". The WinElo is the standard Elo rating, and the LongElo is the Elo rating considering longer outputs are always better than shorter outputs. Then, we present the final adjusted Elo by taking the difference between WinElo and LongElo, i.e., AdjustedElo = WinElo - LengthPenalty * LongElo.

3.3. Checklist-based Evaluation

In our automatic evaluation, we use a checklist (a list of 5~10 questions) for prompting GPT-4 to judge which model output is better. This checklist is example-specific. You can find real examples in "🔍 Explore | 🆚 Evaluate". The checklists help us ensure that GPT-4 uses a rather fixed standard to compare different model pairs on the same examples. Also, they facilitate us to better explain how GPT-4 make the decisions.

3.4. Estimated Win Rates

We estimate the win rates of each model winning GPT-4 by the differences of their Elo Rating versus GPT-4's. The formula can be found in this page.

3.5. Human-Verified Auto Evaluation

Although the current version of our WildBench is purely based on automatic evaluators, we aim to collect human preferences from our demo here ("🔍 Explore | 🆚 Evaluate") and then incorporate these human evaluation for mitigating the bias of GPT-4 based evaluation. We also plan to recruit domain experts for further improving the fairness of our evaluation. Please stay tuned!

4. How can I test my model on WildBench?

Please refer to our Github here and create a PR or issue to tell us the information about your model.

5. How do I know why a particular model is weaker than others?

Please click the tab for "🔍 Explore | 🆚 Evaluate" and select the models and task types that you're interested in. We'll sample an example with two model outputs for you to compare and you can see the model ids after you submit your feedback.

6. Any future plans for WildBench?

We have many todo items! The most important one is to collect human preferences for improving our evaluation. We are also going to recruit domain experts for further improving the fairness of our evaluation. As for auto-evaluation, we will add multiple auto evaluators for mitigating the bias of GPT-4 based evaluation. For example, we aim to use Claude 3 as evaluator to check if the ranking would be different. We're also developing our open-source evaluation models for supporting faster local evaluation.

7. How do I contact you?

Please use the community discussion board here or the Github issues. Also, please feel free to email us at yuchenl@allenai.org and mention "WildBench" in the title.