2. Where are the examples of WildBench from?
++ WildBench was designed with a focus on capturing the real-world complexity and diversity of tasks that large language models (LLMs) encounter. The design process involved several key steps: +
+2.1. Task Collection from WildChat
++ WildChat, a dataset akin to ShareGPT but larger and with user consent, was utilized to gather human-GPT conversations. We filtered the data for English, non-toxic responses and used various popular LLMs to generate responses, which were then scored using reward models such as StarlingRM and PairRM. The examples with the highest score variance were shortlisted, from which 1024 were chosen for curating the WildBench v1.0, ensuring a mix of diversity and quality. +
+2.2. Task Categories
+
+ + The tasks are classified into 12 categories to cover a broad spectrum of real-user scenarios. This categorization helps in maintaining a balanced task distribution, mirroring the task variety in WildChat and differing significantly from traditional benchmarks. +
+2.3. Additional Annotations
++ WildBench includes further annotations like secondary task types, conversation turn counts, user intents, moderation tags, and evaluation checklists, providing deeper insights into the tasks and enhancing response assessments. These annotations are generated by GPT-4. +
+