gradio space

README.md

@@ -1,12 +1,15 @@
 ---
-title: WildBench Leaderboard
+title: AI2 WildBench Leaderboard
 emoji: 🦁 
 colorFrom: blue
 colorTo: yellow
-sdk: static
+sdk: gradio
+sdk_version: 4.19.2
+app_file: app.py
 pinned: true
-fullWidth: false
-tags: ["leaderboard"]
+fullWidth: true
+hf_oauth: true
+api: false
 ---
 
 Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

_about_us.md

@@ -0,0 +1,17 @@
+## About Us
+
+### Team 
+
+We are from [AllenAI](https://allenai.org/) (AI2), a non-profit research organization. Team members include:
+
+[Bill Yuchen Lin](https://yuchenlin.xyz/), [Khyathi Chandu](https://khyathiraghavi.github.io/), [Faeze Brahman](https://fabrahman.github.io/), [Yuntian Deng](https://yuntiandeng.com/), [Abhilasha Ravichander](https://www.cs.cmu.edu/~aravicha/), [Valentina Pyatkin](https://valentinapy.github.io/), [Ronan Le Bras](https://rlebras.github.io/), and [Yejin Choi](https://homes.cs.washington.edu/~yejin/).
+
+(Each of us contributed significantly and nearly equally. Ronan & Yejin are advisors. Bill is the main project lead.)
+ 
+### Contact
+
+Please contact us in the following ways:
+- Github Issues/PRs for adding a new model: [https://github.com/allenai/WildBench](https://github.com/allenai/WildBench)
+- HF Discussions for general questions about the leaderboard: [https://huggingface.co/spaces/allenai/WildBench/discussions](https://huggingface.co/spaces/allenai/WildBench/discussions)
+- Other questions: Please contact Yuchen with email: yuchenl[at]allenai[dot]org
+ 

_header.md

@@ -0,0 +1,4 @@
+<br/>
+
+# 🦁 WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild
+[⚙️ GitHub](https://github.com/allenai/WildBench) | [🤗 HuggingFace](https://huggingface.co/collections/allenai/wildbench-65e8f2fa9c1260a85a933627) | [💬 Discussions](https://huggingface.co/spaces/allenai/WildBench/discussions)

_intro.md

@@ -0,0 +1,131 @@
+
+<details open><summary style="font-size: 1.8em; font-weight: bold;">1. What is WildBench? Why should I use it?</summary>
+<div style="font-size: 1.4em; margin-top: 30px;">
+    🦁 <b>WildBench</b> is a benchmark for evaluating large language models (LLMs) on challenging tasks that are more representative of real-world applications. The examples are collected from real users by the <a href="https://wildchat.allen.ai/"><b>AI2 WildChat</b></a> project.</li>
+    <br>
+    <b>🆕 Motivation</b>: We aim to provide a more <strong>realistic</strong> and <strong>challenging</strong> benchmark for evaluating LLMs, as opposed to existing benchmarks that do not capture the <em>diversity</em> and <em>complexity</em> of <em>real-world</em> tasks.    
+    <h2 style="color: purple">🌠 Key Features:</h2>
+    <ul>
+        <!-- <li><b style="color: purple">🌟 Fine-grained:</b> 
+        We provide a fine-grained annotation for each example, including task types and <b>checklists</b> for evaluating the quality of responses. In addition, we use <b>length-penalized</b> Elo ratings to ensure that the quality of responses is not biased towards longer outputs.</li>
+        <li><b style="color: purple">🌟 Transparent & Fair: </b> We test all LLMs on the SAME set of examples, ensuring a fair  evaluation. You can explore the data and see the difference between two models to analyze the concrete gap between any pair of LLMs. </li>
+        <li><b style="color: purple">🌟 Easy & Fast:</b> WildBench (v1.0) contains 1024 examples, and it is extremely easy to add your own LLMs to our leaderboard! 1️⃣ Let us know your model ID and suggested inference configs; 2️⃣ We'll run inference and evaluation for you; 3️⃣ Voilà! We'll notify you when your results are ready on the leaderboard.</li>
+        <li><b style="color: purple">🌟 Dynamic:</b> WildBench will not be a static dataset. We will continue adding new examples and updating evaluation methods. Our goal is to include new challenging examples from real users over time and provide fast yet reliable evaluations.</li>
+        <li><b style="color: purple">🌟 Human Verification (ongoing):</b> Although we currently use GPT-4 as the automatic evaluator, we are also collecting human preferences here (see the 🔍 🆚 Tab). We plan to update the leaderboard by incorporating human evaluations in the near future.</li>
+        <li><b style="color: purple">🌟 Community-driven:</b> In addition to collecting human preferences for improving our evaluation, we also welcome community users to contribute new examples they find challenging to top LLMs like GPT-4/Claude3. Any feedback and suggestions are welcome, and we'll do our best to upgrade our data and evaluation methods accordingly.    </li> -->
+        <li><b style="color: purple">🌟 Challenging & Real:</b> We carefully curate a collection of 1024 hard tasks from real users, which cover common use cases such as code debugging, creative writing, and data analysis.</li>
+        <li><b style="color: purple">🌟 Reliable AutoEval w/ Checklists:</b> Instead of merely asking GPT-4 to choose between A and B, we provide an instance-specific Checklist (i.e., a list of evaluation questions) for it to reason before making a judgment. It’s similar to CoT. Thus, our eval is highly interpretable and easy-to-verify.</li>
+        <li><b style="color: purple">🌟 Length Penalty:</b> GPT-4 judges tend to prefer longer outputs (although humans do too); to avoid this, we devise a simple method to add length penalty on Elo. You can even slide it on our leaderboard UI!</li>
+        <li><b style="color: purple">🌟 Task Categorization:</b> We tag each example with 12 task types, so we can analyze task-specific performance of LLMs, in addition to their overall ranking.</li>
+        <li><b style="color: purple">🌟 Fair Comparisons:</b> WildBench tests all examples on all LLMs. This is different from arena-style evaluation, where one example is only tested on a single pair of models and never seen again.</li>
+        <li><b style="color: purple">🌟 Easy & Fast:</b> WildBench (v1.0) contains 1024 examples now, and it is extremely easy to add your own LLMs to our leaderboard! We will do the work for you!</li>
+        <li><b style="color: purple">🌟 Dynamic:</b> WildBench will not be a static dataset. We will continue adding new examples and updating evaluation methods based on community feedback.</li>
+        <li><b style="color: purple">🌟 Human Evaluation (ongoing):</b> We are collecting human preferences via our Leaderboard UI (check the 🔍 🆚 tab). Please help us vote! (We’re planning to recruit domain experts too.)</li>
+        <li><b style="color: purple">🌟 Community driven:</b> We welcome everyone to contribute to human evaluation and create challenging examples. We also value your feedback and suggestions, and will continue enhancing our benchmark leaderboard accordingly.</li>
+    </ul>
+</div>
+</details>
+
+
+--- 
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">2. Where are the examples of WildBench from? </summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>
+            <b>WildBench</b> was designed with a focus on capturing the real-world complexity and diversity of tasks that large language models (LLMs) encounter. The design process involved several key steps:
+        </p>
+        <h2>2.1. Task Collection from WildChat</h2>
+        <p>
+            <b>WildChat</b>, a dataset akin to ShareGPT but larger and with user consent, was utilized to gather human-GPT conversations. We filtered the data for English, non-toxic responses and used various popular LLMs to generate responses, which were then scored using reward models such as StarlingRM and PairRM. The examples with the highest score variance were shortlisted, from which 1024 were chosen for curating the <b>WildBench v1.0</b>, ensuring a mix of diversity and quality.
+        </p>
+        <h2>2.2. Task Categories</h2>
+        <img src="https://huggingface.co/spaces/WildEval/WildBench-Leaderboard/resolve/main/task_dist.png" width="80%" />
+        <p>
+            The tasks are classified into 12 categories to cover a broad spectrum of real-user scenarios. This categorization helps in maintaining a balanced task distribution, mirroring the task variety in WildChat and differing significantly from traditional benchmarks.
+        </p>
+        <h2>2.3. Additional Annotations</h2>
+        <p>
+            WildBench includes further annotations like secondary task types, conversation turn counts, user intents, moderation tags, and evaluation checklists, providing deeper insights into the tasks and enhancing response assessments. These annotations are generated by GPT-4.
+        </p>
+    </div>
+</details>
+
+<!-- ---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">3. How is WildBench different from other benchmarks?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <h2>3.1. WildBench vs AlpacaEval</h2>
+        <p>
+            Unlike AlpacaEval's simpler, single-turn prompts, WildBench employs over 1024 multi-turn prompts from genuine user interactions, focusing on challenging and varied tasks. This represents a significant shift towards realism and complexity, aiming to reflect authentic LLM usage.
+        </p>
+        <h2>3.2. WildBench vs MT-Bench</h2>
+        <p>
+            MT-Bench offers two-turn instruction-following tasks, while WildBench provides a broader and more challenging array of multi-turn scenarios, ensuring a comprehensive evaluation across different dimensions.
+        </p>
+        <h2>3.3. WildBench vs Chatbot Arena</h2>
+        <p>
+            Though both benchmarks use real-user data, WildBench is distinct in its focus on challenging content, task diversity, and a structured, transparent evaluation methodology that offers more detailed insights into LLM performance.
+        </p>
+    </div>
+</details> 
+
+-->
+
+---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">3. How do you evaluate the performance of LLMs on WildBench?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <h2>3.1. Elo Rating</h2>
+        <p>We show two Elo rating for each model in our Main table. The "Overall" Elo rating is the standard method of using bootstrap method to compute the Elo scores for each model. The "Task-Avg" Elo is computed by first computing standard Elo on subsets of our data for each task type and then perform the average of them. </p>
+        <h2>3.2. Length Penalty</h2>
+        <p>We know that GPT-based evaluation tends to prefer longer responses, which is also the case for human evaluation. To mitigate this, we use a length penalty to normalize the Elo rating of the responses. Specifically, we compute two versions of Elo ratings for each model: one is based on win rates, and the other is based on "longer rates". The <code>WinElo</code> is the standard Elo rating, and the LongElo is the Elo rating considering longer outputs are always better than shorter outputs.
+Then, we present the final adjusted Elo by taking the difference between <code>WinElo</code> and <code>LongElo</code>, i.e.,
+        <code>AdjustedElo = WinElo - LengthPenalty * LongElo</code>.
+        </p>
+        <h2>3.3. Checklist-based Evaluation</h2>
+        <p>In our automatic evaluation, we use a checklist (a list of 5~10 questions) for prompting GPT-4 to judge which model output is better. This checklist is example-specific. You can find real examples in "🔍 Explore | 🆚 Evaluate". The checklists help us ensure that GPT-4 uses a rather fixed standard to compare different model pairs on the same examples. Also, they facilitate us to better explain how GPT-4 make the decisions.  </p>
+        <h2>3.4. Estimated Win Rates</h2>
+        <p>We estimate the win rates of each model winning GPT-4 by the differences of their Elo Rating versus GPT-4's. The formula can be found in  <a href="https://www.hexwiki.net/index.php/Elo_rating#Definition"> this page</a>. </p>
+        <h2>3.5. Human-Verified Auto Evaluation</h2>
+        <p>Although the current version of our WildBench is purely based on automatic evaluators, we aim to collect human preferences from our demo here ("🔍 Explore | 🆚 Evaluate") and then incorporate these human evaluation for mitigating the bias of GPT-4 based evaluation. We also plan to recruit domain experts for further improving the fairness of our evaluation. Please stay tuned! </p>
+    </div>
+</details>
+
+---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">4. How can I test my model on WildBench?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>Please refer to our Github <a href="https://github.com/allenai/WildBench">here</a> and create a PR or issue to tell us the information about your model. </p>
+    </div>
+</details>
+
+---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">5. How do I know why a particular model is weaker than others?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>Please click the tab for "🔍 Explore | 🆚 Evaluate" and select the models and task types that you're interested in. We'll sample an example with two model outputs for you to compare and you can see the model ids after you submit your feedback. </p>
+    </div>
+</details>
+
+---
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">6. Any future plans for WildBench?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>We have many todo items! The most important one is to collect human preferences for improving our evaluation. We are also going to recruit domain experts for further improving the fairness of our evaluation. As for auto-evaluation, we will add multiple auto evaluators for mitigating the bias of GPT-4 based evaluation. For example, we aim to use Claude 3 as evaluator to check if the ranking would be different. We're also developing our open-source evaluation models for supporting faster local evaluation. </p>
+    </div>
+</details>
+
+--- 
+
+<details>
+    <summary style="font-size: 1.8em; font-weight: bold;">7. How do I contact you?</summary>
+    <div style="font-size: 1.4em; margin-top: 30px;">
+        <p>Please use the community discussion board <a href="https://huggingface.co/spaces/WildEval/WildBench-Leaderboard/discussions">here</a> or the Github issues. Also, please feel free to email us at yuchenl@allenai.org and mention "WildBench" in the title. </p>
+    </div>
+</details>

app.py

@@ -0,0 +1,461 @@
+"""A gradio app that renders a static leaderboard. This is used for Hugging Face Space."""
+import ast
+import argparse
+import glob
+import pickle
+import plotly
+import gradio as gr
+import numpy as np
+import pandas as pd
+import gradio as gr
+import pandas as pd
+from pathlib import Path
+import json
+from constants import BANNER, CITATION_TEXT, WINRATE_HEATMAP, css, js_code, all_task_types, DEFAULT_LP, TASK_TYPE_STR
+from datetime import datetime, timezone
+from data_utils import load_eval_results, sample_an_eval_result, apply_length_penalty, post_processing, add_winrates, add_winrates_tasks
+# from gradio.themes.utils import colors, fonts, sizes
+from themes import Seafoam
+from huggingface_hub import HfApi
+# from datasets import Dataset, load_dataset, concatenate_datasets
+import os, uuid 
+from utils_display import model_info
+
+# get the last updated time from the elo_ranks.all.jsonl file
+LAST_UPDATED = None 
+with open("_intro.md", "r") as f:
+    INTRO_MD = f.read()
+
+with open("_about_us.md", "r") as f:
+    ABOUT_MD = f.read()
+
+with open("_header.md", "r") as f:
+    HEADER_MD = f.read()
+
+original_df, ablation_df = None, None
+eval_results = load_eval_results() 
+ 
+available_models = [] # to be filled in later
+
+  
+def display_chat_history(model_selections, task_selections):
+    eval_item = sample_an_eval_result(eval_results, model_selections, task_selections)
+    session_id = eval_item["session_id"]
+    chats = [x["content"] for x in eval_item['conversation_input']]
+    # form a list of tuples of two adjacent messages in chats 
+    chats_common = chats[:] + [None]
+    # chats_modelA = ["Model A Output"] + [eval_item["model_A_output"]]
+    # chats_modelB = ["Model B Output"] + [eval_item["model_B_output"]]
+    chats_modelA = [None] + [eval_item["model_A_output"]]
+    chats_modelB = [None] + [eval_item["model_B_output"]]
+    message_history_common = [(chats_common[i], chats_common[i+1]) for i in range(0, len(chats_common) -1, 2)]
+    message_history_model_A = [(chats_modelA[i], chats_modelA[i+1]) for i in range(0, len(chats_modelA) -1, 2)]
+    message_history_model_B = [(chats_modelB[i], chats_modelB[i+1]) for i in range(0, len(chats_modelB) -1, 2)]
+    checklist_string = ""
+    for item in eval_item["checklist"]:
+        checklist_string += f"1. {item}\n"    
+    list_reasons = eval_item["reason"].strip().split(". ")
+    # remove the last one if it is empty
+    if list_reasons[-1] == "":
+        list_reasons = list_reasons[:-1]
+    list_reasons = "\n".join([f"- {item}." for item in list_reasons]) 
+    gpt4_reason = f"### Choice: {eval_item['choice']}.  Reason: ⬇️\n"  +  list_reasons
+    assignment_string = f"Model A: {eval_item['model_A']} | Model B: {eval_item['model_B']}"
+    user_intent = f"- 🆔: `{session_id}` \n- 💬 **User Intent:** {eval_item['intent']} \n- ⚙️ **Task category**: {', '.join(eval_item['all_tags'])}"
+    return session_id, user_intent, message_history_common, message_history_model_A, message_history_model_B, gpt4_reason, checklist_string, assignment_string
+
+
+
+
+def slider_change_main(length_penalty):
+    global original_df, ablation_df
+    adjusted_df = apply_length_penalty(original_df, ablation_df, length_penalty) 
+    adjusted_df = adjusted_df[["Model", "Overall Elo", "Task-Avg Elo", "# battles", "Length"]]
+    adjusted_df = adjusted_df.sort_values(by="Overall Elo", ascending=False)
+    adjusted_df = add_winrates(adjusted_df) 
+    adjusted_df = adjusted_df.drop(columns=["Length"])
+    return adjusted_df
+
+def slider_change_full(length_penalty, show_winrate):
+    global original_df, ablation_df
+    adjusted_df = apply_length_penalty(original_df, ablation_df, length_penalty)
+    # sort the model by the "Task-Avg Elo" column
+    adjusted_df = adjusted_df.sort_values(by="Task-Avg Elo", ascending=False)
+    adjusted_df.drop(columns=["Overall Elo", "Task-Avg Elo", "# battles", "Length"], inplace=True)
+    if show_winrate == "none":
+        return adjusted_df
+    elif show_winrate == "gpt-3.5":
+        adjusted_df = add_winrates_tasks(adjusted_df, ref="gpt-3.5")
+    elif show_winrate == "gpt-4":
+        adjusted_df = add_winrates_tasks(adjusted_df, ref="gpt-4")
+    return adjusted_df
+
+seafoam = Seafoam()
+def build_demo(TYPES):
+    global original_df, ablation_df, skip_empty_original_df, skip_empty_ablation_df, available_models
+    with gr.Blocks(theme=gr.themes.Soft(), css=css) as demo:
+    # with gr.Blocks(theme=seafoam, css=css) as demo:
+        gr.HTML(BANNER, elem_id="banner")
+        # gr.Markdown("### Work in progress. Please do not share.", elem_classes="markdown-text") # TODO: remove this later.
+        gr.Markdown(HEADER_MD, elem_classes="markdown-text")
+        
+
+        with gr.Tabs(elem_classes="tab-buttons") as tabs:
+            with gr.TabItem("🏅 Leaderboard", elem_id="od-benchmark-tab-table", id=0):
+                gr.Markdown(f"**Version**: WildBench (v1.0; 2024.03.07) | **# Examples**: 1024 | **# Models**: {len(available_models)} | **# Comparisons**: 26k", elem_classes="markdown-text")
+                
+                with gr.TabItem("Main Table", elem_id="od-benchmark-tab-table-ablation", id=0, elem_classes="subtab"):
+                    # original_df, ablation_df = skip_empty_original_df, skip_empty_ablation_df
+                    default_main_df = apply_length_penalty(original_df, ablation_df, length_penalty=DEFAULT_LP) 
+                    default_main_df = default_main_df[["Model", "Overall Elo", "Task-Avg Elo", "# battles", "Length"]] 
+                    default_main_df = add_winrates(default_main_df)
+                    default_main_df = default_main_df.drop(columns=["Length"])
+                    # TODO: add the win rate for GPT-4 and GPT-3.5T
+                    with gr.Row():
+                        with gr.Column(scale=4):
+                            gr.Markdown("**Overall Elo**: [Standard Elo rating with boostrap.](https://en.wikipedia.org/wiki/Elo_rating_system). | **Task-Avg Elo**: Compute Elo on subsets of each task type and then take avg. | **Win Rates**: [Estimated by Elo differences](https://www.hexwiki.net/index.php/Elo_rating#Definition). | **Length penalty**: Models w/ longer outputs are penalized. (Plz check 📖 **Details**.)", elem_classes="markdown-text-small top-left-LP") 
+                        with gr.Column(scale=0.8):
+                            length_penlty_slider = gr.Slider(minimum=0.1, maximum=1, step=0.1, value=DEFAULT_LP, label="Length Penalty", elem_id="length-penalty-slider") 
+                    # checkbox_skip_empty = gr.Checkbox(label="Skip empty results", value=False, elem_id="skip-empty-checkbox", scale=2)
+                    leaderboard_table = gr.components.Dataframe(
+                        value=default_main_df,
+                        datatype=TYPES,
+                        # max_rows=None,
+                        height=1000,
+                        elem_id="leaderboard-table",
+                        interactive=False,
+                        visible=True,
+                        min_width=60,
+                        ) 
+                    length_penlty_slider.change(fn=slider_change_main, inputs=[length_penlty_slider], outputs=[leaderboard_table])
+                
+
+                with gr.TabItem("All Tasks (Win% vs GPT-3.5T)", elem_id="od-benchmark-tab-table-ablation", id=1):
+                    with gr.Row():
+                        with gr.Column(scale=4):
+                            gr.Markdown(TASK_TYPE_STR, elem_classes="markdown-text-small top-left-LP") 
+                        with gr.Column(scale=0.8):
+                            length_penlty_slider_full = gr.Slider(minimum=0.1, maximum=1, step=0.1, value=DEFAULT_LP, label="Length Penalty", elem_id="length-penalty-slider")
+                    default_full_df = apply_length_penalty(original_df, ablation_df, length_penalty=DEFAULT_LP)
+                    # do not show the "# battles" column here 
+                    default_full_df = default_full_df.drop(columns=["Overall Elo", "Task-Avg Elo", "# battles", "Length"])
+                    default_full_df = add_winrates_tasks(default_full_df, ref="gpt-3.5")
+
+                    leaderboard_table_full = gr.components.Dataframe(
+                        value=default_full_df,
+                        datatype=TYPES,
+                        # max_rows=None,
+                        height=1000,
+                        elem_id="leaderboard-table-full_table",
+                        interactive=False,
+                        visible=True,
+                        min_width=60,
+                        )
+                    show_winrate = gr.Checkbox(value="gpt-3.5", visible=False)
+                    length_penlty_slider_full.change(fn=slider_change_full, inputs=[length_penlty_slider_full, show_winrate], outputs=[leaderboard_table_full])
+
+
+                with gr.TabItem("All Tasks (Win% vs GPT-4)", elem_id="od-benchmark-tab-table-ablation", id=2):
+                    with gr.Row():
+                        with gr.Column(scale=4):
+                            gr.Markdown(TASK_TYPE_STR, elem_classes="markdown-text-small top-left-LP") 
+                        with gr.Column(scale=0.8):
+                            length_penlty_slider_full = gr.Slider(minimum=0.1, maximum=1, step=0.1, value=DEFAULT_LP, label="Length Penalty", elem_id="length-penalty-slider")
+                    default_full_df = apply_length_penalty(original_df, ablation_df, length_penalty=DEFAULT_LP)
+                    # do not show the "# battles" column here 
+                    default_full_df = default_full_df.drop(columns=["Overall Elo", "Task-Avg Elo", "# battles", "Length"])
+                    default_full_df = add_winrates_tasks(default_full_df, ref="gpt-4")
+                    leaderboard_table_full = gr.components.Dataframe(
+                        value=default_full_df,
+                        datatype=TYPES,
+                        # max_rows=None,
+                        height=1000,
+                        elem_id="leaderboard-table-full_table",
+                        interactive=False,
+                        visible=True,
+                        min_width=60,
+                        )
+                    show_winrate = gr.Checkbox(value="gpt-4", visible=False)
+                    length_penlty_slider_full.change(fn=slider_change_full, inputs=[length_penlty_slider_full, show_winrate], outputs=[leaderboard_table_full])
+                    
+
+                with gr.TabItem("All Tasks (Elo)", elem_id="od-benchmark-tab-table-ablation", id=3):
+                    with gr.Row():
+                        with gr.Column(scale=4):
+                            gr.Markdown(TASK_TYPE_STR, elem_classes="markdown-text-small top-left-LP") 
+                        with gr.Column(scale=0.8):
+                            length_penlty_slider_full = gr.Slider(minimum=0.1, maximum=1, step=0.1, value=DEFAULT_LP, label="Length Penalty", elem_id="length-penalty-slider")
+                    default_full_df = apply_length_penalty(original_df, ablation_df, length_penalty=DEFAULT_LP)
+                    # do not show the "# battles" column here 
+                    default_full_df = default_full_df.drop(columns=["Overall Elo", "Task-Avg Elo", "# battles", "Length"])
+                    leaderboard_table_full = gr.components.Dataframe(
+                        value=default_full_df,
+                        datatype=TYPES,
+                        # max_rows=None,
+                        height=1000,
+                        elem_id="leaderboard-table-full_table",
+                        interactive=False,
+                        visible=True,
+                        min_width=60,
+                        )
+                    show_winrate = gr.Checkbox(value="none", visible=False)
+                    length_penlty_slider_full.change(fn=slider_change_full, inputs=[length_penlty_slider_full, show_winrate], outputs=[leaderboard_table_full])
+
+                
+                # with gr.TabItem("Pairwise Win Rates", elem_id="od-benchmark-tab-table-ablation", id=4): 
+                #     # TODO: show all winrate 
+                #     # winrates_heatmap = pickle.load(open("data_dir/pairwise_win_fractions.pkl", "rb"))
+                #     # gr.Plot(value=winrates_heatmap, scale=2, min_width=800, container=False, elem_classes="plotly-plot", visible=True)
+                #     gr.HTML(WINRATE_HEATMAP, visible=True)
+
+            with gr.TabItem("📖 Details", elem_id="od-benchmark-tab-table", id=1):
+                gr.Markdown(INTRO_MD, elem_classes="markdown-text-details")
+
+            with gr.TabItem("🔍 Explore | 🆚 Evaluate", elem_id="od-benchmark-tab-table", id=2): 
+
+                with gr.Row(): 
+                    btn_show_history = gr.Button("🎲  Click here to sample an example + a pair of LLM outputs! ", elem_classes="sample_button")
+
+                with gr.Row():
+                    with gr.Column(scale=1.5):
+                        with gr.Accordion("Choose models to sample from", open=False, elem_classes="accordion-label"):
+                            model_options = available_models  
+                            selected_models = gr.CheckboxGroup(model_options, info="", value=model_options, show_label=False, elem_id="select-models") 
+                            clear_button = gr.Button("Clear", elem_classes="btn_boderline_gray", scale=1)
+                            # clear the selected_models
+                            clear_button.click(lambda: {selected_models: {"value": [], "__type__": "update"}}, inputs=[], outputs=[selected_models])
+                    with gr.Column(scale=1):
+                        with gr.Accordion("Choose task types to sample from", open=False, elem_classes="accordion-label"):
+                            select_tasks = gr.CheckboxGroup(all_task_types, info="", value=all_task_types, show_label=False, elem_id="select-tasks") 
+                            clear_task_button = gr.Button("Clear", elem_classes="btn_boderline_gray", scale=1)
+                            # clear the select_tasks
+                            clear_task_button.click(lambda: {select_tasks: {"value": [], "__type__": "update"}}, inputs=[], outputs=[select_tasks])
+                
+                with gr.Row():
+                    with gr.Column():
+                        gr.Markdown("## 📢 Chat History", elem_classes="markdown-text")
+                        Chatbot_Common = gr.Chatbot(avatar_images=["human_icon.jpeg", "ai_icon.png"], height="auto", container=False, label="Common Chat History", likeable=False, show_share_button=False, show_label=True, elem_classes="chat-common", layout="bubble")
+                        Chatbot_Common.change(lambda x: x, inputs=[], outputs=[], scroll_to_output=False, js=js_code)
+                        with gr.Accordion("✍️ Task Annotation", elem_classes="accordion-label", open=False):
+                            user_intent = gr.Markdown("", elem_classes="markdown-text-small")
+                # two columns for the two models
+                with gr.Row():
+                    # https://www.gradio.app/docs/chatbot
+                    with gr.Column():
+                        gr.Markdown("## ⬅️ Model A Output", elem_classes="markdown-text")
+                        Chatbot_A = gr.Chatbot(height="auto", container=False, label="Model A Output", likeable=False, show_share_button=False, show_label=True, elem_classes="chat-specific", layout="bubble")
+                        Chatbot_A.change(lambda x: x, inputs=[], outputs=[], scroll_to_output=False, js=js_code)
+                    with gr.Column():
+                        # add a Markdown to show this is for Model B 
+                        gr.Markdown("## ➡️ Model B Output", elem_classes="markdown-text")
+                        Chatbot_B = gr.Chatbot(height="auto", container=False, label="Model B Output", likeable=False, show_share_button=False, show_label=True, elem_classes="chat-specific", layout="bubble")
+                        Chatbot_B.change(lambda x: x, inputs=[], outputs=[], scroll_to_output=False, js=js_code)
+                with gr.Row():
+                    # Here we can show the GPT-4 judgement for the model outputs
+                    # show a textarea
+                    with gr.Column():
+                        with gr.Accordion("⏱️ Checklist", open=False, elem_classes="accordion-label"):
+                            checklist = gr.Markdown("### Checklist: \n Will be shown later.", elem_classes="markdown-text-tiny")
+                        with gr.Accordion("⚖️ GPT-4 Judgement", open=False, elem_classes="accordion-label") as gpt4_accordion:
+                            # gpt4_reason = gr.TextArea(label="GPT-4 Judgement", placeholder="Will be shown later.", type="text", elem_classes="", max_lines=10, show_copy_button=True)
+                            gpt4_reason = gr.Markdown("Will be shown later.", elem_classes="markdown-text-tiny")
+            
+                with gr.Row():
+                    # show buttons for user to choose which model output is better or Tie 
+                    btn_model_A = gr.Button("⬅️ Model A is better! ", elem_classes="btn_boderline_gray", scale=2, interactive=False)
+                    btn_tie = gr.Button("🟰 Tie", elem_classes="btn_boderline_gray", scale=2, interactive=False)
+                    btn_model_B = gr.Button("➡️ Model B is better!", elem_classes="btn_boderline_gray", scale=2, interactive=False)
+                    # TODO: if click btn_model_A, btn_model_B, or btn_tie, then show a text box and a submit button for the user to input the reason
+                with gr.Row():
+                    with gr.Column(scale=2):
+                        reason_textbox = gr.Textbox(label="Reason", placeholder="Please input your reason here.", type="text", elem_classes="", max_lines=10, lines=8, show_copy_button=False, visible=True, scale=4, interactive=True)
+                    with gr.Column():
+                        with gr.Row():
+                            user_choice = gr.Markdown("Your choice: N/A", elem_classes="markdown-text", visible=True)
+                            btn_pass = gr.Button("🔁 Next", elem_classes="btn_boderline_next", scale=1)
+                        user_name = gr.Textbox(label="Your HF Username", placeholder="Your HuggingFace username", type="text", elem_classes="", max_lines=1, show_copy_button=False, visible=True, interactive=True, show_label=False)
+                        # login_btn = gr.LoginButton(visible=False, interactive=True, elem_classes="btn_boderline")
+                        submit_button = gr.Button("Submit your feedback! 🚀", elem_classes="btn_boderline", visible=True, interactive=False)
+                        assignment = gr.Markdown("Model A: | Model B: ", elem_classes="markdown-text-tiny-red", visible=False)
+                    
+                
+                session_id = gr.Textbox(label="Session ID", placeholder="N/A.", type="text", elem_classes="", max_lines=10, show_copy_button=False, visible=False)
+                def show_reason_and_submit(session_id, user_name_text, btn, request: gr.Request): 
+
+                    if request.username is not None:
+                        user_name_text = request.username 
+                    result_dict = {
+                        reason_textbox: {"visible": True, "__type__": "update"}, 
+                        submit_button: {"visible": True, "__type__": "update", "interactive": True}, 
+                        user_name: {"visible": True, "__type__": "update", "value": user_name_text},
+                    }
+                    if "Model A" in btn:
+                        choice = "Model A" 
+                        result_dict.update({
+                            user_choice: {"value": f"Your choice: **{choice}**", "__type__": "update", "visible": True},  
+                            btn_model_A: {"elem_classes": "btn_boderline_selected", "__type__": "update"},
+                            btn_model_B: {"elem_classes": "btn_boderline", "__type__": "update"},
+                            btn_tie: {"elem_classes": "btn_boderline", "__type__": "update"},
+                        })
+                    elif "Model B" in btn:
+                        choice = "Model B"
+                        result_dict.update({
+                            user_choice: {"value": f"Your choice: **{choice}**", "__type__": "update", "visible": True},  
+                            btn_model_B: {"elem_classes": "btn_boderline_selected", "__type__": "update"},
+                            btn_model_A: {"elem_classes": "btn_boderline", "__type__": "update"},
+                            btn_tie: {"elem_classes": "btn_boderline", "__type__": "update"},
+                        })
+                    elif "Tie" in btn:
+                        choice = "Tie"
+                        result_dict.update({
+                            user_choice: {"value": f"Your choice: **{choice}**", "__type__": "update", "visible": True},  
+                            btn_tie: {"elem_classes": "btn_boderline_selected", "__type__": "update"},
+                            btn_model_A: {"elem_classes": "btn_boderline", "__type__": "update"},
+                            btn_model_B: {"elem_classes": "btn_boderline", "__type__": "update"},
+                        })
+                    else:
+                        choice = "N/A"
+                        result_dict.update({
+                            user_choice: {"value": f"Your choice: **{choice}**", "__type__": "update", "visible": True},  
+                        }) 
+                    return result_dict
+                
+                btn_model_A.click(show_reason_and_submit, inputs=[session_id, user_name, btn_model_A], outputs=[user_choice, reason_textbox, submit_button, user_name, btn_model_A, btn_tie, btn_model_B])
+                btn_tie.click(show_reason_and_submit, inputs=[session_id, user_name, btn_tie], outputs=[user_choice, reason_textbox, submit_button,  user_name, btn_model_A, btn_tie, btn_model_B])
+                btn_model_B.click(show_reason_and_submit, inputs=[session_id, user_name, btn_model_B], outputs=[user_choice, reason_textbox, submit_button,  user_name, btn_model_A, btn_tie, btn_model_B])
+
+                def submit_feedback(session_id, user_reason, user_choice, user_name_text, request: gr.Request):
+                    if "N/A" in session_id or "N/A" in user_choice:
+                        # send a message to the user to sample an example and select a choice first 
+                        return {
+                            submit_button: {"interactive": True, "__type__": "update", "value": "Submit your feedback! 🚀 Please sample an example and select a choice!"},
+                        } 
+                    # create a jsonl file and upload it to hf 
+                    choice_str = ""
+                    if "Model A" in user_choice:
+                        choice_str = "Model A"
+                    elif "Model B" in user_choice:
+                        choice_str = "Model B"
+                    elif "Tie" in user_choice:
+                        choice_str = "Tie"
+                    else:
+                        choice_str = "N/A"
+                    if user_name_text == "" and request.username is None:
+                        user_name_text = "Anonymous"
+                    if request.username is not None:
+                        user_name_text = request.username
+                    feedback_item = {
+                        "session_id": session_id,
+                        "user_name": user_name_text, 
+                        "user_reason": user_reason,
+                        "user_choice": choice_str,
+                        "ip": request.client.host
+                    }
+                    jsonl_str = json.dumps(feedback_item)
+                    api = HfApi()
+                    token = os.getenv("HF_TOKEN")
+                    if token is None:
+                        raise ValueError("Hugging Face token not found. Ensure the HF_TOKEN environment variable is set.")
+
+                    # Generate a random filename using UUID
+                    filename = f"{uuid.uuid4()}.json"
+
+                    # Define the repository
+                    repo_id = "WildEval/WildBench-HumanFeedback"
+
+                    # Upload the json_str as a file directly to the specified path in your dataset repository
+                    api.upload_file(
+                        token=token,
+                        repo_id=repo_id,
+                        repo_type="dataset",
+                        path_or_fileobj=jsonl_str.encode("utf-8"),  # Convert string to bytes
+                        path_in_repo=filename,
+                        commit_message=f"Add user feedback for session_id: {session_id}",
+                    )
+                    return {
+                        submit_button: {"interactive": False, "__type__": "update", "value": "Submitted! ✅ \n Please click 🔁 Next."},
+                        reason_textbox: {"interactive": False, "__type__": "update"},
+                        btn_model_A: {"interactive": False, "__type__": "update"},
+                        btn_tie: {"interactive": False, "__type__": "update"},
+                        btn_model_B: {"interactive": False, "__type__": "update"},
+                        user_name: {"interactive": False, "__type__": "update"},
+                        assignment: {"visible": True, "__type__": "update"}
+                        }
+                def reset_submission(session_id):
+                    return {
+                        submit_button: {"interactive": False, "__type__": "update", "value": "Submit your feedback! 🚀"},
+                        reason_textbox: {"interactive": True, "__type__": "update", "value": ""},
+                        btn_model_A: {"interactive": True, "__type__": "update", "elem_classes": "btn_boderline_gray"},
+                        btn_tie: {"interactive": True, "__type__": "update", "elem_classes": "btn_boderline_gray"},
+                        btn_model_B: {"interactive": True, "__type__": "update", "elem_classes": "btn_boderline_gray"},
+                        user_name: {"interactive": True, "__type__": "update"},
+                        user_choice: {"value": "Your choice: N/A", "__type__": "update"},
+                        assignment: {"__type__": "update", "visible": False},
+                        gpt4_accordion: {"__type__": "update", "open": False},
+                        }
+                # reset the reason_textbox, submit_button, and btn_model_A
+                session_id.change(reset_submission, inputs=[session_id], outputs=[submit_button, reason_textbox, btn_model_A, btn_tie, btn_model_B, user_name, user_choice, assignment, gpt4_accordion])
+                submit_button.click(submit_feedback, inputs=[session_id, reason_textbox, user_choice, user_name], outputs=[submit_button, reason_textbox, btn_model_A, btn_tie, btn_model_B, user_name, assignment])
+                
+                # Display chat history when button is clicked
+                # TODO: add the model list and tag list
+                btn_show_history.click(fn=display_chat_history, inputs=[selected_models, select_tasks], outputs=[session_id, user_intent, Chatbot_Common, Chatbot_A, Chatbot_B, gpt4_reason, checklist, assignment])
+                btn_pass.click(fn=display_chat_history, inputs=[selected_models, select_tasks], outputs=[session_id, user_intent, Chatbot_Common, Chatbot_A, Chatbot_B, gpt4_reason, checklist, assignment]) # the pass button will be the same function of resampling 
+
+            with gr.TabItem("📮 About Us", elem_id="od-benchmark-tab-table", id=3):
+                gr.Markdown(ABOUT_MD, elem_classes="markdown-text")
+        gr.Markdown(f"Last updated on **{LAST_UPDATED}**", elem_classes="markdown-text-small")
+        
+        with gr.Row():
+            with gr.Accordion("📙 Citation", open=False, elem_classes="accordion-label"):
+                gr.Textbox(
+                    value=CITATION_TEXT, 
+                    lines=7,
+                    label="Copy the BibTeX snippet to cite this source",
+                    elem_id="citation-button",
+                    show_copy_button=True)
+                # ).style(show_copy_button=True)
+
+    return demo
+
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--share", action="store_true")
+    parser.add_argument("--result_file", help="Path to results table", default="data_dir/elo_ranks.all.jsonl")
+    parser.add_argument("--length_balation_file", help="Path to results table", default="data_dir/elo_ranks.length_ablation.all.jsonl")
+    parser.add_argument("--skip_empty_result_file", help="Path to results table", default="data_dir/elo_ranks.skip_empty.all.jsonl")
+    parser.add_argument("--skip_empty_length_balation_file", help="Path to results table", default="data_dir/elo_ranks.skip_empty.length_ablation.all.jsonl")
+    args = parser.parse_args()
+
+    LAST_UPDATED = datetime.fromtimestamp(Path(args.result_file).stat().st_mtime, tz=timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
+ 
+    original_df = pd.read_json(args.result_file , lines=True)
+    ablation_df = pd.read_json(args.length_balation_file, lines=True)
+    skip_empty_original_df = pd.read_json(args.skip_empty_result_file , lines=True)
+    skip_empty_ablation_df = pd.read_json(args.skip_empty_length_balation_file, lines=True)
+   
+    
+    # available_models = sorted(list(set(list(original_df["model name "])))) 
+    available_models = list(model_info.keys())
+    # remove the rows where the model name is not in the available_models
+    original_df = original_df[original_df["model name "].isin(available_models)]
+    ablation_df = ablation_df[ablation_df["model name "].isin(available_models)]
+    skip_empty_ablation_df = skip_empty_ablation_df[skip_empty_ablation_df["model name "].isin(available_models)]
+    skip_empty_original_df = skip_empty_original_df[skip_empty_original_df["model name "].isin(available_models)]
+
+    model_len_info = json.load(open("model_len_info.json", "r"))
+
+    original_df = post_processing(original_df, model_len_info)
+    ablation_df = post_processing(ablation_df, model_len_info)
+    skip_empty_original_df = post_processing(skip_empty_original_df, model_len_info)
+    skip_empty_ablation_df = post_processing(skip_empty_ablation_df, model_len_info)
+    
+
+    
+
+    TYPES = ["markdown", "number"]
+
+    demo = build_demo(TYPES)
+    demo.launch(share=args.share, height=1000)

compute_model_output_len.py

@@ -0,0 +1,24 @@
+from utils_display import model_info
+from data_utils import load_infer_results
+from tqdm import tqdm 
+import json 
+
+length_info = {}
+
+for model_name in tqdm(list(model_info.keys())): 
+    result = load_infer_results(model_name)
+    lens = []
+    for item in result:
+        o = item["output"]
+        if type(o) == list:
+            L = len(o[0].strip())
+        else:
+            L = len(o.strip())
+        if L > 0:
+            lens.append(L)
+    avg_len = sum(lens) / len(lens)
+    print(f"{model_name}: {avg_len}")
+    length_info[model_name] = avg_len
+
+with open("model_len_info.json", "w") as f:
+    json.dump(length_info, f, indent=2)

constants.py

@@ -0,0 +1,194 @@
+from pathlib import Path
+
+DEFAULT_LP = 0.3
+
+banner_url = "https://allenai.github.io/WildBench/gray_banner.png" # the same repo here.
+BANNER = f'<div style="display: flex; justify-content: flex-start;"><img src="{banner_url}" alt="Banner" style="width: 40vw; min-width: 300px; max-width: 800px;"> </div>'
+
+TITLE = "<html> <head> <style> h1 {text-align: center;} </style> </head> <body> <h1> 🦁 AI2 WildBench Leaderboard </b> </body> </html>"
+ 
+WINRATE_HEATMAP = "<div><img src='https://github.com/WildEval/WildBench-Leaderboard/blob/main/gradio/pairwise_win_fractions.png?raw=true' style='width:100%;'></div>"
+
+CITATION_TEXT = """@misc{wildbench2024,
+	title        = {WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild},
+	author       = {Bill Yuchen Lin and Khyathi Chandu and Faeze Brahman and Yuntian Deng and Abhilasha Ravichander and Valentina Pyatkin and Ronan Le Bras and Yejin Choi},
+	year         = 2024,
+	url			 = {https://huggingface.co/spaces/allenai/WildBench},
+}
+"""
+
+
+column_names = {
+    "model name ": "Model",
+    "elo overall": "Overall Elo",
+    'Information seeking': 'InfoSek',
+    'Creative Writing': 'CrtWrt',
+    'Coding & Debugging': 'Code',
+    'Reasoning': 'Reason',
+    'Editing': 'Edit',
+    'Math': 'Math',
+    'Planning': 'Plan',
+    'Brainstorming': 'Brnstrm',
+    'Role playing': 'RolPly',
+    'Advice seeking': 'AdvSek',
+    'Data Analysis': 'DataAna',
+    'Others': 'Misc',
+    "average": "Task-Avg Elo",
+}
+
+all_task_types = [
+    'Information seeking',
+    'Creative Writing',
+    'Coding & Debugging',
+    'Reasoning',
+    'Editing',
+    'Math',
+    'Planning',
+    'Brainstorming',
+    'Role playing',
+    'Advice seeking',
+    'Data Analysis',
+    'Others'
+]
+
+
+
+js_code = """
+function scroll_top() {
+    console.log("Hello from Gradio!");  
+    const bubbles = document.querySelectorAll('.bubble-wrap');
+    bubbles.forEach((bubble, index) => {
+        setTimeout(() => {
+            bubble.scrollTop = 0;
+        }, index * 100); // Delay of 100ms between each iteration
+    });
+} 
+"""
+
+
+TASK_TYPE_STR = "**Tasks**: Info seeking (**InfoSek**), Creative Writing (**CrtWrt**), Coding&Debugging (**Code**), Reasoning (**Reason**), Editing (**Edit**), **Math**, Planning (**Plan**), Brainstorming (**Brnstrm**), Role playing (**RolPly**), Advice seeking (**AdvSek**), Data Analysis (**DataAna**)"
+
+css = """
+code {
+    font-size: large;
+}
+footer {visibility: hidden}
+.top-left-LP{
+    margin-top: 6px;
+    margin-left: 5px;
+}
+.markdown-text{font-size: 14pt}
+.markdown-text-small{font-size: 13pt}
+.markdown-text-tiny{font-size: 12pt}
+.markdown-text-tiny-red{
+    font-size: 12pt;
+    color: red;
+    background-color: yellow;
+    font-color: red;
+    font-weight: bold;
+}
+th {
+  text-align: center;
+  font-size: 17px; /* Adjust the font size as needed */
+}
+td {
+  font-size: 15px; /* Adjust the font size as needed */
+  text-align: center;
+}
+
+.sample_button{
+    border: 1px solid #000000;
+    border-radius: 5px;
+    padding: 5px;
+    font-size: 15pt;
+    font-weight: bold;
+    margin: 5px;
+}
+
+.chat-common{
+    height: auto;
+    max-height: 400px;
+    min-height: 100px; 
+}
+.chat-specific{
+    height: auto;
+    max-height: 600px;
+    min-height: 200px; 
+}
+#od-benchmark-tab-table-button{
+    font-size: 15pt;
+    font-weight: bold;
+} 
+
+.btn_boderline{
+    border: 1px solid #000000;
+    border-radius: 5px;
+    padding: 5px;
+    margin: 5px;
+    font-size: 15pt;
+    font-weight: bold; 
+}
+
+.btn_boderline_next{
+    border: 0.1px solid #000000;
+    border-radius: 5px;
+    padding: 5px;
+    margin: 5px;
+    font-size: 15pt;
+    font-weight: bold; 
+}
+
+.btn_boderline_gray{
+    border: 0.5px solid gray;
+    border-radius: 5px;
+    padding: 5px;
+    margin: 5px;
+    font-size: 15pt;
+    font-weight: italic; 
+}
+.btn_boderline_selected{
+    border: 2px solid purple;
+    background-color: #f2f2f2;
+    border-radius: 5px;
+    padding: 5px;
+    margin: 5px;
+    font-size: 15pt;
+    font-weight: bold;  
+}
+.accordion-label button span{
+    font-size: 14pt;
+    font-weight: bold;
+} 
+
+#select-models span{
+    font-size: 10pt;
+}
+
+#select-tasks span{
+    font-size: 10pt;
+}
+
+
+.markdown-text-details{
+    margin: 10px;
+    padding: 10px;
+}
+
+
+button.selected[role="tab"][aria-selected="true"] {
+    font-size: 18px; /* or any other size you prefer */
+    font-weight: bold;
+}
+
+#od-benchmark-tab-table-ablation-button {
+    font-size: larger; /* Adjust the font size as needed */
+}
+
+
+.plotly-plot{
+    height: auto;
+    max-height: 600px;
+    min-height: 600px; 
+}
+"""
+ 

data_dir/elo_ranks.all.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1309, "Information seeking": 1282, "Creative Writing": 1267, "Coding & Debugging": 1376, "Reasoning": 1366, "Editing": 1211, "Math": 1276, "Planning": 1299, "Brainstorming": 1322, "Role playing": 1240, "Advice seeking": 1295, "Data Analysis": 1286, "Others": 1068, "average": 1274.0, "# battles": 5429}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1155, "Information seeking": 1131, "Creative Writing": 1066, "Coding & Debugging": 1282, "Reasoning": 1204, "Editing": 1146, "Math": 1160, "Planning": 1159, "Brainstorming": 1111, "Role playing": 964, "Advice seeking": 1114, "Data Analysis": 1151, "Others": 1031, "average": 1126.5833333333333, "# battles": 2366}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1140, "Information seeking": 1115, "Creative Writing": 1081, "Coding & Debugging": 1247, "Reasoning": 1170, "Editing": 1152, "Math": 1142, "Planning": 1163, "Brainstorming": 1092, "Role playing": 1005, "Advice seeking": 1081, "Data Analysis": 1170, "Others": 1014, "average": 1119.3333333333333, "# battles": 2608}
+{"model name ": "mistral-large-2402", "elo overall": 1120, "Information seeking": 1090, "Creative Writing": 1107, "Coding & Debugging": 1155, "Reasoning": 1136, "Editing": 1097, "Math": 1092, "Planning": 1081, "Brainstorming": 1078, "Role playing": 1067, "Advice seeking": 1060, "Data Analysis": 1097, "Others": 1021, "average": 1090.0833333333333, "# battles": 1881}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1102, "Information seeking": 1084, "Creative Writing": 1138, "Coding & Debugging": 1140, "Reasoning": 1047, "Editing": 1105, "Math": 1021, "Planning": 1090, "Brainstorming": 1076, "Role playing": 1118, "Advice seeking": 1067, "Data Analysis": 1058, "Others": 1002, "average": 1078.8333333333333, "# battles": 2371}
+{"model name ": "gemini-1.0-pro", "elo overall": 1073, "Information seeking": 1089, "Creative Writing": 1040, "Coding & Debugging": 1083, "Reasoning": 1067, "Editing": 1021, "Math": 1063, "Planning": 1067, "Brainstorming": 1023, "Role playing": 1032, "Advice seeking": 1072, "Data Analysis": 1090, "Others": 1007, "average": 1054.5, "# battles": 1815}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1069, "Information seeking": 1079, "Creative Writing": 1050, "Coding & Debugging": 1125, "Reasoning": 1063, "Editing": 1018, "Math": 1061, "Planning": 1004, "Brainstorming": 1026, "Role playing": 1080, "Advice seeking": 1005, "Data Analysis": 1060, "Others": 993, "average": 1047.0, "# battles": 3417}
+{"model name ": "Yi-34B-Chat", "elo overall": 1058, "Information seeking": 1072, "Creative Writing": 1105, "Coding & Debugging": 939, "Reasoning": 1073, "Editing": 1042, "Math": 1011, "Planning": 1094, "Brainstorming": 1140, "Role playing": 1071, "Advice seeking": 1087, "Data Analysis": 943, "Others": 997, "average": 1047.8333333333333, "# battles": 2428}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1031, "Information seeking": 1092, "Creative Writing": 1065, "Coding & Debugging": 883, "Reasoning": 1022, "Editing": 987, "Math": 975, "Planning": 1003, "Brainstorming": 1049, "Role playing": 1070, "Advice seeking": 1036, "Data Analysis": 959, "Others": 1020, "average": 1013.4166666666666, "# battles": 1942}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1007, "Information seeking": 1056, "Creative Writing": 1058, "Coding & Debugging": 850, "Reasoning": 975, "Editing": 986, "Math": 947, "Planning": 984, "Brainstorming": 1041, "Role playing": 1053, "Advice seeking": 1008, "Data Analysis": 912, "Others": 1014, "average": 990.3333333333334, "# battles": 1888}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 997, "Information seeking": 1003, "Creative Writing": 1024, "Coding & Debugging": 939, "Reasoning": 993, "Editing": 980, "Math": 975, "Planning": 965, "Brainstorming": 941, "Role playing": 1065, "Advice seeking": 976, "Data Analysis": 932, "Others": 991, "average": 982.0, "# battles": 3435}
+{"model name ": "zephyr-7b-beta", "elo overall": 986, "Information seeking": 1002, "Creative Writing": 991, "Coding & Debugging": 991, "Reasoning": 962, "Editing": 981, "Math": 941, "Planning": 967, "Brainstorming": 936, "Role playing": 1022, "Advice seeking": 938, "Data Analysis": 1003, "Others": 962, "average": 974.6666666666666, "# battles": 3367}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 984, "Information seeking": 1033, "Creative Writing": 1026, "Coding & Debugging": 843, "Reasoning": 962, "Editing": 963, "Math": 908, "Planning": 987, "Brainstorming": 1003, "Role playing": 1048, "Advice seeking": 1020, "Data Analysis": 934, "Others": 1011, "average": 978.1666666666666, "# battles": 1898}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 983, "Information seeking": 935, "Creative Writing": 934, "Coding & Debugging": 1155, "Reasoning": 1018, "Editing": 994, "Math": 1148, "Planning": 986, "Brainstorming": 929, "Role playing": 924, "Advice seeking": 961, "Data Analysis": 1071, "Others": 977, "average": 1002.6666666666666, "# battles": 14003}
+{"model name ": "gemma-7b-it", "elo overall": 951, "Information seeking": 931, "Creative Writing": 938, "Coding & Debugging": 993, "Reasoning": 957, "Editing": 928, "Math": 985, "Planning": 981, "Brainstorming": 984, "Role playing": 954, "Advice seeking": 973, "Data Analysis": 979, "Others": 992, "average": 966.25, "# battles": 2518}
+{"model name ": "command", "elo overall": 948, "Information seeking": 939, "Creative Writing": 942, "Coding & Debugging": 969, "Reasoning": 925, "Editing": 943, "Math": 940, "Planning": 979, "Brainstorming": 988, "Role playing": 948, "Advice seeking": 964, "Data Analysis": 966, "Others": 979, "average": 956.8333333333334, "# battles": 1733}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 890, "Information seeking": 902, "Creative Writing": 902, "Coding & Debugging": 824, "Reasoning": 903, "Editing": 949, "Math": 896, "Planning": 907, "Brainstorming": 946, "Role playing": 905, "Advice seeking": 946, "Data Analysis": 927, "Others": 999, "average": 917.1666666666666, "# battles": 2391}
+{"model name ": "gemma-2b-it", "elo overall": 874, "Information seeking": 843, "Creative Writing": 898, "Coding & Debugging": 894, "Reasoning": 829, "Editing": 888, "Math": 896, "Planning": 906, "Brainstorming": 919, "Role playing": 906, "Advice seeking": 891, "Data Analysis": 914, "Others": 977, "average": 896.75, "# battles": 2484}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 860, "Information seeking": 878, "Creative Writing": 866, "Coding & Debugging": 802, "Reasoning": 883, "Editing": 955, "Math": 915, "Planning": 897, "Brainstorming": 891, "Role playing": 865, "Advice seeking": 938, "Data Analysis": 885, "Others": 1007, "average": 898.5, "# battles": 2350}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 847, "Information seeking": 849, "Creative Writing": 862, "Coding & Debugging": 817, "Reasoning": 847, "Editing": 870, "Math": 897, "Planning": 830, "Brainstorming": 873, "Role playing": 911, "Advice seeking": 852, "Data Analysis": 879, "Others": 973, "average": 871.6666666666666, "# battles": 2478}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 821, "Information seeking": 790, "Creative Writing": 800, "Coding & Debugging": 947, "Reasoning": 798, "Editing": 871, "Math": 892, "Planning": 806, "Brainstorming": 783, "Role playing": 892, "Advice seeking": 823, "Data Analysis": 922, "Others": 968, "average": 857.6666666666666, "# battles": 2479}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 806, "Information seeking": 805, "Creative Writing": 834, "Coding & Debugging": 754, "Reasoning": 806, "Editing": 917, "Math": 856, "Planning": 836, "Brainstorming": 856, "Role playing": 853, "Advice seeking": 883, "Data Analysis": 861, "Others": 997, "average": 854.8333333333334, "# battles": 2297}

data_dir/elo_ranks.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1309, "# battles": 5429}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1155, "# battles": 2366}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1140, "# battles": 2608}
+{"model name ": "mistral-large-2402", "elo overall": 1120, "# battles": 1881}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1102, "# battles": 2371}
+{"model name ": "gemini-1.0-pro", "elo overall": 1073, "# battles": 1815}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1069, "# battles": 3417}
+{"model name ": "Yi-34B-Chat", "elo overall": 1058, "# battles": 2428}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1031, "# battles": 1942}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1007, "# battles": 1888}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 997, "# battles": 3435}
+{"model name ": "zephyr-7b-beta", "elo overall": 986, "# battles": 3367}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 984, "# battles": 1898}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 983, "# battles": 14003}
+{"model name ": "gemma-7b-it", "elo overall": 951, "# battles": 2518}
+{"model name ": "command", "elo overall": 948, "# battles": 1733}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 890, "# battles": 2391}
+{"model name ": "gemma-2b-it", "elo overall": 874, "# battles": 2484}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 860, "# battles": 2350}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 847, "# battles": 2478}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 821, "# battles": 2479}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 806, "# battles": 2297}

data_dir/elo_ranks.length_ablation.all.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1228, "Information seeking": 1274, "Creative Writing": 1202, "Coding & Debugging": 1224, "Reasoning": 1285, "Editing": 1132, "Math": 1202, "Planning": 1198, "Brainstorming": 1181, "Role playing": 1225, "Advice seeking": 1274, "Data Analysis": 1198, "Others": 1033, "average": 1202.3333333333333, "# battles": 5430}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1112, "Information seeking": 1102, "Creative Writing": 1136, "Coding & Debugging": 1119, "Reasoning": 1071, "Editing": 1080, "Math": 1094, "Planning": 1057, "Brainstorming": 1038, "Role playing": 1126, "Advice seeking": 1050, "Data Analysis": 1083, "Others": 1010, "average": 1080.5, "# battles": 2371}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1112, "Information seeking": 1206, "Creative Writing": 1171, "Coding & Debugging": 938, "Reasoning": 1091, "Editing": 1057, "Math": 993, "Planning": 1105, "Brainstorming": 1136, "Role playing": 1128, "Advice seeking": 1110, "Data Analysis": 965, "Others": 1017, "average": 1076.4166666666667, "# battles": 1942}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1107, "Information seeking": 1184, "Creative Writing": 1146, "Coding & Debugging": 937, "Reasoning": 1095, "Editing": 1017, "Math": 993, "Planning": 1113, "Brainstorming": 1140, "Role playing": 1141, "Advice seeking": 1136, "Data Analysis": 996, "Others": 1008, "average": 1075.5, "# battles": 1898}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1096, "Information seeking": 1182, "Creative Writing": 1135, "Coding & Debugging": 924, "Reasoning": 1075, "Editing": 1038, "Math": 1004, "Planning": 1093, "Brainstorming": 1141, "Role playing": 1103, "Advice seeking": 1116, "Data Analysis": 969, "Others": 1032, "average": 1067.6666666666667, "# battles": 1888}
+{"model name ": "zephyr-7b-beta", "elo overall": 1079, "Information seeking": 1027, "Creative Writing": 1094, "Coding & Debugging": 1134, "Reasoning": 1047, "Editing": 1091, "Math": 1051, "Planning": 1064, "Brainstorming": 1075, "Role playing": 1038, "Advice seeking": 985, "Data Analysis": 1092, "Others": 1017, "average": 1059.5833333333333, "# battles": 3367}
+{"model name ": "Yi-34B-Chat", "elo overall": 1047, "Information seeking": 1082, "Creative Writing": 1068, "Coding & Debugging": 952, "Reasoning": 1086, "Editing": 987, "Math": 1064, "Planning": 1090, "Brainstorming": 1119, "Role playing": 1055, "Advice seeking": 1117, "Data Analysis": 970, "Others": 1004, "average": 1049.5, "# battles": 2428}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 1037, "Information seeking": 1012, "Creative Writing": 1068, "Coding & Debugging": 1040, "Reasoning": 1023, "Editing": 1066, "Math": 1044, "Planning": 1027, "Brainstorming": 1007, "Role playing": 1060, "Advice seeking": 1016, "Data Analysis": 1030, "Others": 1026, "average": 1034.9166666666667, "# battles": 3435}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1014, "Information seeking": 979, "Creative Writing": 944, "Coding & Debugging": 1162, "Reasoning": 1056, "Editing": 1056, "Math": 1076, "Planning": 1037, "Brainstorming": 953, "Role playing": 899, "Advice seeking": 985, "Data Analysis": 1080, "Others": 988, "average": 1017.9166666666666, "# battles": 2608}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1011, "Information seeking": 1002, "Creative Writing": 1001, "Coding & Debugging": 1055, "Reasoning": 1004, "Editing": 1001, "Math": 1008, "Planning": 975, "Brainstorming": 967, "Role playing": 1029, "Advice seeking": 965, "Data Analysis": 1028, "Others": 1021, "average": 1004.6666666666666, "# battles": 3418}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1008, "Information seeking": 965, "Creative Writing": 941, "Coding & Debugging": 1191, "Reasoning": 1044, "Editing": 1082, "Math": 1045, "Planning": 1028, "Brainstorming": 945, "Role playing": 886, "Advice seeking": 1014, "Data Analysis": 1089, "Others": 999, "average": 1019.0833333333334, "# battles": 2367}
+{"model name ": "command", "elo overall": 1000, "Information seeking": 969, "Creative Writing": 982, "Coding & Debugging": 1071, "Reasoning": 996, "Editing": 1024, "Math": 1000, "Planning": 1059, "Brainstorming": 1033, "Role playing": 951, "Advice seeking": 978, "Data Analysis": 1018, "Others": 994, "average": 1006.25, "# battles": 1733}
+{"model name ": "mistral-large-2402", "elo overall": 986, "Information seeking": 944, "Creative Writing": 993, "Coding & Debugging": 1011, "Reasoning": 978, "Editing": 993, "Math": 996, "Planning": 930, "Brainstorming": 987, "Role playing": 985, "Advice seeking": 951, "Data Analysis": 1015, "Others": 1004, "average": 982.25, "# battles": 1881}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 955, "Information seeking": 978, "Creative Writing": 968, "Coding & Debugging": 861, "Reasoning": 988, "Editing": 987, "Math": 950, "Planning": 994, "Brainstorming": 1017, "Role playing": 974, "Advice seeking": 1034, "Data Analysis": 923, "Others": 1025, "average": 974.9166666666666, "# battles": 2391}
+{"model name ": "gemini-1.0-pro", "elo overall": 950, "Information seeking": 940, "Creative Writing": 942, "Coding & Debugging": 998, "Reasoning": 964, "Editing": 958, "Math": 972, "Planning": 912, "Brainstorming": 946, "Role playing": 983, "Advice seeking": 923, "Data Analysis": 1004, "Others": 998, "average": 961.6666666666666, "# battles": 1816}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 934, "Information seeking": 956, "Creative Writing": 959, "Coding & Debugging": 838, "Reasoning": 940, "Editing": 983, "Math": 991, "Planning": 967, "Brainstorming": 1001, "Role playing": 978, "Advice seeking": 1013, "Data Analysis": 926, "Others": 1001, "average": 962.75, "# battles": 2350}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 924, "Information seeking": 943, "Creative Writing": 951, "Coding & Debugging": 817, "Reasoning": 943, "Editing": 961, "Math": 938, "Planning": 979, "Brainstorming": 982, "Role playing": 953, "Advice seeking": 985, "Data Analysis": 913, "Others": 1007, "average": 947.6666666666666, "# battles": 2297}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 917, "Information seeking": 867, "Creative Writing": 917, "Coding & Debugging": 1006, "Reasoning": 894, "Editing": 963, "Math": 969, "Planning": 883, "Brainstorming": 870, "Role playing": 958, "Advice seeking": 857, "Data Analysis": 1011, "Others": 988, "average": 931.9166666666666, "# battles": 2480}
+{"model name ": "gemma-7b-it", "elo overall": 915, "Information seeking": 914, "Creative Writing": 864, "Coding & Debugging": 1028, "Reasoning": 894, "Editing": 880, "Math": 939, "Planning": 887, "Brainstorming": 875, "Role playing": 954, "Advice seeking": 906, "Data Analysis": 988, "Others": 1002, "average": 927.5833333333334, "# battles": 2519}
+{"model name ": "gemma-2b-it", "elo overall": 884, "Information seeking": 885, "Creative Writing": 890, "Coding & Debugging": 925, "Reasoning": 837, "Editing": 895, "Math": 836, "Planning": 923, "Brainstorming": 902, "Role playing": 910, "Advice seeking": 897, "Data Analysis": 911, "Others": 987, "average": 899.8333333333334, "# battles": 2484}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 846, "Information seeking": 837, "Creative Writing": 837, "Coding & Debugging": 830, "Reasoning": 851, "Editing": 874, "Math": 931, "Planning": 852, "Brainstorming": 879, "Role playing": 891, "Advice seeking": 883, "Data Analysis": 882, "Others": 993, "average": 878.3333333333334, "# battles": 2478}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 822, "Information seeking": 754, "Creative Writing": 789, "Coding & Debugging": 955, "Reasoning": 844, "Editing": 880, "Math": 902, "Planning": 826, "Brainstorming": 815, "Role playing": 777, "Advice seeking": 803, "Data Analysis": 918, "Others": 844, "average": 842.25, "# battles": 14003}

data_dir/elo_ranks.length_ablation.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1228, "# battles": 5430}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1112, "# battles": 2371}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1112, "# battles": 1942}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1107, "# battles": 1898}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1096, "# battles": 1888}
+{"model name ": "zephyr-7b-beta", "elo overall": 1079, "# battles": 3367}
+{"model name ": "Yi-34B-Chat", "elo overall": 1047, "# battles": 2428}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 1037, "# battles": 3435}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1014, "# battles": 2608}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1011, "# battles": 3418}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1008, "# battles": 2367}
+{"model name ": "command", "elo overall": 1000, "# battles": 1733}
+{"model name ": "mistral-large-2402", "elo overall": 986, "# battles": 1881}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 955, "# battles": 2391}
+{"model name ": "gemini-1.0-pro", "elo overall": 950, "# battles": 1816}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 934, "# battles": 2350}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 924, "# battles": 2297}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 917, "# battles": 2480}
+{"model name ": "gemma-7b-it", "elo overall": 915, "# battles": 2519}
+{"model name ": "gemma-2b-it", "elo overall": 884, "# battles": 2484}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 846, "# battles": 2478}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 822, "# battles": 14003}

data_dir/elo_ranks.skip_empty.all.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1296, "Information seeking": 1276, "Creative Writing": 1265, "Coding & Debugging": 1327, "Reasoning": 1359, "Editing": 1200, "Math": 1269, "Planning": 1286, "Brainstorming": 1320, "Role playing": 1237, "Advice seeking": 1298, "Data Analysis": 1250, "Others": 1068, "average": 1262.9166666666667, "# battles": 5362}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1139, "Information seeking": 1124, "Creative Writing": 1065, "Coding & Debugging": 1236, "Reasoning": 1191, "Editing": 1132, "Math": 1148, "Planning": 1149, "Brainstorming": 1109, "Role playing": 965, "Advice seeking": 1110, "Data Analysis": 1117, "Others": 1030, "average": 1114.6666666666667, "# battles": 2258}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1127, "Information seeking": 1112, "Creative Writing": 1081, "Coding & Debugging": 1197, "Reasoning": 1168, "Editing": 1140, "Math": 1134, "Planning": 1157, "Brainstorming": 1091, "Role playing": 1003, "Advice seeking": 1079, "Data Analysis": 1137, "Others": 1014, "average": 1109.4166666666667, "# battles": 2483}
+{"model name ": "mistral-large-2402", "elo overall": 1109, "Information seeking": 1087, "Creative Writing": 1111, "Coding & Debugging": 1105, "Reasoning": 1126, "Editing": 1088, "Math": 1085, "Planning": 1076, "Brainstorming": 1078, "Role playing": 1069, "Advice seeking": 1060, "Data Analysis": 1061, "Others": 1021, "average": 1080.5833333333333, "# battles": 1794}
+{"model name ": "Yi-34B-Chat", "elo overall": 1094, "Information seeking": 1078, "Creative Writing": 1100, "Coding & Debugging": 1059, "Reasoning": 1091, "Editing": 1067, "Math": 1032, "Planning": 1127, "Brainstorming": 1138, "Role playing": 1070, "Advice seeking": 1099, "Data Analysis": 1019, "Others": 997, "average": 1073.0833333333333, "# battles": 2134}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1089, "Information seeking": 1076, "Creative Writing": 1136, "Coding & Debugging": 1089, "Reasoning": 1041, "Editing": 1094, "Math": 1011, "Planning": 1077, "Brainstorming": 1078, "Role playing": 1115, "Advice seeking": 1062, "Data Analysis": 1029, "Others": 1003, "average": 1067.5833333333333, "# battles": 2313}
+{"model name ": "gemini-1.0-pro", "elo overall": 1082, "Information seeking": 1113, "Creative Writing": 1061, "Coding & Debugging": 1056, "Reasoning": 1062, "Editing": 1016, "Math": 1062, "Planning": 1071, "Brainstorming": 1039, "Role playing": 1053, "Advice seeking": 1078, "Data Analysis": 1064, "Others": 1008, "average": 1056.9166666666667, "# battles": 1643}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1069, "Information seeking": 1109, "Creative Writing": 1068, "Coding & Debugging": 996, "Reasoning": 1056, "Editing": 1011, "Math": 994, "Planning": 1025, "Brainstorming": 1050, "Role playing": 1074, "Advice seeking": 1042, "Data Analysis": 1029, "Others": 1018, "average": 1039.3333333333333, "# battles": 1715}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1057, "Information seeking": 1076, "Creative Writing": 1046, "Coding & Debugging": 1073, "Reasoning": 1050, "Editing": 1003, "Math": 1048, "Planning": 996, "Brainstorming": 1025, "Role playing": 1080, "Advice seeking": 1003, "Data Analysis": 1023, "Others": 993, "average": 1034.6666666666667, "# battles": 3349}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1035, "Information seeking": 1061, "Creative Writing": 1061, "Coding & Debugging": 943, "Reasoning": 997, "Editing": 1008, "Math": 967, "Planning": 1006, "Brainstorming": 1037, "Role playing": 1048, "Advice seeking": 1014, "Data Analysis": 977, "Others": 1016, "average": 1011.25, "# battles": 1657}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1015, "Information seeking": 1043, "Creative Writing": 1025, "Coding & Debugging": 927, "Reasoning": 980, "Editing": 986, "Math": 918, "Planning": 1006, "Brainstorming": 1004, "Role playing": 1046, "Advice seeking": 1026, "Data Analysis": 983, "Others": 1011, "average": 996.25, "# battles": 1667}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 983, "Information seeking": 996, "Creative Writing": 1022, "Coding & Debugging": 884, "Reasoning": 971, "Editing": 966, "Math": 965, "Planning": 949, "Brainstorming": 940, "Role playing": 1059, "Advice seeking": 973, "Data Analysis": 902, "Others": 992, "average": 968.25, "# battles": 3342}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 968, "Information seeking": 930, "Creative Writing": 934, "Coding & Debugging": 1101, "Reasoning": 1009, "Editing": 976, "Math": 1142, "Planning": 970, "Brainstorming": 929, "Role playing": 923, "Advice seeking": 954, "Data Analysis": 1021, "Others": 978, "average": 988.9166666666666, "# battles": 13333}
+{"model name ": "zephyr-7b-beta", "elo overall": 968, "Information seeking": 993, "Creative Writing": 999, "Coding & Debugging": 927, "Reasoning": 949, "Editing": 966, "Math": 924, "Planning": 954, "Brainstorming": 939, "Role playing": 1021, "Advice seeking": 931, "Data Analysis": 965, "Others": 960, "average": 960.6666666666666, "# battles": 3310}
+{"model name ": "gemma-7b-it", "elo overall": 928, "Information seeking": 928, "Creative Writing": 935, "Coding & Debugging": 938, "Reasoning": 951, "Editing": 905, "Math": 982, "Planning": 966, "Brainstorming": 979, "Role playing": 953, "Advice seeking": 968, "Data Analysis": 950, "Others": 994, "average": 954.0833333333334, "# battles": 2429}
+{"model name ": "command", "elo overall": 926, "Information seeking": 937, "Creative Writing": 941, "Coding & Debugging": 906, "Reasoning": 914, "Editing": 930, "Math": 927, "Planning": 974, "Brainstorming": 984, "Role playing": 942, "Advice seeking": 961, "Data Analysis": 915, "Others": 980, "average": 942.5833333333334, "# battles": 1655}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 912, "Information seeking": 904, "Creative Writing": 901, "Coding & Debugging": 923, "Reasoning": 922, "Editing": 974, "Math": 911, "Planning": 927, "Brainstorming": 943, "Role playing": 904, "Advice seeking": 954, "Data Analysis": 1005, "Others": 999, "average": 938.9166666666666, "# battles": 2090}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 877, "Information seeking": 877, "Creative Writing": 865, "Coding & Debugging": 893, "Reasoning": 900, "Editing": 985, "Math": 929, "Planning": 917, "Brainstorming": 891, "Role playing": 865, "Advice seeking": 954, "Data Analysis": 944, "Others": 1006, "average": 918.8333333333334, "# battles": 2045}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 856, "Information seeking": 849, "Creative Writing": 862, "Coding & Debugging": 902, "Reasoning": 861, "Editing": 881, "Math": 916, "Planning": 841, "Brainstorming": 867, "Role playing": 911, "Advice seeking": 857, "Data Analysis": 938, "Others": 969, "average": 887.8333333333334, "# battles": 2170}
+{"model name ": "gemma-2b-it", "elo overall": 849, "Information seeking": 836, "Creative Writing": 893, "Coding & Debugging": 822, "Reasoning": 807, "Editing": 875, "Math": 886, "Planning": 895, "Brainstorming": 919, "Role playing": 909, "Advice seeking": 877, "Data Analysis": 870, "Others": 979, "average": 880.6666666666666, "# battles": 2410}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 821, "Information seeking": 807, "Creative Writing": 834, "Coding & Debugging": 820, "Reasoning": 819, "Editing": 938, "Math": 868, "Planning": 848, "Brainstorming": 854, "Role playing": 856, "Advice seeking": 890, "Data Analysis": 912, "Others": 996, "average": 870.1666666666666, "# battles": 2036}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 798, "Information seeking": 786, "Creative Writing": 797, "Coding & Debugging": 881, "Reasoning": 782, "Editing": 857, "Math": 880, "Planning": 789, "Brainstorming": 782, "Role playing": 891, "Advice seeking": 812, "Data Analysis": 883, "Others": 970, "average": 842.5, "# battles": 2415}

data_dir/elo_ranks.skip_empty.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1296, "# battles": 5362}
+{"model name ": "claude-3-opus-20240229", "elo overall": 1139, "# battles": 2258}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 1127, "# battles": 2483}
+{"model name ": "mistral-large-2402", "elo overall": 1109, "# battles": 1794}
+{"model name ": "Yi-34B-Chat", "elo overall": 1094, "# battles": 2134}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1089, "# battles": 2313}
+{"model name ": "gemini-1.0-pro", "elo overall": 1082, "# battles": 1643}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1069, "# battles": 1715}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 1057, "# battles": 3349}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1035, "# battles": 1657}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1015, "# battles": 1667}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 983, "# battles": 3342}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 968, "# battles": 13333}
+{"model name ": "zephyr-7b-beta", "elo overall": 968, "# battles": 3310}
+{"model name ": "gemma-7b-it", "elo overall": 928, "# battles": 2429}
+{"model name ": "command", "elo overall": 926, "# battles": 1655}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 912, "# battles": 2090}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 877, "# battles": 2045}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 856, "# battles": 2170}
+{"model name ": "gemma-2b-it", "elo overall": 849, "# battles": 2410}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 821, "# battles": 2036}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 798, "# battles": 2415}

data_dir/elo_ranks.skip_empty.length_ablation.all.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1219, "Information seeking": 1266, "Creative Writing": 1199, "Coding & Debugging": 1154, "Reasoning": 1275, "Editing": 1120, "Math": 1192, "Planning": 1188, "Brainstorming": 1179, "Role playing": 1218, "Advice seeking": 1277, "Data Analysis": 1161, "Others": 1033, "average": 1188.5, "# battles": 5363}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1166, "Information seeking": 1225, "Creative Writing": 1172, "Coding & Debugging": 1063, "Reasoning": 1130, "Editing": 1078, "Math": 1008, "Planning": 1135, "Brainstorming": 1139, "Role playing": 1130, "Advice seeking": 1115, "Data Analysis": 1043, "Others": 1016, "average": 1104.5, "# battles": 1715}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1151, "Information seeking": 1195, "Creative Writing": 1149, "Coding & Debugging": 1051, "Reasoning": 1120, "Editing": 1042, "Math": 1009, "Planning": 1141, "Brainstorming": 1142, "Role playing": 1139, "Advice seeking": 1149, "Data Analysis": 1059, "Others": 1008, "average": 1100.3333333333333, "# battles": 1667}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1148, "Information seeking": 1192, "Creative Writing": 1137, "Coding & Debugging": 1032, "Reasoning": 1102, "Editing": 1064, "Math": 1021, "Planning": 1121, "Brainstorming": 1140, "Role playing": 1102, "Advice seeking": 1126, "Data Analysis": 1040, "Others": 1032, "average": 1092.4166666666667, "# battles": 1657}
+{"model name ": "Yi-34B-Chat", "elo overall": 1106, "Information seeking": 1096, "Creative Writing": 1066, "Coding & Debugging": 1126, "Reasoning": 1118, "Editing": 1013, "Math": 1095, "Planning": 1137, "Brainstorming": 1118, "Role playing": 1052, "Advice seeking": 1133, "Data Analysis": 1072, "Others": 1003, "average": 1085.75, "# battles": 2134}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1088, "Information seeking": 1092, "Creative Writing": 1132, "Coding & Debugging": 1053, "Reasoning": 1057, "Editing": 1068, "Math": 1086, "Planning": 1042, "Brainstorming": 1037, "Role playing": 1123, "Advice seeking": 1041, "Data Analysis": 1048, "Others": 1010, "average": 1065.75, "# battles": 2313}
+{"model name ": "zephyr-7b-beta", "elo overall": 1060, "Information seeking": 1022, "Creative Writing": 1093, "Coding & Debugging": 1065, "Reasoning": 1036, "Editing": 1074, "Math": 1040, "Planning": 1051, "Brainstorming": 1072, "Role playing": 1035, "Advice seeking": 976, "Data Analysis": 1047, "Others": 1014, "average": 1043.75, "# battles": 3310}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 1020, "Information seeking": 1004, "Creative Writing": 1065, "Coding & Debugging": 970, "Reasoning": 998, "Editing": 1052, "Math": 1030, "Planning": 1005, "Brainstorming": 1008, "Role playing": 1061, "Advice seeking": 1007, "Data Analysis": 998, "Others": 1026, "average": 1018.6666666666666, "# battles": 3342}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 994, "Information seeking": 975, "Creative Writing": 944, "Coding & Debugging": 1099, "Reasoning": 1039, "Editing": 1045, "Math": 1069, "Planning": 1017, "Brainstorming": 953, "Role playing": 896, "Advice seeking": 976, "Data Analysis": 1041, "Others": 988, "average": 1003.5, "# battles": 2483}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 991, "Information seeking": 993, "Creative Writing": 1003, "Coding & Debugging": 980, "Reasoning": 987, "Editing": 991, "Math": 994, "Planning": 960, "Brainstorming": 965, "Role playing": 1024, "Advice seeking": 959, "Data Analysis": 983, "Others": 1020, "average": 988.25, "# battles": 3350}
+{"model name ": "claude-3-opus-20240229", "elo overall": 986, "Information seeking": 951, "Creative Writing": 940, "Coding & Debugging": 1137, "Reasoning": 1034, "Editing": 1067, "Math": 1028, "Planning": 1010, "Brainstorming": 945, "Role playing": 885, "Advice seeking": 1009, "Data Analysis": 1055, "Others": 1000, "average": 1005.0833333333334, "# battles": 2259}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 985, "Information seeking": 988, "Creative Writing": 969, "Coding & Debugging": 998, "Reasoning": 1017, "Editing": 1021, "Math": 960, "Planning": 1026, "Brainstorming": 1018, "Role playing": 972, "Advice seeking": 1049, "Data Analysis": 1014, "Others": 1025, "average": 1004.75, "# battles": 2090}
+{"model name ": "command", "elo overall": 982, "Information seeking": 965, "Creative Writing": 983, "Coding & Debugging": 1004, "Reasoning": 980, "Editing": 1005, "Math": 992, "Planning": 1052, "Brainstorming": 1030, "Role playing": 954, "Advice seeking": 974, "Data Analysis": 967, "Others": 995, "average": 991.75, "# battles": 1655}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 966, "Information seeking": 968, "Creative Writing": 956, "Coding & Debugging": 956, "Reasoning": 965, "Editing": 1019, "Math": 1011, "Planning": 999, "Brainstorming": 999, "Role playing": 976, "Advice seeking": 1027, "Data Analysis": 1012, "Others": 1000, "average": 990.6666666666666, "# battles": 2045}
+{"model name ": "mistral-large-2402", "elo overall": 958, "Information seeking": 939, "Creative Writing": 995, "Coding & Debugging": 941, "Reasoning": 969, "Editing": 985, "Math": 985, "Planning": 919, "Brainstorming": 986, "Role playing": 988, "Advice seeking": 952, "Data Analysis": 968, "Others": 1005, "average": 969.3333333333334, "# battles": 1794}
+{"model name ": "gemini-1.0-pro", "elo overall": 951, "Information seeking": 957, "Creative Writing": 960, "Coding & Debugging": 942, "Reasoning": 955, "Editing": 947, "Math": 969, "Planning": 904, "Brainstorming": 953, "Role playing": 1000, "Advice seeking": 926, "Data Analysis": 964, "Others": 998, "average": 956.25, "# battles": 1644}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 948, "Information seeking": 952, "Creative Writing": 957, "Coding & Debugging": 918, "Reasoning": 962, "Editing": 990, "Math": 962, "Planning": 995, "Brainstorming": 982, "Role playing": 958, "Advice seeking": 998, "Data Analysis": 993, "Others": 1006, "average": 972.75, "# battles": 2036}
+{"model name ": "gemma-7b-it", "elo overall": 889, "Information seeking": 907, "Creative Writing": 858, "Coding & Debugging": 953, "Reasoning": 881, "Editing": 854, "Math": 929, "Planning": 870, "Brainstorming": 872, "Role playing": 949, "Advice seeking": 899, "Data Analysis": 943, "Others": 1003, "average": 909.8333333333334, "# battles": 2430}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 887, "Information seeking": 862, "Creative Writing": 913, "Coding & Debugging": 930, "Reasoning": 875, "Editing": 948, "Math": 964, "Planning": 862, "Brainstorming": 870, "Role playing": 956, "Advice seeking": 845, "Data Analysis": 964, "Others": 987, "average": 914.6666666666666, "# battles": 2416}
+{"model name ": "gemma-2b-it", "elo overall": 854, "Information seeking": 876, "Creative Writing": 883, "Coding & Debugging": 838, "Reasoning": 813, "Editing": 878, "Math": 820, "Planning": 904, "Brainstorming": 902, "Role playing": 909, "Advice seeking": 881, "Data Analysis": 861, "Others": 985, "average": 879.1666666666666, "# battles": 2410}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 854, "Information seeking": 830, "Creative Writing": 837, "Coding & Debugging": 929, "Reasoning": 864, "Editing": 889, "Math": 953, "Planning": 863, "Brainstorming": 874, "Role playing": 894, "Advice seeking": 883, "Data Analysis": 954, "Others": 993, "average": 896.9166666666666, "# battles": 2170}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 789, "Information seeking": 744, "Creative Writing": 788, "Coding & Debugging": 860, "Reasoning": 822, "Editing": 857, "Math": 874, "Planning": 797, "Brainstorming": 815, "Role playing": 775, "Advice seeking": 788, "Data Analysis": 832, "Others": 845, "average": 816.4166666666666, "# battles": 13333}

data_dir/elo_ranks.skip_empty.length_ablation.jsonl

@@ -0,0 +1,22 @@
+{"model name ": "gpt-4-0125-preview", "elo overall": 1219, "# battles": 5363}
+{"model name ": "Llama-2-70b-chat-hf.nosp", "elo overall": 1166, "# battles": 1715}
+{"model name ": "Llama-2-7b-chat-hf.nosp", "elo overall": 1151, "# battles": 1667}
+{"model name ": "Llama-2-13b-chat-hf.nosp", "elo overall": 1148, "# battles": 1657}
+{"model name ": "Yi-34B-Chat", "elo overall": 1106, "# battles": 2134}
+{"model name ": "Mistral-7B-Instruct-v0.2", "elo overall": 1088, "# battles": 2313}
+{"model name ": "zephyr-7b-beta", "elo overall": 1060, "# battles": 3310}
+{"model name ": "tulu-2-dpo-70b", "elo overall": 1020, "# battles": 3342}
+{"model name ": "claude-3-sonnet-20240229", "elo overall": 994, "# battles": 2483}
+{"model name ": "Mixtral-8x7B-Instruct-v0.1", "elo overall": 991, "# battles": 3350}
+{"model name ": "claude-3-opus-20240229", "elo overall": 986, "# battles": 2259}
+{"model name ": "Llama-2-70b-chat-hf", "elo overall": 985, "# battles": 2090}
+{"model name ": "command", "elo overall": 982, "# battles": 1655}
+{"model name ": "Llama-2-13b-chat-hf", "elo overall": 966, "# battles": 2045}
+{"model name ": "mistral-large-2402", "elo overall": 958, "# battles": 1794}
+{"model name ": "gemini-1.0-pro", "elo overall": 951, "# battles": 1644}
+{"model name ": "Llama-2-7b-chat-hf", "elo overall": 948, "# battles": 2036}
+{"model name ": "gemma-7b-it", "elo overall": 889, "# battles": 2430}
+{"model name ": "Mistral-7B-Instruct-v0.1", "elo overall": 887, "# battles": 2416}
+{"model name ": "gemma-2b-it", "elo overall": 854, "# battles": 2410}
+{"model name ": "vicuna-13b-v1.5", "elo overall": 854, "# battles": 2170}
+{"model name ": "gpt-3.5-turbo-0125", "elo overall": 789, "# battles": 13333}

data_dir/leaderboard_data.jsonl

@@ -0,0 +1,13 @@
+{"model name": "gpt-4-0125-preview", "win rate": 0.833984375, "win+tie rate": 0.9423828125, "num votes": 1024, "longer rate": 0.9296875, "charlen": 3190.716796875, "elo": 0}
+{"model name": "Mixtral-8x7B-Instruct-v0.1", "win rate": 0.568359375, "win+tie rate": 0.6767578125, "num votes": 1024, "longer rate": 0.7724609375, "charlen": 2483.9638671875, "elo": 0}
+{"model name": "Yi-34B-Chat", "win rate": 0.5322265625, "win+tie rate": 0.658203125, "num votes": 1024, "longer rate": 0.7412109375, "charlen": 2567.92578125, "elo": 0}
+{"model name": "gpt-3.5-turbo-0125", "win rate": 0.5, "win+tie rate": 0.5, "num votes": 12288, "longer rate": 0.3017578125, "charlen": 1725.7216796875, "elo": 0}
+{"model name": "tulu-2-dpo-70b", "win rate": 0.44921875, "win+tie rate": 0.5771484375, "num votes": 1024, "longer rate": 0.7919921875, "charlen": 2614.822265625, "elo": 0}
+{"model name": "zephyr-7b-beta", "win rate": 0.4404296875, "win+tie rate": 0.5478515625, "num votes": 1024, "longer rate": 0.822265625, "charlen": 3011.2529296875, "elo": 0}
+{"model name": "gemma-7b-it", "win rate": 0.4150390625, "win+tie rate": 0.49609375, "num votes": 1024, "longer rate": 0.6064453125, "charlen": 1951.2548828125, "elo": 0}
+{"model name": "Llama-2-70b-chat-hf", "win rate": 0.3125, "win+tie rate": 0.4462890625, "num votes": 1024, "longer rate": 0.6845703125, "charlen": 2007.029296875, "elo": 0}
+{"model name": "gemma-2b-it", "win rate": 0.296875, "win+tie rate": 0.375, "num votes": 1024, "longer rate": 0.560546875, "charlen": 1862.732421875, "elo": 0}
+{"model name": "Llama-2-13b-chat-hf", "win rate": 0.2666015625, "win+tie rate": 0.4130859375, "num votes": 1024, "longer rate": 0.6826171875, "charlen": 1945.42578125, "elo": 0}
+{"model name": "vicuna-13b-v1.5", "win rate": 0.2548828125, "win+tie rate": 0.3759765625, "num votes": 1024, "longer rate": 0.509765625, "charlen": 1642.1640625, "elo": 0}
+{"model name": "Mistral-7B-Instruct-v0.1", "win rate": 0.23046875, "win+tie rate": 0.3232421875, "num votes": 1024, "longer rate": 0.6171875, "charlen": 2208.8115234375, "elo": 0}
+{"model name": "Llama-2-7b-chat-hf", "win rate": 0.2216796875, "win+tie rate": 0.345703125, "num votes": 1024, "longer rate": 0.65234375, "charlen": 1878.521484375, "elo": 0}

data_dir/pairwise_win_fractions.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53b8c101b7473e39e483771317b1bd34c9feda733d4b5cdc67b3508b2ca968b1
+size 11827

data_dir/winrate_data.jsonl

@@ -0,0 +1,13 @@
+{"model name": "gpt-4-0125-preview", "win rate": 0.5, "win+tie rate": 0.5, "num votes": 4611, "longer rate": 0.7931034482758621, "charlen": 3195.4654088050315, "elo": 0}
+{"model name": "Mistral-7B-Instruct-v0.2", "win rate": 0.23167252145204986, "win+tie rate": 0.2624417522212806, "num votes": 65, "longer rate": 0.2153846153846154, "charlen": 2808.230769230769, "elo": 0}
+{"model name": "Yi-34B-Chat", "win rate": 0.19769275417519458, "win+tie rate": 0.27965996728994863, "num votes": 61, "longer rate": 0.36065573770491804, "charlen": 2712.4590163934427, "elo": 0}
+{"model name": "Mixtral-8x7B-Instruct-v0.1", "win rate": 0.16477418902065052, "win+tie rate": 0.23508668902065052, "num votes": 1024, "longer rate": 0.203125, "charlen": 2483.9638671875, "elo": 0}
+{"model name": "zephyr-7b-beta", "win rate": 0.12713105860443502, "win+tie rate": 0.17791230860443502, "num votes": 1024, "longer rate": 0.341796875, "charlen": 3011.2529296875, "elo": 0}
+{"model name": "gemma-7b-it", "win rate": 0.12270420594225764, "win+tie rate": 0.1611657444037961, "num votes": 52, "longer rate": 0.1346153846153846, "charlen": 1973.8076923076924, "elo": 0}
+{"model name": "tulu-2-dpo-70b", "win rate": 0.11486627676908645, "win+tie rate": 0.17736627676908645, "num votes": 1024, "longer rate": 0.240234375, "charlen": 2614.822265625, "elo": 0}
+{"model name": "Llama-2-13b-chat-hf", "win rate": 0.09671362313232809, "win+tie rate": 0.1493452020796965, "num votes": 57, "longer rate": 0.17543859649122806, "charlen": 1949.6666666666667, "elo": 0}
+{"model name": "Mistral-7B-Instruct-v0.1", "win rate": 0.09374492823671128, "win+tie rate": 0.13010856460034764, "num votes": 55, "longer rate": 0.12727272727272726, "charlen": 2199.8, "elo": 0}
+{"model name": "gemma-2b-it", "win rate": 0.08728338828088478, "win+tie rate": 0.1550799984503763, "num votes": 59, "longer rate": 0.0847457627118644, "charlen": 1707.084745762712, "elo": 0}
+{"model name": "vicuna-13b-v1.5", "win rate": 0.05894420249306174, "win+tie rate": 0.05894420249306174, "num votes": 45, "longer rate": 0.1111111111111111, "charlen": 1800.911111111111, "elo": 0}
+{"model name": "Llama-2-70b-chat-hf", "win rate": 0.03594723537048477, "win+tie rate": 0.1073758067990562, "num votes": 56, "longer rate": 0.10714285714285714, "charlen": 2110.4285714285716, "elo": 0}
+{"model name": "Llama-2-7b-chat-hf", "win rate": 0.01868885019984797, "win+tie rate": 0.09561192712292489, "num votes": 65, "longer rate": 0.03076923076923077, "charlen": 1663.7076923076922, "elo": 0}

data_utils.py

@@ -0,0 +1,191 @@
+from datasets import load_dataset, Dataset
+import os 
+from datasets import load_dataset
+from datasets.utils.logging import disable_progress_bar
+from constants import column_names, all_task_types
+from utils_display import make_clickable_model
+import random 
+disable_progress_bar()
+
+id_to_data = None 
+model_len_info = None 
+
+def estimated_win_rate(elo_a, elo_b):
+    """
+    Calculate the estimated win rate for player A against player B using their Elo ratings.
+
+    :param elo_a: Elo rating of player A
+    :param elo_b: Elo rating of player B
+    :return: Estimated win rate for player A
+    """
+    exponent = (elo_b - elo_a) / 400
+    probability_a_wins = 1 / (1 + 10 ** exponent)
+    return (1-probability_a_wins)*100
+
+
+
+# Formats the columns
+def formatter(x):
+    if type(x) is str:
+        x = x
+    else: 
+        x = round(x, 2)
+    return x
+
+
+def add_winrates(current_df):
+    df = current_df.copy()
+    elo_column = "Overall Elo" 
+
+    # Correct way to filter the DataFrame and get the Elo rating for "gpt-4-0125-preview"
+    model_a_elo = df[df["Model"].str.contains("gpt-4")][elo_column].iloc[0]
+
+    # Correct way to filter the DataFrame and get the Elo rating for "gpt-3.5-turbo-0125"
+    model_b_elo = df[df["Model"].str.contains("gpt-3.5")][elo_column].iloc[0]
+
+    
+    # Calculate the win rate of "gpt-4-0125-preview" against all models
+    df['Win% vs GPT-4'] = df[elo_column].apply(lambda x: estimated_win_rate(model_a_elo, x)).apply(formatter)    
+    df['Win% vs GPT-3.5T'] = df[elo_column].apply(lambda x: estimated_win_rate(model_b_elo, x)).apply(formatter)    
+    # apply the formatter for the two new columns 
+    cols = list(df.columns)
+    cols.remove("# battles"); cols.append("# battles")
+    cols.remove("Length"); cols.append("Length")
+    df = df[cols]
+    return df
+
+def add_winrates_tasks(current_df, ref="gpt-4"):
+    new_df = current_df.copy()
+    for t in all_task_types:
+        column = column_names[t]
+        model_a_elo = current_df[current_df["Model"].str.contains(ref)][column].iloc[0]
+        new_df[column] = current_df[column].apply(lambda x: estimated_win_rate(model_a_elo, x)).apply(formatter)
+    return new_df
+        
+
+def post_processing(df, model_len_info):
+    if model_len_info:
+        df["Length"] = df["model name "].apply(lambda x: model_len_info[x])
+
+    for col in df.columns:
+        if col == "model name ":
+            df[col] = df[col].apply(lambda x: x.replace(x, make_clickable_model(x)))
+        else:
+            df[col] = df[col].apply(formatter) # For numerical values 
+    df.rename(columns=column_names, inplace=True)
+    df.sort_values(by="Overall Elo", inplace=True, ascending=False)
+    # put the "Overall Elo" and "Task-Avg Elo" column to the front
+    # add the length info
+    df = df[["Model", "Overall Elo", "Task-Avg Elo"] + [col for col in df.columns if col not in ["Model", "Overall Elo", "Task-Avg Elo"]]]
+    return df
+
+def apply_length_penalty(original_df, ablation_df, length_penalty=0.2):
+    original_df = original_df.copy()
+    ablation_df = ablation_df.copy()
+    # replace all values in original_df with the values as z = x - y * length_penalty where y is from ablation_df at the same row and column
+    # except for the "Model" column and the "# battles" column 
+    # do not assume the order of the rows are the same in both dataframes
+    for i, row in original_df.iterrows():
+        for col in original_df.columns:
+            if col == "Model" or col == "# battles" or col == "Length":
+                continue
+            # assert that the model names are the same in both dataframes
+            assert original_df.at[i, "Model"] == ablation_df[ablation_df["Model"] == row["Model"]]["Model"].values[0]
+            original_df[col] = original_df[col].astype(float)
+            original_df.at[i, col] = original_df.at[i, col] - ablation_df[ablation_df["Model"] == row["Model"]][col].values[0] * length_penalty
+    # post_processing
+    original_df = post_processing(original_df, model_len_info=None)
+    return original_df
+
+
+def load_benchdata():
+    print("Loading WildBench data...")
+    bench_data = load_dataset("WildEval/WildBench", split="test")
+    return bench_data
+
+def load_benchdata_dict():
+    print("Loading WildBench data....")
+    bench_data = load_dataset("WildEval/WildBench", split="test")
+    id_to_data = {}
+    for item in bench_data:
+        id_to_data[item["session_id"]] = item
+    return id_to_data
+
+def load_eval_results():
+    print("Loading WildBench Evaluation data...")
+    eval_results = load_dataset("WildEval/WildBench-Evaluation", "all", split="train")
+    return eval_results
+
+def load_infer_results(model_name):
+    print(f"Loading WildBench Results for {model_name}...")
+    infer_results = load_dataset("WildEval/WildBench-Results", model_name, split="train")
+    return infer_results
+
+
+
+def sample_an_eval_result(eval_results, model_list=[], tag_list=[]):
+    global id_to_data          
+    eval_results = list(eval_results)
+    random.shuffle(eval_results)
+    for eval_item in eval_results:  
+        # print(json.dumps(eval_item, indent=2))
+        # print(f"## Session ID: {eval_item['session_id']}")
+        # eval_item["eval_id"]
+        assignment = eval_item['assignment']
+        model_1, model_2 = eval_item['model_1'], eval_item['model_2']
+        model_A = model_1 if assignment['A'] == model_1 else model_2
+        model_B = model_2 if assignment['B'] == model_2 else model_1
+        if len(model_list) >= 2:
+            if model_A not in model_list or model_B not in model_list:
+                continue
+        elif len(model_list) == 1:
+            if model_A != model_list[0] and model_B != model_list[0]:
+                continue
+        else:
+            pass 
+        if tag_list:
+            if set(tag_list).isdisjoint(set(eval_item['tags'])):
+                continue
+        winner = eval_item['winner']
+        # print(f"## Model A: {model_A} | Model B: {model_B} | Winner: {winner}")
+        task_type = eval_item['tags'][0] # primary task type
+        chat_history = eval_item['history']
+        last_query = eval_item['last_query']
+        # print(f"## Task Type: {task_type}")
+        # print(f"## Chat History: {chat_history}")
+        # print(f"## Last Query -->  USER: {last_query}")
+
+        model_A_output = eval_item['model_1_output'] if model_1 == model_A else eval_item['model_2_output']
+        model_B_output = eval_item['model_2_output'] if model_2 == model_B else eval_item['model_1_output']
+
+        if len(model_A_output.strip()) == 0 or len(model_B_output.strip()) == 0:
+            continue
+
+        conversation_input = id_to_data[eval_item['session_id']]["conversation_input"]
+        # print(f"\n\n\n## Model A ({model_A}) Output ##\n{model_A_output}")
+        # print(f"\n\n\n## Model B ({model_B}) Output ##\n{model_B_output}")
+
+        # print(f"\n\n\n## Winner ##\n{winner}")
+        # print(f"\n\n\n## GPT-4 Judgement ##\n{eval_item['parsed_result']}")
+
+        result_dict = {
+            "session_id": eval_item['session_id'],
+            "model_A": model_A,
+            "model_B": model_B,
+            "winner": winner,
+            "intent": id_to_data[eval_item['session_id']]["intent"],
+            "task_type": task_type,
+            "all_tags": eval_item['tags'],
+            "chat_history": chat_history,
+            "last_query": last_query,
+            "conversation_input": conversation_input,
+            "model_A_output": model_A_output,
+            "model_B_output": model_B_output,
+            "reason": eval_item['parsed_result']["reason"],
+            "choice": eval_item['parsed_result']["choice"],
+            "checklist": id_to_data[eval_item['session_id']]["checklist"],
+        }
+        break 
+    return result_dict
+
+id_to_data = load_benchdata_dict()

legacy.txt

@@ -0,0 +1,7 @@
+// "Llama-2-70b-chat-hf": {"pretty_name": "Llama-2-70B (+sys_prmpt)", "hf_model_id": "meta-llama/Llama-2-70b-hf"},
+// "Llama-2-7b-chat-hf": {"pretty_name": "Llama-2-7B (+sys_prmpt)", "hf_model_id": "meta-llama/Llama-2-7b-hf"},
+// "Llama-2-13b-chat-hf": {"pretty_name": "Llama-2-13B (+sys_prmpt)", "hf_model_id": "meta-llama/Llama-2-13b-hf"},
+// "Mistral-7B-Instruct-v0.2": {"pretty_name": "Mistral-7B (v0.2)", "hf_model_id": "mistralai/Mistral-7B-v0.2"},
+// "command": {"pretty_name": "Cohere Command", "hf_model_id": "https://cohere.com/models/command"},
+// "gemma-2b-it": {"pretty_name": "Gemma-2B-it", "hf_model_id": "google/gemma-2b"},
+// "claude-3-sonnet-20240229": {"pretty_name": "Claude 3 Sonnet", "hf_model_id": "https://www.anthropic.com/claude"}

model_info.json

@@ -0,0 +1,19 @@
+{
+    "Llama-2-13b-chat-hf.nosp": {"pretty_name": "Llama-2-13B-chat", "hf_model_id": "meta-llama/Llama-2-13b-chat-hf"},
+    "Llama-2-70b-chat-hf.nosp": {"pretty_name": "Llama-2-70B-chat", "hf_model_id": "meta-llama/Llama-2-70b-chat-hf"},
+    "Llama-2-7b-chat-hf.nosp": {"pretty_name": "Llama-2-7B-chat", "hf_model_id": "meta-llama/Llama-2-7b-chat-hf"},
+    "Llama-2-7b-chat-hf": {"pretty_name": "Llama-2-7B-chat (+sys prmpt)", "hf_model_id": "meta-llama/Llama-2-7b-chat-hf"},
+    "Mistral-7B-Instruct-v0.1": {"pretty_name": "Mistral-7B-Instruct", "hf_model_id": "mistralai/Mistral-7B-Instruct-v0.1"},
+    "Mixtral-8x7B-Instruct-v0.1": {"pretty_name": "Mixtral-8x7B-Instruct", "hf_model_id": "mistralai/Mixtral-8x7B-Instruct-v0.1"},
+    "Yi-34B-Chat": {"pretty_name": "Yi-34B-Chat", "hf_model_id": "01-ai/Yi-34B"},
+    "gemini-1.0-pro": {"pretty_name": "gemini-1.0-pro", "hf_model_id": "https://blog.google/technology/ai/google-gemini-ai/"},
+    "gemma-7b-it": {"pretty_name": "Gemma-7B-it", "hf_model_id": "google/gemma-7b"},
+    "gpt-3.5-turbo-0125": {"pretty_name": "gpt-3.5-turbo-0125", "hf_model_id": "https://platform.openai.com/"},
+    "gpt-4-0125-preview": {"pretty_name": "gpt-4-0125-preview", "hf_model_id": "https://platform.openai.com/"},
+    "tulu-2-dpo-70b": {"pretty_name": "Tulu-2-dpo-70b", "hf_model_id": "allenai/tulu-2-dpo-70b"},
+    "vicuna-13b-v1.5": {"pretty_name": "Vicuna-13b-v1.5", "hf_model_id": "lmsys/vicuna-13b-v1.5"},
+    "zephyr-7b-beta": {"pretty_name": "Zephyr-7b-beta", "hf_model_id": "HuggingFaceH4/zephyr-7b-beta"},
+    "mistral-large-2402": {"pretty_name": "Mistral-Large", "hf_model_id": "https://mistral.ai/news/mistral-large/"},
+    "claude-3-opus-20240229": {"pretty_name": "Claude 3 Opus", "hf_model_id": "https://www.anthropic.com/claude"},
+    "claude-3-sonnet-20240229": {"pretty_name": "Claude 3 Sonnet", "hf_model_id": "https://www.anthropic.com/claude"}
+}

model_len_info.json

@@ -0,0 +1,19 @@
+{
+  "Llama-2-13b-chat-hf.nosp": 2943.346238938053,
+  "Llama-2-70b-chat-hf.nosp": 3077.0840707964603,
+  "Llama-2-7b-chat-hf.nosp": 2965.4059734513276,
+  "Llama-2-7b-chat-hf": 2137.34,
+  "Mistral-7B-Instruct-v0.1": 2208.8115234375,
+  "Mixtral-8x7B-Instruct-v0.1": 2483.9638671875,
+  "Yi-34B-Chat": 2899.1797133406835,
+  "gemini-1.0-pro": 2407.559462254395,
+  "gemma-7b-it": 1960.829244357213,
+  "gpt-3.5-turbo-0125": 1725.7216796875,
+  "gpt-4-0125-preview": 3190.716796875,
+  "tulu-2-dpo-70b": 2630.2337917485265,
+  "vicuna-13b-v1.5": 1864.2749445676275,
+  "zephyr-7b-beta": 3011.2529296875,
+  "mistral-large-2402": 2352.189453125,
+  "claude-3-opus-20240229": 2460.330078125,
+  "claude-3-sonnet-20240229": 2456.21484375
+}

requirements.txt

@@ -0,0 +1,4 @@
+gradio[oauth]==4.19.2
+datasets
+toolz==0.12.1
+plotly

themes.py

@@ -0,0 +1,45 @@
+from __future__ import annotations
+from typing import Iterable
+import gradio as gr
+from gradio.themes.base import Base
+from gradio.themes.utils import colors, fonts, sizes
+import time
+
+class Seafoam(Base):
+    def __init__(
+        self,
+        *,
+        primary_hue: colors.Color | str = colors.blue,
+        secondary_hue: colors.Color | str = colors.gray,
+        neutral_hue: colors.Color | str = colors.gray,
+        spacing_size: sizes.Size | str = sizes.spacing_md,
+        radius_size: sizes.Size | str = sizes.radius_md,
+        text_size: sizes.Size | str = sizes.text_lg,
+        font: fonts.Font
+        | str
+        | Iterable[fonts.Font | str] = (
+            fonts.GoogleFont("Quicksand"),
+            "ui-sans-serif",
+            "sans-serif",
+        ),
+        font_mono: fonts.Font
+        | str
+        | Iterable[fonts.Font | str] = (
+            fonts.GoogleFont("IBM Plex Mono"),
+            "ui-monospace",
+            "monospace",
+        ),
+    ):
+        super().__init__(
+            primary_hue=primary_hue,
+            secondary_hue=secondary_hue,
+            neutral_hue=neutral_hue,
+            spacing_size=spacing_size,
+            radius_size=radius_size,
+            text_size=text_size,
+            font=font,
+            font_mono=font_mono,
+        )
+
+
+seafoam = Seafoam()

update_data.sh

@@ -0,0 +1,15 @@
+# download https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/wget -O $data_dir/  gradio/elo_ranks.all.jsonl to the data_dir
+
+data_dir="data_dir"
+rm -rf $data_dir
+mkdir $data_dir 
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/pairwise_win_fractions.png
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/pairwise_win_fractions.pkl
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/elo_ranks.jsonl
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/elo_ranks.length_ablation.jsonl
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/elo_ranks.skip_empty.jsonl
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/elo_ranks.skip_empty.length_ablation.jsonl
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/elo_ranks.all.jsonl
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/elo_ranks.length_ablation.all.jsonl
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/elo_ranks.skip_empty.all.jsonl
+wget -P $data_dir https://raw.githubusercontent.com/WildEval/WildBench-Leaderboard/main/gradio/elo_ranks.skip_empty.length_ablation.all.jsonl

utils_display.py

@@ -0,0 +1,26 @@
+import json 
+
+with open("model_info.json", "r") as f:
+    model_info = json.load(f)
+
+def make_clickable_model(model_name):
+    global model_info
+    if model_name in model_info:
+        if model_info[model_name]["hf_model_id"].startswith("http"):
+            link = model_info[model_name]["hf_model_id"]
+            return f'🔒 <a target="_blank" href="{link}" style="color: var(--link-text-color); text-decoration: underline;text-decoration-style: dotted;">{model_info[model_name]["pretty_name"]}</a>'
+        else:
+            link = f"https://huggingface.co/{model_info[model_name]['hf_model_id']}"
+            return f'🔥 <a target="_blank" href="{link}" style="color: var(--link-text-color); text-decoration: underline;text-decoration-style: dotted;">{model_info[model_name]["pretty_name"]}</a>'
+    else:
+        return model_name
+    
+
+def styled_error(error):
+    return f"<p style='color: red; font-size: 20px; text-align: center;'>{error}</p>"
+
+def styled_warning(warn):
+    return f"<p style='color: orange; font-size: 20px; text-align: center;'>{warn}</p>"
+
+def styled_message(message):
+    return f"<p style='color: green; font-size: 20px; text-align: center;'>{message}</p>"