Merge master and web

.github/workflows/push_spaces.yaml

@@ -0,0 +1,35 @@
+name: Deploy
+
+on:
+  push:
+    branches:
+      - master
+    paths:
+      - 'data/**'
+      - 'app.py'
+      - 'LEADERBOARD.md'
+      - 'README.md'
+      - 'requirements.txt'
+
+concurrency:
+  group: ${{ github.ref }}-hfdeploy
+  cancel-in-progress: true
+
+jobs:
+  push:
+    runs-on: ubuntu-latest
+    if: github.event.repository.fork == false
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+          lfs: true
+          ref: master
+      - name: Push to Space
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+        run: |
+          for i in 1 2 3 4 5; do
+            git push -f https://jaywonchung:$HF_TOKEN@huggingface.co/spaces/symbioticlab/ml-energy-leaderboard master:main && break || sleep 5;
+          done

Dockerfile

@@ -32,7 +32,7 @@ RUN git clone https://github.com/SymbioticLab/Zeus.git zeus \
 # Install requirements for benchmarking
 ADD . /workspace/leaderboard
 RUN cd leaderboard \
-      && pip install -r requirements.txt \
+      && pip install -r requirements-benchmark.txt \
       && cd ..
 
 ENV TRANSFORMERS_CACHE=/data/leaderboard/hfcache

LEADERBOARD.md

@@ -0,0 +1,68 @@
+The goal of the ML.ENERGY Leaderboard is to give people a sense of how much **energy** LLMs would consume.
+
+## How is energy different?
+
+Even between models with the exact same architecture and size, the average energy consumption per prompt is different because they have **different verbosity**.
+That is, when asked the same thing, they answer in different lengths.
+
+## Metrics
+
+- `gpu`: NVIDIA GPU model name
+- `task`: Name of the task. See *Tasks* below for details.
+- `throughput` (token/s): The average number of tokens generated per second.
+- `response_length` (token): The average number of tokens in the model's response.
+- `latency` (s): The average time it took for the model to generate a response.
+- `energy` (J): The average energy consumed by the model to generate a response.
+
+## Tasks
+
+For each task, every model uses the same system prompt. We still account for differences in roles, e.g. `USER`, `HUMAN`, `ASSISTANT`, `GPT`.
+
+| Name | System prompt |
+|--|--|
+| chat | A chat between a human user (prompter) and an artificial intelligence (AI) assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. |
+| chat-concise | A chat between a human user (prompter) and an artificial intelligence (AI) assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. The assistant's answers are very concise. |
+| instruct | Below is an instruction that describes a task. Write a response that appropriately completes the request. |
+| instruct-concise | Below is an instruction that describes a task. Write a response that appropriately completes the request. The response should be very concise. |
+
+## Setup
+
+Find our benchmark script for one model [here](https://github.com/ml-energy/leaderboard/blob/master/benchmark.py).
+
+### Software
+
+- PyTorch 2.0.1
+- [FastChat](https://github.com/lm-sys/fastchat) -- For various model support
+- [Zeus](https://ml.energy/zeus) -- For GPU energy measurement
+
+### Hardware
+
+- NVIDIA A40 GPU
+
+### Parameters
+
+- Model
+  - Batch size 1
+  - FP16
+- Sampling (decoding)
+  - Greedy sampling from multinomial distribution
+  - Temperature 0.7
+  - Repetition penalty 1.0
+
+## Data
+
+We randomly sampled around 3000 prompts from the [cleaned ShareGPT dataset](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered).
+See [here](https://github.com/ml-energy/leaderboard/tree/master/sharegpt) for more detail on how we created the benchmark dataset.
+
+We used identical system prompts for all models (while respecting their own *role* tokens):
+```
+A chat between a human user (prompter) and an artificial intelligence (AI) assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
+```
+
+## Upcoming
+
+- Compare against more optimized inference runtimes, like TensorRT.
+- Other GPUs
+- Other model/sampling parameters
+- More models
+- Model quality evaluation numbers (e.g., AI2 Reasoning Challenge, HellaSwag)

README.md

@@ -1,3 +1,13 @@
+---
+title: "ML.ENERGY Leaderboard"
+python_version: "3.9"
+app_file: "app.py"
+sdk: "gradio"
+sdk_version: "3.35.2"
+pinned: true
+tags: ["energy", "leaderboard"]
+---
+
 # ML.ENERGY Leaderboard
 
 [![Leaderboard](https://custom-icon-badges.herokuapp.com/badge/ML.ENERGY-Leaderboard-blue.svg?logo=ml-energy)](https://ml.energy/leaderboard)

app.py

@@ -0,0 +1,360 @@
+from __future__ import annotations
+
+import os
+import json
+import yaml
+import itertools
+import contextlib
+
+import numpy as np
+import gradio as gr
+import pandas as pd
+import plotly.io as pio
+import plotly.express as px
+pio.templates.default = "plotly_white"
+
+
+class TableManager:
+    def __init__(self, data_dir: str) -> None:
+        """Load leaderboard data from CSV files in data_dir."""
+        # Load and merge CSV files.
+        df = self._read_tables(data_dir)
+        models = json.load(open(f"{data_dir}/models.json"))
+
+        # Add the #params column.
+        df["parameters"] = df["model"].apply(lambda x: models[x]["params"])
+
+        # Make the first column (model) an HTML anchor to the model's website.
+        def format_model_link(model_name: str) -> str:
+            url = models[model_name]["url"]
+            nickname = models[model_name]["nickname"]
+            return (
+                f'<a style="text-decoration: underline; text-decoration-style: dotted" '
+                f'target="_blank" href="{url}">{nickname}</a>'
+            )
+        df["model"] = df["model"].apply(format_model_link)
+
+        # Sort by energy.
+        df = df.sort_values(by="energy", ascending=True)
+
+        # The full table where all the data are.
+        self.full_df = df
+        # The currently visible table after filtering.
+        self.cur_df = df
+        # The current index of the visible table after filtering.
+        self.cur_index = df.index.to_numpy()
+
+    def _read_tables(self, data_dir: str) -> pd.DataFrame:
+        """Read tables."""
+        df_score = pd.read_csv(f"{data_dir}/score.csv")
+
+        with open(f"{data_dir}/schema.yaml") as file:
+            self.schema: dict[str, list] = yaml.safe_load(file)
+
+        res_df = pd.DataFrame()
+
+        # Do a cartesian product of all the choices in the schema
+        # and try to read the corresponding CSV files.
+        for choice in itertools.product(*self.schema.values()):
+            filepath = f"{data_dir}/{'_'.join(choice)}_benchmark.csv"
+            with contextlib.suppress(FileNotFoundError):
+                df = pd.read_csv(filepath)
+                for key, val in zip(self.schema.keys(), choice):
+                    df.insert(1, key, val)
+                res_df = pd.concat([res_df, df])
+
+        if res_df.empty:
+            raise ValueError(f"No benchmark CSV files were read from {data_dir=}.")
+
+        return pd.merge(res_df, df_score, on=["model"]).round(2)
+
+    def _format_msg(self, text: str) -> str:
+        """Formats into HTML that prints in Monospace font."""
+        return f"<pre style='font-family: monospace'>{text}</pre>"
+
+    def add_column(self, column_name: str, formula: str):
+        """Create and add a new column with the given formula."""
+        # If the user did not provide the name of the new column,
+        # generate a unique name for them.
+        if not column_name:
+            counter = 1
+            while (column_name := f"custom{counter}") in self.full_df.columns:
+                counter += 1
+
+        # If the user did not provide a formula, return an error message.
+        if not formula:
+            return self.cur_df, self._format_msg("Please enter a formula.")
+
+        # If there is an equal sign in the formula, `df.eval` will
+        # return an entire DataFrame with the new column, instead of
+        # just the new column. This is not what we want, so we check
+        # for this case and return an error message.
+        if "=" in formula:
+            return self.cur_df, self._format_msg("Invalid formula: expr cannot contain '='.")
+
+        # The user may want to update an existing column.
+        verb = "Updated" if column_name in self.full_df.columns else "Added"
+
+        # Evaluate the formula and catch any error.
+        try:
+            col = self.full_df.eval(formula)
+            if isinstance(col, pd.Series):
+                col = col.round(2)
+            self.full_df[column_name] = col
+        except Exception as exc:
+            return self.cur_df, self._format_msg(f"Invalid formula: {exc}")
+
+        # If adding a column succeeded, `self.cur_df` should also be updated.
+        self.cur_df = self.full_df.loc[self.cur_index]
+        return self.cur_df, self._format_msg(f"{verb} column '{column_name}'.")
+
+    def get_dropdown(self):
+        columns = self.full_df.columns.tolist()[1:] # include gpu and task in the dropdown
+        return [
+            gr.Dropdown(choices=columns, label="X"),
+            gr.Dropdown(choices=columns, label="Y"),
+            gr.Dropdown(choices=columns, label="Z (optional)"),
+        ]
+
+    def update_dropdown(self):
+        columns = self.full_df.columns.tolist()[1:]
+        dropdown_update = gr.Dropdown.update(choices=columns)
+        return [dropdown_update] * 3
+
+    def set_filter_get_df(self, *filters):
+        """Set the current set of filters and return the filtered DataFrame."""
+        index = np.full(len(self.full_df), True)
+        for setup, choice in zip(self.schema, filters):
+            index = index & self.full_df[setup].isin(choice)
+        self.cur_df = self.full_df.loc[index]
+        self.cur_index = index
+        return self.cur_df
+
+    def plot_scatter(self, width, height, x, y, z):
+        # The user did not select either x or y.
+        if not x or not y:
+            return None, width, height, self._format_msg("Please select both X and Y.")
+
+        # Width and height may be an empty string. Then we set them to 600.
+        if not width and not height:
+            width, height = "600", "600"
+        elif not width:
+            width = height
+        elif not height:
+            height = width
+        try:
+            width, height = int(width), int(height)
+        except ValueError:
+            return None, width, height, self._format_msg("Width and height should be positive integers.")
+
+        # Strip the <a> tag from model names.
+        text = self.cur_df["model"].apply(lambda x: x.split(">")[1].split("<")[0])
+        if z is None or z == "None" or z == "":
+            fig = px.scatter(self.cur_df, x=x, y=y, text=text)
+        else:
+            fig = px.scatter_3d(self.cur_df, x=x, y=y, z=z, text=text)
+        fig.update_traces(textposition="top center")
+        fig.update_layout(width=width, height=height)
+
+        return fig, width, height, ""
+
+
+# Find the latest version of the CSV files in data/
+# and initialize the global TableManager.
+latest_date = sorted(os.listdir("data/"))[-1]
+
+# The global instance of the TableManager should only be used when
+# initializing components in the Gradio interface. If the global instance
+# is mutated while handling user sessions, the change will be reflected
+# in every user session. Instead, the instance provided by gr.State should
+# be used.
+global_tbm = TableManager(f"data/{latest_date}")
+
+# Custom JS.
+# XXX: This is a hack to make the model names clickable.
+#      Ideally, we should set `datatype` in the constructor of `gr.DataFrame` to
+#      `["markdown"] + ["number"] * (len(df.columns) - 1)` and format models names
+#      as an HTML <a> tag. However, because we also want to dynamically add new
+#      columns to the table and Gradio < 4.0 does not support updating `datatype` with
+#      `gr.DataFrame.update` yet, we need to manually walk into the DOM and replace
+#      the innerHTML of the model name cells with dynamically interpreted HTML.
+#      Desired feature tracked at https://github.com/gradio-app/gradio/issues/3732
+dataframe_update_js = f"""
+function format_model_link() {{
+    // Iterate over the cells of the first column of the leaderboard table.
+    for (let index = 1; index <= {len(global_tbm.full_df)}; index++) {{
+        // Get the cell.
+        var cell = document.querySelector(
+            `#tab-leaderboard > div > div > div > table > tbody > tr:nth-child(${{index}}) > td:nth-child(1) > div > span`
+        );
+
+        // If nothing was found, it likely means that now the visible table has less rows
+        // than the full table. This happens when the user filters the table. In this case,
+        // we should just return.
+        if (cell == null) break;
+
+        // This check exists to make this function idempotent.
+        // Multiple changes to the Dataframe component may invoke this function,
+        // multiple times to the same HTML table (e.g., adding and sorting cols).
+        // Thus, we check whether we already formatted the model names by seeing
+        // whether the child of the cell is a text node. If it is not,
+        // it means we already parsed it into HTML, so we should just return.
+        if (cell.firstChild.nodeType != 3) break;
+
+        // Decode and interpret the innerHTML of the cell as HTML.
+        var decoded_string = new DOMParser().parseFromString(cell.innerHTML, "text/html").documentElement.textContent;
+        var temp = document.createElement("template");
+        temp.innerHTML = decoded_string;
+        var model_anchor = temp.content.firstChild;
+
+        // Replace the innerHTML of the cell with the interpreted HTML.
+        cell.replaceChildren(model_anchor);
+    }}
+
+    // Return all arguments as is.
+    return arguments
+}}
+"""
+
+# Custom CSS.
+css = """
+/* Make ML.ENERGY look like a clickable logo. */
+.text-logo {
+    color: #27cb63 !important;
+    text-decoration: none !important;
+}
+
+/* Make the submit button the same color as the logo. */
+.btn-submit {
+    background: #27cb63 !important;
+    color: white !important;
+    border: 0 !important;
+}
+
+/* Center the plotly plot inside its container. */
+.plotly > div {
+    margin: auto !important;
+}
+
+/* Limit the width of the first column to 300 px. */
+table td:first-child,
+table th:first-child {
+    max-width: 300px;
+    overflow: auto;
+    white-space: nowrap;
+}
+"""
+
+block = gr.Blocks(css=css)
+with block:
+    tbm = gr.State(global_tbm)  # type: ignore
+    gr.HTML("<h1><a href='https://ml.energy' class='text-logo'>ML.ENERGY</a> Leaderboard</h1>")
+
+    with gr.Tabs():
+        # Tab 1: Leaderboard.
+        with gr.TabItem("Leaderboard"):
+            with gr.Row():
+                with gr.Box():
+                    gr.Markdown("## Select benchmark parameters")
+                    checkboxes = []
+                    for key, choices in global_tbm.schema.items():
+                        # Specifying `value` makes everything checked by default.
+                        checkboxes.append(gr.CheckboxGroup(choices=choices, value=choices, label=key))
+
+            # Block 1: Leaderboard table.
+            with gr.Row():
+                dataframe = gr.Dataframe(type="pandas", elem_id="tab-leaderboard")
+            # Make sure the models have clickable links.
+            dataframe.change(None, None, None, _js=dataframe_update_js)
+            # Table automatically updates when users check or uncheck any checkbox.
+            for checkbox in checkboxes:
+                checkbox.change(TableManager.set_filter_get_df, inputs=[tbm, *checkboxes], outputs=dataframe)
+
+            # Block 2: Allow users to add new columns.
+            with gr.Row():
+                with gr.Column(scale=3):
+                    with gr.Row():
+                        colname_input = gr.Textbox("power", lines=1, label="Custom column name")
+                        formula_input = gr.Textbox("energy/latency", lines=1, label="Formula")
+                with gr.Column(scale=1):
+                    with gr.Row():
+                        add_col_btn = gr.Button("Add to table (⏎)", elem_classes=["btn-submit"])
+                    with gr.Row():
+                        clear_input_btn = gr.Button("Clear")
+            with gr.Row():
+                add_col_message = gr.HTML("")
+            colname_input.submit(
+                TableManager.add_column,
+                inputs=[tbm, colname_input, formula_input],
+                outputs=[dataframe, add_col_message],
+            )
+            formula_input.submit(
+                TableManager.add_column,
+                inputs=[tbm, colname_input, formula_input],
+                outputs=[dataframe, add_col_message],
+            )
+            add_col_btn.click(
+                TableManager.add_column,
+                inputs=[tbm, colname_input, formula_input],
+                outputs=[dataframe, add_col_message],
+            )
+            clear_input_btn.click(
+                lambda: (None, None, None),
+                inputs=None,
+                outputs=[colname_input, formula_input, add_col_message],
+            )
+
+            # Block 3: Allow users to plot 2D and 3D scatter plots.
+            with gr.Row():
+                with gr.Column(scale=3):
+                    with gr.Row():
+                        # Initialize the dropdown choices with the global TableManager with just the original columns.
+                        axis_dropdowns = global_tbm.get_dropdown()
+                with gr.Column(scale=1):
+                    with gr.Row():
+                        plot_btn = gr.Button("Plot", elem_classes=["btn-submit"])
+                    with gr.Row():
+                        clear_plot_btn = gr.Button("Clear")
+            with gr.Accordion("Plot size (600 x 600 by default)", open=False):
+                with gr.Row():
+                    plot_width_input = gr.Textbox("600", lines=1, label="Width (px)")
+                    plot_height_input = gr.Textbox("600", lines=1, label="Height (px)")
+            with gr.Row():
+                plot = gr.Plot()
+            with gr.Row():
+                plot_message = gr.HTML("")
+            add_col_btn.click(TableManager.update_dropdown, inputs=tbm, outputs=axis_dropdowns)  # type: ignore
+            plot_width_input.submit(
+                TableManager.plot_scatter,
+                inputs=[tbm, plot_width_input, plot_height_input, *axis_dropdowns],
+                outputs=[plot, plot_width_input, plot_height_input, plot_message],
+            )
+            plot_height_input.submit(
+                TableManager.plot_scatter,
+                inputs=[tbm, plot_width_input, plot_height_input, *axis_dropdowns],
+                outputs=[plot, plot_width_input, plot_height_input, plot_message],
+            )
+            plot_btn.click(
+                TableManager.plot_scatter,
+                inputs=[tbm, plot_width_input, plot_height_input, *axis_dropdowns],
+                outputs=[plot, plot_width_input, plot_height_input, plot_message],
+            )
+            clear_plot_btn.click(
+                lambda: (None,) * 7,
+                None,
+                outputs=[*axis_dropdowns, plot, plot_width_input, plot_height_input, plot_message],
+            )
+
+            # Block 4: Leaderboard date.
+            with gr.Row():
+                gr.HTML(f"<h3 style='color: gray'>Date: {latest_date}</h3>")
+
+        # Tab 2: About page.
+        with gr.TabItem("About"):
+            # Read in LEADERBOARD.md
+            gr.Markdown(open("LEADERBOARD.md").read())
+
+    # Load the table on page load.
+    block.load(lambda tbm: tbm.full_df, inputs=tbm, outputs=dataframe)
+
+block.launch()

data/2023-06-17/A40_chat-concise_benchmark.csv

@@ -0,0 +1,21 @@
+model,throughput,response_length,latency,energy
+lmsys/vicuna-7B,29.829026631206286,271.9100067159167,9.132098360456242,2143.52561215591
+StabilityAI/stablelm-tuned-alpha-7b,26.57393014131587,255.54365345869712,9.397491535841587,2439.293790463411
+databricks/dolly-v2-12b,15.273711386107992,141.4445936870383,8.891394107289802,2095.3922259908463
+tatsu-lab/alpaca-7B,29.852723929321897,121.29281396910679,4.0361613936664735,1080.58840899923
+camel-ai/CAMEL-13B-Combined-Data,17.460563027552272,283.4543317662861,16.28615281731328,4262.537929482914
+BAIR/koala-7b,29.888796737616882,251.45399597044997,8.385359924898546,1940.5972622565405
+h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b-preview-300bt-v2,29.57083939186151,212.1212222968435,7.140702722639097,1524.9156188717134
+lmsys/vicuna-13B,17.282398545945153,269.62491605104094,15.6926107484118,4203.671348891897
+togethercomputer/RedPajama-INCITE-7B-Chat,14.451529449594792,275.07991940899933,18.294811550193185,2937.839604096736
+metaai/llama-13B,15.493654667854246,81.26796507723304,4.881042191492302,1264.815973472223
+BAIR/koala-13b,17.393931641363825,252.56816655473472,14.499323956849762,3747.8785020146615
+nomic-ai/gpt4all-13b-snoozy,17.45953616124214,217.35325721961047,12.440528350608277,3263.628521155058
+BlinkDL/RWKV-4-Raven-7B-v12-Eng98%-Other2%-20230521-ctx8192.pth,32.86455306586691,235.19274680993956,6.718876629108356,1661.2857568838062
+lmsys/fastchat-t5-3b-v1.0,21.09615171109894,313.09905977165886,18.366778339359637,1807.6800728676938
+project-baize/baize-v2-7B,28.92598212176896,321.06010745466756,10.940218308832323,2644.9160527197046
+OpenAssistant/oasst-sft-1-pythia-12b,16.01484723680571,249.1007387508395,15.153340834740217,3829.1071417058643
+metaai/llama-7B,25.80475014752762,63.463734049697784,2.2525196486312047,539.0479066487654
+Neutralzz/BiLLa-7B-SFT,29.382300021941255,141.6155137676293,4.84122748247456,1131.9990564138398
+openaccess-ai-collective/manticore-13b-chat-pyg,17.220798012743607,268.91269308260576,15.692034786355059,4051.8244570182064
+FreedomIntelligence/phoenix-inst-chat-7b,32.33242374435414,229.95869711215582,6.910495058340042,2049.7076356614534

data/2023-06-17/A40_chat_benchmark.csv

@@ -0,0 +1,21 @@
+model,throughput,response_length,latency,energy
+lmsys/vicuna-7B,30.071255281840546,284.27266621893887,9.474226236086707,2239.068904633984
+lmsys/vicuna-13B,17.50774908972375,281.298522498321,16.096842177613397,4265.287245130957
+tatsu-lab/alpaca-7B,30.09713731797294,125.20013431833445,4.129986896187982,916.045386501007
+metaai/llama-7B,25.768609507174105,64.59032907991941,2.284814629996714,525.7081235728675
+metaai/llama-13B,15.699146010424393,80.32236400268637,4.757332595030835,1293.689832437891
+camel-ai/CAMEL-13B-Combined-Data,17.40620018812374,292.3438549361988,16.834190191676036,4466.796722968406
+BlinkDL/RWKV-4-Raven-7B-v12-Eng98%-Other2%-20230521-ctx8192.pth,33.10830960148045,243.21793149764943,6.9481068778416555,1833.7241615177682
+databricks/dolly-v2-12b,15.597444626791148,148.3270651443922,9.168758730287117,2362.087664204047
+FreedomIntelligence/phoenix-inst-chat-7b,32.663340053939855,243.14909335124244,7.271332307256473,2149.2483156478947
+h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b-preview-300bt-v2,28.851651162429675,216.66286098052385,7.544740398256815,1636.1981326393268
+lmsys/fastchat-t5-3b-v1.0,21.694893194918894,312.84116856950976,17.951570049172194,1787.5060366017656
+Neutralzz/BiLLa-7B-SFT,29.49201862368961,159.29986568166555,5.443799112468728,1218.644757555166
+nomic-ai/gpt4all-13b-snoozy,17.46230398293782,250.1742780389523,14.322901371942146,4093.901904969787
+openaccess-ai-collective/manticore-13b-chat-pyg,17.485883513135143,289.58697112155807,16.594830177599892,4316.488665547325
+OpenAssistant/oasst-sft-1-pythia-12b,16.056643548610985,254.26259234385495,15.462307354021265,3891.8823989257867
+project-baize/baize-v2-7B,29.004360284420006,324.24546675621224,11.011670755046683,2621.3502615853154
+BAIR/koala-7b,29.723806931945834,260.7196104768301,8.720630589929986,2017.3295624580246
+BAIR/koala-13b,17.451436035057224,262.5295500335796,15.030911340299886,3827.6102800537265
+StabilityAI/stablelm-tuned-alpha-7b,26.413142361637988,255.34687709872398,9.454673889303727,2319.91146675621
+togethercomputer/RedPajama-INCITE-7B-Chat,21.410571862447824,279.5094022834117,12.506414288534286,2541.441298522497

data/2023-06-17/A40_instruct-concise_benchmark.csv

@@ -0,0 +1,21 @@
+model,throughput,response_length,latency,energy
+openaccess-ai-collective/manticore-13b-chat-pyg,17.4993855646115,229.5795836131632,13.132503049058466,3501.182491605137
+lmsys/vicuna-7B,29.046593546528904,212.760241773002,7.3296203452423265,1706.1712568838818
+BlinkDL/RWKV-4-Raven-7B-v12-Eng98%-Other2%-20230521-ctx8192.pth,33.07481929862108,242.74177300201478,6.943508281124177,1787.5870651446628
+tatsu-lab/alpaca-7B,29.475397017987323,117.95399597044997,3.9749026008906023,927.7634805239352
+metaai/llama-13B,15.786345111463364,102.35762256548018,6.009854743435423,1590.8409496307413
+OpenAssistant/oasst-sft-1-pythia-12b,16.03459728484094,241.31732706514438,14.686811677200872,3673.6327222969167
+BAIR/koala-7b,29.75360600658546,200.51544660846204,6.658448294727219,1426.3522693082625
+databricks/dolly-v2-12b,15.330621441395213,149.7411014103425,9.354867176525545,2240.8669214236447
+togethercomputer/RedPajama-INCITE-7B-Chat,20.849945627496787,275.1017461383479,12.625316554842463,2521.5761004031265
+BAIR/koala-13b,17.3129078938621,185.43653458697113,10.592529783475095,3058.9324654130387
+metaai/llama-7B,26.263780879593444,96.94089993284084,3.35252434620665,871.9958969106643
+lmsys/vicuna-13B,17.391771626218006,199.80960376091338,11.55835909586615,3031.3180846202845
+camel-ai/CAMEL-13B-Combined-Data,17.27426964902266,194.22028206850234,11.174119858559074,2956.610076225849
+FreedomIntelligence/phoenix-inst-chat-7b,32.63089093979437,197.44492948287441,5.895621189552368,1736.5409079919812
+StabilityAI/stablelm-tuned-alpha-7b,26.523547972773766,244.16588314304903,8.984752658064695,2305.969679315211
+h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b-preview-300bt-v2,29.926231643904146,223.85426460711886,7.510739674653152,1676.2203126259078
+Neutralzz/BiLLa-7B-SFT,29.118626503392385,104.97817327065144,3.5443721553023035,818.274197783844
+nomic-ai/gpt4all-13b-snoozy,17.423064750595767,135.3938885157824,7.734149922101941,1871.6546057756862
+project-baize/baize-v2-7B,28.13796712305154,262.9902619207522,9.250474432119292,2105.324460711873
+lmsys/fastchat-t5-3b-v1.0,40.20822673632634,281.74110141034254,10.492163513616964,1110.3276249158694

data/2023-06-17/A40_instruct_benchmark.csv

@@ -0,0 +1,21 @@
+model,throughput,response_length,latency,energy
+FreedomIntelligence/phoenix-inst-chat-7b,32.795664087070854,221.2484889187374,6.588933942256567,1863.514234721291
+tatsu-lab/alpaca-7B,30.107577299286163,126.36030893216925,4.161682809197595,973.6026363331109
+togethercomputer/RedPajama-INCITE-7B-Chat,17.009700321585225,282.3190060443251,15.98330062659441,2834.287281396827
+lmsys/vicuna-7B,29.417977025894693,267.841840161182,9.164755312435684,2131.3740241775145
+BlinkDL/RWKV-4-Raven-7B-v12-Eng98%-Other2%-20230521-ctx8192.pth,33.80171881884355,264.9563465413029,7.560664676534496,2049.7698284082962
+databricks/dolly-v2-12b,15.67950302952103,155.61316319677636,9.582122375200395,2369.283402619204
+camel-ai/CAMEL-13B-Combined-Data,17.522554791478672,245.7824042981867,14.081241566503387,3646.9116689053053
+BAIR/koala-7b,29.350583449996343,253.7239758226998,8.64835721658589,1918.897159502941
+openaccess-ai-collective/manticore-13b-chat-pyg,17.267666593018745,276.03559435862996,16.03621509688224,4113.539149429272
+OpenAssistant/oasst-sft-1-pythia-12b,14.438332974888972,253.76796507723304,17.249506058312747,3936.226839825601
+h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b-preview-300bt-v2,29.207313503768304,233.47951645399598,8.096591130254916,1804.4821860309694
+project-baize/baize-v2-7B,28.47516339265779,306.76561450638013,10.688193155492014,2415.801835795991
+StabilityAI/stablelm-tuned-alpha-7b,23.120196395716167,244.85930154466084,10.369934308136857,2445.444213566122
+lmsys/vicuna-13B,17.595105665816288,263.95567494963063,15.050040050311223,3967.4957498321023
+Neutralzz/BiLLa-7B-SFT,28.937231313361377,142.33848220282067,4.7632941655637016,1177.3565590999485
+metaai/llama-13B,15.747651109641996,101.69375419744796,5.970782866386873,1693.432888515849
+lmsys/fastchat-t5-3b-v1.0,31.014371537480102,357.13734049697786,17.964342393854206,1758.7082199462513
+nomic-ai/gpt4all-13b-snoozy,17.558360268154225,232.67461383478846,13.290953806575821,3411.2449123573792
+BAIR/koala-13b,17.468010116614902,254.08529214237743,14.4913390549458,3858.416870718604
+metaai/llama-7B,26.40244189851013,104.19308260577569,3.608983782098236,864.4181752854275

data/2023-06-17/models.json

@@ -0,0 +1,102 @@
+{
+    "lmsys/vicuna-7B": {
+        "url": "https://lmsys.org/blog/2023-03-30-vicuna/",
+        "nickname": "LMSys/vicuna-7B",
+        "params": 7
+    },
+    "lmsys/vicuna-13B": {
+        "url": "https://lmsys.org/blog/2023-03-30-vicuna/",
+        "nickname": "LMSys/vicuna-13B",
+        "params": 13
+    },
+    "tatsu-lab/alpaca-7B": {
+        "url": "https://huggingface.co/tatsu-lab/alpaca-7b-wdiff",
+        "nickname": "tatsu-lab/alpaca-7B",
+        "params": 7
+    },
+    "metaai/llama-7B": {
+        "url": "https://github.com/facebookresearch/llama",
+        "nickname": "MetaAI/LLaMA-7B",
+        "params": 7
+    },
+    "metaai/llama-13B": {
+        "url": "https://github.com/facebookresearch/llama",
+        "nickname": "MetaAI/LLaMA-13B",
+        "params": 13
+    },
+    "camel-ai/CAMEL-13B-Combined-Data": {
+        "url": "https://huggingface.co/camel-ai/CAMEL-13B-Combined-Data",
+        "nickname": "Camel-AI/CAMEL-13B-Combined-Data",
+        "params": 13
+    },
+    "BlinkDL/RWKV-4-Raven-7B-v12-Eng98%-Other2%-20230521-ctx8192.pth": {
+        "url": "https://huggingface.co/BlinkDL/rwkv-4-raven",
+        "nickname": "BlinkDL/RWKV-4-Raven-7B",
+        "params": 7
+    },
+    "databricks/dolly-v2-12b": {
+        "url": "https://huggingface.co/databricks/dolly-v2-12b",
+        "nickname": "databricks/dolly-v2-12B",
+        "params": 12
+    },
+    "FreedomIntelligence/phoenix-inst-chat-7b": {
+        "url": "https://huggingface.co/FreedomIntelligence/phoenix-inst-chat-7b",
+        "nickname": "FreedomIntelligence/phoenix-inst-chat-7b",
+        "params": 7
+    },
+    "h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b-preview-300bt-v2": {
+        "url": "https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b-preview-300bt-v2",
+        "nickname": "H2OAI/H2OGPT-oasst1-7B",
+        "params": 7
+    },
+    "lmsys/fastchat-t5-3b-v1.0": {
+        "url": "https://huggingface.co/lmsys/fastchat-t5-3b-v1.0",
+        "nickname": "LMSys/fastchat-t5-3b-v1.0",
+        "params": 3
+    },
+    "Neutralzz/BiLLa-7B-SFT": {
+        "url": "https://huggingface.co/Neutralzz/BiLLa-7B-SFT",
+        "nickname": "Neutralzz/BiLLa-7B-SFT",
+        "params": 7
+    },
+    "nomic-ai/gpt4all-13b-snoozy": {
+        "url": "https://huggingface.co/nomic-ai/gpt4all-13b-snoozy",
+        "nickname": "nomic-ai/gpt4all-13b-snoozy",
+        "params": 13
+    },
+    "openaccess-ai-collective/manticore-13b-chat-pyg": {
+        "url": "https://huggingface.co/openaccess-ai-collective/manticore-13b-chat-pyg",
+        "nickname": "openaccess-ai-collective/manticore-13b-chat-pyg",
+        "params": 13
+    },
+    "OpenAssistant/oasst-sft-1-pythia-12b": {
+        "url": "https://huggingface.co/OpenAssistant/oasst-sft-1-pythia-12b",
+        "nickname": "OpenAssistant/oasst-sft-1-pythia-12b",
+        "params": 12
+    },
+    "project-baize/baize-v2-7B": {
+        "url": "https://huggingface.co/project-baize/baize-v2-7B",
+        "nickname": "project-baize/baize-v2-7B",
+        "params": 7
+    },
+    "BAIR/koala-7b": {
+        "url": "https://bair.berkeley.edu/blog/2023/04/03/koala/",
+        "nickname": "BAIR/koala-7b",
+        "params": 7
+    },
+    "BAIR/koala-13b": {
+        "url": "https://bair.berkeley.edu/blog/2023/04/03/koala/",
+        "nickname": "BAIR/koala-13b",
+        "params": 13
+    },
+    "StabilityAI/stablelm-tuned-alpha-7b": {
+        "url": "https://huggingface.co/StabilityAI/stablelm-tuned-alpha-7b",
+        "nickname": "StabilityAI/stablelm-tuned-alpha-7b",
+        "params": 7
+    },
+    "togethercomputer/RedPajama-INCITE-7B-Chat": {
+        "url": "https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Chat",
+        "nickname": "togethercomputer/RedPajama-INCITE-7B-Chat",
+        "params": 7
+    }
+}

data/2023-06-17/schema.yaml

@@ -0,0 +1,2 @@
+gpu: ["A40"]
+task: ["chat", "chat-concise", "instruct", "instruct-concise"]

data/2023-06-17/score.csv

@@ -0,0 +1,21 @@
+model,lmsys_elo
+lmsys/vicuna-7B,1007
+lmsys/vicuna-13B,1054
+tatsu-lab/alpaca-7B,NaN
+metaai/llama-7B,NaN
+metaai/llama-13B,854
+camel-ai/CAMEL-13B-Combined-Data,NaN
+BlinkDL/RWKV-4-Raven-7B-v12-Eng98%-Other2%-20230521-ctx8192.pth,NaN
+databricks/dolly-v2-12b,866
+FreedomIntelligence/phoenix-inst-chat-7b,NaN
+h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b-preview-300bt-v2,NaN
+lmsys/fastchat-t5-3b-v1.0,941
+Neutralzz/BiLLa-7B-SFT,NaN
+nomic-ai/gpt4all-13b-snoozy,NaN
+openaccess-ai-collective/manticore-13b-chat-pyg,NaN
+OpenAssistant/oasst-sft-1-pythia-12b,921
+project-baize/baize-v2-7B,NaN
+BAIR/koala-7b,NaN
+BAIR/koala-13b,980
+StabilityAI/stablelm-tuned-alpha-7b,882
+togethercomputer/RedPajama-INCITE-7B-Chat,NaN

index.html

@@ -0,0 +1,12 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <title>ML.ENERGY Leaderboard</title>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+    <script type="module" src="https://gradio.s3-us-west-2.amazonaws.com/3.23.0/gradio.js"></script>
+  </head>
+  <body>
+    <gradio-app src="https://symbioticlab-ml-energy-leaderboard.hf.space?__theme=light"></gradio-app>
+  </body>
+</html>

requirements-benchmark.txt

@@ -0,0 +1,5 @@
+zeus-ml
+fschat==0.2.14
+rwkv==0.7.5
+einops
+tyro

requirements.txt

@@ -1,5 +1,2 @@
-zeus-ml
-fschat==0.2.14
-rwkv==0.7.5
-einops
-tyro
+plotly==5.15.0
+gradio==3.35.2