| TITLE = """<h1 align="center" id="space-title">🤗 Open LLM-Perf Leaderboard 🏋️</h1>""" | |
| INTRODUCTION_TEXT = f""" | |
| The 🤗 Open LLM-Perf Leaderboard 🏋️ aims to benchmark the performance (latency & throughput) of Large Language Models (LLMs) on different hardwares and backends using [Optimum-Benchmark](https://github.com/huggingface/optimum-benchmark) and [Optimum](https://github.com/huggingface/optimum) flavors. | |
| Anyone from the community can request a model or a hardware+backend configuration for automated benchmarking: | |
| - Model requests should be made in the [🤗 Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) and will be added to the 🤗 Open LLM-Perf Leaderboard 🏋️ once they're publicly available. | |
| - Hardware+Backend requests should be made in the 🤗 Open LLM-Perf Leaderboard 🏋️ [community discussions](https://huggingface.co/spaces/optimum/llm-perf-leaderboard/discussions). | |
| [Config files](https://github.com/huggingface/optimum-benchmark/blob/main/examples/bert.yaml) (which can be used with Optimum-Benchmark) will be available soon for reproduction, questioning and correction of our results. | |
| """ | |
| SINGLE_A100_TEXT = """<h3>Single-GPU (1xA100):</h3> | |
| <ul> | |
| <li>Singleton Batch (1)</li> | |
| <li>Thousand Tokens (1000)</li> | |
| </ul> | |
| """ | |
| CITATION_BUTTON_LABEL = "Copy the following snippet to cite these results." | |
| CITATION_BUTTON_TEXT = r"""@misc{open-llm-perf-leaderboard, | |
| author = {Ilyas Moutawwakil}, | |
| title = {Open LLM-Perf Leaderboard}, | |
| year = {2023}, | |
| publisher = {Hugging Face}, | |
| howpublished = "\url{https://huggingface.co/spaces/optimum/llm-perf-leaderboard}", | |
| @software{optimum-benchmark, | |
| author = {Ilyas Moutawwakil}, | |
| publisher = {Hugging Face}, | |
| title = {A framework for benchmarking the performance of Transformers models on different hardwares and backends}, | |
| } | |
| """ | |