🏆 llm-leaderboard

A joint community effort to create one central leaderboard for LLMs. Contributions and corrections welcome!

Sources for the numbers are added as links to the table.

Interactive Dashboard

https://llm-leaderboard.streamlit.app/

How to Contribute

We are always happy for contributions! You can contribute by the following:

table work (don't forget the links):
- filling missing entries
- adding a new model as a new row to the leaderboard and add the source of the evaluation to the sources table. Please keep alphabetic order.
- adding a new benchmark as a new column in the leaderboard and add the benchmark to the benchmarks table. Please keep alphabetic order.
code work:
- improving the existing code
- requesting and implementing new features

Leaderboard

Model Name	Chatbot Arena Elo	HumanEval-Python (pass@1)	LAMBADA (zero-shot)	TriviaQA (zero-shot)
alpaca-13b	1008
cerebras-gpt-7b			0.636	0.141
cerebras-gpt-13b			0.635	0.146
chatglm-6b	985
code-cushman-001		33.5
code-davinci-002		65.8
codegen-16B-mono		29.3
codegen-16B-multi		18.3
codegx		22.9
dolly-v2-12b	944
eleuther-pythia-7b			0.667	0.198
eleuther-pythia-12b			0.704	0.233
fastchat-t5-3b	951
gpt-3.5		48.1
gpt-4		67.0
gpt-neox-20b			0.719	0.347
gptj-6b			0.683	0.234
koala-13b	1082
llama-7b		10.5	0.738	0.443
llama-13b	932	15.8
llama-33b		21.7
llama-65b		23.7
mpt-7b			0.702	0.343
oasst-pythia-12b	1065
opt-7b			0.677	0.227
opt-13b			0.692	0.282
palm-540b		26.2
stablelm-base-alpha-7b			0.533	0.049
stablelm-tuned-alpha-7b	858
starcoder-base-16B		30.4
starcoder-16B		33.6
starcoder-16B (prompted)		40.8
vicuna-13b	1169

Benchmarks

Benchmark Name	Author	Link	Description
Chatbot Arena Elo	LMSYS	https://lmsys.org/blog/2023-05-03-arena/	"In this blog post, we introduce Chatbot Arena, an LLM benchmark platform featuring anonymous randomized battles in a crowdsourced manner. Chatbot Arena adopts the Elo rating system, which is a widely-used rating system in chess and other competitive games." (Source: https://lmsys.org/blog/2023-05-03-arena/)
HumanEval	Chen et al.	https://arxiv.org/abs/2107.03374v2	"It used to measure functional correctness for synthesizing programs from docstrings. It consists of 164 original programming problems, assessing language comprehension, algorithms, and simple mathematics, with some comparable to simple software interview questions." (Source: https://paperswithcode.com/dataset/humaneval)
LAMBADA	Paperno et al.	https://arxiv.org/abs/1606.06031	"The LAMBADA evaluates the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse." (Source: https://huggingface.co/datasets/lambada)
TriviaQA	Joshi et al.	https://arxiv.org/abs/1705.03551v2	"We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions." (Source: https://arxiv.org/abs/1705.03551v2)

Sources

Author	Link
LMSYS	https://lmsys.org/blog/2023-05-03-arena/
MOSAICML	https://www.mosaicml.com/blog/mpt-7b
BigCode	https://www.bigcode-project.org/