Repository: stanford-crfm/helm
Issue: Create a Leaderboard
Root cause: Unimplemented Feature Gap

It is an amazing framework for evaluating LLMs. But there isn't a default Leaderboard, this makes the framework less popular compared to Open LLM Leaderboard, Chat Arena, etc.

I would recommend creating a Leaderboard to increase the popularity of this framework. One possible solution would be to use an average of AR2_Challenge, Hellaswag, MMLU, and TruthfulQA for the leaderboard similar to Open LLM Leaderboard. It would be great to add the AR2_Challenge as proposed in https://github.com/stanford-crfm/helm/issues/1695.

Open LLM Leaderboard is updating their MMLU evaluation https://huggingface.co/blog/evaluating-mmlu-leaderboard. I found that MMLU (original) and MMLU (helm) scores are almost the same even though the evaluation methods are different. I am wondering if this is the same for other tasks. If yes, then the results from this leaderboard and Open LLM Leaderboard could be directly compared, or at least as a good reference.



That's a great suggestion!

We already have pseudo-leaderboards: the main one is [this page](https://crfm.stanford.edu/helm/latest/?group=core_scenarios), where each of the aspects (Accuracy, Calibration, etc.) can be considered a separate leaderboard where the models are ranked by that aspect.

We are missing some things that a leaderboard would have, e.g.:

- Visual design and visualizations that makes it look like a leaderboard (see: Papers With Code, Quac, Dynabench, etc.)
- Single top-level metric per aspect (win-rate does not fit the bill because it's a relative metric)
- Better explanatory notes
- Better discoverability

In terms of comparability - this is tricky because we've made lots of small decisions in HELM that diverge from other implementations.

For instance, for MMLU, reproduction would mean running all the subjects (#1335), using the same adapter (#1557), and macro-averaging in the same way (#1328).

I am quite surprised that the scores are so similar though, especially since subject subset choice has a massive impact on the metrics.

@yifanmai This pseudo-leaderboard is way too complex. It is a good reference for professionals, but not good for the general public. It would be good if the leaderboard could be clean and nice.

It is not that surprising that the scores are similar. MMLU (original) calculates the highest accuracy in ["A", "B", "C", "D"], while MMLU (HELM) calculates the highest accuracy in all tokens. A decent LLM should almost predict multiple choice questions in ["A", "B", "C", "D"], so the scores are similar. Actually, MMLU (HELM) should be slightly smaller than MMLU (original).

Definitely agreed that we need to do a better job of summarizing information for the general public.

I'll have to take a look at what OpenLM leaderboard does, but we currently only evaluate 5 of the MMLU subjects, rather than doing a complete evaluation. The accuracy can be change drastically from subject to subject.

@yifanmai Can I know which 5 subjects are used for MMLU HELM evaluation? Open LLM Leaderboard evaluates all 57 subjects. I mean the scores are similar if MMLU (original) and MMLU (HELM) are evaluated on the same subject. 

Sure, the 5 subjects are the ones labelled as priority 1 and 2 in `run_specs.conf` [here](https://github.com/stanford-crfm/helm/blob/cffe38eb2c814d054c778064859b6e1551e5e106/src/helm/benchmark/presentation/run_specs.conf#L32-L88): `abstract_algebra`, `college_chemistry`, `computer_security`, `econometrics`, `us_foreign_policy`

You can also get this information from the [MMLU page on the website](https://crfm.stanford.edu/helm/latest/?group=mmlu).

Closed because we now have an actual leaderboard: https://crfm.stanford.edu/helm/lite/latest/#/leaderboard