open-llm-leaderboard/open_llm_leaderboard · Wrong number of parameters reported in Open LLM Leaderboard

Fapb

6 days ago

Maybe I missed something or found a bug, but it seems that recently, since about December 17th 2024, the number of parameters is not being reported correctly in Open LLM Leaderboard:

https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?params=1%2C16&columns=rank%2Cmodel.type_icon%2Cid%2Cmodel.average_score%2Cevaluations.ifeval.normalized_score%2Cevaluations.bbh.normalized_score%2Cevaluations.math.normalized_score%2Cevaluations.gpqa.normalized_score%2Cevaluations.musr.normalized_score%2Cevaluations.mmlu_pro.normalized_score%2Cmetadata.submission_date%2Cmetadata.params_billions

It seems that the number reported in the table is 2 times too low.

Can you confirm or fix it?

Thanks :slight_smile:

Fapb changed discussion title from Open LLM Leaderboard - wrong number of parameters to Wrong number of parameters reported in Open LLM Leaderboard 6 days ago

brankor-mcom

6 days ago

See #1049.

clefourrier

Open LLM Leaderboard org 5 days ago

•

edited 5 days ago

Hi! Closing this issue as it's a duplicate of the other one :)
We're aware of this and Alina will work on a fix once she is back (the whole team is on holidays and I've taken over all our support across libs, so I'm focusing on the critical stuff for now)

clefourrier changed discussion status to closed 5 days ago