Models incorrectly labeled as gated/private/deleted

#377
by Quster - opened

I have noticed that a lot of models that are publicly available are only shown on the leaderboard with gated/private/deleted models. Some examples:

I am not sure what the reason is but with the recent change that gated/private/deleted models are now not shown by default this has become an issue. Especially with a lot of these models being recent additions that are among the best 7b parameter models on the leaderboard right now.

Good catch! I was wondering what happened to SynthIA. It's definitely available as a public model.

deleted

Can confirm. I tested several LLMs that came to mind and they're all hidden behind gated/private/deleted, including Zephyr, Dolphin, OpenHermes and even the base Mistral.

Open LLM Leaderboard org

Very good catch!
I'll investigate asap

Open LLM Leaderboard org

It should be good! Thanks for your report!

We had a requirements conflict between the version of hf_hub allowing Collections and the version of transformers allowing mistral models - so transformers had been downgraded, and any mistral model was unable to register as having a valid configuration - fixed now :)

clefourrier changed discussion status to closed

Sign up or log in to comment