Feature request: Hide models with insufficient model card from default view in leaderboard

#762
by ThiloteE - opened

Define certain criteria and then hide them. Please. Just requiring a few simple criteria that should go a long way to clean up this mess:

  • Submitting an empty default readme template should not be sufficient.
  • The name of the base model should be somewhere in the configuration files. → Add to search feature
  • The parameter count should be somewhere in the configuration files
  • The number of experts should be somewhere in the configuration files → Add to search feature
    ...
Open LLM Leaderboard org
edited May 29

Hi!
We already request model READMEs to have a minimal number of lines - finding more detailed information is out of scope on our side for this version of the leaderboard, but you are welcome to open a PR if you want to contribute this! :)
For the parameter count, it's not needed if the model is uploaded in safetensors as it can easily be extracted from there. For the rest, it will need to have a good logic, as not all models have experts for example.

clefourrier changed discussion status to closed

My first point was refering to model cards like these: https://huggingface.co/netcat420/MFANN3bv0.3
image.png

Is there not a way to detect, if the automatic generated model card is used as is without modification? Or is this actually a good way to detect model for flagging?

Open LLM Leaderboard org

We could add this to our checks for flagging and look for the [optional] and [More information Needed] for example

clefourrier changed discussion status to open

Might be worth a try.

clefourrier changed discussion title from Hide models with insufficient model card from default view in leaderboard to Feature request: Hide models with insufficient model card from default view in leaderboard

Sign up or log in to comment