Spaces:

HuggingFaceH4

/

open_llm_leaderboard

like 9.69k

Running on CPU Upgrade

Inspired by Brainstorming: Suggestions for improving the leaderboard #477

• "Submit your model here" should feature a short section about decontamination efforts. In particular refer to ways how model authors can test their model for contamination.
• "Submit your model here" should feature "Read the FAQ and "Explanation of icons" in the "About" section of the leaderboard, before you submit a model" or a similar sentence. This sentence should direct to the current "knowledge base" of the leaderboard.
• The knowledge base of the leaderboard, e.g. the "About" section or the "Submit your model here" section of the leaderboard should be updated with more information about the main selectable sections of the leaderboard, but in particular some information about how the type of finetuning or model architecture (Mergers, Moergers, Pretrained, etc.) will have an effect on the leaderboard scores and hence requires differentiation. E.g. Mergers scoring consistently higher, often because of (intentional/unintentional) data contamination and its compounding effects and therefore a "merge" label is required, as comparison is exceedingly difficult. Another point would be to explain the importance of base or foundational models (called "Pretrained" on this leaderboard) and why their scores cannot easily be compared with any of the finetunes and that this is something that is ok. I (We?) probably do not want base models that are already finetuned simply to achieve high scores on the leaderboard, as that would make it harder to finetune them further, but I digress.

In short: I believe at least SOME model authors upload their models to huggingface without following the discussion section, hence are unaware of model contamination and are even more unaware, which model might potentially be contaminated. They just pick the top scoring models on the leaderboard, then do some merging and get a higher score... in in those cases the "Submit your model here" section is the easiest and most crucial place to distribute knowledge. The "About" section is important too, but less so than "Submit your model here". In general, I call for the expansion of the existing or a newly introduced and very very easy to access knowledge base.