Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.8k

Running on CPU Upgrade

Some other models also have the same availability issue. I wrote a simple script to check:

from huggingface_hub import HfApi
from datasets import load_dataset

hf_api = HfApi()

dataset = load_dataset("open-llm-leaderboard/contents", split="train", streaming=True)

for model in dataset.filter(lambda x: not x["Available on the hub"]):
    try:
        model_info = hf_api.model_info(model["fullname"], revision=model["Model sha"])
        print(model_info.private)
    except:
        print("Not Found")

As I understand, at the moment only 3 of all marked as "Available on the hub: false" are actually not on the hub.

And there seems to be quite a few models categorized in "🤝 base merges and moerges" while "Not_Merged: true", and vice versa.

Hi @CoolSpring ,

Thank you for reporting this bug, I'm going to investigate it, since your models shouldn't be under deleted/incomplete flag

Hi @CoolSpring ,

Could you please check if your models have licenses? We currently consider models without licenses as incomplete models

Hi @CoolSpring ,

Could you please check if your models have licenses? We currently consider models without licenses as incomplete models

That makes sense, thanks! I did not assign licenses to them.

A quick glimpse into open-llm-leaderboard/contents shows they are indeed marked as Available on the hub: false, Not_Merged: true (both should be the opposite).

And there seems to be quite a few models categorized in "🤝 base merges and moerges" while "Not_Merged: true", and vice versa.

Another thing I find curious is I have selected my models as "base merges and moerges" when submitting, but they are also given a contradictory label Not_Merged. Could you please also spare some time to investigate this matter?

Hi @CoolSpring ,

Thanks for adding licenses! Could you please check the Leaderboard now? Both model are hidden correctly under "Merge/Moerge" filter

Seems like the problem is solved, I close the issue. Please, feel free to ping me here in case of any questions about these models, or open a new discussion

Just see it now, thank you for your great work!