Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 10.1k

Running on CPU Upgrade

Same! My guess is the queue is so long it doesn’t show on the page but it’s still in the queue

Hi @jaspercatapang @ariellee, what are your model names?

Hello @clefourrier , my model name is MayaPH/GodziLLa-30B. Thank you.

Hi @jaspercatapang @ariellee, what are your model names?

ariellee/SuperPlatty-30B
lilloukas/GPlatty-30B
lilloukas/Platypus-30B

@clefourrier Same here. The name is upstage/llama-30b-instruct.

Hi @jaspercatapang @ariellee @win7785
Can you make sure your models can be loaded properly using the AutoClasses? The following should not return an error for your models to be loadable.

config =  AutoConfig.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Hi @jaspercatapang @ariellee @win7785
Can you make sure your models can be loaded properly using the AutoClasses? The following should not return an error for your models to be loadable.

config =  AutoConfig.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Hello @clefourrier , modified our config files and ran the three lines with no errors. Please tell me how else I could resolve it on my end if it doesn't work as expected. Thank you for keeping us updated 😊

@jaspercatapang Did you try resubmitting after your modifications? :)

@clefourrier Yes, I resubmitted right after.

Hi @jaspercatapang @ariellee @win7785
Can you make sure your models can be loaded properly using the AutoClasses? The following should not return an error for your models to be loadable.

config =  AutoConfig.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Worked with no issues. When I submitted awhile ago I received the notice that the models were submitted for evaluation, and I figure they’re in the queue just can’t be seen on the leaderboard page since the queue is so large?

@clefourrier Thanks for checking but there are no issues to load model, tokenizer, and config from huggingface hub.

Here are examples what i used:

model_name_or_path = "upstage/llama-30b-instruct"

# load config
config = AutoConfig.from_pretrained(
    model_name_or_path, 
    use_cache=False,
)

# load model
model = AutoModel.from_pretrained(
    model_name_or_path,
    config=config,
    device_map='auto',
    torch_dtype=torch.float16,
    low_cpu_mem_usage=True,
)

# load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    model_name_or_path,
)

@clefourrier I think the display in evaluation queue doesn't seem to be working properly.
I did check from the following process:

1. git clone cerebras/Cerebras-GPT-111M model that have already been successfully evaluated.
2. upload above model to my own workspace (win7785/test)
3. submit win7785/test model but there are no updates in Pending Evaluation Queue
4. I think this model has already been evaluated successfully, so I don't think there will be any errors like loading the model, config and tokenizer using Auto*.from_pretrained method.

Also, if I try to resubmit a model that has already been submitted (cerebras/Cerebras-GPT-111M), the system displays the message "This model has already been submitted".
However, if I submit the new model (win7785/test) twice, the system will display a message like "Your request has been submitted to the evaluation queue!" both times.
Can I add a new model to the evaluation queue now?

Hi @win7785 @arielnlee @jaspercatapang !
We just pushed the update of the leaderboard (annoucement, and made sure to fix the submission queue, can you try resubmitting ?
Thank you all very much for your patience :)

@clefourrier WOO!! we are in the queue! thanks for all your hard work and the updates :)

Hi @win7785 @arielnlee @jaspercatapang !
We just pushed the update of the leaderboard (annoucement, and made sure to fix the submission queue, can you try resubmitting ?
Thank you all very much for your patience :)

@clefourrier Everything's working perfectly. Thank you for your hard work! ✺◟(＾∇＾)◞✺

I'm happy to hear that! Closing the issue :)