Spaces:

HuggingFaceH4

/

open_llm_leaderboard

like 9.7k

Running on CPU Upgrade

Let me not open a new issue since it's related. Is 141B too big for the leaderboard?
https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/MaziyarPanahi/Goku-8x22B-v0.1_eval_request_False_float16_Original.json

Hi all!
As you can see from the job ids (-1), the jobs were not launched - this is because our backend assumes that the models have 140B activated parameters (which is too big for the cluster, hence skipped), not 140B total parameters with considerably less activated. I'm unsure there is an easy way for us to make the difference automatically at the moment, but we'll gladly update our backend and re-submit your models once we can get this information.

Hey, is this fixed now or still waiting?

Hi everyone!

Thanks to @SaylorTwift , now we can submit moe models bigger than 140B for evaluation, thus I resubmitted this one for @MaziyarPanahi

Please, provide me with the requests files for similar models to resubmit

Fantastic! Thanks @alozowski and @SaylorTwift

Here's mine: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/migtissera/Tess-2.0-Mixtral-8x22B_eval_request_False_float16_Original.json

Thanks!

Could you please rerun this as well
https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/0-hero/Matter-0.2-8x22B_eval_request_False_bfloat16_Original.json

Resubmitted both migtissera/Tess-2.0-Mixtral-8x22B and 0-hero/Matter-0.2-8x22B 👍

In that case I close this discussion, if there are any problems with models evaluations please open new ones for each model

Says FAILED

I think all 3 failed again

Yes, I created a separate discussion for my models. 2 of the failed models were 8B, so something else might have happened.

Hi everyone!

Hmm, I see, all these models have indeed failed, let me investigate

Hi @alozowski
Just as an FYI, the last attempt at my model just failed: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/MaziyarPanahi/Goku-8x22B-v0.1_eval_request_False_float16_Original.json

Hey, any update here on the Tess model? Do you want me to open a separate ticket to track it? This is the model: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/migtissera/Tess-2.0-Mixtral-8x22B_eval_request_False_float16_Original.json

There seems to be something going on with the LB eval cluster, at least for some large models. Even my Llama-3-70B submission has been running for the last 2 days. https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/MaziyarPanahi/Llama-3-70B-Instruct-DPO-v0.1_eval_request_False_bfloat16_Original.json

Hi!

We're still looking up ways to launch moe models correctly on our backend - and we've also had network failures on our cluster last week. We'll keep you posted as soon as we have updates.

@MaziyarPanahi , what you are reporting is normal and unrelated to the current issue :) When the research cluster is full, the evaluation jobs are cancelled and rescheduled, but we keep the status to "running" to keep it simple for end users. It's likely your model was "running, cancelled, rescheduled, running, ..."

Hi @clefourrier

Thanks for the update regarding MoE models, appreciate it.

but we keep the status to "running" to keep it simple for end users. It's likely your model was "running, cancelled, rescheduled, running, ..."

I didn't know that, it makes sense now. Thank you :)

Could you please resubmit Tess? https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/migtissera/Tess-2.0-Mixtral-8x22B_eval_request_False_float16_Original.json

Doing it right now, tell me if it works.

To this day, the only 8x22B models in the Leaderboard are from MistralAI. I don't believe we have ever had a successful eval on any 8x22B fine-tuned models. @clefourrier is the issue resolved and the only limitation is to find free resources? Or we still don't know if the MoE models with this size might get rejected?

These ones we launched manually when they came out because they were important for the community.
Good question, I think @SaylorTwift took a look at the backend side so I'll let him answer.
(The main problem we had was (as indicated in the title) identifying the number of activated params in MoEs.)