Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 10.9k

Running on CPU Upgrade

Hi,
Thanks for your issue!
There was a problem loading models which seems to be on our side, I added your models back to the queue.

@clefourrier tyvm, but they failed again. :/

Just fyi, I also have problems loading them (and also other models like Pallas-0.5, which are already in this leaderboard) with current latest transformers release, but all works ok with the release from ~3 weeks ago: https://github.com/huggingface/transformers/tree/v4.37.2

I hope this helps...

I can't reopen: Error: Cannot change the status to open because this discussion has been closed by someone else, but you can still comment.

Interesting!
Your models failed (both times) because of an OOM error at the last step of the shards download. (One time can happen sometimes, two times at the exact same spot could be a bug).

We switched our backend to 4.38 a couple days ago, tagging @ArthurZ who works on transformers to let him know that this happen.
It's likely this will have to be moved to a discussion on transformers though.

@clefourrier Thank you! Hope it's ok to ping you again for a rerun once a new release with the fix is made.

@ArthurZ I tried other models too in order to provide better bug report and I opened an issue here: https://github.com/huggingface/transformers/issues/29252

All models that are YI 200K or derived fail in latest transformers library.
Standard Yi does not fail.
I haven't tried other models (non Yi) with very long context length.

Ofc!

@clefourrier A new release of transformers was made. If you guys updated the version used for the leaderboard, could you please rerun the above mentioned models? Thank you!

Hi @Mihaiii !
Sorry about the delay! We're going to wait for the transformers release which goes with the integration of the Cohere code (this PR), as we avoid updating the code base of the leaderboard too often (we've seen edge failure cases with silent library incompatibilities in the past).

Thanks for the update! 🙏

@Mihaiii We upgraded the front to the latest release! Should be good once it finishes building

@clefourrier That's awesome to hear! Could we please retry the Cluj models, in this case? The links are in the first message. :)

Thanks for reminding me! Passed them to pending again :)