Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 10.1k

Running on CPU Upgrade

Yeah, seems like this has failed as well: https://huggingface.co/migtissera/SynthIA-7B-v1.3

This also failed: https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/akjindal53244/Mistral-7B-v0.1-Open-Platypus_eval_request_False_bfloat16_Original.json

Hi ! The problem has been fixed and your models are being re-run :)

Hi @SaylorTwift , can you rerun these too. Thanks.

• https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/Weyaxi/OpenOrca-Nebula-7B_eval_request_False_float16_Original.json

• https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/Weyaxi/TekniumAiroboros-Nebula-7B_eval_request_False_float16_Original.json

• https://huggingface.co/datasets/open-llm-leaderboard/requests/blob/main/Weyaxi/Dolphin-Nebula-7B_eval_request_False_float16_Original.json

@SaylorTwift : To understand this mystery behind Mistral specific fine tuned model failing in evaluation. Could you please provide some details why and how the Mistral specific fine tuned models are failing in evaluation on LB submission.
I submitted my model same time with others and looks like it failed too and as usual, I can’t submit it again.
pankajmathur/Mistral-7B-model_45k6e2e4

It didnt get added to the leaderboard and now says I already submitted when trying to resubmit:
https://huggingface.co/teknium/CollectiveCognition-v1.1-Mistral-7B
https://huggingface.co/teknium/CollectiveCognition-v1-Mistral-7B
https://huggingface.co/teknium/Mistral-Trismegistus-7B

Same for me.

@pankajmathur Sure ! It is because mistral uses a new model architecture that was added in transformers 4.34.0 and the leaderboard was running on 4.33.0, simple as that :)

@Weyaxi @teknium your models were in fact already submitted but failed for the reason mentioned above, I change your models' status to PENDING so they should be in the queue again. However, because we are conducting experiments on the leaderboard right now, it might take some time for your models to be evaluated. Thanks for your patience :)

I'm going to close this issue in the meantime, but feel free to reopen if your models have another problem after evaluation!