Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 10.2k

Running on CPU Upgrade

same model with different precisions? fp16 bf16 int8 int4 gptq...

@JosephusCheung Thank you for your sharing. really helped.
however, I think one model should be shown once, not shown for every possible precisions.
I wonder how admins think

Hi @leejunhyeok ,

As indicated in our FAQ (in the About tab) of the leaderboard, some models appear several times because we evaluated them in different precision.

I think that rule should be modified @clefourrier , as some submitters could mess up leaderboard!

• if model is trained with different precisions and parameters are different, that is fine and good to go
  • in this case, model version will be different and their effort should be respected
• if model is trained with one precision and multiple submission of different precision is done, that is not fine.
  • in this case, only one(probably the best) submission should be shown

thanks

You are talking nonsense, a model can be eval with different precisions, with not pre-quantized weight and pre-quantized GPTQ, no matter how it was trained. This should not be a problem.

Hi @leejunhyeok ,

Thanks a lot for your interest!
We convert the weight precision on the fly during the evaluation (as would people wanting to use a given model at a lower precision would do). We think it's important for the community to know if a model is still performing well at a lower precision than the one it was trained on, so we'll keep the feature.

Closing this issue.