Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.8k

Running on CPU Upgrade

Hi, unless collaborating with the model's creators directly, we will not evaluate this model.

Why not?

@Nanbeige Want to be evaluated on HF leaderboard?

For all other models that do have their own code, submissions can be done by everybody.

Models that need trust_remote_code need to be carefully inspected before being added and ran by hand. This can quickly become a lot of added work for the Leaderboard team. We therefore chose to not evaluate those models unless we already partner with the team behind it.

For all other models that do have their own code, submissions can be done by everybody.

What do you mean ?

I get that. I hear there is work on enabling evaluation of trust_remote_code models, too.
To my knowledge everybody can submit models other than their own.

I think the latter makes sense. Most models are open source. You could copy the model and submit that, too, if submissions were only to be done by the publisher. This is more direct.

oh wait, I now see the typo.

For all other models that do NOT have their own code, submissions can be done by everybody.
I guess it all makes sense now.

Oh yeah sorry if it wasn't clear, you can submit anyone's models, however, for models that need special care, we only evaluate if the model is pretrained and if we have a collaboration with the team. Hope that's clearer !

Hi everyone! Thanks for your interest in our Nanbeige model.
We would love to see Nanbeige-16B-Base integrated into OpenLLM. Please see https://huggingface.co/Nanbeige/Nanbeige-16B-Base for more information. @SaylorTwift

Would you please load our Nanbeige-16B-Base by setting "trust_remote_code=True", and run the OpenLLM evaluations?
Many thanks 😊! @SaylorTwift