Spaces:

upstage

/

open-ko-llm-leaderboard

like 361

Running on CPU Upgrade

Hi! Thank you for your interest in the 🚀Open Ko-LLM Leaderboard!
Below are some common questions - if this FAQ does not answer what you need, feel free to create a new issue, and we'll take care of it as soon as we can!

• ❓The leaderboard has crashed with a connection error, help!
  This happens from time to time, and is normal, don't worry. The leaderboard will be automatically restarted in less than an hour (or earlier if one of the maintainers notices it). Please only open an issue if the leaderboard is down for longer than an hour.

• ❓Why do models appear several times in the leaderboard?
  We run evaluations with user selected precision and model commit. Sometimes, users submit specific models at different commits and at different precisions (for example, in float16 and 4bit to see how quantization affects performance). You should be able to verify this by displaying the precision and model sha columns in the display. If, however, you see models appearing several time with the same precision and hash commit, this is not normal.

• ❓Why don't you display closed source model scores?
  This is a leaderboard for Open models, both for philosophical reasons (openness is cool) and for practical reasons: we want to ensure that the results we display are accurate and reproducible, but 1) commercial closed models can change their API thus rendering any scoring at a given time incorrect 2) we re-run everything on our cluster to ensure all models are run on the same setup and you can't do that for these models.

• ❓What about models of type X?
  We only support models that have been integrated in a stable version of the transformers library for automatic submission. We are doing our best to extend the leaderboard to new models and evaluations.

• ❓My model disappeared from all the queues, what happened?
  A model disappearing from all the queues usually means that there has been a failure. You can check if that is the case by looking at your model here.

• ❓What causes an evaluation failure?
  Most of the failures we get come from problems in the submissions (corrupted files, config problems, wrong parameters selected for eval ...), so we'll be grateful if you first make sure you have followed the steps in About. However, from time to time, we have failures on our side (hardware/node failures, problem with an update of our backend, connectivity problem ending up in the results not being saved, ...). As we store the logs for all models, feel free to create an issue, where you link to the requests file of your model (look for it here), so we can investigate! If the model failed due to a problem on our side, we'll relaunch it right away!
  Note: Please do not re-upload your model under a different name, it will not help

• ❓I upgraded my model and want to re-submit, how can I do that?
  Please open an issue with the precise name of your model, and we'll remove your model from the leaderboard so you can resubmit.

• ❓What is this concept of "flagging"?
  This mechanism allows user to report models that have unfair performance on the leaderboard. This contains several categories: exceedingly good results on the leaderboard because the model was (maybe accidentally) trained on the evaluation data, models that are copy of other models not attributed properly, etc.

• ❓My model has been flagged improperly, what can I do?
  Every flagged model has a discussion associated with it - feel free to plead your case there, and we'll see what to do.