Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.7k

Running on CPU Upgrade

Hi @pankajmathur !

Thanks for your appreciation of the leaderboard!

1) Reproducibility

@SaylorTwift will be able to give you the full command to reproduce, I can already see that yours misses the "few-shot-as-multiturn" keyword argument. It's highly possible that our doc is not perfectly up to date at the moment, for which we apologize - hopefully all our PRs will be merged soon which will make everything easier to run. You also should not manually select the fewshot sample size as it's hardcoded for all our tasks.

I think you could open issues on the harness (to indicate the bugs you found with the release, if it's the latest) in order to help them with the next version, it will be very helpful :)

2) "For all the current submission (FINISHED and PENDING) there should be option to resubmit the models without loosing vote count."

I don't think we will do this, as we expect users to be careful when submitting their models to avoid us wasting compute on evaluating models in an improper fashion. Loosing the associated votes will force users to be careful, and will prevent people from submitting the same model 10 times at different precisions/with without chat templates, etc. We do not want people to abuse this system.

At the moment, the leaderboard assumes that chat models require chat templates, but if you know your model does not require them, you are free to untoggle the category at submission time.

Thank you @clefourrier for the prompt response and for the effort in going through this discussion, it's appreciated.

1. Reproducibility
   Hi @SaylorTwift , nice to meet you. when you get a chance, could you please share the full command to reproduce the v2 LB evals? I'm bit confused about what impact "few-shot-as-multiturn" argument has on the zero-shot group of tasks like leaderboard_ifeval. Why having this argument will evaluate model wrongly on zero shot task. Hence, I believe, the issue remains the same: orca_mini_v6_8b is evaluated incorrectly on LB

Here are the ifeval results and logs without the "few-shot-as-multiturn" argument:
For orca_mini_v6_8b, the LB shows a score of 1.11 out of 100.
When I ran this on 1xA100 using the command below, it shows:
Instance-Level Strict Accuracy: 0.32532347504621073 out of 1
Prompt-Level Strict Accuracy: 0.4580335731414868 out of 1

Here is the command:

pretrained_model="pankajmathur/orca_mini_v6_8b"
lm_eval --model hf --model_args pretrained=$pretrained_model,dtype=bfloat16 --tasks leaderboard_ifeval --num_fewshot 0 --batch_size=auto --device cuda:0 --output_path hf_open_llm_lb_2/leaderboard_ifeval --use_cache cache_$pretrained_model/leaderboard_ifeval

Here is the screenshot of the result:

Here is the output log:
https://huggingface.co/pankajmathur/orca_mini_v6_8b/blob/main/open-llm-lb-2-evals-log/leaderboard_ifeval/results_2024-07-09T18-34-22.230254.json

Opening Issues with lm-evaluation-harness: Yup, good idea. I'll do that for the issues which I found while running the latest (v0.4.3) release.

2. "For all the current submissions (FINISHED and PENDING), there should be an option to resubmit the models without losing vote count."
   @clefourrier : Understood, However, How do you expect any random user (including anonymous ones), who can submit models for evaluation, to understand which model is best to evaluate with chat_template and which are not?

As a matter of fact, the models in discussion "orca_mini_v6_8b" were not submitted by me so it got evaluated with the chat_template flag set to True, which was wrong way to do the evaluation for this model.
So, the enforcement of not removing wrongly submitted models for evaluation and not preserving vote counts goes against the author of the model, not against people who want to abuse the system.

It was always possible that any person, knowingly or unknowingly, could submit any model incorrectly for evaluation, including with the wrong precision, wrong model type, and now with V2, the wrong chat_template flag.
I know, We do not want people to abuse this system. but the whole voting system to prioritize the evaluation of a wrongly submitted models has to be bit more thought through and may be with more edge cases.

one very quick idea, could be to just send a confirmation notification to original author of the model that your model is submitted for evaluation with these arguments, please approve before it goes to voting or evaluation and let author decide what are the best arguments for model evaluation.
Again, There could be many other ways to handle this.

Thanks again for all the hard work.
Pankaj

Thanks for your answer!

The confirmation email is an interesting idea! We'll think about it and see how feasible it is.

Thanks @clefourrier

@SaylorTwift : I just submitted multiple PR's https://huggingface.co/datasets/open-llm-leaderboard/requests/discussions

for many of my models with:

 "use_chat_template": false

Let me know if you have questions or there is any issues accepting this PR's.

https://huggingface.co/datasets/open-llm-leaderboard/requests/tree/main/pankajmathur

Hi @pankajmathur ,
We'll be merging your PRs for the models which have not already started evaluating.
For all the models which were already launched, this will be an issue, as we will need to cancel the job, and relaunch, which will be a waste of compute.
I think we'll indeed add a logging system to force people to be connected and only submit their own models to avoid these kind of issues (cc @alozowski since you wanted to add this).

Btw, you opened several PRs several times, please try to open a single PR next time.
I'm also not accepting your PRs for finished models, as we already evaluated them, and changing the request file would mean providing an incorrect display to users.

Thanks @clefourrier truly appreciated. Yup will do a single PR next time (hopefully not many next time needed after adding logging system).
Also, Understood about already evaluated model, Make sense. I will submit them again with proper arguments.
Let me know if you have any questions in meanwhile.

@SaylorTwift : Could you please share the full command to reproduce the v2 LB evals?

Hi @pankajmathur ,

I think I can help you with the command to reproduce the evals – here is the Reproducibility info in our documentation, I hope you will find it useful.

Let me close this discussion now, feel free to open a new one in case of any other questions!