Repository: EleutherAI/lm-evaluation-harness
Issue: not getting same accuracy as on the leaderboard when evaluating locally.
Root cause: Documentation Deficiency

Hello, how to reproduce the results of leaderboard 2. here is my evaluation command and results with phi-3-mini-4k

` lm_eval --model hf \
    --model_args pretrained=microsoft/Phi-3-mini-4k-instruct,dtype="bfloat16"\
    --tasks leaderboard_gpqa\
    --device cuda:0 \
    --num_fewshot 0\
    --batch_size auto:2

|           Tasks            |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|----------------------------|-------|------|-----:|--------|---|-----:|---|-----:|
|leaderboard_gpqa            |N/A    |none  |     0|acc_norm|↑  |0.3297|±  |0.0136|
| - leaderboard_gpqa_diamond |      1|none  |     0|acc_norm|↑  |0.3333|±  |0.0336|
| - leaderboard_gpqa_extended|      1|none  |     0|acc_norm|↑  |0.3168|±  |0.0199|
| - leaderboard_gpqa_main    |      1|none  |     0|acc_norm|↑  |0.3438|±  |0.0225|

|     Groups     |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|----------------|-------|------|-----:|--------|---|-----:|---|-----:|
|leaderboard_gpqa|N/A    |none  |     0|acc_norm|↑  |0.3297|±  |0.0136|`

the leaderboard accuracy is 9.28 which is much lower than these results.
what is wrong with my evaluation
thank you


Same problem when evaluate qwen2-7B model, got a quit similar result: 32.38. 
This is similar to the qwen2-7b's [official report](https://huggingface.co/Qwen/Qwen2-7B) and lm-eval-harness's gpqa zeroshot task (gpqa_main_zeroshot, gpqa_extended_zeroshot, gpqa_diamond_zeroshot). 
But is quit far from the leaderboard which is  7.27.
I found that the gpqa raw score on the leaderboard is 0.3 which is quit close to the result I got.
I'm not sure whether we should compare the local eval result to the raw scores? like gpqa raw, musr raw, bbh raw etc.

You are right. the raw datasets results are closer to the one obtained with lm_eval on leaderboard 2. but do not always match.  

Hi! Yes, you should compare local results to the raw score, the scores we display by default have been normalized (it's explained in the [release blog](https://huggingface.co/spaces/open-llm-leaderboard/blog) if you need more information).

The scores do not always match 100% as changing the batch size and hardware can change the results, so you won't get exactly the same numbers as we do.

Hi, I meet the same problem when evaluate LLaMa2-7B with hellaswag. Here is the result:
![image](https://github.com/user-attachments/assets/85c14569-69cc-44e9-a8b0-f318287d9c09)
The result is quite different from the results obtained from other papers (70+), so why?

'python /public/ly/SBF/lm-evaluation-harness/main.py --model hf-causal-experimental --model_args pretrained=/public/MountData/yaolu/LLM_pretrained/LLAMA2_7B  --tasks hellaswag --batch_size auto  --device cuda:0'

Hi! No idea, but to reproduce results of the leaderboard v1 you should follow the reproducibility steps in the leaderboard documentation and use a specific commit of the harness.

Closing because 

**scores reported by `leaderboard` tasks in lm_eval should be compared to the _raw_ scores for Open LLM Leaderboard 2 tasks** is the intended interpretation--thanks @clefourrier for chiming in on this!

@yaolu-zjut we don't have the bandwidth to extensively provide support for older lm eval harness versions, but I'd note that HellaSwag in particular seems to have very variable scores when one runs with a small `--limit` value for debugging--often `--limit 10` results in a score that's 40% lower than expected. Beyond that, I can't speak to why your model might be underperforming without more info