Repository: EleutherAI/lm-evaluation-harness
Issue: Different in MMLU Scores in lm-eval 0.4.2
Root cause: Algorithmic Error

Hi team,
I think the MMLU Scores are a bit off in the newest version of lm-eval (0.4.2). Here is the command I used to reproduce the issue:
```
lm-eval --model hf \
    --model_args pretrained=HuggingFaceH4/zephyr-7b-beta \
    --tasks mmlu
```
For 0.4.2 I got:
```
HuggingFaceH4/zephyr-7b-beta
|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.5810|±  |0.0040|
| - humanities     |N/A    |none  |     0|acc   |0.5290|±  |0.0069|
| - other          |N/A    |none  |     0|acc   |0.6575|±  |0.0082|
| - social_sciences|N/A    |none  |     0|acc   |0.6747|±  |0.0083|
| - stem           |N/A    |none  |     0|acc   |0.4919|±  |0.0087|
```

However, for previous versions
0.4.1 and 0.4.0:
```
HuggingFaceH4/zephyr-7b-beta
|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.5855|±  |0.1186|
| - humanities     |N/A    |none  |     0|acc   |0.5288|±  |0.1102|
| - other          |N/A    |none  |     0|acc   |0.6633|±  |0.1007|
| - social_sciences|N/A    |none  |     0|acc   |0.6828|±  |0.0865|
| - stem           |N/A    |none  |     0|acc   |0.4986|±  |0.1175|
```

Is this an expected behavior? as the Stderr is smaller for 0.4.2

Hi! Stderr is an expected change in v0.4.2--previous stderrs were too large due to being improperly aggregated for task groupings.

Regarding the scores changing, no this isn't expected and I'll try to replicate this. It does look like they are mostly within standard error, and some tiny deviations like this are sometimes unsurprising (e.g. from batch size or other small nondeterminacies, or sometimes two choices' loglikelihoods have the same floating point value as each other), but still, worth investigating.

On my end v0.4.2 gives
```
|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.5802|±  |0.0040|
| - humanities     |N/A    |none  |     0|acc   |0.5262|±  |0.0069|
| - other          |N/A    |none  |     0|acc   |0.6579|±  |0.0082|
| - social_sciences|N/A    |none  |     0|acc   |0.6744|±  |0.0083|
| - stem           |N/A    |none  |     0|acc   |0.4922|±  |0.0087|
```
--running now on v0.4.1 and will dump samples to see if there are differences in outputs.

Oh, sorry, this is because of https://github.com/EleutherAI/lm-evaluation-harness/pull/1425 . 

TLDR: for tasks like MMLU where each target string is one token, we can run one model call per document and extract the corresponding choice's token's loglikelihood for each, instead of having a separate call for each answer choice's log probability. This saves on runtime by a factor of the number of choices, and should not affect results more than, say, choosing a different batch size. It also more closely matches precisely what is done in the original MMLU github repository.

This behavior can be turned off with `--model_args logits_cache=False` ! 

oh thanks for pointing this out, does this difference in score happen to Mistral-based models only or does this vary that much for other models? 

Running 
```
lm-eval --model hf \
    --model_args pretrained=HuggingFaceH4/zephyr-7b-beta \
    --tasks mmlu \
    --batch_size=16 \
    --model_args logits_cache=False
````

Gives me random scores:

```
|      Groups      |Version|Filter|n-shot|Metric|Value |   |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu              |N/A    |none  |     0|acc   |0.2292|±  |0.0035|
| - humanities     |N/A    |none  |     0|acc   |0.2423|±  |0.0062|
| - other          |N/A    |none  |     0|acc   |0.2382|±  |0.0076|
| - social_sciences|N/A    |none  |     0|acc   |0.2168|±  |0.0074|
| - stem           |N/A    |none  |     0|acc   |0.2131|±  |0.0073|
```

You'll need to run

```
lm-eval --model hf \
    --model_args pretrained=HuggingFaceH4/zephyr-7b-beta,logits_cache=False \
    --tasks mmlu \
    --batch_size=16 
```

for both model args to be respected (this is a footgun, will update in future to ensure cases like these explicitly error / behave as intended)


> difference in score happen to Mistral-based models only or does this vary that much for other models?

It should happen for other models too but should be within stderr for them! (in an idealized world the two approaches should be precisely the same). I would expect it to have variance on the order of things like batch size (aka, very small deviations caused by maybe a couple documents' predictions flipping, but not a lot)--if it does introduce more than that, then there is likely some bug.



Hi @justinphan3110cais , just wanted to check if you were seeing equivalent results with `logits_cache=False` after fixing the `--model_args` issue!

Sorry for not updating this, I tried `logits_cache=False` and it still gives the same score as the default it (which is lower than 0.4.1) while being xN times slower.

No problem! Awesome :)