Repository: EleutherAI/lm-evaluation-harness
Issue: Gemma-2 different result with/without flash attention 2
Root cause: Environment Incompatibility

I ran `ifeval` with different batch sizes using gemma-2-9b-it model and got quite different results:

1) With Auto batch size ( it turned out to be 1 after detection)
```
hf (pretrained=google/gemma-2-9b-it), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: auto
|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      2|none  |     0|inst_level_loose_acc   |↑  |0.7674|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.7566|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.6784|±  |0.0201|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.6654|±  |0.0203|
```

2) With batch size 32
```
~/lm-evaluation-harness$ accelerate launch -m lm_eval --model hf     --model_args pretrained=google/gemma-2-9b-it   --tasks ifeval  --batch_size 32
hf (pretrained=google/gemma-2-9b-it), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 32
|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      2|none  |     0|inst_level_loose_acc   |↑  |0.0528|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.0528|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.0462|±  |0.0090|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.0462|±  |0.0090|
```

It seems to be a bug inside the implementation of the batch inference. Did I miss something?

Hi! Thanks for reporting this. Looks like a bug to me. A small difference is expected due to floating point precision issues but this magnitude far too large. Will look into it!

Thank you, @baberabb !

It is worth noting that [vLLM currently lacks sliding window support](https://github.com/vllm-project/vllm/blob/6aa33cb2ddd769e764a3312627cab5bffaa383cc/vllm/model_executor/models/gemma2.py#L143-L147), which is used inside Gemma 2 models. I am not sure if this will impact the final result in batch inference. 

If it is using vLLM to serve underneath, it is likely due to an underlying problem inside vLLM. Gemma 2 models work fine with transformers using `eager` and `flash_attention_2` (not FSDP) as the attention implementation.

BTW, here is the evaluation results of Llama

### llama-3.1-8b-instruct

1) Auto

|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      2|none  |     0|inst_level_loose_acc   |↑  |0.6223|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.5935|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.4843|±  |0.0215|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.4455|±  |0.0214|

2) Batch size 32

hf (pretrained=meta-llama/Meta-Llama-3.1-8B-Instruct), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 32
|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      2|none  |     0|inst_level_loose_acc   |↑  |0.6163|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.5815|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.4861|±  |0.0215|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.4399|±  |0.0214|

@tanliboy Can't seem to reproduce the issue. I used:

`accelerate launch lm_eval -model hf --model_args pretrained=google/gemma-2-9b-it,attn_implementation=flash_attention_2  --tasks ifeval -b 16`

hf (pretrained=google/gemma-2-9b-it,attn_implementation=flash_attention_2), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 16
|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      3|none  |     0|inst_level_loose_acc   |↑  |0.7554|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.7482|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.6710|±  |0.0202|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.6617|±  |0.0204|

Maybe a dependency issue? Could you try in a new environment?

@baberabb I found that flash attention wasn't installed during my evaluation. So I tested the evaluation with and without `flash_attn` installed using the same command:

```
accelerate launch lm_eval --model hf --model_args pretrained=google/gemma-2-9b-it  --tasks ifeval --batch_size 16/32
```

A) With flash attention installed:

hf (pretrained=google/gemma-2-9b-it,attn_implementation=eager), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 16
|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      3|none  |     0|inst_level_loose_acc   |↑  |0.7662|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.7578|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.6821|±  |0.0200|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.6710|±  |0.0202|

B) Without flash attention installed (after `pip uninstall flash_attn`)

hf (pretrained=google/gemma-2-9b-it), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 32
|Tasks |Version|Filter|n-shot|        Metric         |   |Value |   |Stderr|
|------|------:|------|-----:|-----------------------|---|-----:|---|------|
|ifeval|      3|none  |     0|inst_level_loose_acc   |↑  |0.0528|±  |   N/A|
|      |       |none  |     0|inst_level_strict_acc  |↑  |0.0528|±  |   N/A|
|      |       |none  |     0|prompt_level_loose_acc |↑  |0.0462|±  |0.0090|
|      |       |none  |     0|prompt_level_strict_acc|↑  |0.0462|±  |0.0090|


It seems that the default flash attention implementation might be the root cause. Could you confirm whether we use FSDP for attention during batch inference when flash attention isn't available?

We can close this issue given it is not a problem with specified attention implementation.

Hi FSDP isn't actually supported in our library. `accelerate launch` runs model parallel by creating a replica on each GPU and splitting the data. From the [readme](https://github.com/EleutherAI/lm-evaluation-harness#hugging-face-transformers):
> WARNING: This setup does not work with FSDP model sharding, so in accelerate config FSDP must be disabled, or the NO_SHARD FSDP option must be used.