Repository: EleutherAI/lm-evaluation-harness
Issue: Lack of reproducability despite set batch size, seed, greedy sampling
Root cause: Resource Mishandling

Hi, I am seeing numerical differences between different runs evaluating the same model using greedy decoding, see https://github.com/EleutherAI/lm-evaluation-harness/blob/0c9072191e790204c4dfeb3285925b586a548c2e/lm_eval/tasks/gsm8k/gsm8k.yaml#L23-L29:

```bash
export OUT_PATH="."
export TIMESTAMP=$(date +"%Y-%m-%d_%H-%M-%S")
export EXP_NAME="${TIMESTAMP}_noquant_lmeval_repro_bs24"
CUDA_VISIBLE_DEVICES=7 nohup lm_eval --model hf \
    --model_args pretrained=Qwen/Qwen3-8B \
    --tasks gsm8k \
    --device cuda:0 \
    --batch_size 24 \
    --seed 42 &> ${OUT_PATH}/${EXP_NAME}.log &
```

I am seeing the following results over repeated experiments on 1xH100 with `transformers==4.57.1`, `torch==2.9.0+cu126` and lm-eval-harness 0c9072191e790204c4dfeb3285925b586a548c2e:

```bash
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8893|±  |0.0086|
|     |       |strict-match    |     5|exact_match|↑  |0.8848|±  |0.0088|

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8863|±  |0.0087|
|     |       |strict-match    |     5|exact_match|↑  |0.8832|±  |0.0088|

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8855|±  |0.0088|
|     |       |strict-match    |     5|exact_match|↑  |0.8825|±  |0.0089|

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.8878|±  |0.0087|
|     |       |strict-match    |     5|exact_match|↑  |0.8863|±  |0.0087|


# with gsm8k_platinum:

|    Tasks     |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|--------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_platinum|      3|flexible-extract|     5|exact_match|↑  |0.9173|±  |0.0079|
|              |       |strict-match    |     5|exact_match|↑  |0.9156|±  |0.0080|

|    Tasks     |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|--------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_platinum|      3|flexible-extract|     5|exact_match|↑  |0.9165|±  |0.0080|
|              |       |strict-match    |     5|exact_match|↑  |0.9140|±  |0.0081|

|    Tasks     |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|--------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_platinum|      3|flexible-extract|     5|exact_match|↑  |0.9181|±  |0.0079|
|              |       |strict-match    |     5|exact_match|↑  |0.9148|±  |0.0080|

|    Tasks     |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|--------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k_platinum|      3|flexible-extract|     5|exact_match|↑  |0.9173|±  |0.0079|
|              |       |strict-match    |     5|exact_match|↑  |0.9132|±  |0.0081|
```

Printing the context at https://github.com/EleutherAI/lm-evaluation-harness/blob/0c9072191e790204c4dfeb3285925b586a548c2e/lm_eval/models/huggingface.py#L998-L1005 and comparing the different runs, it is clear that the batching is the same over different runs.

Still, we see significant difference of ~0.4% at worse in end metrics over different runs.

As greedy decoding is used, batching is consistently the same, I was expecting a much smaller difference. Do you know where this variability comes from? AFAIK some variability can come from atomicAdd & the reduction order, but even accounting for that ~0.4% difference sounds huge.

Although this is within the returned `stderr`, this has nothing to do with it as `stderr` accounts for the standard error of the mean depending on the number of samples.

WDYT?

Hi! Would you happen to have the logged samples handy? Would be easy to check how much different the generated strings are. I ran this on A40s, where the scores are the same (same environment, back to back). Could be a difference of ops between GPUs? Also have you tried with `VLLM_BATCH_INVARIANT=1` yet? 

Thanks a lot @baberabb 

I'll try to log/save the generated sequences to see how they diverge.