Repository: EleutherAI/lm-evaluation-harness
Issue: Generative tasks don't run with batch_size > 1
Root cause: Validation Gap

Can someone confirm that this is indeed the case, (for the hf backend, ie with --model hf) ? Anyways, if this is expected behavior, better error handling is apt

@AbhinavDutta you'd have better answers if you provide a better issue description :)

You can print the context size at https://github.com/EleutherAI/lm-evaluation-harness/blob/68b03658ace40fa93221518b30096485a2387c58/lm_eval/models/huggingface.py#L997 to verify your input shapes.

I apologize for my terseness. I just fixed the bug for my  private fork and moved on (I was actually writing a patch to evaluate models quantized with TE which is remarkably similar to normal HF models, when I came across this but I fixed it simply by setting using eager mode, and disabling the part where lm_eval forcefully sets batch_size=1). lm_eval actually forces batch_size to 1 which was unacceptable for my case, due to some idiosyncracies of TE. This is not necessary in cases when someone is using the eager mode, but as far as correctness goes, it still works, despite being *extremely* inefficient.



INFO 11-13 09:07:22 [__init__.py:216] Automatically detected platform cuda.
2025-11-13:09:08:26 INFO     [__main__:446] Selected Tasks: ['gsm8k']
2025-11-13:09:08:26 INFO     [evaluator:202] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-11-13:09:08:26 INFO     [evaluator:240] Initializing hf model, with arguments: {'pretrained': '/pvcdata/t-abdutta/t-abdutta/huggingface/qwen3/Qwen3-0.6B'}
2025-11-13:09:08:26 INFO     [models.huggingface:148] Using device 'cuda'
2025-11-13:09:08:27 INFO     [models.huggingface:418] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda'}
`torch_dtype` is deprecated! Use `dtype` instead!
2025-11-13:09:08:34 WARNING  [models.huggingface:568] Falling back to attention implementation 'eager' to avoid a known CUDA index bug with SDPA masks on padded batches in PyTorch >= 2.5. Override via `--model_args attn_implementation=sdpa` to force SDPA.
2025-11-13:09:08:34 INFO     [models.huggingface:576] Using attention implementation 'eager' for Hugging Face generation.
2025-11-13:09:08:36 INFO     [evaluator:305] gsm8k: Using gen_kwargs: {'until': ['Question:', '
', '
'], 'do_sample': False, 'temperature': 0.0}
2025-11-13:09:08:36 WARNING  [evaluator:324] Overwriting default num_fewshot of gsm8k from 5 to 1
2025-11-13:09:08:36 INFO     [api.task:434] Building contexts for gsm8k on rank 0...
100%|██████████████████████████████████████████████████████████████████| 1319/1319 [00:01<00:00, 1105.87it/s]
2025-11-13:09:08:37 INFO     [evaluator:574] Running generate_until requests
Running generate_until requests:   0%|                                              | 0/1319 [00:00<?, ?it/s]2025-11-13:09:08:37 WARNING  [models.huggingface:1438] PyTorch>=2.5 SDPA currently crashes on padded multi-request batches; forcing generation batch_size=1. Re-run with `--batch_size 1` to silence this warning.
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.