Repository: EleutherAI/lm-evaluation-harness
Issue: vllm: Sampled token IDs exceed the max model length
Root cause: Validation Gap

Hi,

When evaluating with vllm v1:

```
lm_eval --model vllm \
    --model_args pretrained=${MODEL_ID},dtype=auto,add_bos_token=True,tensor_parallel_size=1,gpu_memory_utilization=0.7,max_model_len=6000,enforce_eager=True \
    --tasks wikitext \
    --gen_kwargs "do_sample=False,num_beams=1,top_p=1,temperature=0" \
    --batch_size 64 \
    --device cuda
```

I get:

```
ERROR 07-11 12:33:55 [core.py:588] EngineCore encountered a fatal error.
ERROR 07-11 12:33:55 [core.py:588] Traceback (most recent call last):
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/engine/core.py", line 579, in run_engine_core
ERROR 07-11 12:33:55 [core.py:588]     engine_core.run_busy_loop()
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/engine/core.py", line 606, in run_busy_loop
ERROR 07-11 12:33:55 [core.py:588]     self._process_engine_step()
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/engine/core.py", line 631, in _process_engine_step
ERROR 07-11 12:33:55 [core.py:588]     outputs, model_executed = self.step_fn()
ERROR 07-11 12:33:55 [core.py:588]                               ^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/engine/core.py", line 235, in step
ERROR 07-11 12:33:55 [core.py:588]     model_output = self.execute_model(scheduler_output)
ERROR 07-11 12:33:55 [core.py:588]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/engine/core.py", line 221, in execute_model
ERROR 07-11 12:33:55 [core.py:588]     raise err
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/engine/core.py", line 212, in execute_model
ERROR 07-11 12:33:55 [core.py:588]     return self.model_executor.execute_model(scheduler_output)
ERROR 07-11 12:33:55 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/executor/abstract.py", line 87, in execute_model
ERROR 07-11 12:33:55 [core.py:588]     output = self.collective_rpc("execute_model",
ERROR 07-11 12:33:55 [core.py:588]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
ERROR 07-11 12:33:55 [core.py:588]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 07-11 12:33:55 [core.py:588]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/utils/__init__.py", line 2943, in run_method
ERROR 07-11 12:33:55 [core.py:588]     return func(*args, **kwargs)
ERROR 07-11 12:33:55 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/root/miniforge3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 07-11 12:33:55 [core.py:588]     return func(*args, **kwargs)
ERROR 07-11 12:33:55 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/worker/gpu_worker.py", line 320, in execute_model
ERROR 07-11 12:33:55 [core.py:588]     output = self.model_runner.execute_model(scheduler_output,
ERROR 07-11 12:33:55 [core.py:588]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/root/miniforge3/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 07-11 12:33:55 [core.py:588]     return func(*args, **kwargs)
ERROR 07-11 12:33:55 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588]   File "/shared_volume/repos/vllm/vllm/v1/worker/gpu_model_runner.py", line 1514, in execute_model
ERROR 07-11 12:33:55 [core.py:588]     assert end_idx <= self.max_model_len, (
ERROR 07-11 12:33:55 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-11 12:33:55 [core.py:588] AssertionError: Sampled token IDs exceed the max model length. Total number of tokens: 6001 > max_model_len: 6000
```

I think restricting max length with `max_model_len=6000` used to work before, maybe something is wrong with lm-eval-harness x v1. Maybe this is easy to fix, maybe just the command to modify.

Thank you

+1 I am seeing the same issue.

I'm hitting this as well. Is it a matter of incrementing the max_model_len that is passed into vllm [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/vllm_causallms.py#L170)?



+1 This is still an issue on latest, seemingly for most multichoice evals. Consistently fails by exceeding configured model length by 1 token. `Sampled token IDs exceed the max model length. Total number of tokens: 4097 > max_model_len: 4096`

Seems to have been a problem for 3-6+ months and blocks running evals with vllm?

e.g. just running "leaderboard" tasks.

<details>
<summary>
cmd and trace
</summary>

```bash
NCCL_DEBUG=WARN VLLM_WORKER_MULTIPROC_METHOD=spawn  VLLM_USE_V1=1 HF_DATASETS_TRUST_REMOTE_CODE=1 HF_ALLOW_CODE_EVAL=1 HF_TOKEN="" lm_eval --model vllm     --model_args pretrained=Qwen/Qwen3-8B-Base,dtype="bfloat16",gpu_memory_utilization=0.8,data_parallel_size=2,max_model_len=4096     --tasks leaderboard    --batch_size auto     --output_path output/    --log_samples --confirm_run_unsafe_code
```

```
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710] EngineCore encountered a fatal error.
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710] Traceback (most recent call last):
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 701, in run_engine_core
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     engine_core.run_busy_loop()
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1045, in run_busy_loop
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     executed = self._process_engine_step()
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 754, in _process_engine_step
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     outputs, model_executed = self.step_fn()
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 284, in step
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     model_output = self.execute_model_with_error_logging(
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 270, in execute_model_with_error_logging
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     raise err
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 261, in execute_model_with_error_logging
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     return model_fn(scheduler_output)
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 103, in execute_model
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     output = self.collective_rpc("execute_model",
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/utils/__init__.py", line 3122, in run_method
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     return func(*args, **kwargs)
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     return func(*args, **kwargs)
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 447, in execute_model
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     output = self.model_runner.execute_model(scheduler_output,
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     return func(*args, **kwargs)
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2413, in execute_model
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     ) = self._bookkeeping_sync(scheduler_output, sampler_output,
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]   File "/mnt/home/rob/lm-evaluation-harness/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 2190, in _bookkeeping_sync
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710]     assert end_idx <= self.max_model_len, (
(EngineCore_DP1 pid=3709656) ERROR 10-10 23:05:38 [core.py:710] AssertionError: Sampled token IDs exceed the max model length. Total number of tokens: 4097 > max_model_len: 4096
```
</details>

I am still seeing this, do we have a solution yet?

Same here