Repository: EleutherAI/lm-evaluation-harness
Issue: [VLLM V1] Prompt length of xxx is longer than the maximum model length of xxx
Root cause: Validation Gap

vLLM switched to V1 since 0.8.0, the behaviour of the max prompt length seems changed. lm-eval will generate the request with prompt length = max-length, which will cause an error in vLLM 0.8.0+. 

I have tried the latest release of lm-eval and the main branch version, both fail.

Workaround: disable the v1 for vLLM 0.8.0+ by `VLLM_USE_V1=0`.

# Reproduce
```bash
# success
VLLM_USE_V1=0 lm-eval \
  --model vllm \
  --model_args pretrained=meta-llama/Llama-3.2-1B,max_model_len=2048,gpu_memory_utilization=0.5 \
  --tasks wikitext \
  --batch_size 1

# fail with message: ValueError: Prompt length of 2048 is longer than the maximum model length of 2048.
lm-eval \
  --model vllm \
  --model_args pretrained=meta-llama/Llama-3.2-1B,max_model_len=2048,gpu_memory_utilization=0.5 \
  --tasks wikitext \
  --batch_size 1
```
Log:
```
Traceback (most recent call last):
  File "/home/yaoyao/miniconda3/envs/aureq/bin/lm-eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/yaoyao/repos/vllm-aureq/3rdparty/lm-evaluation-harness/lm_eval/__main__.py", line 432, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/yaoyao/repos/vllm-aureq/3rdparty/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/home/yaoyao/repos/vllm-aureq/3rdparty/lm-evaluation-harness/lm_eval/evaluator.py", line 333, in simple_evaluate
    results = evaluate(
  File "/home/yaoyao/repos/vllm-aureq/3rdparty/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
    return fn(*args, **kwargs)
  File "/home/yaoyao/repos/vllm-aureq/3rdparty/lm-evaluation-harness/lm_eval/evaluator.py", line 553, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/yaoyao/repos/vllm-aureq/3rdparty/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 330, in loglikelihood_rolling
    batch_nlls = self._loglikelihood_tokens(
  File "/home/yaoyao/repos/vllm-aureq/3rdparty/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 474, in _loglikelihood_tokens
    outputs = self._model_generate(requests=inputs, generate=False)
  File "/home/yaoyao/repos/vllm-aureq/3rdparty/lm-evaluation-harness/lm_eval/models/vllm_causallms.py", line 278, in _model_generate
    outputs = self.model.generate(
  File "/home/yaoyao/miniconda3/envs/aureq/lib/python3.10/site-packages/vllm/utils.py", line 1072, in inner
    return fn(*args, **kwargs)
  File "/home/yaoyao/miniconda3/envs/aureq/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 457, in generate
    self._validate_and_add_requests(
  File "/home/yaoyao/miniconda3/envs/aureq/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1308, in _validate_and_add_requests
    self._add_request(
  File "/home/yaoyao/miniconda3/envs/aureq/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 1326, in _add_request
    self.llm_engine.add_request(
  File "/home/yaoyao/miniconda3/envs/aureq/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 184, in add_request
    request = self.processor.process_inputs(request_id, prompt, params,
  File "/home/yaoyao/miniconda3/envs/aureq/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 207, in process_inputs
    self._validate_model_inputs(processed_inputs, lora_request)
  File "/home/yaoyao/miniconda3/envs/aureq/lib/python3.10/site-packages/vllm/v1/engine/processor.py", line 322, in _validate_model_inputs
    raise ValueError(
ValueError: Prompt length of 2048 is longer than the maximum model length of 2048.
```

Also encountered this error. This is caused by the VLLM assuming that at least 1 token should be left for generation, so the prompt must be at most max_model_len -1 in length. 


For wikitext, you can patch by making the following revision in `lm-evaluation-harness/lm_eval/models/vllm_causallms.py`


```
def loglikelihood_rolling(
        self, requests: List[Instance], disable_tqdm: bool = False
    ) -> List[float]:
        adaptive_batch_size = None
        if self.batch_size == "auto":
            adaptive_batch_size = len(requests)

        # First, collect all windows from all requests
        all_windows = []  # List of (request_idx, window) tuples
        request_window_counts = []  # Track number of windows per request

        for req_idx, (string,) in enumerate(
            tqdm(
                [req.args for req in requests],
                disable=(disable_tqdm or (self.rank != 0)),
            )
        ):
            rolling_token_windows: List[Tuple[List[int], List[int]]] = list(
                map(
                    make_disjoint_window,
                    get_rolling_token_windows(
                        token_list=self.tok_encode(string),
                        prefix_token=self.prefix_token_id,
                        # max_seq_len - (1 for context) - (at least 1 token for generation)
                        max_seq_len=self.max_length - 2, # CHANGE THIS 
                        context_len=1,
                    ),
                )
            )

```

I use ```max_num_seqs=1``` and ```max_num_batched_tokens=2048``` not oom happen. ```max_model_len``` truncate prompt like ```[-max_model_len:]```