Repository: EleutherAI/lm-evaluation-harness
Issue: [BUG] lm-eval does not know how to control context length when using the OpenAI API
Root cause: Validation Gap

Hello! lm-eval does not know how to control context length when using the OpenAI API.

If the length of the input context is about `8000` tokens and the number of new tokens is `256` (this is the default value in lm_eval), then lm_eval will attempt to send such a request via the `OpenAI API`, resulting in a `“400 Bad Request”` error if the maximum possible length on the server is `8192` tokens. And my task completely falls down.

When I use lm_eval through hf or vllm engine (`--model hf` or `--model vllm`), this problem does not occur, because there is some protection for this case.

Below I've given a simple example for reproducibility of the problem via the OpenAI API. I specifically create one sample that contains `8000` tokens for the tokenizer from `openchat/openchat-3.6-8b-20240522`. I create a simple task to generate the tokens and run lm_eval. In the background I have the `openchat/openchat-3.6-8b-20240522` model running via `VLLM API Server` (vllm==0.6.4.post1, `python -m vllm.entrypoints.openai.api_server --model openchat/openchat-3.6-8b-20240522`).

```bash
git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness.git

cd lm-evaluation-harness
git checkout v0.4.6
pip install -e .[api,vllm]
cd ..

long_string=""
for i in {1..8000}
do
    long_string="${long_string}
"
done
echo '{"inputs": "'${long_string}'"}' > simple.json

mkdir -p lm_eval_bug_demonstration
echo '
task: bug_demonstration
output_type: generate_until
dataset_path: json
dataset_kwargs:
  data_files:
    validation: simple.json

validation_split: validation
doc_to_text: "{{inputs}}"
doc_to_target: ""
' > lm_eval_bug_demonstration/bug_demostration.yaml

MODEL_ARGS="model=openchat/openchat-3.6-8b-20240522"
MODEL_ARGS="${MODEL_ARGS},base_url=http://localhost:8000/v1/completions"
MODEL_ARGS="${MODEL_ARGS},tokenizer_backend=huggingface"
MODEL_ARGS="${MODEL_ARGS},tokenizer=openchat/openchat-3.6-8b-20240522"
MODEL_ARGS="${MODEL_ARGS},max_length=8192"
MODEL_ARGS="${MODEL_ARGS},tokenized_requests=true"

lm_eval \
    --include_path lm_eval_bug_demonstration \
    --tasks bug_demonstration \
    --model local-completions \
    --model_args "${MODEL_ARGS}" \
    --limit 1
```

VLLM API Server produces logs when such a task is performed:
```text
ERROR 12-09 05:54:31 serving_completion.py:112] Error in preprocessing prompt inputs
ERROR 12-09 05:54:31 serving_completion.py:112] Traceback (most recent call last):
ERROR 12-09 05:54:31 serving_completion.py:112]   File "/root/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_completion.py", line 104, in create_completion
ERROR 12-09 05:54:31 serving_completion.py:112]     request_prompts, engine_prompts = self._preprocess_completion(
ERROR 12-09 05:54:31 serving_completion.py:112]   File "/root/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 406, in _preprocess_completion
ERROR 12-09 05:54:31 serving_completion.py:112]     request_prompts = [
ERROR 12-09 05:54:31 serving_completion.py:112]   File "/root/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 406, in <listcomp>
ERROR 12-09 05:54:31 serving_completion.py:112]     request_prompts = [
ERROR 12-09 05:54:31 serving_completion.py:112]   File "/root/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 391, in _tokenize_prompt_input_or_inputs
ERROR 12-09 05:54:31 serving_completion.py:112]     yield self._normalize_prompt_tokens_to_input(
ERROR 12-09 05:54:31 serving_completion.py:112]   File "/root/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 259, in _normalize_prompt_tokens_to_input
ERROR 12-09 05:54:31 serving_completion.py:112]     return self._validate_input(request, input_ids, input_text)
ERROR 12-09 05:54:31 serving_completion.py:112]   File "/root/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/serving_engine.py", line 302, in _validate_input
ERROR 12-09 05:54:31 serving_completion.py:112]     raise ValueError(
ERROR 12-09 05:54:31 serving_completion.py:112] ValueError: This model's maximum context length is 8192 tokens. However, you requested 8256 tokens (8000 in the messages, 256 in the completion). Please reduce the length of the messages or completion.
INFO:     ::1:34944 - "POST /v1/completions HTTP/1.1" 400 Bad Request
```

lm-eval produces logs when performing such a task:
```text
2024-12-09:05:54:20,443 INFO     [__main__.py:279] Verbosity set to INFO
2024-12-09:05:54:20,443 INFO     [__main__.py:303] Including path: lm_eval_bug_demonstration
2024-12-09:05:54:28,408 WARNING  [__main__.py:312]  --limit SHOULD ONLY BE USED FOR TESTING.REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2024-12-09:05:54:28,409 INFO     [__main__.py:376] Selected Tasks: ['bug_demonstration']
2024-12-09:05:54:28,412 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-12-09:05:54:28,412 INFO     [evaluator.py:201] Initializing local-completions model, with arguments: {'model': 'openchat/openchat-3.6-8b-20240522', 'base_url': 'http://localhost:8000/v1/completions', 'tokenizer_backend': 'huggingface', 'tokenizer': 'openchat/openchat-3.6-8b-20240522', 'max_length': 8192, 'tokenized_requests': True}
2024-12-09:05:54:28,413 INFO     [api_models.py:115] Using max length 8192 - 1
2024-12-09:05:54:28,413 INFO     [api_models.py:118] Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2024-12-09:05:54:28,413 INFO     [api_models.py:133] Using tokenizer huggingface
Generating validation split: 1 examples [00:00, 357.14 examples/s]
2024-12-09:05:54:29,528 INFO     [task.py:415] Building contexts for bug_demonstration on rank 0...
100%|███████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2233.39it/s]
2024-12-09:05:54:29,529 INFO     [evaluator.py:496] Running generate_until requests
Requesting API:   0%|                                                                                 | 0/1 [00:00<?, ?it/s]2024-12-09:05:54:29,596 WARNING  [api_models.py:374] API request failed with error message: {"object":"error","message":"This model's maximum context length is 8192 tokens. However, you requested 8256 tokens (8000 in the messages, 256 in the completion). Please reduce the length of the messages or completion.","type":"BadRequestError","param":null,"code":400}. Retrying...
2024-12-09:05:54:30,640 WARNING  [api_models.py:374] API request failed with error message: {"object":"error","message":"This model's maximum context length is 8192 tokens. However, you requested 8256 tokens (8000 in the messages, 256 in the completion). Please reduce the length of the messages or completion.","type":"BadRequestError","param":null,"code":400}. Retrying...
2024-12-09:05:54:31,685 WARNING  [api_models.py:374] API request failed with error message: {"object":"error","message":"This model's maximum context length is 8192 tokens. However, you requested 8256 tokens (8000 in the messages, 256 in the completion). Please reduce the length of the messages or completion.","type":"BadRequestError","param":null,"code":400}. Retrying...
Traceback (most recent call last):
  File "/root/.local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/root/llms_benchmark/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/root/llms_benchmark/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/root/llms_benchmark/lm-evaluation-harness/lm_eval/evaluator.py", line 303, in simple_evaluate
    results = evaluate(
  File "/root/llms_benchmark/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
  File "/root/llms_benchmark/lm-evaluation-harness/lm_eval/evaluator.py", line 507, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/root/llms_benchmark/lm-evaluation-harness/lm_eval/models/api_models.py", line 598, in generate_until
    outputs = retry(
  File "/root/.local/lib/python3.10/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
  File "/root/.local/lib/python3.10/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
  File "/root/.local/lib/python3.10/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/root/.local/lib/python3.10/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
  File "/root/.local/lib/python3.10/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/root/.local/lib/python3.10/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
  File "/root/llms_benchmark/lm-evaluation-harness/lm_eval/models/api_models.py", line 377, in model_call
    response.raise_for_status()
  File "/root/.local/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://localhost:8000/v1/completions
```

Thanks for reporting this! added truncation in the PR