Repository: EleutherAI/lm-evaluation-harness
Issue: Eval breaking on openai-compatible model run - AssertionError.
Root cause: Validation Gap

I'm trying to run evaluation using local openai-compatible endpoint with tokenization on the server side.
Both these examples run vllm, with the second being behind llama-swap. 
Not sure what the error i'm getting is but it seems to me that the bug is not related to issues with the API.
The assertion error doesn't provide much meaningful info either, but it seems that it is related to reordering of respo

```
lm_eval --model local-chat-completions --tasks noridiom_nob_p0 --model_args base_url=http://localhost:7000/v1/chat/completions,num_concurrent=1,max_retries=3,tokenized_requests=False,model="openai/gpt-oss-20b" --apply_chat_template
```
```bash
Welcome to Ubuntu 24.04.3 LTS (GNU/Linux 6.8.0-85-generic x86_64)
Requesting API: 1%|▍ | 19/3170 [05:14<7:56:48, 9.08s/it]Requesting API: 2%|█▎ | 59/3170 [23:17<17:14:13, 19.95s/it]Requesting API: 11%|██████▉ | 335/3170 [1:35:25<15:20:45, 19.49s/it]Traceback (most recent call last):
File "/home/me/Repos/lm-evaluation-harness/.venv/bin/lm_eval", line 10, in
sys.exit(cli_evaluate())
File "/home/me/Repos/lm-evaluation-harness/lm_eval/main.py", line 459, in cli_evaluate
results = evaluator.simple_evaluate(
File "/home/me/Repos/lm-evaluation-harness/lm_eval/utils.py", line 458, in _wrapper
return fn(*args, **kwargs)
File "/home/me/Repos/lm-evaluation-harness/lm_eval/evaluator.py", line 357, in simple_evaluate
results = evaluate(
File "/home/me/Repos/lm-evaluation-harness/lm_eval/utils.py", line 458, in _wrapper
return fn(*args, **kwargs)
File "/home/me/Repos/lm-evaluation-harness/lm_eval/evaluator.py", line 585, in evaluate
resps = getattr(lm, reqtype)(cloned_reqs)
File "/home/me/Repos/lm-evaluation-harness/lm_eval/models/api_models.py", line 814, in generate_until
return re_ord.get_original(res)
File "/home/me/Repos/lm-evaluation-harness/lm_eval/models/utils.py", line 541, in get_original
assert all(cov)
AssertionError
Requesting API: 11%|██████▉ | 335/3170 [1:35:36<13:29:10, 17.13s/it]
```
Command:
```
lm_eval --model local-chat-completions --tasks nortruthfulqa_gen_nob_p0 --model_args base_url=http://localhost:8000/v1/chat/completions,num_concurrent=1,max_retries=3,tokenized_requests=False,model="GPTOSS-20B" --apply_chat_template
```
```
2025-10-18:23:54:34 WARNING [models.api_models:376] Cannot determine EOS string to pass to stop sequence. Manually set by passing eos_string to model_args.
Requesting API: 38%|███████████████████████████████████████████ | 132/346 [07:42<18:02, 5.06s/it]
Traceback (most recent call last):
File "/home/me/Repos/lm-evaluation-harness/.venv/bin/lm_eval", line 10, in
sys.exit(cli_evaluate())
^^^^^^^^^^^^^^
File "/home/me/Repos/lm-evaluation-harness/lm_eval/main.py", line 459, in cli_evaluate
results = evaluator.simple_evaluate(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/me/Repos/lm-evaluation-harness/lm_eval/utils.py", line 458, in _wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/me/Repos/lm-evaluation-harness/lm_eval/evaluator.py", line 357, in simple_evaluate
results = evaluate(
^^^^^^^^^
File "/home/me/Repos/lm-evaluation-harness/lm_eval/utils.py", line 458, in _wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/me/Repos/lm-evaluation-harness/lm_eval/evaluator.py", line 585, in evaluate
resps = getattr(lm, reqtype)(cloned_reqs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/me/Repos/lm-evaluation-harness/lm_eval/models/api_models.py", line 814, in generate_until
return re_ord.get_original(res)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/me/Repos/lm-evaluation-harness/lm_eval/models/utils.py", line 541, in get_original
assert all(cov)
^^^^^^^^
AssertionError
Requesting API: 38%|███████████████████████████████████████████ | 132/346 [07:43<12:31, 3.51s/it]
```

Looking into this myself, but any help is appreciated!

Ok so I've found out that VLLM returns `content: null`
when the reasoning model stops generating due to max tokens constraint. 
Therefore it gets skipped in the collator and there's a mismatch for the response count triggering the assertion error.
I will make a PR to resolve this, which would treat the null value returned by the API as an empty string (to stop it from breaking). This should come with an appropriate warning.

But this outcome is probably something one should expect to happen when using greedy decoding on reasoning models anyway.