Repository: EleutherAI/lm-evaluation-harness
Issue: Reasoning models and token limits
Root cause: Validation Gap

I'm trying lm-eval for the first time and running into an issue where the eval fails when the response is None. I think this is happening for Qwen3 because the reasoning tokens exceed max_gen_toks. What's the best way to handle this?

```
 lm-eval \
  --model local-chat-completions \
  --model_args "model=Qwen3-4B,base_url=http://127.0.0.1:8080/v1/chat/completions,num_concurrent=10,max_gen_toks=512" \
  --tasks gsm8k \
  --num_fewshot 0 \
  --apply_chat_template \
  --batch_size auto
```

```
  File "lm-eval/bin/lm-eval", line 10, in <module>
    sys.exit(cli_evaluate())
  File "lm-eval/lib/python3.9/site-packages/lm_eval/__main__.py", line 389, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "lm-eval/lib/python3.9/site-packages/lm_eval/utils.py", line 422, in _wrapper
    return fn(*args, **kwargs)
  File "lm-eval/lib/python3.9/site-packages/lm_eval/evaluator.py", line 308, in simple_evaluate
    results = evaluate(
  File "lm-eval/lib/python3.9/site-packages/lm_eval/utils.py", line 422, in _wrapper
    return fn(*args, **kwargs)
  File "lm-eval/lib/python3.9/site-packages/lm_eval/evaluator.py", line 546, in evaluate
    task.apply_filters()
  File "lm-eval/lib/python3.9/site-packages/lm_eval/api/task.py", line 1183, in apply_filters
    f.apply(self._instances)                                                                                                        File "lm-eval/lib/python3.9/site-packages/lm_eval/api/filter.py", line 51, in apply
    resps = f().apply(resps, docs)
  File "lm-eval/lib/python3.9/site-packages/lm_eval/filters/extraction.py", line 56, in apply    filtered_resps = list(map(lambda x: filter_set(x), resps))
  File "lm-eval/lib/python3.9/site-packages/lm_eval/filters/extraction.py", line 56, in <lambda>
    filtered_resps = list(map(lambda x: filter_set(x), resps))                                                                      File "lm-eval/lib/python3.9/site-packages/lm_eval/filters/extraction.py", line 41, in filter_set                                                                                                                                 match = self.regex.findall(resp)
TypeError: expected string or bytes-like object
```

What local inference engine are you using? 
512 toks is really low for a reasoning model; your final benchmark scores, even if the command succeeded to run, will be heavily underscored. 
There is a --write_out argument that can be passed (I'm not sure if it works with local-chat-completions, I've tested it with vllm) that prints out some samples with formatted chat template and some good info that might help you debug the error. 
With vllm generation, I did not face this issue even when the model response got truncated due to max_gen_toks, the extraction and verification methods implemented could handle this properly and report the predicted_answer as [invalid]. 

I would say the easiest workaround is to modify the max_gen_tokens = 8k or even 32k. 


> I would say the easiest workaround is to modify the max_gen_tokens = 8k or even 32k.

got GSM8K i would say 1-2k are more than enough even for reasoning model, the questions are direct, no complexity, just requires just arithmetic steps