Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval harness with mmlu_pro and thinking models
Root cause: Validation Gap

Hi, I am trying to run the mmlu_pro eval on [moonshotai/Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking) deployed with SGLang. However, since this is a thinking model similar to https://github.com/EleutherAI/lm-evaluation-harness/issues/3044 the response is being returned as none. This even happens when setting `max_gen_tokens ` to a large value like 32k.

I am running this command `lm_eval --model local-chat-completions --tasks mmlu_pro_biology --model_args model=moonshotai/Kimi-K2-Thinking,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=128,timeout=12000 --log_samples --output_path kimi-mmlu --fewshot_as_multiturn --apply_chat_template --gen_kwargs=max_gen_toks=32000` and see this stack trace

```
2025-11-09:17:21:01 INFO     [__main__:450] Selected Tasks: ['mmlu_pro_biology']
2025-11-09:17:21:01 INFO     [evaluator:202] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2025-11-09:17:21:01 WARNING  [evaluator:214] generation_kwargs: {'max_gen_toks': 32000} specified through cli, these settings will update set parameters in yaml tasks. Ensure 'do_sample=True' for non-greedy decoding!
2025-11-09:17:21:01 INFO     [evaluator:240] Initializing local-chat-completions model, with arguments: {'model': 'moonshotai/Kimi-K2-Thinking', 'base_url':
        'http://127.0.0.1:8000/v1/chat/completions', 'num_concurrent': 128, 'timeout': 12000}
2025-11-09:17:21:01 INFO     [models.api_models:172] Using max length 2048 - 1
2025-11-09:17:21:01 INFO     [models.api_models:193] Using tokenizer None
2025-11-09:17:21:02 INFO     [evaluator:305] mmlu_pro_biology: Using gen_kwargs: {'until': ['Question:'], 'max_gen_toks': 32000, 'do_sample': False, 'temperature': 0.0}
2025-11-09:17:21:02 WARNING  [evaluator:480] Chat template formatting change affects loglikelihood and multiple-choice tasks. See docs/chat-template-readme.md for details.
2025-11-09:17:21:02 INFO     [api.task:434] Building contexts for mmlu_pro_biology on rank 0...
100%|████████████████████████████████████████| 717/717 [00:00<00:00, 958.77it/s]
2025-11-09:17:21:03 INFO     [evaluator:574] Running generate_until requests
2025-11-09:17:21:03 INFO     [models.api_models:733] Tokenized requests are disabled. Context + generation length is not checked.
Requesting API: 100%|█████████████████████████| 717/717 [08:30<00:00,  1.40it/s]
Traceback (most recent call last):
  File "/home/prithudasgupta_google_com/venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/__main__.py", line 459, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/utils.py", line 458, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/evaluator.py", line 357, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/utils.py", line 458, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/evaluator.py", line 600, in evaluate
    task.apply_filters()
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/api/task.py", line 1246, in apply_filters
    f.apply(self._instances)
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/api/filter.py", line 51, in apply
    resps = f().apply(resps, docs)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/filters/extraction.py", line 56, in apply
    filtered_resps = list(map(lambda x: filter_set(x), resps))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/filters/extraction.py", line 56, in <lambda>
    filtered_resps = list(map(lambda x: filter_set(x), resps))
                                        ^^^^^^^^^^^^^
  File "/home/prithudasgupta_google_com/lm-evaluation-harness/lm_eval/filters/extraction.py", line 41, in filter_set
    match = self.regex.findall(resp)
            ^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'NoneType'
```

For reference, I am serving the model with SGLang like so.

```
docker run --gpus all \
-v /home/prithudasgupta_google_com/models:/models \
--shm-size=32g \
-e NVIDIA_DISABLE_REQUIRE=1 \
--rm --name "sgl" \
-p 8000:8000 \
--entrypoint /bin/bash \
-it lmsysorg/sglang:v0.5.5

python -m sglang.launch_server --model-path moonshotai/Kimi-K2-Thinking --tp 8 --trust-remote-code  --tool-call-parser kimi_k2 --reasoning-parser kimi_k2 --port=8000 --host=0.0.0.0
```


+1, same issue with gpt-oss-120B on gms8k, ran with vllm. Reverting to `--local-completions` "fixes" it

Should be fixed with PR #3395