zhiminy/EvalEng huggingface dataset:
Repository: EleutherAI/lm-evaluation-harness
Issue: Caching requests does not work for model gguf
Root cause: Configuration Error

I am encountering an issue, where the lm_eval evaluation with `--model gguf` does not cache the requests. The caching works for --local-completions on multiple machines, but not for `--model gguf` on any machine. When I set a limit it caches the requests once its done with the whole evaluation, but not during the evaluation process. Permissions are set correctly and the `database` is created at the beginning of the process, but nothing written to it. Would appreciate any help on this :)
Repository: EleutherAI/lm-evaluation-harness
Issue: hellaswag and arc_challenge produce lower accuracy than expected
Root cause: Documentation Deficiency

Hi! I'm using lm_evaluation_harness to evaluate my model on hellaswag and arc_challenge. However, the accuracy scores I obtained are significantly lower than reported. Below is the code I'm using:
```python
from transformers import AutoModelForCausalLM
from lm_eval import evaluator, tasks, models
model = AutoModel.from_pretrained('meta-llama/Llama-3.1-8B-Instruct', device_map='auto')
m_eval_model = models.huggingface.HFLM(pretrained=model)
acc = evaluator.simple_evaluate(
    model=lm_eval_model,
    tasks="hellaswag",
    num_fewshot=10,
    max_batch_size=64,
    limit=1024,
)['results']['hellaswag']['acc_norm,none']
print(acc)
```
The result I obtained for hellaswag and arc_challenge are 72.07 and 60.94, respectively, which are significantly lower than the [reported score](https://www.toolify.ai/ai-model/neuralmagic-meta-llama-3-1-8b-instruct-quantized-w8a16?utm_source=chatgpt.com) on Llama-3.1-8B-Instruct (80.01 and 81.57). Any idea why that happens?

Nevermind, I noticed the issue was related to my usage. Please refer to [this page](https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a16?utm_source=chatgpt.com) if you encountered similar issues.
Repository: EleutherAI/lm-evaluation-harness
Issue: clean up process results
Root cause: Algorithmic Error

The current implementation of `process_results()` contains logic (for `generate_until`) that can be counterintuitive in certain use cases:

1. If we pass targets as list, the `process_results()` logic will follow the multiple-target path in the [conditional logic here](https://github.com/EleutherAI/lm-evaluation-harness/blob/d693dcd2b46c59d5478960d0fba52fdc1174763d/lm_eval/api/task.py#L1659).  This becomes problematic in case we are using a custom metric which can handle list ground truth correctly.
2. The function always [casts](https://github.com/EleutherAI/lm-evaluation-harness/blob/82fe48ec1b400ceaba5e4b25c962ba62302641d6/lm_eval/api/task.py#L1654) targets to match the result type, which creates similar issues.
Repository: EleutherAI/lm-evaluation-harness
Issue: eval bbh failed
Root cause: Configuration Error

when evaluating LLama-3.1-8B finetuned by LLama-factory, the bbh's result is 0, however, when evaluating MMLU, everything is ok. suppose that the LLama-factory finetuned model's "eos_token" is  "
" while the offical version of meta-llama is "
", when i change the "
" to "
", the result is normal.
base model's tokenizer_config:
"bos_token": "
",
  "clean_up_tokenization_spaces": true,
  "eos_token": "
",

sft model finetuned by llama-factory:
"bos_token": "
",
  "chat_template": ...
  "clean_up_tokenization_spaces": true,
  "eos_token": "
",

meta-llama-3.1-8B-instruct:
"eos_token": "
",

+1, same issue with `Qwen/Qwen3-8B`

seems related https://github.com/EleutherAI/lm-evaluation-harness/issues/2095
Repository: EleutherAI/lm-evaluation-harness
Issue: Do we have plan to add BFCL into the tasks?
Root cause: Unimplemented Feature Gap

Given BFCL is being frequently used as benchmark for tool calling, I'm wondering if we have plan to include the tasks? Thanks.

+1 for this feature request. 

+1 for this feature request.

+1 for this feature request.
Repository: EleutherAI/lm-evaluation-harness
Issue: What is the correct way to save result file?
Root cause: Validation Gap

I use `output_path` arg to specify the saved file.
```bash
lm_eval --model sglang \
    --model_args "pretrained=${model_to_evaluate},dp_size=4,dtype=auto,mem_fraction_static=0.8" \
    --tasks bbh,mmlu_pro,hellaswag,gpqa_main_cot_n_shot,gsm8k_cot,arc_challenge_chat,truthfulqa_gen,mmlu_flan_cot_fewshot,ifeval\
    --batch_size 32\
    --output_path /mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/tests/clean_result/${model_name}_dirty.json
```
But I got:
```bash
/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/tests/clean_result/llama3.2-3b-ndp-422-olmo_dirty.json/__mnt__dolphinfs__ssd_pool__docker__user__hadoop-aipnlp__INS__ruanjunhao04__ndp__trained_model__llama3.2-3b-ndp-422-olmo__checkpoint-14676/results_2025-04-23T21-51-32.829014.json
```

The actually thing I want to get:
```bash
/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/tests/clean_result/llama3.2-3b-ndp-422-olmo_dirty.json
```


The following logic is different from the arg description as it could only be a directory instead of dir/file.json
https://github.com/EleutherAI/lm-evaluation-harness/blob/e4a7b69fe0fc6cb430e12cf15c4109bf28185124/lm_eval/__main__.py#L132-L139


https://github.com/EleutherAI/lm-evaluation-harness/blob/e4a7b69fe0fc6cb430e12cf15c4109bf28185124/lm_eval/loggers/evaluation_tracker.py#L231-L233
Repository: EleutherAI/lm-evaluation-harness
Issue: How to use T5 related models for evaluation?
Root cause: Documentation Deficiency

Hi, thank you for your effort to make this tool ! I am curious if there is any possibility to use T5 related models for evaluation. It seems that this tool only support GPT related models. 

It is possible. We haven't added support in this library yet, but the BigScience Workshop fork of this library does have it. You can find their repo [here](https://github.com/bigscience-workshop/lm-evaluation-harness)

For those coming from Google, I had no problems evaluating a T5 model on `lm-eval` v2 (i.e. the `big-refactor` branch if it hasn't been merged to `master` by the time you read this).

We upstreamed this months ago and forgot to close this issue I suppose. We now support all models that are loaded with the either `transformers.AutoModelForCausalLM` or `transformers.AutoModelForSeq2SeqLM`. This includes custom local architectures that are not in the main library.
Repository: EleutherAI/lm-evaluation-harness
Issue: Support for OlympiadBench and AMC
Root cause: Documentation Deficiency

Thank you for your contributions to the open source community

I would like to ask if the current evaluation framework supports the two mathematical task sets of OlympiadBench_MATH and AMC?

Hello, I have a simple suggestion:

You can try copying the aime folder and tweaking the yaml file within it. Generally, you just need to modify the following fields based on your target task and dataset structure:

task

dataset_path

test_split

doc_to_text

doc_to_target

I have already easily completed the evaluation task on AMC23 this way. Although I haven't specifically tested it on the OlympiadBench_MATH dataset, based on my observation of it, the principle should be the same.
Repository: EleutherAI/lm-evaluation-harness
Issue: llama3-8b-instruct 0-shot mbpp pass@1=0
Root cause: Unimplemented Feature Gap

**issue**
llama3-8b-instruct 0-shot mbpp pass@1=0

**reproduce:**
command:
lm_eval --model hf \
    --model_args pretrained=meta-llama/Meta-Llama-3-8B-Instruct \
    --tasks mbpp,mbpp_plus \
    --device cuda:0 \
    --batch_size 8 \
    --output_path outputs \
    --num_fewshot 0 \

output:
pass_at_1 value: 0 stderr: 0

What could be the problem?

same for qwen

Hi! added instruct variants in the PR. closing this issue, and let's discuss in #2990 or the PR.
Repository: EleutherAI/lm-evaluation-harness
Issue: how do evaluate mmlu by local-chat-completions or local-completions?
Root cause: Documentation Deficiency

command：
lm_eval --model gguf --tasks mmlu --model_args model=/workspace/model/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf,tokenizer=/mnt/f/deepseek-r1-tokenizer/,base_url=http://0.0.0.0:8084,tokenized_requests=False --apply_chat_template=default_chat_template --batch_size 16 --limit 1

I deploy a model by llamacpp server by openai interface, add tokenized_requests=False, but post message also is input_id. 


no reply, I complete it by myself.
correct command:
lm_eval --model local-chat-completions --tasks mmlu --model_args model=/workspace/model/DeepSeek-R1-UD-IQ1_S/DeepSeek-R1-UD-IQ1_S-00001-of-00003.gguf,tokenizer=/mnt/f/deepseek-r1-tokenizer/,base_url=http://0.0.0.0:8084,tokenized_requests=False/ --apply_chat_template --batch_size 16 --limit 1
Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError: top_k_top_p_sampling_from_probs() missing 1 required positional argument: 'top_p'
Root cause: Interface Contract Mismatch

```bash
lm_eval --model sglang \
    --model_args "pretrained=${model_to_evaluate},dp_size=4,dtype=auto,mem_fraction_static=0.8" \
    --tasks bbh,mmlu_pro,hellaswag,gpqa_main_cot_n_shot,gsm8k_cot,arc_challenge_chat,truthfulqa_gen,mmlu_flan_cot_fewshot,ifeval\
    --batch_size 32\
    --output_path /mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/ndp/tests/clean_result/${model_name}_dirty.json
```

```bash
TpModelWorkerClient hit an exception: Traceback (most recent call last):
  File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/miniforge3/envs/sglang/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 112, in forward_thread_func
    self.forward_thread_func_()
  File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/miniforge3/envs/sglang/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/miniforge3/envs/sglang/lib/python3.12/site-packages/sglang/srt/managers/tp_worker_overlap_thread.py", line 143, in forward_thread_func_
    logits_output, next_token_ids = self.worker.forward_batch_generation(
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/miniforge3/envs/sglang/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 184, in forward_batch_generation
    next_token_ids = self.model_runner.sample(logits_output, model_worker_batch)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/miniforge3/envs/sglang/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1093, in sample
    next_token_ids = self.sampler(
                     ^^^^^^^^^^^^^
  File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/miniforge3/envs/sglang/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/miniforge3/envs/sglang/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/dolphinfs/ssd_pool/docker/user/hadoop-aipnlp/INS/ruanjunhao04/miniforge3/envs/sglang/lib/python3.12/site-packages/sglang/srt/layers/sampler.py", line 105, in forward
    batch_next_token_ids = top_k_top_p_sampling_from_probs(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: top_k_top_p_sampling_from_probs() missing 1 required positional argument: 'top_p'
```
Repository: EleutherAI/lm-evaluation-harness
Issue: please add model phi-4-multimodal, Qwen omni
Root cause: Unimplemented Feature Gap

<img width="982" alt="Image" src="https://github.com/user-attachments/assets/9a41522c-b4b5-40aa-b6f5-13867fe5f4e7" />

I'm hitting this for Phi multimodal as well:

```
lm_eval --model hf-multimodal --model_args pretrained="microsoft/Phi-3.5-vision-instruct",dtype=auto,add_bos_token=True,convert_img_format=True,trust_remote_code=True --tasks mmmu_val --num_fewshot 0
...
    self._model = self.AUTO_MODEL_CLASS.from_pretrained(
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 574, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.microsoft.Phi-3.5-vision-instruct.4a0d683eba9f1d0cbfb6151705d1ee73c25a80ca.configuration_phi3_v.Phi3VConfig'> for this kind of AutoModel: AutoModelForVision2Seq.
Model type should be one of BlipConfig, Blip2Config, ChameleonConfig, GitConfig, Idefics2Config, Idefics3Config, InstructBlipConfig, InstructBlipVideoConfig, Kosmos2Config, LlavaConfig, LlavaNextConfig, LlavaNextVideoConfig, LlavaOnevisionConfig, Mistral3Config, MllamaConfig, PaliGemmaConfig, Pix2StructConfig, Qwen2_5_VLConfig, Qwen2VLConfig, VideoLlavaConfig, VipLlavaConfig, VisionEncoderDecoderConfig.
```

95% sure this is because `lm_eval` is attempting to use `AutoModelForVision2Seq`, when it needs to be `AutoModelForCausalLM` as shown [here](https://huggingface.co/microsoft/Phi-4-multimodal-instruct/blob/main/sample_inference_phi4mm.py) (not sure why Microsoft put it in CausalLM and not Vision2Seq 🤷 ).

Is this a setting we have control over?
Repository: EleutherAI/lm-evaluation-harness
Issue: How to test those think/no_think model like Qwen3
Root cause: Documentation Deficiency

Run some test on Qwen-3-32B with vllm and gsm8k

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.6285|±  |0.0133|
|     |       |strict-match    |     5|exact_match|↑  |0.7460|±  |0.0120|

which I think is much below it should be.

I'm running with follow cmd
```
export MODEL=./Qwen3-32B
lm_eval \
  --model vllm \
  --model_args pretrained=$MODEL,add_bos_token=True,tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.9 \
  --tasks gsm8k  --num_fewshot 5 --batch_size auto \
  --fewshot_as_multiturn --apply_chat_template
```

Should I try fix the chat template to make Qwen-3 don't use reason?

I weigh in on this comment too.

I‘ve add ```--system_instruction "You are a helpful assistant. /no_think.``` to get qwen3 no_think mode.

Some benchmark use logprob need low gpu_memory_utilization to 0.5 which cause lm_eval run very slowly.
Repository: EleutherAI/lm-evaluation-harness
Issue: Livecodebench,AIME24 datasets
Root cause: Unimplemented Feature Gap

Hi,
The recent benchmarks are Livecodebench,AIME24,Codeforces etc.
I want to test my Qwen3 models on these benchmarks by giving API endpoint.
I dont see these benchmarks are supported in tasks.
Can you please guide.

I am having the same needs

The same

+1 for Livecodebench
Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval multimodal failing for any num_fewshot>0
Root cause: Interface Contract Mismatch

This seemed pretty consistent across Qwen VL models. On python 3.10.12, `lm_eval==0.4.8`, `torch==2.7.0`, `transformers==4.51.3`, the command
```
lm_eval \
  --model hf-multimodal \
  --model_args pretrained="Qwen/Qwen2.5-VL-7B-Instruct",dtype=auto,add_bos_token=True,convert_img_format=True \
  --tasks mmmu_val \
  --num_fewshot 5
```

triggers the following error

```
2025-05-01:21:30:56,104 INFO     [lm_eval.evaluator:517] Running generate_until requests
Running generate_until requests with text+image input:   0%|                                                                                             | 0/900 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/bdellabe/projects/.venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/lm_eval/__main__.py", line 389, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/lm_eval/utils.py", line 422, in _wrapper
    return fn(*args, **kwargs)
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/lm_eval/evaluator.py", line 308, in simple_evaluate
    results = evaluate(
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/lm_eval/utils.py", line 422, in _wrapper
    return fn(*args, **kwargs)
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/lm_eval/evaluator.py", line 528, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/lm_eval/models/hf_vlms.py", line 671, in generate_until
    inputs = self.tok_batch_multimodal_encode(
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/lm_eval/models/hf_vlms.py", line 306, in tok_batch_multimodal_encode
    encoding = self.processor(
  File "/home/bdellabe/projects/.venv/lib/python3.10/site-packages/transformers/models/qwen2_5_vl/processing_qwen2_5_vl.py", line 167, in __call__
    "
" * (image_grid_thw[index].prod() // merge_length),
IndexError: index 1 is out of bounds for dimension 0 with size 1
```
The processors is expecting several images as input, but only ever gets 1. 

Seems like somewhere in the code more images need passed in. 

Happy to work on this if you can point me to where this needs to be added!
Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate input, response pair data saved with log_samples
Root cause: Documentation Deficiency

Hi!

I am trying to save the outputs of running a task (IFEval) on a given model through the log_samples=True. The json seems pretty detailed and it should be possible to get the metrics from the json directly without running the evaluation again. I need this to evaluate a subset of the prompts based on several filters.

Is there an interface to do this?

Hi! A quick way would be:

```python
from lm_eval.tasks import TaskManager, get_task_dict
from collections import defaultdict

resps = {"ifeval": [[["hello world"]], [["hello"]]]} # from samples jsonl, sorted by doc_id

res = defaultdict(list)
task_manager = TaskManager()
tasks = get_task_dict(["ifeval"])
for task_name, task in tasks.items():
    task.build_all_requests(limit=2)
    for i, instance in enumerate(task.instances):
        instance.resps = resps[task_name][i][0] # instance.resps are list[str]
    task.apply_filters()
    for i, (doc_id, doc) in enumerate(task.doc_iterator(limit=2)):
        # Process each filter for this document
        for filter_key in task.instances[i].filtered_resps.keys():
            filtered_resp = task.instances[i].filtered_resps[filter_key]
            metric = task.process_results(doc, [filtered_resp]) # dict of each metric for this sample
            res[filter_key].append(metric)
```
Hope this helps!
Repository: EleutherAI/lm-evaluation-harness
Issue: Add textsynth.com API support
Root cause: Unimplemented Feature Gap

The attached patch adds support for the models supported by the textsynth.com API.

[lm_evaluation_harness_textsynth.tar.gz](https://github.com/EleutherAI/lm-evaluation-harness/files/8193236/lm_evaluation_harness_textsynth.tar.gz)
Repository: EleutherAI/lm-evaluation-harness
Issue: Allow tasks to register a metric
Root cause: Documentation Deficiency

I'm a noob with respect to lm_eval, but here's an attempt at a feature request that might make sense.

It would be pretty cool if tasks could use the @register_metric decorator to register their own custom metrics. I was trying to copy the "contains" metric used in the squad_completion task. It would be very slick if you could include a utils.py in your task directory with something vaguely like:

```python
@register_metric(
    metric="contains",
    higher_is_better=True,
    output_type="generate_until",
    aggregation="mean",
)
def contains_score(
    predictions: List[str],
    references: List[str],
    ignore_case = True,
):
    flags = re.IGNORECASE if ignore_case else 0
    return np.mean([
        max(
            int(bool(re.search(re.compile(re.escape(label), flags), pred)))
            for label in references
        )
        for pred in predictions
    ])
```

Hi! We do allow tasks to use their own metric function using the `!function` constructor (at least for `yaml` based tasks; [example](https://github.com/EleutherAI/lm-evaluation-harness/blob/a96085f1515fe87af44350b01094bee248515356/lm_eval/tasks/humaneval/humaneval.yaml#L9)). Python based tasks like `squad_completion` overload the `process_resuts` method (where the metric function is used), so the expectation so the expectation is it will be handled directly in the task class.

Otherwise a global metric can be added in `lm_eval/api/metrics.py`

Hope this answers your issue!
Repository: EleutherAI/lm-evaluation-harness
Issue: Is this result reasonable, please?

 I evaluated the llama2-7b-hf model using the SQuADv2 dataset in a 1-shot setting. From what I can see, the original Llama 2 documentation (or paper) does not use the SQuADv2 dataset, only the SQuAD dataset.
![Image](https://github.com/user-attachments/assets/6f8d27a3-196b-429b-8e52-9ce400e06146)
Repository: EleutherAI/lm-evaluation-harness
Issue: Log truncation/max_length to logged samples
Root cause: Unimplemented Feature Gap

Is there any way to know if  input tokens are truncated  due to max_length? 

We generally add warning logs (currently in HF, adding to vllm in #2962). Probably also a good idea to add it to saved samples. Will look into it!

thanks a lot!
Repository: EleutherAI/lm-evaluation-harness
Issue: Performance bottleneck: consider multiprocessing for cached request checking
Root cause: Algorithmic Error


Currently, the following loop runs in a single process, which can be a bottleneck when checking a large number of cached requests:

https://github.com/EleutherAI/lm-evaluation-harness/blob/a96085f1515fe87af44350b01094bee248515356/lm_eval/api/model.py#L264-L284

Since this part of the code is only computing hashes and checking for existence in self.dbdict, it seems like a good candidate for parallelization using multiprocessing.

Would it be possible to explore using Python’s multiprocessing module or similar (e.g., concurrent.futures.ProcessPoolExecutor) to speed up this step?

This could significantly reduce the latency during evaluation or repeated runs, especially when dealing with a large number of requests.

Let me know if this is something you’d be open to – I’d be happy to explore or contribute a PR if helpful.
Repository: EleutherAI/lm-evaluation-harness
Issue: Enable thinking
Root cause: Documentation Deficiency

Hi

I'm now trying to test one of the Qwen's CoT models. But, I wanted to compare it with and without the thinking process. 
It should be able to disable that function in the apply_chat_template but I don't know where to modify. 

Can you guys point out which part I could change so that I can turn on/off the function? 

Thanks!

Hi! It was recently merged for vllm in #2947 (`enable_thinking` in `model_args`). For hf the corresponding method to modify is [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/2f03271d25db3c19e5552e19f59816bcbba07357/lm_eval/models/huggingface.py#L1413)

Amazing! Just what I was looking for haha. Thanks for the comment. Closing issue!