Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval on squadv2 and meta-llama/Meta-Llama-3.1-8B fails with TypeError: Instance.__init__() got an unexpected keyword argument 'apply_chat_template'
Root cause: Interface Contract Mismatch

```
lm_eval --model hf --model_args pretrained=meta-llama/Meta-Llama-3.1-8B --tasks squadv2 --batch_size 8
```

causes
```
File "/workspace/lm-evaluation-harness/lm_eval/tasks/squadv2/task.py", line 121, in construct_requests
   Instance(
TypeError: Instance.__init__() got an unexpected keyword argument 'apply_chat_template'
```


It is caused by latest [MR](https://github.com/EleutherAI/lm-evaluation-harness/pull/2475).

[`Instance`](https://github.com/EleutherAI/lm-evaluation-harness/blob/9d36354e6422db94c374dc476c093b1495370983/lm_eval/api/instance.py#L11) does not expect `apply_chat_template` but it is provided [here](https://github.com/EleutherAI/lm-evaluation-harness/blob/9d36354e6422db94c374dc476c093b1495370983/lm_eval/api/task.py#L452)

Apparently this is still an open issue, I cannot run squadv2 as well.

Any updates on this?

What I did was to solve this is to pop `apply_chat_template` out of `kwargs` before passing it to `Instance`: `apply_chat_template = kwargs.pop("apply_chat_template", False)` inside of the `construct_requests` method in `/workspace/lm-evaluation-harness/lm_eval/tasks/squadv2/task.py`

can you share your final `construct_requests ` method? I did what you mentioned above but still get the same error. TIA!

I use this to eval Qwen2.5 and encountered this error
lm_eval/evaluator.py at simple_evaluate, before Line 196, between ############ 

https://github.com/EleutherAI/lm-evaluation-harness/blob/29971faaede3e0459eef6d1b4ec0f58460bed510/lm_eval/evaluator.py#L196

```
    if isinstance(model, str):
        if model_args is None:
            eval_logger.warning("model_args not specified. Using defaults.")
            model_args = ""

        if isinstance(model_args, dict):
            eval_logger.info(
                f"Initializing {model} model, with arguments: {model_args}"
            )
            
            ##############################################################
            print('here is a patching at /usr/local/lib/python3.10/site-packages/lm_eval/evaluator.py')
            print('before',model_args)
            model_args = {k: v for k, v in model_args.items() if k not in ['apply_chat_template','fewshot_as_multiturn']}
            print('after',model_args)
            # before {'pretrained': './saves/merged/mergedjgo30mjk', 'dtype': 'bfloat16', 'apply_chat_template': True, 'fewshot_as_multiturn': True, 'use_cache': True}
            # after {'pretrained': './saves/merged/mergedjgo30mjk', 'dtype': 'bfloat16', 'use_cache': True}
            ##############################################################

            lm = lm_eval.api.registry.get_model(model).create_from_arg_obj(
                model_args,
                {
                    "batch_size": batch_size,
                    "max_batch_size": max_batch_size,
                    "device": device,
                },
            )

        else:
            eval_logger.info(
                f"Initializing {model} model, with arguments: {simple_parse_args_string(model_args)}"
            )
```