Repository: EleutherAI/lm-evaluation-harness
Issue: lm_eval incompatible with mistralai/Mistral-Small-3.2-24B-Instruct-2506
Root cause: Interface Contract Mismatch

Hello!
I am trying to evaluate Mistral Small in the following way:

- Spin up a vLLM server as in the [model card ](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506)
- Launch the evaluation: `lm_eval --model openai-chat-completions --model_args model=mistralai/Mistral-Small-3.2-24B-Instruct-2506,base_url=http://localhost:8000/v1/chat/completions --tasks mcqa --include tasks --apply_chat_template`

I am getting this Warning from lm_eval:
```2025-08-28:13:54:34 WARNING  [models.api_models:438] API request failed with error message: {"error":{"message":"3 validation errors for ChatCompletionRequest\nmessages.0.system.type\n  Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden\nmessages.1.user.type\n  Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden\nmessages.2.assistant.type\n  Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden 3 validation errors for ChatCompletionRequest\nmessages.0.system.type\n  Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden\nmessages.1.user.type\n  Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden\nmessages.2.assistant.type\n  Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden","type":"BadRequestError","param":null,"code":400}}. Retrying...```
vLLM side instead I got this:
```
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245] messages.0.system.type
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245]   Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245]     For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245] messages.1.user.type
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245]   Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245]     For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245] messages.2.assistant.type
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245]   Extra inputs are not permitted [type=extra_forbidden, input_value='text', input_type=str]
(APIServer pid=294697) ERROR 08-28 13:54:34 [serving_chat.py:245]     For further information visit https://errors.pydantic.dev/2.11/v/extra_forbidden
```
My task code is the following:
```
task: mcqa
dataset_path: 'my_dataset'
test_split: 'train'
doc_to_text: !function utils_mcqa.doc_to_text
doc_to_target: 'output'
process_docs: !function utils_mcqa.process_dataset
process_results: !function utils_mcqa.process_results
gen_prefix: 'Answer: '
description: 'The following are multiple choice questions (with answers)\n'
metric_list:
  - metric: "IoU"
    aggregation: mean
    higher_is_better: true
  - metric: "acc"
    aggregation: mean
    higher_is_better: true
metadata:
  version: 2.1
dataset_kwargs:
  trust_remote_code: true
num_fewshot: 2
fewshot_split: 'train'
fewshot_config:
  sampler: first_n

```

I have already tried the task with openai gpt-4o and it is working fine. Any idea on what could be the problem?