Repository: UKGovernmentBEIS/inspect_ai
Issue: Questions About Evaluating Local Open-source Models with Tool-using Capabilities via vLLM Serving Using inspect_ai
Root cause: External Dependency Breakage

Hi there,

I'm reaching out because I've encountered some issues while using inspect_ai-based framework ([AgentHarm](https://huggingface.co/datasets/ai-safety-institute/AgentHarm/tree/main)) to evaluate local open-source models, and I was hoping you might have some insights into the potential causes.

I initially set up vLLM serving for the Qwen-2.5 series models following the approach described in the inspect_ai documentation (https://inspect.ai-safety-institute.org.uk/models.html#sec-vllm-server). This setup works fine with basic tool-using example programs.

However, when I try to run tests using [AgentHarm](https://huggingface.co/datasets/ai-safety-institute/AgentHarm/tree/main) (which is built on inspect_ai) with the command:

```bash
python inspect_eval.py --model_name openai/Qwen/Qwen2.5-14B-Instruct --prompt_template_name empty
```

All Qwen models, regardless of size, return the same error:

```
"error": "{'object': 'error', 'message': "[{'type': 'extra_forbidden', 'loc': ('body', 'tools', 0, 'function', 'strict'), 'msg': 'Extra inputs are not permitted', 'input': True, 'url': 'https://errors.pydantic.dev/2.9/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'tools', 1, 'function', 'strict'), 'msg': 'Extra inputs are not permitted', 'input': True, 'url': 'https://errors.pydantic.dev/2.9/v/extra_forbidden'}, {'type': 'extra_forbidden', 'loc': ('body', 'tools', 2, 'function', 'strict'), 'msg': 'Extra inputs are not permitted', 'input': True, 'url': 'https://errors.pydantic.dev/2.9/v/extra_forbidden'}]", 'type': 'BadRequestError', 'param': None, 'code': 400}"
```

I suspect this might be related to how vLLM serving handles tool-using support, but despite spending considerable time debugging, I haven't been able to identify the root cause. Do you have any thoughts on potential directions I could investigate?

Looking forward to your response!

Best,
Jinchuan

I am not at all sure how to debug this! This looks like a Pydantic validation error complaining about fields it doesn't expect. I would generally say that `extra_forbidden` isn't a great default when dealing with clients trying to make things work across services so one tact would be to convince vLLM to not validate that way (given that people will be taking code and libraries meant for use with OpenAI then pointing them at vLLM.



> I am not at all sure how to debug this! This looks like a Pydantic validation error complaining about fields it doesn't expect. I would generally say that `extra_forbidden` isn't a great default when dealing with clients trying to make things work across services so one tact would be to convince vLLM to not validate that way (given that people will be taking code and libraries meant for use with OpenAI then pointing them at vLLM.

Thanks! I just found the root cause - vLLM doesn't currently support the strict structured output feature (strict=True) for tools in the OpenAI API. I'll raise an issue or PR with their team about this.