Repository: pydantic/pydantic-ai

Issue #224: Tool calling issues with vllm
Created: 2024-12-12T13:14:19Z
Hi,
Is there a way where we can add the vLLM OpenAI compatibility support.
vllm OpenAI Support
So that anyone can use any llm model calls.

Duplicate issue #927: result_type does not work with an OpenAI compatible endpoint/model
Created: 2025-02-14T14:46:13Z
Hi folks,
I am trying to run pydantic-ai with Qwen family of models and encountered the following error:
When using a result_type in an Agent definition, the call fails.
This is a minimal code example to reproduce the issue. An example input for a prompt is "capital of Germany":
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from rich.prompt import Prompt

from pydantic import BaseModel

class MyModel(BaseModel):
    city: str
    country: str

model3 = OpenAIModel(
    model_name='qwen-max-latest',
    base_url='https://dashscope-intl.aliyuncs.com/compatible-mode/v1',
    api_key='sk-xxx',
)

query = Prompt.ask("Enter a query")

agent = Agent(model3, result_type=MyModel)
result = agent.run_sync(query)
print(result.data)
print(result.all_messages)
This leads to the following error:
  File "/Users/wladi/Documents/git/pydantic-demo/.venv/lib/python3.12/site-packages/pydantic_ai/agent.py", line 432, in run_sync
    return asyncio.get_event_loop().run_until_complete(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.5/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/wladi/Documents/git/pydantic-demo/.venv/lib/python3.12/site-packages/pydantic_ai/agent.py", line 340, in run
    end_result, _ = await graph.run(
                    ^^^^^^^^^^^^^^^^
  File "/Users/wladi/Documents/git/pydantic-demo/.venv/lib/python3.12/site-packages/pydantic_graph/graph.py", line 187, in run
    next_node = await self.next(next_node, history, state=state, deps=deps, infer_name=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wladi/Documents/git/pydantic-demo/.venv/lib/python3.12/site-packages/pydantic_graph/graph.py", line 263, in next
    next_node = await node.run(ctx)
                ^^^^^^^^^^^^^^^^^^^
  File "/Users/wladi/Documents/git/pydantic-demo/.venv/lib/python3.12/site-packages/pydantic_ai/_agent_graph.py", line 299, in run
    return await self._handle_text_response(ctx, texts, handle_span)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wladi/Documents/git/pydantic-demo/.venv/lib/python3.12/site-packages/pydantic_ai/_agent_graph.py", line 364, in _handle_text_response
    ctx.state.increment_retries(ctx.deps.max_result_retries)
  File "/Users/wladi/Documents/git/pydantic-demo/.venv/lib/python3.12/site-packages/pydantic_ai/_agent_graph.py", line 86, in increment_retries
    raise exceptions.UnexpectedModelBehavior(
pydantic_ai.exceptions.UnexpectedModelBehavior: Exceeded maximum retries (1) for result validation

If I remove the result_type parameter, or set it to str as the default, the code works fine (to exclude that the general setup works.)
I also tried the code with an OpenAI model and a Gemini model, and they work with result_type other than str.
What am I missing? Do I need to specify the final tool manually when using an OpenAI-compatible endpoint?

Closing comment (2025-02-14T20:03:49Z): This looks like a duplicate of issues reported in #224, going to close as such