Repository: pydantic/pydantic-ai

Issue #242: Use ollama's structured outputs feature
Created: 2024-12-13T15:35:45Z
I'm looking into pydantic-ai with small and locally running ollama models as backbones.
I'm noticing that sometimes even for simple models it's possible to run into unexpected ValidationErrors.
Here's what I mean: With a pydantic model as simple as
class Answer(BaseModel):
    value: str = ""
I can see pytandic-ai sometimes retrying and failing in validation.
Having experience with llama.cpp's grammars this was unexpected to me. I was under the assumption that pydantic-ai would transform the pydantic model into a grammar or json schema to hard-restrict the llm's output accordingly. Then validation could never fail by design since the llm's output is restricted to the specific grammar.
Instead when I debug the request pydantic-ai sends to the locally running ollama with
nc -l -p 11434

I can see pydantic-ai turning the pydantic model into a tool use invocation.
With ollama v0.5.0 structured output via json schema is now supported:
https://github.com/ollama/ollama/releases/tag/v0.5.0
I was wondering if that would solve the issue of small locally running models sometimes running into validation errors, since we hard-restrict the output to the shape of our pydantic model.
Any thoughts on this, or ideas why validation can fail with tool usage as implemented right now? Any pointers in terms of for which model providers validation might fail and for what reason? Thanks!

Duplicate issue #607: Validation with nested Pydantic models (ollama, llama3.1)
Created: 2025-01-03T16:13:25Z
When using the pydantic_ai library with a nested BaseModel, an UnexpectedModelBehavior error occurs, despite the underlying model (e.g., ollama:llama3.1) being capable of handling the requested structure and providing valid output.
The example here https://ai.pydantic.dev/examples/pydantic-model/#running-the-example is fully functional with ollama:llama3.1, but this slight modification to include nested models fails to work:
import os
from typing import cast

import logfire
from pydantic import BaseModel

from pydantic_ai import Agent
from pydantic_ai.models import KnownModelName

logfire.configure(send_to_logfire='if-token-present')

class Country(BaseModel):
    name: str

class MyModel(BaseModel):
    city: str
    country: Country

model = cast(KnownModelName, os.getenv('PYDANTIC_AI_MODEL', 'ollama:llama3.1'))
print(f'Using model: {model}')
agent = Agent(model, result_type=MyModel)

if __name__ == '__main__':
    result = agent.run_sync('The windy city in the US of A.')
    print(result.data)
    print(result.usage())
I get this error:
UnexpectedModelBehavior: Exceeded maximum retries (1) for result validation

It seems that the model fails to generate the expected nested structure.
Validation of the Model's Capability:
To confirm that the underlying model (ollama:llama3.1) supports this functionality, the following custom implementation was tested:
from ollama import chat, Options
from typing import Type, TypeVar
from pydantic import BaseModel

T = TypeVar('T', bound=BaseModel)

class Agent:
    def __init__(self, model: str, result_type: Type[T], options: Options = None):
        self.model = model.split(":")[1]
        self.result_type = result_type
        self.options = options

    def run_sync(self, prompt: str) -> T:
        response = chat(
            messages=[
                {
                    'role': 'user',
                    'content': prompt
                }
            ],
            model=self.model,
            format=self.result_type.model_json_schema(),
            options=self.options
        )
        if response.message.content is None:
            raise Exception("No response from the model")
        else:
            o_data: str = response.message.content
            o: T = self.result_type.model_validate_json(o_data)
            return o

class Country(BaseModel):
    name: str

class MyModel(BaseModel):
    city: str
    country: Country

model = cast(KnownModelName, os.getenv('PYDANTIC_AI_MODEL', 'ollama:llama3.1'))
print(f'Using model: {model}')
agent = Agent(model, result_type=MyModel)

if __name__ == '__main__':
    result = agent.run_sync('The windy city in the US of A.')
    print(result)
This implementation returned the expected result without any errors:
city='Chicago' country=Country(name='United States')

Environment:

Python version: 3.10+
pydantic_ai[examples]>=0.0.17

Could I be misunderstanding or misusing the library?

Closing comment (2025-01-07T08:49:50Z): I think it's a partial duplicate. #242 seems more focused on the fact that Ollama doesn't always decide to use the tool when provided a response model but this issue is running into issues with Ollama providing a response that adheres to the provided response model but the OpenAI model isn't set up to decode properly.