Repository: ollama
Issue: StructuredOutputs Schema Missing in Prompt [Unlike OpenAI API Default Behavior]
Labels: feature request

When using StructuredOutputs, I noticed that the model's outputs were nonsensical and didn't align with expectations.

After debugging, I discovered that the output schema isn't included in the prompt, leaving the model unaware of its options and what it should generate. While developers could manually add the schema to the prompt, this isn't a common practice. For instance, OpenAI's API automatically includes the schema (easily verified by counting input prompt tokens), and I believe this behavior should be standard for all inference engines.

I tested Ollama in three different ways, and all of them exhibited the same behavior. Below is the code to reproduce the issue:

```python
import aiohttp
from pydantic import BaseModel, Field
from typing import Union, List, Literal

from typing import Annotated


class Action1(BaseModel):
    type: Literal["action1"]
    __doc__: str = "store in memory"

class Action2(BaseModel):
    type: Literal["action2"]
    __doc__: str = "call someone"

class Action3(BaseModel):
    type: Literal["action3"]
    __doc__: str = "move to a location"

class Response(BaseModel):
    steps: Annotated[
        List[Union[Action1, Action2, Action3]],
        Field(..., description='Sequence of steps to perform', discriminator='type')
    ]

messages = [
     {"role": "system", "content": "Decompose input request into sequence of steps. Possible steps are listed in the response schema."},
     {"role": "user", "content": "I want to go to the park."},
]

payload = {
    "model": 'qwen2.5-coder:7b',
    "messages": messages,
    "stream": False,
    "options": {"temperature" : 0.0},
    "format": Response.model_json_schema()
}

async def get_response(payload):
    async with aiohttp.ClientSession() as session:
        async with session.post('http://localhost:11434/api/chat', json=payload) as response:
            json_response = await response.json()
            raw_string = json_response['message']['content']
            res = Response.model_validate_json(raw_string)
            return json_response, res

json_response, res = await get_response(payload)

# json_response['prompt_eval_count'] shows 40 prompt tokens
# res.model_dump() shows `{'steps': [{'type': 'action1'}, {'type': 'action2'}]}`, which doesn't make any sense
# the correct answer for "I want to go to the park." is Action 3 ("move to a location"), not 1 or 2

messages_w_schema = [
     {
         "role": "system",
         "content":
          "Decompose input request into sequence of steps. Possible steps are listed in the response schema."
          + f'\nSCHEMA:\n{Response.model_json_schema()}'
     },
     {"role": "user", "content": "I want to go to the park."},
]

payload_new = {
    "model": 'qwen2.5-coder:7b',
    "messages": messages_w_schema,
    "stream": False,
    "options": {"temperature" : 0.0},
    "format": Response.model_json_schema()
}

json_response_2, res_2 = await get_response(payload_new)

# json_response_2['prompt_eval_count'] shows 328 prompt tokens
# res_2.model_dump() correclty suggests action3

#============

from ollama import chat

chat_res_1 = chat(
    model=payload['model'],
    messages=messages,
    format=Response.model_json_schema(),
    options={'temperature': 0}
)

chat_res_2 = chat(
    model=payload['model'],
    messages=messages_w_schema,
    format=Response.model_json_schema(),
    options={'temperature': 0}
)

# chat_res_1.prompt_eval_count = 40, chat_res_2.prompt_eval_count = 328
# Response.model_validate_json(chat_res_1.message.content) again points to actions 1 & 2, not 3

#================
from openai import OpenAI
client = OpenAI(
    base_url = 'http://localhost:11434/v1',
    api_key='ollama',
)

oai_client_1 = client.beta.chat.completions.parse(
    model=payload['model'],
    messages=payload['messages'],
    response_format=Response,
    temperature=0
)

oai_client_2 = client.beta.chat.completions.parse(
    model=payload_new['model'],
    messages=payload_new['messages'],
    response_format=Response,
    temperature=0
)

# same here: oai_client_1.choices[0].message.parsed contains actions 1 & 2, not 3
# and input prompt token counts are the same
```

As you might see, we have 3 actions: memory, call and navigation. We ask an llm to decompose input query into sequence of actions. For this example (`I want to go to the park.`) we expect model to output 3rd action. There is no way for the model to understand which action to choose other than parsing that info from the output schema. Without it, model just randomly guesses, and is not able to produce anything meaningful. 

====
The biggest issue I see is that even though Ollama is OpenAI compatible, it doesn't have the same logic under the hood, which might surprise a lot of developers when switching from proprietary LLMs to local ones. 

Proposal: by default, add output schema to inputs, and maybe allow to disable that behaviour with a flag. Given that the functionality was added not so long ago, I believe it is important to fix it RN, before the workaround logic with system prompt patching will be widely spread. 

===

My library versions:
- pip: ollama==0.4.4
- ollama CLI version is 0.5.1
- pip: openai=='1.57.3'