Repository: ollama
Issue: Better Tool Call parsing
Labels: feature request

Currently tool call patterns are defined in go templates. this is fine for cases [e.g. in this comment](https://github.com/ollama/ollama/issues/6061#issuecomment-2257137350). However, it is not ideal.

## Problems

1. Content loss

To say, the model responds this text:
```plaintext
Yes, I can help you compute 3+4 with python
<tool_call>
{"name":"python", "args": {"expr":"3+4"}}
</tool_call>
```
In [this line](https://github.com/ollama/ollama/blob/cd5c8f6471abf32965289f0226016a78f0c5c938/server/routes.go#L1480), all content are removed. So if the model provided some information like the first sentence, it is ignored arbitrarily.

2. Streaming

Tool calls does NOT support streaming, it only works when it got the full content. However we all know we can process for tool calls even dont know all content.

3. Other format support

[here](https://github.com/ollama/ollama/blob/d05da2991245cfa0cd8da0bda476c626e26caaec/server/model.go#L301), this function only supports json, shall we support other formats, for example XML, in the future?

## Solution

**TL;DR:** For 1. and 2., we can use [Aho–Corasick algorithm](https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm) for parsing. Define the pattern such as `@@json{name, args}@@` means `{"name":"python", "args": {"expr":"3+4"}}` (e.g. llama3.2) and the pattern `<tool_call>@@json{name,args}@@</tool_call>` could be used for the output above.

When you are not in stream mode we can still process it like a stream. So each time a token received from the state machine, we should do some process to see if the token is part of the tool call syntax. This fundamentally resolves the first problem, because we only remove the part of output that we are sure it is part of tool call.

So I think a state machine should be very nice to "guess" if it is a valid tool call. We should keep in mind that the model may also put out some normal jsons (e.g. user ask it to process a json file), so never regard a json as tool call arbitrarily, we are "guessing" it. When we got a new token, we try to match it with our state machine. **If it matches successfully, it is possible a tool call, so dont send the token to the client and put on hold temporarily.** When we reach the end of the pattern string, e.g. finally matched `</tool_call>` means it is a valid tool call, we can drop all tokens, otherwise, when the state machine fails to match, this json is not a tool call and it should be sent to the client, we then send all tokens and reset the state machine.

Then lets talk about the pattern string. I currently design it like regular expression, all chars not wrapped with `@@` should be matched as is(spaces, \t \r and \n are allowed everywhere in pattern and match string and will be ignored). So `<tool_call>` and `</tool_call>` can be matched.

The model output is NOT reliable. When we test on the qwen model, it sometimes dont output `<tool_call>`, but a random token, though the json is still valid. for these situation we can use `@<match as is>@?` in the pattern string. for example 
`@<tool_call>@? @@json{name,args}@@ @</tool_call>@?`

Then lets talk how to ensure the json object is a valid tool call, instead of what the user ask it to put out. Im designing it like this:
`@@json@@` means it is json and matches all josn. you can also specify type of the object to validate it. e.g. `@@json{name}@@` means the `name` field of the json object should neither be `undefined` nor `null`. `@@json{name:string}@@` further specifies the type must be string.

All supported types listed:
|pattern|example|
|---|---|
|any|
|string| "text"|
|number| 114514 |
| [`type`] | `[number]` => [1, 2, 3, 4] |
| {`type of values`} | `{number}` => {"name1": 1, "name2": 5} |
| {`name of field`: `type`} | `{name:string,args:any}` |

Welcome to share your opinions here.

Related: #5796