Repository: UKGovernmentBEIS/inspect_ai
Issue: Tool calls for json output
Root cause: Unimplemented Feature Gap

I frequently use tool call functions as a cheat to get structured (json) output (usually using pydantic). Pretty similar to what e.g. instructor (and somewhat outlines, for openai at least) does under the hood. From what I can tell this isn't really straightforward or completely possible with inspect using the current tools within `TaskState` and `Generate`.

(Caveat, I may just be misunderstanding the tool interface here, I did my best based on the documentation and reviewing some of the source code)

1. the `@tool` decorator doesn't seem to consider structure deeply. I'm not sure that the type is correctly specified in the json schema of the `complex` argument below (I usually get a list of any type as a response)
- Could the tool call interface perhaps be generalized such that you can specify a specific schema?
2. the tool response always seems to be passed back to the model for analysis. For my use case, the tool response is for the program, not the model, so automatically passing it back as a tool message isn't helpful
- Could this be made configurable such that the tool result is not automatically passed back to the model for further multi-turn generation?


This is an example of what I've currently tried:

```python
from typing import Literal
from pydantic import BaseModel
from inspect_ai.solver import tool, use_tools
from inspect_ai import task, Task


class Word(BaseModel):
  type: Literal["adjective", "noun"]
  word: str

@tool(prompt="A defined schema for the structured extraction of words from the input", name="extract_words")
def my_tool():
  async def extract(extracted: List[Word]):
    """
    Accepts the extracted nouns and adjectives from the sentence

    Args:
      extracted: the structured output from the LLM's analysis
    Returns: the same structured output passed, for consumption by the application
    """
    print("got input extracted", extracted)
    return extracted
  return extract

@task
def eval_word_extract():
  return Task(
     dataset=MemoryDataset([Sample(input="Extract the nouns and adjectives from the following sentence.\nSentence:\nThe quick brown fox jumped over the lazy dog.")]),
     plan=[
        use_tools([my_tool()], tool_choice="any"),
        generate(cache=True),
     ]
  ) 
```

Example of response. Note that the arguments passed to the tool just seem to be a string here.

![image](https://github.com/user-attachments/assets/0dd27358-dd33-4bcb-b348-879361e075c2)


It may be that we're not correctly extracting the typeinfo for `Word`. If you look in the log at the underlying JSON, what is the schema for the tool that we are sending to the model?

w/r/t To custom logic around tool calling, you can indeed do this by passing `tool_calls="none"` to `generate()`. See docs on this here: https://inspect.ai-safety-institute.org.uk/agents.html#sec-tool-calls

Ah! Thanks for the tip about `tool_calls`, that makes perfect sense.

This is the debug log from the request. It is indeed not interpreting the parameter type. I also tried a couple other simpler structured types, but those didn't work either. 

All of `List[Words]`, `List[str]`, `list` show up as 

```python
{'words': {'type': 'string', 'description': '...'}
```

Full response

```

Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'system', 'content': 'A defined schema for the structured extraction of words from the input'}, {'role': 'user', 'content': 'Extract the nouns and adjectives from the following sentence.\nSentence:\nThe quick brown fox jumped over the lazy dog.'}], 'model': 'gpt-4o-mini', 'stream': False, 'tool_choice': 'auto', 'tools': [{'type': 'function', 'function': {'name': 'extract_words', 'description': 'Accepts the extracted nouns and adjectives from the sentence', 'parameters': {'type': 'object', 'properties': {'words': {'type': 'string', 'description': 'the extract json object words'}}, 'required': ['words']}}}]}}
--

```

Okay, that's kind of what I thought it was doing. We are soon going to invest some more time in richer typing for tool declarations. One thing you could try is just declaring the type to be a plain `dict` (that might work, but looking at the code it seems like it could). Another thing to try (for now, we will make this better!) is to tell the model to give you a JSON string which you can then parse or alternatively some other structured string that is more consistently able to form.

FYI there is work in progress on this here: https://github.com/UKGovernmentBEIS/inspect_ai/tree/feature/tool-json-schema

Should be merged within the next few days.

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/158