Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] stateful tool use
Root cause: Unimplemented Feature Gap

One of the biggest limitations of the current tool implementation is that it is very hard to make them stateful (i.e. if you want to make its behavior change over multiple calls in a message chain). For instance, if you wanted to implement a tool
```python
@tool
def increment() -> Tool:
   async def execute():
       ...
   return execute
```
that everytime a model calls it return 1 more than the previous time, I don't know of a good way to implement this. Of course in this contrived example you could make the model pass in the previous result, but this isn't an option when you want to keep certain information hidden from the agent. My current solution is to store the state in a file hidden away in the docker container, but this isn't ideal and there is a slight chance the model will come across it.

There are two possible ways I can see implementing this functionality. The first is a function similar to `sandbox` or `get_model` that returns the current state. This might look like
```python
@tool
def increment() -> Tool:
   async def execute():
      state = get_state()
      state.metadata['increment'] += 1
      return state.metadata['increment']
   return execute
```
The other option (which might be simpler to implement but a little less clean) is to let tools define an optional state parameter. This might look like
```python
@tool
def increment() -> Tool:
   # the state only gets passed to the tool if the tool defines it as a param and is excluded from the spec given to the models
   async def execute(state):
      state.metadata['increment'] += 1
      return state.metadata['increment']
   return execute
```

We have a solution for this coming soon! There will be a `state.store` that tools can access with the context bound `store()` function. Here is the in-flight PR: https://github.com/UKGovernmentBEIS/inspect_ai/pull/82

This should get merged in the next week or so. Note that if you want a shorter term solution you can always create local state like this:

```python
@tool
def increment() -> Tool:

   increment = 0
   
   async def execute(state):
      nonlocal increment
      increment += 1
      return increment

   return execute
```

Note that this might suffice for many use cases, but the `store()` will have the benefit of some automatic diffing we'll do for the log as well as the ability to share it across multiple tools.


I think this local state idea almost works, but only if you have a single sample per task since the tool is created once per task not once per sample. This implementation shares the state across all samples.

Just had a look at #82. I'm very excited for this PR! The transcripts and subtasks also solve another issue I was having related to needing to run multiple parallel subtasks (and the constraints of the current system with only a single message chain).

Ah, you are indeed correct about the samples constraint. The `store()` should solve it properly though as that is per-sample. Will ping back on this thread when the PR is merged!

We have now merged the store API which should provide what's needed for stateful tool use (docs here: https://inspect.ai-safety-institute.org.uk/agents-api.html#sharing-state)