Repository: UKGovernmentBEIS/inspect_ai
Issue: `bash` tool is now unusable by Anthropic models
Root cause: Interface Contract Mismatch

As of #1431, Anthropic models are no longer able to use the existing [bash](https://github.com/UKGovernmentBEIS/inspect_ai/blob/8710a15dd4099112d33e5e385de6991e9cbd9914/src/inspect_ai/tool/_tools/_execute.py#L22-L59) tool. If you give an Anthropic model access to `bash`, but not the new `bash_session`, when the model attempts to call `bash` its call is silently rewritten to `bash_session`, which fails.

The model then also gets gaslit because on its next turn it is informed that it tried to use the `bash_session` tool, and that such tool does not exist:

```
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "Let me try with the correct function name and format:"
        },
        {
          "type": "tool_use",
          "id": "toolu_019xyLTxKrTC36p9tFeu7seg",
          "name": "bash_session",    <----  ** NOT what the model actually wrote! **
          "input": {
            "cmd": "ls -la"
          }
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "tool_use_id": "toolu_019xyLTxKrTC36p9tFeu7seg",
          "type": "tool_result",
          "content": "Tool bash_session not found",
          "is_error": true,
          "cache_control": {
            "type": "ephemeral"
          }
        }
      ]
    }
```

Seeing that it "used the incorrect syntax last time", it attempts to call `bash` instead, which again fails for the same reason, and again causes a gaslight on its next attempt. The model never gets to see that it has already tried this before.

cc @epatey @jjallaire 

I should mention that the relevant code is: https://github.com/UKGovernmentBEIS/inspect_ai/blob/8710a15dd4099112d33e5e385de6991e9cbd9914/src/inspect_ai/model/_providers/anthropic.py#L76-L80 and: https://github.com/UKGovernmentBEIS/inspect_ai/blob/8710a15dd4099112d33e5e385de6991e9cbd9914/src/inspect_ai/model/_providers/anthropic.py#L812-L814


Thanks so much for reporting this and pinpointing the issue! Fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1508

Thanks, @jjallaire! I see a couple problems remaining after the fix in #1508 (both apply only to Anthropic models):

1. When the tool name mapping does happen (e.g. when using `bash_session`), this code still rewrites the model's output history. e.g. The model calls `bash`, the command succeeds, but then in future turns it will be informed that it called `bash_session`, not `bash`, where `bash_session` is not a tool that it knows about or would have had reason to call. I predict this will be a little confusing for the models, and will artificially decrease their performance a little.
2. It's not possible to use both `bash` and `bash_session`. I don't know of a reason for trying to do so, but from the perspective of an Inspect user this seems like an arbitrary restriction. I get that it's complicated because Anthropic requires that the bash session tool is named `bash`, but this seems to me like something that the Inspect library could work around somehow.

Okay, thanks! I think both of these are issues we can resolve without much trouble. Will re-open this issue to track follow up.

cc @epatey 

It would also be helpful to explain the situation to users in the docs and maybe also in the logs. It's pretty surprising that the model sees a different tool name than the user does.

We do explain the tool binding in the docs here: https://inspect.aisi.org.uk/tools-standard.html#tool-binding. Let us know if you think this bears improvement! 



Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1524