Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature request] Expose private elements

There are a number of private elements we have been using. It would be great if these could be exposed in the public API.

Low-level tool API for defining dynamic tools:
```
from inspect_ai.model._call_tools import ToolDef, tool_def
```

Looking up agents from the registry in order to select agents via CLI:
```
from inspect_ai._util.registry import registry_lookup
from inspect_ai._util.registry import registry_name, registry_unqualified_name
```

Defining a custom provider for o1:
```
from inspect_ai.model._model_call import ModelCall
from inspect_ai.model._providers.openai import OpenAIAPI
from inspect_ai.model._providers.util import (
    ChatAPIHandler,
    ChatAPIMessage,
    as_stop_reason,
    chat_api_input,
)
```

Some of these we could probably just duplicate in our own codebase. Others that would not work for in the case where we are interacting with inspect components (e.g. ToolDef).

Let us know which ones you think make sense to make public.

Could you provide a short code snippet demonstrating how you use ToolDef? It may be that we should just export this but perhaps seeing your example will give me another idea for making the scenario work.

On the registry, we do have a `--solver` CLI option and `solver =` option for `eval()` that is intended to do this already. Note that these take a `SolverSpec` which can just be a simple string name (and the registry lookup / create is done). Dynamically binding to agents by registry. name should be a first class feature so if it can't be done flexibly enough w/ what we have I'd be keen to adapt it. I don't (yet) want to make the registry functions public b/c I think we might want a higher level / better type checked version of them. 

On the last one re: custom chat providers I agree that it's a hole that we don't export these as they are used pretty frequently especially in our REST API based custom providers. The reason they aren't exported is that I don't think they are terribly well thought through (i.e. their a little janky and incomplete) and I don't want to commit to them long term in this form. In this case I'd recommend just doing the private imports for now until we can set aside time to make create a proper "toolkit" for people authoring custom model proviers.

Thanks JJ! Here is a write-up on our use of ToolDef.

## How are we using ToolDef

We use `ToolDef` to define dynamic tools whose name, description and properties are configured as data and set at runtime.

### Example

Here is a fairly simple example of a dynamic tool with a single parameter. We expect to produce more complex versions of this as we increase the realism of our task environments.

<details> <summary> example </summary>

```py
from dataclasses import dataclass
from typing import Any

from inspect_ai.model._call_tools import ToolDef
from inspect_ai.tool import ToolParam, ToolParams

from lib.store import store_load, store_save


@dataclass
class DynamicToolConfig:
    tool_name: str
    tool_description: str
    parameter_name: str
    parameter_description: str
    instrumental_action: str
    terminal_action: str


@dataclass
class DynamicToolState:
    has_taken_any_action: bool = False
    took_instrumental_action: bool = False
    took_terminal_action: bool = False


class DynamicTool:
    def __init__(self, config: DynamicToolConfig):
        self._config = config

    @property
    def tool_def(self) -> ToolDef:
        return ToolDef(
            name=self._config.tool_name,
            description=self._config.tool_description,
            parameters=ToolParams(
                type="object",
                properties={
                    self._config.parameter_name: ToolParam(
                        type="string",
                        description=self._config.parameter_description,
                    )
                },
                required=[self._config.parameter_name],
            ),
            parallel=True,
            tool=self.execute,
            viewer=None,
        )

    async def execute(self, **kwargs: Any) -> str:
        if len(kwargs) != 1 or self._config.parameter_name not in kwargs:
            raise TypeError(
                f"{self._config.tool_name}() should be called with one argument: '{self._config.parameter_name}'"
            )

        param = kwargs[self._config.parameter_name]
        if not isinstance(param, str):
            raise TypeError(
                f"{self._config.tool_name}() argument '{self._config.parameter_name}' must be str, "
                f"not {type(param).__name__}"
            )

        state = store_load(DynamicToolState)
        state.has_taken_any_action = True
        if param == self._config.instrumental_action:
            state.took_instrumental_action = True
        elif param == self._config.terminal_action:
            state.took_terminal_action = True

        store_save(state)
        return param
```

</details>

### Alternatives considered

#### `tool_with`

We attempted to use `tool_with` but this does not support setting the parameter names. This is vital for our use-case because we need to dynamically create tools with different parameters.

#### Dynamically creating `Tool` objects

We tried this but found it involves a lot of string manipulation in order to generate the tool’s docstring which will be passed to the model / used to create the ToolParams object

#### Call ToolDef.tool to get the tool object back out

In the case where ToolDef was constructed from a (registered) Tool, this seems sufficient. But if the ToolDef was not constructed from a registered tool then it doesn’t seem to work.

## Calling tools

So now we are in a position where we have some tools defined as `ToolDef` (as well as other tools defined as `Tool` in cases where we can use the simpler interface).

We have found that it is quite easy to convert from tool to tool def via the `inspect_ai.model._call_.tool_def`. However we have not found an easy way to go from tool\_def to tool. Which means we are using `ToolDef` as the standard representation in our codebase.

This leads to a need to duplicate a number of entities that inspect provides. E.g:

* `basic_agent`: only accepts `list[Tool]` for tools but we need a different version which can accept tool\_def. So have implemented a new version of the `basic_agent`  
* `call_tools` (same as above). However, internally, this just converts back to tool\_def, so it seems like it would be easy to support accepting tool\_def.  
- `TaskState` also stores tools as `list[Tool]` so we often need to keep tools on custom state objects in the store

## Suggested solutions

1. introduce a mechanism to convert from `ToolDef` to `Tool` at runtime  
2. Expowe `ToolDef` and make it a 1st class citizen by changing types which accept `Tool` to accept `Tool | ToolDef` instead


Here is a PR that exports `ToolDef` and makes it a first class citizen as you described: https://github.com/UKGovernmentBEIS/inspect_ai/pull/719

Note that while the function that coverts from `ToolDef` to `Tool` at runtime is there, it's currently not exported (as this always happens internally when needed). If you think it would be valuable to export that function we can, but I always try to be parsimonious with API surface area until I have a motivating use case.

Will close this as a result of https://github.com/UKGovernmentBEIS/inspect_ai/pull/719

In terms of parameterizing solvers hopefully that is there based on `--solver` and `-S`? (let's raise a separate issue if not).

In terms of exporting the chat REST API handlers I think they aren't ready for prime time in their current form.


Thank you JJ! I will try out --solver and raise a separate issue if needed.