Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Headless browser tool
Root cause: Unimplemented Feature Gap

Inspect currently does not support any web interaction apart from web search. 

As part of the effort of migrating [GDM Dangerous Capability evaluations](https://github.com/google-deepmind/dangerous-capability-evaluations) to Inspect we’d like to propose implementing a simple stateful headless web browser tool. While there is a separate feature request for [supporting desktop agents in Inspect](https://github.com/UKGovernmentBEIS/inspect_ai/issues/290), it would be solving a more complex task, while GDM evaluations do not require GUI and a headless browser will be sufficient for their needs.


## Requirements
* A browser should be running in a SandboxEnvironment (e.g. Docker).
* A browser should retain its state throughout the execution of a task.
* An agent should be able to perform the following actions:
   * Get a list of available web elements on the page;
   * Click on a web element;
   * Input text into a web element,
* The interface for the agent to interact with the browser must be text based: 
   * Ideally it should be accomplished through the usage of accessibility trees. 

#### Nice-to-have features
* Agent processing the browser page screenshot in additional to its test representation with DOM/accessibility tree.

## Missing functionality in Inspect
* A Docker container with a headless browser and corresponding interface to interact with it.
* An Inspect tool to send commands to the browser and parse the results.


## Proposed high-level design

![Screenshot 2024-09-13 at 13 43 30](https://github.com/user-attachments/assets/9a6468f1-468d-426d-9419-156961bedf71)


Elements in green are the ones introduced in this design, specifically:
* __WebServer__ - a simple python server with 2 components:
  * __gRPC service__  - processes incoming gRPC requests from the Inspect agent.
  * __[Playwright library](https://playwright.dev/python/docs/library)__ - instantiates and communicates with the headless browser.
* __web_browser__ - a regular Inspect tool which sends model browsing requests to the WebServer.


## Open design questions
The definition of gRPC service.

## Next steps
Our team is happy to implement this feature in the Inspect framework if there are no objections from the UK AISI team. Any comments/suggestions will be much appreciated! 

Next we’ll create subtasks with more implementation details.

Yes, it would be great to work with you on this!

Ideally, we could make this work over the existing `SandboxEnvironment` interface. This supports `exec()` but no general purpose RPC endpoint. It seems like we should be able to define a stub client within the container that is able to maintain the stateful browser session? The reason we prefer to not extend the interface is that there will be many implementations of sandboxes and the more requirements we place on them the harder conformance will be. For example, we currently have  k8s and ssh sandboxes in development and adding the requirement of hosting an RPC service might be tricky for one or both.

Some other questions/considerations:

1. Playright and Chromium dependencies within the container. I think we will want to at a minimum check for these when the `web_browser()` tool fires up and provide a nice exception message indicating how to meet the requirements. We might also consider publishing a Docker container that already has these to the registry? We currently use `python:3.12-bookworm` as our "default" Docker container, perhaps this is enough motivation to have an Inspect specific default container maintained?

2. Does the `web_browser()` strictly need to live in a Docker container? (i.e. is this a security driven requirement?). An Inspect agent can execute either within the host machine's context or within the sandbox context. The host machine would be simpler to get working and not require special dependencies inside the container. 

As things get rolling I think it would be good to have this posted as a Draft PR so we can provide feedback as close to development as possible. 

Happy to jump on a call to discuss all of the above!


Thank you for a quick response!

0. Reg the main point about the SandboxEnvironment: yeah, I thought you'd like to keep SandboxEnvironment interface as is.. Just to make sure: you **don't** mean something like that, right? I believe it's too crude but doublechecking just in case.

```py

# tool/_tools/_web_browser.py
@tool
def web_browser(...)  -> Tool:
  async def execute(query: str) -> ToolResult:
    result = await sandbox().exec(cmd=["_BROWSE_WEB_", query])
    return result
  return execute


# util/_sandbox/docker.py
class DockerSandboxEnvironment(SandboxEnvironment):
    ...
    async def sample_init(...) -> dict[str, SandboxEnvironment]:
    	...
    	# Docker container should be up and running at this point
    	if (rpc_endpoint_port not None):
    		rpc_client = cls._create_rpc_endpoint(rpc_endpoint_port)

    	# create the docker sandbox environment
        docker_env = DockerSandboxEnvironment(service, project, working_dir, rpc_client)
        ...


    def __init__(self, service: str, project: ComposeProject, working_dir: str, rpc_client) -> None:
        super().__init__()
        ...
        self._rpc_client = rpc_client


    async def exec(
        self,
        cmd: list[str],
        ...
    ) -> ExecResult[str]:
    	if cmd[0] != '_BROWSE_WEB_':
    		# Current implementation calling `docker compose exec ...`
    		...
    	else:
    		query = cmd[1:]
    		response = self._rpc_client.send_request(query)
    		return response
```

1. Reg the dependencies: for our evaluations we'll need custom docker containers in any case since most self-proliferation evaluations require custom setup: https://github.com/google-deepmind/dangerous-capability-evaluations/blob/main/self_proliferation/environments/base/Dockerfile. So agree on providing a custom Docker container with all the required dependencies.


2. Yes, I believe it's security-driven requirement for us.




Okay, sounds good re: running in the sandbox and just documenting the required dependencies. I think we just need to show a sane error (with a tip on how to resolve) when the dependencies aren't present.

I am saying let's see if we can push it all through `exec()`. The problem with extending the interface is that this will become a requirement for every sandbox implementation, and in some cases may be impossible to fulfil. Even if its crude I think finding a way to use `exec()` will save a huge amount of grief for sandbox implementors. If you want to chat about this realtime LMK (as I certainly could be missing something important in the requirements!).

Based on our offline discussion updating the proposed design for future clarity:

![Screenshot 2024-09-18 at 19 43 34](https://github.com/user-attachments/assets/3fb92644-e206-4bff-88b4-b7c01918b02d)

Again, elements in green are the ones introduced in this design, specifically:

* WebServer - a simple _stateful_ python server with 2 components:
  * gRPC service - processes incoming gRPC requests from the Inspect agent.
  * [Playwright library](https://playwright.dev/python/docs/library) - instantiates and communicates with the headless browser.
* WebClient - a python binary client which establishes connection with the WebServer, sends the request and outputs the response to the stdout so it can be used through the Inspect _bash_ tool. (A separate tool can be introduced as well but we won't be doing it for now). 

Will be able to separate the implementation in the following steps:
- [ ] Implement web_client and web_server with gRPC service.
- [ ] Implement playwright crawler library. 

I wrote an implementation of an HTTP proxy that lives outside the sandbox and uses `exec` to forwards HTTP calls from inspect code into the sandbox (using curl). Is it possible to layer gRPC over an HTTP proxy? It's a very crude proxy; HTTP/1.0 or so.

Hi @art-dsit! Sorry, could you please clarify: is it an alternative solution for this feature request or you're solving a similar problem and wondering how to use gRPC for it?

> Hi @art-dsit! Sorry, could you please clarify: is it an alternative solution for this feature request or you're solving a similar problem and wondering how to use gRPC for it?

I was using my HTTP proxy to forward browser automation calls from the Inspect process to the sandbox. It means you can use Selenium's python bindings in your browser tool, while the browser lives in the sandbox. But I never got further than a proof of concept.

> > Hi @art-dsit! Sorry, could you please clarify: is it an alternative solution for this feature request or you're solving a similar problem and wondering how to use gRPC for it?
> 
> I was using my HTTP proxy to forward browser automation calls from the Inspect process to the sandbox. It means you can use Selenium's python bindings in your browser tool, while the browser lives in the sandbox. But I never got further than a proof of concept.

I'm not sure I understand how the proxy works along with the Inspect process.. But please let me know if you think your approach has more benifits than the original design! 

I think I can close this issue now since the tool was fully merged and is ready to be used. It was put under evals/gdm_capabilities/ for now but we'd be happy for everyone else to use it as well and if needed, can help move it to another location in Inspect. Please reopen if something else is needed.