Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] add support for approvers

## Overview

We’d like to implement a pluggable approvals system within Inspect to provide oversight and control over agent actions, particularly for tool calls and potentially dangerous operations.

## Problem Statement

There is a need to implement a flexible approval mechanism for agent actions within Inspect AI to enable real-time monitoring of large agent eval runs, and to allow for human oversight where malice or otherwise interesting behavior is detected.

This will become increasingly useful as agents become more capable and can successfully pursue long horizon tasks using novel approaches. For example, in a task like “Earn 0.01 BTC” that may take a long time and involve many thousands of actions, we would like to avoid having to review each action but would also like to have potentially malicious actions flagged, e.g. if the agent appears to be spinning up a phishing website or writing malware. 

In order to have this level of oversight, we need Inspect to support this concept of action approval. These approvals can include set of rules, human-in-the-loop or AI.

## Proposed Solution

We do not take opinionated approach whether the approval mechanism should be implemented on the tool or solver level as there are benefits to both. We design the approval mechanism to be  flexible and possible to be used both inside of the tool or or on the solver level with the main interface being the AprroverManager that executes individual Approvers associated with tool calls.

### Key Components:

1. Approvers
    1. Functions that decide whether to accept, reject, terminate or escalate the action.
    2. `Approver` functions receive `TaskState` + `ToolCall` as inputs and return `ApproverDecision` which is one of the enums:
        1. `approve` = this actions seems fine, proceed
        - `reject` = this action seems malicious or otherwise erroneous, don’t execute it, sample again from the model
        - `escalate` = defer the decision to the next Approver
        - `terminate` = stop the agent
2. Approval Manager
    1. `ApprovalManager` class initialized with a list of one or more `Approvers`, that it will run when the associated tool is requested, but before the tool is actually executed. 
    2. The job of an `ApprovalManager` is to handle the escalations between approvers, and  logging.
    3. `ApprovalManager` contains the `get_approval(..)` method which returns a boolean. 
        1. This method executes the first Approver, then:, 
            1. if result is `approve` then return `true` 
            2. if result is `reject` then return `false` 
            3. if result is `terminate` then terminate the program (sigterm or return sentinel error or something) 
            4. if result is `escalate` then call the next `Approver` in the chain
    4. If the Manager returns true then the action will be executed. If the Manager returns false then the action will not be executed and a message will propagate up explaining why
        1. The explanatory message is passed backed to the LLM as output of the tool call.

### Technical Considerations

This proposed solution does not need any changes to the inspect source code. In the longer term, to better integrate Inspect properly it’ll be necessary to: 

- Create async implementation of `utils.input_screen()` and make the UI async friendly
- Implement an Approval extension point so that we can implement approval at higher levels (solver-level, sandbox-level) instead of on (just) on tasks
- Hook the approver into the core of tool calling rather than needing new approver-specific tools
- Add support for multiple humans concurrently reviewing actions that have gone to the Human-in-the-loop queue (this will be a special Approver that we’ll probably create next)

We might also want to think about:

1. Allowing `n` actions after an Approval: We might want to not run all of the Approvers each time a tool is called, especially if the Approver is something like AI-in-the-loop and definitely not if it’s Human-in-the-loop. In this case we may want to allow for a ‘runway’ where a single Approval enables some number of actions to be executed before Approval is requested again. 
- Different modes of operation for the Approval Managers: By default the Approval Manager will return whatever the first Approver says unless it says says Escalate, in which case it would defer to the next Approver. It might be required in future to have different `mode`s for the Approval Manager. Perhaps we want a mode where we run all the Approvers and check that they all return `accept` in order to allow the action to execute, or maybe it’s sufficient for a single Approver to `accept`.

## **Next steps**

We’re happy to implement the first version of approvers integrated into a new bash tool which will enable basic CLI approval for bash commands. We can put an example inside the examples folder. Any comments/suggestions will be much appreciated!

Longer term, we’ve discussed with J.J. that AISI will replace synchronous input with async implementation and then consider implementing improvements to Inspect (Approval extension point, core tool calling integration, async-friendly UI)

Support has been added here https://github.com/UKGovernmentBEIS/inspect_ai/pull/654

FYI we've implemented support for async approver UI here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/958