zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] make `default` sandbox not required
Root cause: Architectural Constraint

**Problem Description**

Currently, when defining tasks with multiple custom sandbox environments, it's not possible to remove the 'default' sandbox due to a constraint in `context.py`. The `validate_sandbox_environments` function asserts that there must be a sandbox named `'default'`, which prevents the deletion of the default sandbox even if it's not needed.

```python
# context.py
if environments.get("default", None) is None:
    raise RuntimeError(f"No 'default' service provided for {type.__name__}")
```

This constraint hinders tasks where we want to prevent accidental usage of the default sandbox and ensure that only specified custom sandboxes are used.

**Example Use Case**

We have a task that compares the model's behavior in two slightly different sandbox environments: "oversight" and "non_oversight". Here's how the task is structured:

```python
@solver
def my_solver():
    async def solve(...):
        with_oversight, without_oversight = await gather(
            run_scenario(sandbox="oversight"),
            run_scenario(sandbox="non_oversight")
        )
       
        _store = store()
        _store("with_oversight") = with_oversight
        _store("without_oversight") = without_oversight

@subtask
async def run_scenario(sandbox: SandboxEnvironment):
    ...

@scorer
def my_scorer():
    async def score(...):
        _store = store()
        result = _store("with_oversight") and not _store("without_oversight")
        return Score(result)
```

When attempting to remove the default sandbox, we encounter the following error:

```
RuntimeError: No 'default' service provided for <TaskType>
```

**Workarounds Attempted**

- **Renaming the Default Sandbox with `x-default`:** Adding an `x-default` key to the service definition to use a different sandbox as the default. However, this leads to issues when referencing the custom sandbox by its specific name, as it seems to get renamed to 'default'.

- **Using the Default Sandbox as One of the Custom Sandboxes:** Keeping the 'default' sandbox and using it as either "oversight" or "non_oversight". This is not ideal as it can lead to confusion and accidental misuse.


Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/516
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] add option to disable `rich`
Root cause: Unimplemented Feature Gap

[rich](https://github.com/Textualize/rich) output seems very tightly baked into the UI of inspect.

It does make the UI nicer in some cases, but in a lot of others I find it quite a nuisance. Specifically
* Sometimes when trying to debug errors I find rich output less readable / introduces a lot of extra output to sift through (and sometimes maybe hides details of stack traces)
* When trying to share CLI output of inspect for debugging purposes it can mess with the formatting

I'm not sure if this is feasible, but it would be cool to have the option to run inspect with a `disable-rich` flag.

Rich tries to provide some of this capability (so you can still use it and have it disable all of the ANSI and overwriting stuff). See https://rich.readthedocs.io/en/stable/console.html#terminal-detection

It looks like you you could get an approximation of this now with:

```bash
export TERM="dumb"
export NO_COLOR=1
```

I agree that this should be a feature of Inspect rather than something you need to rig up with environment variables, but I'm just curious how you find the output with those variables defined.

Thanks, I will experiment with those and let you know. 

cc @jeremy-apollo

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/515
Repository: UKGovernmentBEIS/inspect_ai
Issue: Link error

https://ukgovernmentbeis.github.io/inspect_ai/workflow.html

If you aren’t familiar with the __name__ == "__main__" idiom, see the docs on [__main__](https://docs.python.org/3/library/main.html) for additional details.

The link for __main__ returns an error

Thanks! Fixed with https://github.com/UKGovernmentBEIS/inspect_ai/commit/671a239f13fe2706f89ffccaf5ac5eef86958fc3
Repository: UKGovernmentBEIS/inspect_ai
Issue: Task metadata
Root cause: Documentation Deficiency

It would be nice to be able to store arbitrary task metadata surfaced somewhere like under task information:
![CleanShot 2024-06-17 at 22 19 53@2x](https://github.com/UKGovernmentBEIS/inspect_ai/assets/125105/a1e9cbcc-92a8-445a-9d97-43772f5d5cb0)

In our case we'd store the version numbers of different packages, who ran the eval, and a hostname.  

Interestingly we have a `metadata` field in the `EvalSpec` (which is what this is drawing from) but no way to populate it! I wonder if should add a `metadata` field to `Task` and forward it here? (@dragonstyle LMK what you think)

If what were displayed here were the task args, that would be swell and probably work well! As these are used to pass to the solver and define what runs and so worth surfacing in and of themselves. We could also use them to capture additional metadata.

@dragonstyle Don't we at least try to display the `task_args`? If we don't then we absolutely should (even if the are also displayed in `config` above)

We are displaying `task_args` on the 'Info' tab (we render this metadata, so it should render simple values, but also things like dictionaries or arrays). Are you not seeing a 'Task Args' section (if you aren't, I should try to reproduce and figure out why!)?

Yeh unfortunately I'm not seeing a Task Args section at all on the Info tab.

Sorry to have let this languish! That section definitely should be present...

- Could you share the inspect version that you're using (`inspect --version` from the terminal)? 
- If possible is there a sample log that you'd be able to share (I can share my email address if you'd rather not post)?

Thanks and sorry for the issue...

Huh, I can't seem to reproduce this anymore. Happy to close this, sorry for the bother!

Ok thanks for the followup and glad it appears to be working again!
Repository: UKGovernmentBEIS/inspect_ai
Issue: `Sample`'s `files` is unusable for binary files
Root cause: Validation Gap

It gives pydantic errors because `Sample` has hardcode file contents to be `str`

Ah, I see, the files should be listed by path not by contents.  This should be made clear in the documentation.

Thanks! Here are some improvements to those docs: https://github.com/UKGovernmentBEIS/inspect_ai/pull/513

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/513
Repository: UKGovernmentBEIS/inspect_ai
Issue: Documentation claims that `sandbox=("docker", "Dockerfile")` is supported, but the code treats the `Dockerfile` as a `compose.yaml` file
Root cause: Documentation Deficiency

The code should automatically create the docker compose file, as the documentation claims

I couldn't find a spot in the documentation that claims that this should work (our only example of this syntax uses the compose file). Likely the confusion arises from the table that lists `Dockerfile` and `compose.yaml` and then later shows an example of `sandbox=("docker", "compose.yaml")` (which definitely might lead one to infer that the `Dockerfile` should also work there).

Hopefully this change to the docs will serve to clarify: https://github.com/UKGovernmentBEIS/inspect_ai/pull/512




Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/512
Repository: UKGovernmentBEIS/inspect_ai
Issue: tool, scorer, etc should preseve function annotations

VSCode tooltips for wrapped functions are missing argument information in the type, e.g.,:
![image](https://github.com/user-attachments/assets/09733894-03e5-4ef0-88ce-83347d57c734)
By contrast, annotating functions with `@functools.cache` or `@torch.no_grad()` does not remove type arguments.  It seems the issue is the `cast` in https://github.com/UKGovernmentBEIS/inspect_ai/blob/5eb6190631bd9687ed4812816cc303d78c1f6976/src/inspect_ai/tool/_tool.py#L154, though I'm not 100% sure this diagnosis is right

Thanks for this! I think there is a way to do this with `functools.wraps` I just need to make sure that we aren't somehow relying downstream on the wrapper (in one case with solvers I know that we are, but I think I have a workaround for that).
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Support video content as input and video-LLM models
Root cause: Unimplemented Feature Gap

In order to be able to contribute the [EgoSchema](https://arxiv.org/abs/2308.09126) benchmark, Inspect has to support video content as input as well as video-LLM models e.g. [VideoLlama2](https://arxiv.org/abs/2406.07476v2).

Credit to Aleksey Bieneman (ASET program participant) for flagging this.

The current model chat APIs don't for the most part accept videos, so we may want to wait until these APIs come online before implementing this benchmark.

We've added support for audio and video inputs here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1102

Currently model support for this is pretty thin, however it's likely that more providers (including hosts of open source models) will be adding support soon.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Bug] Inspect view errors out when port forwarding to laptop's web browser
Root cause: Resource Mishandling

(As with https://github.com/UKGovernmentBEIS/inspect_ai/issues/51 I am working on an SSH server in a devcontainer which may complicate things)

When I run `inspect view` then port forward from vscode to my laptop, I see the following when I open http://localhost:7575 in chrome

```log
127.0.0.1/:8 
        
       GET http://127.0.0.1:7575/libs/bootstrap/css/bootstrap.min.css net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)
127.0.0.1/:48 
        
       GET http://127.0.0.1:7575/libs/bootstrap/js/bootstrap.bundle.min.js net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)
127.0.0.1/:15 
        
       GET http://127.0.0.1:7575/libs/bootstrap/css/bootstrap-icons.min.css net::ERR_CONTENT_LENGTH_MISMATCH 200 (OK)
AppErrorBoundary.mjs:24 {e: ReferenceError: bootstrap is not defined
    at http://127.0.0.1:7575/src/samples/SamplesTab.mjs:60…}
AppErrorBoundary.mjs:24 {e: ReferenceError: bootstrap is not defined
    at http://127.0.0.1:7575/src/samples/SamplesTab.mjs:60…}
```

The page will eventually render this error into HTML (but above is from the console)

This happens intermittently on Safari also, but firefox seems to work fine.

My guess is that this is the same XDG directory thing but we'll see once we have that resolved!

I've seen this issue raised a few times over in microsoft/vscode-remote-release eg: https://github.com/microsoft/vscode-remote-release/issues/9969

this error persists after fixing the XDG directory thing...good catch @tekumara, we can probably close this
Repository: UKGovernmentBEIS/inspect_ai
Issue: `combine_messages` raises pydantic validation errors by casting `str` to `list`
Root cause: Validation Gap

https://github.com/UKGovernmentBEIS/inspect_ai/blob/37d358ac1b024533ebec42f52929d49c6adcbc70/src/inspect_ai/model/_model.py#L683-L695
is wrong and raises pydantic validation errors

In particular, at
https://github.com/UKGovernmentBEIS/inspect_ai/blob/37d358ac1b024533ebec42f52929d49c6adcbc70/src/inspect_ai/model/_model.py#L694
`b.content` is `str` not `list[Content]`

Also,
https://github.com/UKGovernmentBEIS/inspect_ai/blob/37d358ac1b024533ebec42f52929d49c6adcbc70/src/inspect_ai/model/_model.py#L691
swaps the order of `a` and `b`
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support retrieval as a first-class element
Root cause: Unimplemented Feature Gap

I'm using inspect_ai to evaluate a RAG pipeline and it would be nice to be able to expose the retrieved docs and metadata (eg: their score) in the results and UI.

For now I've been using a solver that runs retrieval and generation, and adds the retrieved documents and their metadata as `state.metadata`. 

Thanks for this, you are currently doing it the "right" way but I understand that the the UI display leaves a lot to be desired. We made message histories first class in metadata, @dragonstyle I wonder if there is a way to make documents (and their score) similarly first class?

Hi @tekumara, note that the new `Store` API is now the preferred way to handle this sort of side data: https://inspect.ai-safety-institute.org.uk/agents-api.html#sharing-state

Note that the `Store` is accessible from tools so it's possible to define tools that provide RAG content. Changes to the store also show up as diffs in the Inspect log. Soon (in the next few days) we'll also show the final state of the `Store` in the log viewer alongside the `Metadata` (cc @dragonstyle).
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] support passing `-H` / `--host` to `docker` sandbox
Root cause: Unimplemented Feature Gap

I'd like to be able to specify that some tasks / samples should run on a remote machine by passing `-H` to the docker commands.  Right now, I am using `os.environ["DOCKER_HOST"] = docker_host`, but it seems like this might interfere with other tasks if I try to run them in parallel.  Could support be added to bind `-H` on a per-task basis?

This is a good suggestion -- right our now SandboxSpec type is a simple tuple and I think it needs to become a richer data structure.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Gemini - too many requests causing Internal Server Error (500)
Root cause: Validation Gap

I'm trying to run Gemini 1.5 pro on various evals and keep encountering an `Internal Server Error (500)`. I opened an issue about this in the [google-gemini/generative-ai-python](https://github.com/google-gemini/generative-ai-python/issues/573) repo and best I can tell this seems to be caused by too many incoming API requests (full trackback and environment details in attached there)

I'm unsure whether it's in the purview of Inspect to handle this, but I wanted to flag the issue.

The thing that controls whether we backoff and retry API errors is this function:

```python
@override
def is_rate_limit(self, ex: BaseException) -> bool:
    return isinstance(
        ex,
        TooManyRequests | InternalServerError | ServiceUnavailable | GatewayTimeout,
    )
```

https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_providers/google.py#L188-L192

You could play with this to see if there is another exception type that would pickup this error. 

You can also use `--max-connections` to throttle down the number of active connections.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Support the -F flag for `inspect eval`
Root cause: Unimplemented Feature Gap

Currently you can use the -F flag to filter tasks when running `inspect list tasks` (see https://inspect.ai-safety-institute.org.uk/eval-sets.html#attributes-and-filters).

It would be nice to also support this flag on the `inspect eval` subcommand, so instead of having to run this:
```bash
inspect list tasks security -F light=true | xargs inspect eval-set --log-dir logs-security-42
```
you can run this:
```bash
inspect eval-set --log-dir logs-security-42 -F light=true
```


Yes, that would be so much nicer! Will try to take care of this soon.
Repository: UKGovernmentBEIS/inspect_ai
Issue: `docker compose build` gets run without access to any sample variables
Root cause: Documentation Deficiency

The docs claim that `SAMPLE_METADATA_*` variables in docker-compose get substituted when running docker compose, but `docker compose build` is run per-task, and does not have access to this variable.  I can't quite tell if this is intended, but at the very least it seems a bit unfortunate that warnings will always be emitted about missing variables.

The way around this is to provide a default value for the variable as described here: https://inspect.ai-safety-institute.org.uk/agents.html#sample-metadata
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] live/realtime `inspect view`
Root cause: Unimplemented Feature Gap

I'd like to be able to see messages as they are sent and responded to, but it seems the transcript is not written to until the very end.  Can there be an option `--live-transcript` or similar to `inspect eval` to write to the transcript on every message update (or have a function that is `update_transcript` or similar), so I can see partial progress?

We are working on this, but it will come in stages. First will be to make the log/transcript streaming (likely moving it into SQLite so we can handle much larger log file sizes). Then we'll hookup UI to the streamed transcript (likely we'll first do this in the terminal and then eventually in inspect view)

This is now available on `main`: https://github.com/UKGovernmentBEIS/inspect_ai/pull/865 (this is the terminal version of live transcripts, Inspect View may get this eventually but hopefully this provides for the main use case of seeing partial progress)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Tool environment link is broken

 the "tool environment" link near the bottom of [this page](https://ukgovernmentbeis.github.io/inspect_ai/tools.html#sec-tool-environment) does not go anywhere

Thx! Will fix that in the next push of the docs site.

Fixed in sync (https://github.com/UKGovernmentBEIS/inspect_ai/commit/210e5fa7d8e07547f99d34ffc1caa3f03dbe8252). Note the doc site will be redeployed when we push PyPI (next 2-3 days).
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Broader HuggingFace model compatibility
Root cause: Unimplemented Feature Gap

I'd like to use the framework to obtain model behaviors from a T5 model (flan-alpaca-gpt4-xl) but currently, the framework's HuggingFaceAPI class is designed to work only with AutoModelForCausalLM, which limits its use to decoder-only models and not encoder-decoder models like T5

Request: Extend the HuggingFaceAPI class to support a broader range of HuggingFace model architectures, e.g. Seq2Seq models (e.g., T5, BART) using AutoModelForSeq2SeqLM

Seq2Seq models aren't especially frontier but it could still be useful to collect their responses and allow a wider range of models to be compared

We don't have plans to do this as the main use cases for the library are centered on decoder only chat models. Would certainly be open to a PR for this though.

Note that we have lots of users also using Ollama (which can use HF models: https://huggingface.co/docs/hub/en/ollama) and vLLM (which can also use HF models: https://docs.vllm.ai/en/latest/models/supported_models.html). Not sure whether this helps at all for these model types though.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Typo in _generate_config.py

`"""Random seed. OpenAI only. OpenAI and Mistral only."""`

"OpenAI only" should be removed from [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_generate_config.py#L48)

Yikes, good catch :-) Will be addressed in the next sync to the public repo.

Fixed in latest sync: https://github.com/UKGovernmentBEIS/inspect_ai/commit/210e5fa7d8e07547f99d34ffc1caa3f03dbe8252
Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature Request: Support Anthropic on Vertex models.
Root cause: Unimplemented Feature Gap

Currently Anthropic models on AWS Bedrock is supported but not Google Vertex. 

"As a user of Inspect eval framework and also GCP, I'd like to use Anthropic models through Vertex AI in the Google Cloud project and region of my choice, rather than having to create a new billing process to use through the Anthropic API" 

Details on all anthropic models:
https://docs.anthropic.com/en/docs/about-claude/models

I don't have access to Vertex to test, however I just made a change that _should_ enable you to put together a PR for this. The Anthropic client already supports Bedrock, and I just made it possible to use an `anthropic/bedrock/` model name prefix to opt into using this client: https://github.com/UKGovernmentBEIS/inspect_ai/pull/593/commits/ca04ef5ef5c0fe6497fa1b9d90e1d7a174123377

Anthropic also has a built-in Vertex client, so you could add a `vertex` flag (and service name check) that binds to Vertex rather than Bedrock or the default Anthropic endpoint.

> Currently Anthropic models on AWS Bedrock is supported but not Google Vertex.
> 
> "As a user of Inspect eval framework and also GCP, I'd like to use Anthropic models through Vertex AI in the Google Cloud project and region of my choice, rather than having to create a new billing process to use through the Anthropic API"
> 
> Details on all anthropic models: https://docs.anthropic.com/en/docs/about-claude/models

Hi, Simon, have you solved this issue?


I have attempted to implement this here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1122

There are docs on how to make this work in the PR notes. Unfortunately when I try to make this work on my own account I get 429 errors (hard to know how to overcome these, billing and other config looks fine).

If others could install this branch and try it out I'd be appreciative.

I have merged this as it appears to all work correctly (save for the fact that all of my requests are rate limited and attempting to request an increase in rate limits for Anthropic models on Vertex results in an immediate "can't increase your quota above 0 at this time"). It may just be difficult to get Anthropic API resources provisioned through Vertex right now (I'm sure it depends on the customer though). 

If people want to try this out on their own accounts here are the basic docs: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1122

You might also want to add `--log-level=http` so you can see the HTTP rate limits in the console.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Use of paid API for `web_search()` tool
Root cause: Documentation Deficiency

Hi! :)

Your docs explain:

> The web_search() tool uses [Google Programmable Search Engine](https://programmablesearchengine.google.com/about/). To use it you will therefore need to setup your own Google Programmable Search Engine and also enable the [Programmable Search Element Paid API](https://developers.google.com/custom-search/docs/paid_element). 

The Programmable Search Element Paid API page says that:

> The [Programmable Search Element](https://developers.google.com/custom-search/docs/tutorial/implementingsearchbox) now offers a paid ads-free version. By enabling the Programmable Search Element Paid API your Programmable Search Element will continue to function exactly the same as the free ad-supported version, except that instead of ads being displayed along with search results you'll instead be charged a small fee for each search.

To clarify: is the reason you require the paid API entirely to avoid ads? Or is there some other reason you are using the paid API?

AFAIK the paid API is the only way to get the REST API that we use in Inspect (they don't have a free REST API for search). This may have changed recently but that was the state of things when we implemented this.

Okay, I checked, and it does look like you can now get 100 queries per day for free: https://developers.google.com/custom-search/v1/overview (not sure if this is enough for your evals?). This should work now with Inspect.

A couple of other search APIs that people like are Bing and Brave (don't know if they have free tiers). If you look at the source code for `web_browser()` you'll see that it is well set up for additional providers, so if you know of a free provider you want to add support for we'd be glad to take a PR.

Makes sense, thanks!

I had looked at https://developers.google.com/custom-search/v1/overview but hadn't seen that they limit API access to 100 queries per day.

Since that's so few in our context (e.g. [GAIA](https://huggingface.co/datasets/gaia-benchmark/GAIA) has 450 samples), it's probably fine to keep your docs as written, and require the two environment variables as you currently do.