zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Filter by sample metadata in log viewer
Root cause: Unimplemented Feature Gap

## Current behaviour
The log viewer offers some filtering, including on the contents of the `input` and `target` fields:

<img width="439" alt="Image" src="https://github.com/user-attachments/assets/53e81963-9a0b-4f93-ac31-a62ff882b5c7" />

## Feature suggestion
Offer filtering on sample metadata. 

## Open questions
`metadata` is in general:
* an untyped dictionary
* that may be nested

There may be UI challenges.

Not supporting nesting in the filtering feature would be acceptable, and I think a good trade-off?
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect_ai._eval.score.score` with Sandbox scorer raises "ProcessLookupError: No sandbox environment has been provided..."
Root cause: Documentation Deficiency

I am following the docs here https://inspect.ai-safety-institute.org.uk/scorers.html#score-function, and accordingly calling `inspect_ai._eval.score.score`

Here is the traceback:

```
  File "/Users/t/.asdf/installs/python/3.11.9/lib/python3.11/asyncio/tasks.py", line 279, in __step
    result = coro.throw(exc)
             ^^^^^^^^^^^^^^^
  File "/Users/t/Library/Caches/pypoetry/virtualenvs/benchmarks-QWdJvsQ2-py3.11/lib/python3.11/site-packages/inspect_ai/_eval/score.py", line 114, in score_async
    scores: list[dict[str, SampleScore]] = await asyncio.gather(*tasks)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/t/.asdf/installs/python/3.11.9/lib/python3.11/asyncio/tasks.py", line 349, in __wakeup
    future.result()
  File "/Users/t/.asdf/installs/python/3.11.9/lib/python3.11/asyncio/tasks.py", line 277, in __step
    result = coro.send(None)
             ^^^^^^^^^^^^^^^
  File "/Users/t/Library/Caches/pypoetry/virtualenvs/benchmarks-QWdJvsQ2-py3.11/lib/python3.11/site-packages/inspect_ai/_eval/score.py", line 213, in run_score_task
    result = await scorer(state, target)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/t/repos/benchmarks/bench/task/frontiermath/scorer.py", line 108, in score
    result = await sandbox().exec(
                   ^^^^^^^^^
  File "/Users/t/Library/Caches/pypoetry/virtualenvs/benchmarks-QWdJvsQ2-py3.11/lib/python3.11/site-packages/inspect_ai/util/_sandbox/context.py", line 37, in sandbox
    raise raise_no_sandbox()
          ^^^^^^^^^^^^^^^^^^
  File "/Users/t/Library/Caches/pypoetry/virtualenvs/benchmarks-QWdJvsQ2-py3.11/lib/python3.11/site-packages/inspect_ai/util/_sandbox/context.py", line 111, in raise_no_sandbox
    raise ProcessLookupError(
ProcessLookupError: No sandbox environment has been provided for the current sample or task. Please specify a sandbox for the sample or a global default sandbox for the task
```

Thanks! We have this one logged here with an MRE (https://github.com/UKGovernmentBEIS/inspect_ai/issues/679) so will close this issue.

One very important thing to know and one reason we haven't done this yet is that this obviously will not bring the sandbox back to life in the correct form/state for e.g. scoring a SWE bench run. So it is of pretty narrow (but still potentially handy) utility: it's just for scorers that can operate completely stateless (e.g. HumanEval). It looks like this is for FrontierMath so it probably fits the bill and would be useful/applicable.

Note that we will implement this very soon (and use the other issue as the reference for when it is complete/closed).

> just for scorers that can operate completely stateless (e.g. HumanEval)

Yes, makes sense. 

Sorry for duplicate, this is indeed just #679
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support for Anthropic "extended thinking"
Root cause: Unimplemented Feature Gap

https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking

Thanks! Will try to get this in tomorrow.



Done here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1400

Docs here: https://inspect.ai-safety-institute.org.uk/reasoning.html
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view` does not know how to render tool calls with arguments of type `dict[str, str]`
Root cause: Validation Gap

I see 

`
tool: transpilation_tool(lean_code: "def plus_0_r_statement := ∀ n : Nat, n + 0 = n def plus_comm_statement := ∀ a b : Nat, a + b = b + a",coq_lean_identifiers: [object Object])
`

![Image](https://github.com/user-attachments/assets/7eec706d-1e28-4728-b885-37cef8160227)

The inner tool is defined with `async def translate(lean_code: str, coq_lean_identifiers: dict[str, str])`

This has been resolved. Here is my test case:

```python

@tool
def associate():
    async def execute(associations: dict[str, str]):
        """Associate synonyms with each other.

        You should pass a dictionary of words to associate().
        For example `{ "happy": "elated", "sad": "despondent" }`

        Args:
             associations: Dictionary of synonmyms to associate

        Returns:
             The associations as a string
        """
        return str(associations)

    return execute


@task
def association():
    return Task(
        dataset=[
            Sample(
                input="Please use the associate() tool to associate three synonyms with each other"
            )
        ],
        solver=[use_tools(associate()), generate()],
    )
```

Which now gives:

<img width="708" alt="Image" src="https://github.com/user-attachments/assets/4f9f6844-8184-461d-8dc4-9690df43a808" />
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect view fails to render long steps of the transcript
Root cause: Algorithmic Error

This happens both on web and in VSCode. I'm happy to share a log over Slack but won't upload here so it doesn't appear in training data.


https://github.com/user-attachments/assets/e740043d-fff5-44ae-93ce-d9b19569d280


Note this is distinct from #1375
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer hides part of Assistant response when response contains string "Final Answer"
Root cause: Documentation Deficiency

I encountered this bug while using 484131e0 for the log viewer and dd1e12f95cb3809 to run the eval.

For gemini-2.0-flash-001 (and probably all Gemini models), apparently when the response contains the string "Final Answer", the log viewer hides the rest of the message! :(

# Example

## Link to log viewer
See Sample ID `2024-Mock-AIME-Problem-02`, epoch 6, in https://logs.epoch.ai/inspect-viewer/484131e0/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FZnGSRrKbiEnuju4kk4sgVb.eval

## Screenshots
<img width="1203" alt="Image" src="https://github.com/user-attachments/assets/8b134a34-f545-4f3f-8ba9-d7219a279f0e" />

<img width="1174" alt="Image" src="https://github.com/user-attachments/assets/53b88316-2cfc-489c-901c-f43f04b821de" />

Ah okay, this actually makes sense. The fields are there, and probably log viewer https://github.com/UKGovernmentBEIS/inspect_ai/commit/484131e03529c3d3f8b1f58e72a84ced1f2a61b3 doesn't know about them.

<img width="1209" alt="Image" src="https://github.com/user-attachments/assets/b7f740dc-c15b-4a5c-9acf-856641a80f9b" />
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer: Transcript shows multiple empty assistant blocks, possibly due to rate limits
Root cause: Validation Gap

# Example
See Sample 1, Epoch 1 of https://logs.epoch.ai/inspect-viewer/484131e0/viewer.html?log_file=https%3A%2F%2Flogs.epoch.ai%2Finspect_ai_logs%2FPyNEkFCfMEReodeE2oiPhh.eval

<img width="1058" alt="Image" src="https://github.com/user-attachments/assets/e96c0e2b-444b-4f89-9a0e-d15c3ef324a1" />

# Hypothesis
I suspect this happens when hitting API rate limits. Each call to generate gets recorded in the transcript, even if it didn't return an assistant message. Note that there is no "API" button on the calls where the empty assistant response appears. 

This is caused by rate limits or other interruptions to the model call which results in a pending event being logged. That is addressed here:

https://github.com/UKGovernmentBEIS/inspect_ai/pull/1338/commits/975e335ce93b65689cac48aba683104fa0b77471

Updating to a more recent version of the viewer will resolve this issue.

Thank you, perfect!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Langfuse integration

Hey! It's only been a few days since this was publicised, but I'm liking the dev experience, functional decorator patterns. I'm keen to further explore wrapping eval metrics from RAGAs, as well as lightweight utils for exports to Langfuse. Looks very extensible to LangChain and other frameworks too!

I wonder what's the best way to contribute/ add to the project structure for these features:

- Langfuse integration
- LangChain/ LlamaIndex integration
- RAGAs integration

Thanks! Glad you are finding things navigable so far. 

We do indeed aim to be highly extensible. One motivation for this is to keep the core inspect package fairly lean -- given how much innovation is going on in LLMs if we natively supported "all the things" then the package could get to be too large for its own good.

So our hope is that integrations like you have mentioned here could be done in external packages (which we'd then link to from an "Extensions" page). In some cases it might be better to integrate things directly, but our opening bid will always be "can we do this externally and if not is there an API we can add to make it possible".



Closing this as resolved, feel free to post back here (or create a new issue) if you want to continue the discussion.
Repository: UKGovernmentBEIS/inspect_ai
Issue: support for custom command line parameters (e.g. `-U foo=bar`)
Root cause: Unimplemented Feature Gap

Would be great if in addition to -T, -S and -M inspect allowed custom command line parameters, say -U for user/universal/unspecified, which could be retrieved manually.

Motivation: models and solvers are examples of components that can be shared by different evals and parametrised independently of the rest of the eval. But if I want to create a common component which doesn't conform to either model or solver API, then there is no way to make it parametrisable without updating every task to read and propagate component parameters.



Note, equivalent (but way less elegant!) way of doing this now is through CLI bound environment variables, e.g.

```bash
FOO=bar BAZ=qux inspect eval math.py
```

Then from code:

```python
os.environ.get("FOO")
```

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1485
Repository: UKGovernmentBEIS/inspect_ai
Issue: Anthropic SDK doesn't allow requests with large max_tokens without streaming
Root cause: Validation Gap

Due to 

https://github.com/anthropics/anthropic-sdk-python/blob/599f2b9a9501b8c98fb3132043c3ec71e3026f84/src/anthropic/_base_client.py#L674-L687

```python
    def _calculate_nonstreaming_timeout(self, max_tokens: int) -> Timeout:
        maximum_time = 60 * 60
        default_time = 60 * 10

        expected_time = maximum_time * max_tokens / 128_000
        if expected_time > default_time:
            raise ValueError(
                "Streaming is strongly recommended for operations that may take longer than 10 minutes. "
                + "See https://github.com/anthropics/anthropic-sdk-python#long-requests for more details",
            )
        return Timeout(
            default_time,
            connect=5.0,
        )
```

Looks like this is covered in the docs:

> You should limit the total output tokens (`max_tokens` + `reasoning_tokens`) to no more than 21,333 tokens (as the streaming API is required above that—this will soon be supported in Inspect but is not currently). Even with streaming, a maximum of 32,000 is recommended (above that they advise using batch mode).



Closing in favour of more specific https://github.com/UKGovernmentBEIS/inspect_ai/issues/1408
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support Anthropic streaming API
Root cause: Unimplemented Feature Gap

Just an issue to track this. 

Docs say:

> You should limit the total output tokens (max_tokens + reasoning_tokens) to no more than 21,333 tokens (as the streaming API is required above that—this will soon be supported in Inspect but is not currently). Even with streaming, a maximum of 32,000 is recommended (above that they advise using batch mode).

Done here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1421
Repository: UKGovernmentBEIS/inspect_ai
Issue: Scoring tool calls with transcript
Root cause: Documentation Deficiency

I'm wondering about the best way to make a Scorer that checks whether the solver's tool calls match an expected sequence of tool calls for the sample.  Is there a recommended pattern for this in Inspect?

I see a benchmark in `inspect_evals` [using `state.messages`](https://github.com/UKGovernmentBEIS/inspect_evals/blob/fc9deb7dca1f3c3c0520da33faa12a750c0021fb/src/inspect_evals/agentharm/benchmark/grading_utils.py#L29) to check calls, but this risks missing calls if the solver uses history truncation/compression (something I plan to experiment with).  My other idea was to scrape `ToolEvent`s from `inspect_ai.log.transcript().events`, but I wanted to check if this is an intended/supported mechanic for the Scorer to use (since it isn't directly passed to score() and I didn't see any existing scorers using it); wouldn't want a change to logging format to inadvertently affect the behavior of scoring.  Is scoring `transcript`s supported or should I be doing something else?

(p.s. Inspect has been super useful so far for a new eval suite we're building; thanks for creating it!)

The `transcript()` is a supported API so you could use it (but to your point its a bit more intended for observability than scoring). Another thing you could is whenever you truncate history just record the tool calls in `state.store` (which you can of course get to from the scorer as well)

Thanks, this is helpful; sounds like maybe there's no perfect solution but I'll try using transcript for now (we're aiming to support generalist/multi-task agents, so in this case prefer to avoid adding `state.store` logic on each solver/history util to support this one eval)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support Claude Sonnet 3.7 computer use
Root cause: External Dependency Breakage

Sonnet 3.7 uses the `computer_20250124` tool with some new functionality, but Inspect uses `computer_20241022` (3.5 only)

Reference: https://github.com/UKGovernmentBEIS/inspect_ai/blob/4a573099c88fad9d04f8a3456c5f4f80834836db/src/inspect_ai/model/_providers/anthropic.py#L512

https://docs.anthropic.com/en/docs/agents-and-tools/computer-use#claude-3-7-sonnet-tools

```
│ RuntimeError: BadRequestError('Error code: 400 - {\'type\': \'error\',          │
│ \'error\': {\'type\': \'invalid_request_error\', \'message\':                   │
│ "\'claude-3-7-sonnet-20250219\' does not support tool types: computer_20241022. │
│ Did you mean one of bash_20250124, computer_20250124,                           │
│ text_editor_20250124?"}}')                                                      │
```

#1413 fixes this

https://github.com/UKGovernmentBEIS/inspect_ai/pull/1413 has been merged.
Repository: UKGovernmentBEIS/inspect_ai
Issue: How do I suppress Cache trace messages from my stdout?
Root cause: Configuration Error

No matter what I set any log level to (INSPECT_LOG_LEVEL, INSPECT_TRACE_LEVEL), I am seeing cache trace messages when I set my display to "plain". Is that expected? I would have thought I could suppress these messages somehow.

In particular, I am seeing messages like "[Cache] Computed cache key from components:" 
And of course, they are quite verbose.

Ah, I just realized that this is only happening when I run `eval` programmatically, rather than from the built-in CLI. It appears that I must manually call `init_logger` very early on in my program, before importing other inspect_ai code, or a logger appears to get configured that logs the trace information to the console by default.

(My knowledge of python logging is pretty rough, so I'm not sure what I'm doing wrong such that I don't get the same logging behavior that the inspect eval CLI does.)

Putting this at the very top of my program correctly suprresses trace messages from the console:
```python
from inspect_ai._util.logger import init_logger
from inspect_ai._util.constants import DEFAULT_LOG_LEVEL, DEFAULT_LOG_LEVEL_TRANSCRIPT

init_logger(DEFAULT_LOG_LEVEL, DEFAULT_LOG_LEVEL_TRANSCRIPT)
```

There may be a global logger getting configured by some other code in your environment -- in Inspect v0.3.70 (from roughly a week ago) we explicitly disable these loggers when starting up Inspect logging. 

We just merged a change (https://github.com/UKGovernmentBEIS/inspect_ai/pull/1430) that reduces the coupling between Inspect's log and the global logger, which I'm expecting will resolve this issue. If you could try installing from `main` to confirm that would be great:

```bash
pip install git+https://github.com/UKGovernmentBEIS/inspect_ai
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: Docker build hangs while awaiting user input but gives no corresponding indication to the user
Root cause: Validation Gap

After a Docker build within an Inspect run was taking suspiciously long, I went into the Docker Desktop logs and found that the build process was expecting an input. However, the terminal output gave no indication of this.

The Docker Desktop logs show this:
```
Configuring tzdata
------------------
Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.
1. Africa 4. Australia 7. Atlantic 10. Pacific 13. Legacy
2. America 5. Arctic 8. Europe 11. US
3. Antarctica 6. Asia 9. Indian 12. Etc
Geographic area:
```

Meanwhile, the terminal simply shows the high-level progress of the build, e.g.
```
 => [default 16/16] RUN apt-get install -y jupyter  
```

The terminal output gives no indication that user input is being requested, and entering an input does not do anything.

You can reproduce this by creating the following files and running `inspect eval reproduce.py`.

`Dockerfile`:
```dockerfile
FROM ubuntu:22.04

RUN apt-get update

RUN apt-get install -y build-essential
RUN apt-get install -y gcc
RUN apt-get install -y cmake
RUN apt-get install -y ccache
RUN apt-get install -y curl
RUN apt-get install -y git
RUN apt-get install -y wget
RUN apt-get install -y unzip
RUN apt-get install -y libreadline-dev
RUN apt-get install -y clang
RUN apt-get install -y zlib1g
RUN apt-get install -y zlib1g-dev
RUN apt-get install -y pkg-config
RUN apt-get install -y jupyter
```

`reproduce.py`:
```python
from inspect_ai import task, Task
from inspect_ai.dataset import Sample

@task
def reproduce():
    return Task(dataset=[Sample("")], sandbox="docker")
```

Oddly, the following Dockerfile does not reproduce the problem:

```dockerfile
FROM ubuntu:22.04
RUN apt-get update
RUN apt-get install -y jupyter
```

Furthermore, moving `RUN apt-get install -y jupyter` to the top of the list of `apt-get` installs in the original Dockerfile also does not reproduce the problem.

This is not especially related to Inspect, but it's why the example isn't more minimal.

Thanks for posting this for future reference, AFAICS I don't think there is anything we can do here as our docker build's are definitely meant to be non-interactive.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Recording internal calls of external systems
Root cause: Documentation Deficiency

I'm trying to compare with an external system that makes calls to gpt internally; I can get a log of its total internal gpt usage from its api, and I'd like to record the usage in Inspect so that I can compare cost against other agents.  If I add an event to the transcript:

```python
event = ModelEvent(
    model=model_name,  # openai/gpt-4o
    input=[],
    tools=[],
    tool_choice="auto",
    config=GenerateConfig(),
    output=ModelOutput(model=model_name, usage=usage),
    cache=None,
    call=None,
    pending=False,
)
transcript()._event(event)
```

The call shows up in `inspect view` but not in the cli output, e.g.:
```
╭─ arithmetic_demo (4 samples): openai/gpt-4o ──────────────────────────────────────────╮
│ message_limit: 50                                                       dataset: data │
│                                                                                       │
│                                                                                       │
│ total time:                                      0:00:10                              │
│                                                                                       │
│ accuracy: 1.0  stderr: 0                                                              │
│                                                                                       │
│ Log: logs/2025-03-05T20-54-04+00-00_arithmetic-demo_E9JaXpHPnttSQ6EvjouQA2.eval       │
╰───────────────────────────────────────────────────────────────────────────────────────╯
```

should have gpt-4o usage listed like it does when I use the `generate` solver instead:

```
╭─ arithmetic_demo (4 samples): openai/gpt-4o ──────────────────────────────────────────╮
│ message_limit: 50                                                       dataset: data │
│                                                                                       │
│                                                                                       │
│ total time:                  0:00:02                                                  │
│ openai/gpt-4o                300 tokens [I: 150, O: 150]                              │
│                                                                                       │
│ accuracy: 1.0  stderr: 0                                                              │
│                                                                                       │
│ Log: logs/2025-03-05T20-54-26+00-00_arithmetic-demo_hw7DpG7FyUEJduDttbE2KA.eval       │
╰───────────────────────────────────────────────────────────────────────────────────────╯
```

Is there a recommended way to make it show up?

We don't total up the model events but rather we track that globally. We don't have a public API for this, but you can call the private `record_model_usage()` function in the `model` module to do this.

Thanks! `record_model_usage()` does the trick