zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: New benchmarks should go under benchmarks directory or in a separate package?

Do I understand correctly that AISI prefers other benchmarks to be in a separate package?

Or do you expect users to contribute benchmarks under `benchmarks/` similar to [OpenAI's registry](https://github.com/openai/evals/tree/main/evals/registry/evals) or [lm-evaluation-harness's tasks](https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks)?

The benchmarks in Inspect are not intended as a formal suite, but more example uses of the framework. We will generally check them on 2 or 3 models to ensure that the results are comparable to what is published, however a formal suite would actually ideally be run regularly (and adapted as required) for many more models, and we'd also want to have published results. Not sure that this is something we will do in the short term.

Net: I think if there is a broader effort to build out a suite of Inspect evals comparable to OpenAI/EvalHarness it would at this stage be better done as a community effort. This is something we _might_ do in the future but I have no way of handicapping likelihood or timetable.



Thanks for the reply! Makes sense. I would be happy to get this started for an initial set of benchmarks and encourage the community to add more. Do you think a separate package makes sense or would you recommend incorporating the registry into this repository?

I think it would ideally be in its own repository. Something like this: https://github.com/openai/simple-evals 

Let us know here if/when you put up the repo! (and of course feel free to use the benchmarks we've already published as a starting point). Note that we have BoolQ (https://arxiv.org/abs/1905.10044) and PIQA (https://arxiv.org/abs/1911.11641) benchmarks coming in our next release (~ next few days)

@alexandraabbas we've some additional internal discussion on this and do think it would be beneficial if we make `benchmarks/` more than just examples. That would mean running them on some models and publishing the results in the README as well as providing scripts to run them all against any model (generating the same output).

@sdtblckgov Is going to spend some time working on this. We will post back to this issue once we have something more to share. In the meantime, if you are working on benchmarks let us know the repo URL here and we can provide feedback, etc. (eventually we hope to be able to merge these as PRs, but we are still working on the process for that).

I think making `benchmarks/` more than just examples would be very beneficial!

No problem! Sounds good, I'll get back with the repo once I have something to show.

Hi @alexandraabbas, an update. We have now sorted things for taking contributions into `benchmarks/`. If you are still interested in contributing here please coordinate with @jwilles from Vector Institute as they are leading this effort. Thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view bundle` reverses the order of experiments
Root cause: Algorithmic Error

## Repro

Assuming you have a directory of inspect logs called `logs`:

Run `inspect view --log-dir logs`. Open the sidebar that shows all of the logs in that directory, and observe their order. They are ordered from **newest to oldest**.

Run `inspect view bundle --output-dir bundle --log-dir logs`. Serve the resulting `bundle` directory with `npx serve bundle` (or any other local web server that supports HTTP range requests, which `python -m http.server` notably does not). Open that web view in your browser. Open the sidebar that shows all of the logs in the directory, and observe their order. They are ordered from **oldest to newest**.

## Cause

Experiments are ordered by `mtime`. [src/inspect_ai/log/_bundle.py:copy_log_files](https://github.com/UKGovernmentBEIS/inspect_ai/blob/17aa2a19caf22ecf37c940ccc6831c9cab8b7ce0/src/inspect_ai/log/_bundle.py#L140-L170) reverses the mtime-ordering of the logs. It first reads the logs, sorting them from newest to oldest:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/17aa2a19caf22ecf37c940ccc6831c9cab8b7ce0/src/inspect_ai/log/_bundle.py#L148-L150

Then in that same order, newest to oldest, it copies the logs to an output directory:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/17aa2a19caf22ecf37c940ccc6831c9cab8b7ce0/src/inspect_ai/log/_bundle.py#L165-L166

This has the effect of reversing the `mtime` ordering: the newest files are copied first, thereby making them the oldest files in the output directory (by milliseconds or less, but all that matters for sorting is the relative order).
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view bundle` does not preserve mtime-ordering of the logs directory
Root cause: Algorithmic Error

## Summary

When you run `inspect view bundle --output-dir <output> --log-dir <input>`,  it copies log files from `<input>` to `<output>/logs`. This copy does not preserve relative mtime-ordering, meaning that the oldest log in `<input>` (as measured by mtime) will not necessarily be the oldest log in `<output>/logs`.

Why might this matter? Because `inspect view` uses mtime to determine ordering in the sidebar that shows all of your experiments.

In the typical case, this actually does not cause a problem. `inspect view bundle` also produces `output/logs/logs.json`. This file ([almost!](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1902)) respects the mtime order of `<input>`, and this is the file that the web view uses to determine the order of experiments in the sidebar.

However, it becomes a problem if you ever use `<output>/logs` in a subsequent `inspect view` call. If you do, that view will show the experiments in essentially a random order. This makes it a lot harder to find what you are looking for if you have many experiments.

I imagine most users won't reuse `<output>/logs` in this way. But I did, and I assume I won't be the only one.

## Cause

The code responsible for this copy is here:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/17aa2a19caf22ecf37c940ccc6831c9cab8b7ce0/src/inspect_ai/log/_bundle.py#L204-L215

It copies files in the order they are returned by `os.walk`, which does not guarantee sorting the files in any particular order.

## Fix

The most straightforward fix is to sort the files by mtime when doing that copy: #1905

An alternative that I would nominate for consideration is to have `inspect view` sort experiments by something other than mtime, eg the `started_at` or `completed_at` fields, or the timestamp in the experiment's filename. Sorting by the content or name of the files seems like a less-surprising behavior to me than sorting by mtime. mtimes are sort of fragile, and easy to change without even noticing it. That happened [here](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1902), for example. Sorting by name or contents would have prevented that issue.
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view` uses mtime to sort experiments, which is fragile
Root cause: Architectural Constraint

`inspect view` has a sidebar that shows all of the experiments in an experiment log directory. These experiments are sorted by the `mtime` of their corresponding log file.

I think `mtime` is too fragile to be used as a sorting key like this. For example, both #1902 and #1904 were caused by the `mtime` changing.

Maybe it would be better to use the log's `started_at` or `completed_at` fields or the timestamp in the log's filename?

One of the problems we have is that for listing logs (which can run into the hundreds) we don't want to have to crack the log file headers. That leaves us with 1 of 2 possible approaches: Use the timestamp embedded in the log file name (which is basically `started_at`); or (2) Use the mtime as we do now.

@dragonstyle What do you think about uisng the timestamp embedded in the file name here?

This problem is made more painful on s3, where the Last-Modified timestamp only has granularity of one second. So if you ever need to bulk-edit your eval files, you cannot do even trivial edits any faster than one second per file otherwise you lose your sorting order.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Async custom scorers error out if type annotation missing
Root cause: Validation Gap

Tried out inspect evals yesterday and things worked great out of the box - thanks for all the hard work!


I did run into some sharp edges though: There seems to be some unintuitive behavior when defining custom scorers, where code behavior changes depending on whether you have annotated your scorer function with the `Scorer` return type.

Consider this (bug_repro.py):
```python
from inspect_ai.scorer import Score, Scorer, Target, scorer, accuracy
from inspect_ai.solver import TaskState


@scorer(metrics=[accuracy()])
def working_score() -> Scorer:
    async def _score(state: TaskState, target: Target):
        return Score(
            value=1,
        )

    return _score


@scorer(metrics=[accuracy()])
def broken_score():
    async def _score(state: TaskState, target: Target):
        return Score(
            value=1,
        )

    return _score
```

Now, if I run `inspect score path/to/some.eval --scorer bug_repro.py@working_score`, it works. Conversely, `broken_score` does not.

While the docs _do_ provide an example with the annotation, if it is not provided, the error is somewhat cryptic: 
```
    scorer = scorer_type(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: broken_score() takes 0 positional arguments but 2 were given
```

This led me to think that I needed the outer function to also take in the state+target args, but that raises this error:
```
core_task
    result = await scorer(state, target)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: object function can't be used in 'await' expression
```

Personally, I would avoid relying on type hints to change behavior in an implicit way like this. If it is non-trivial to do without, it feels like it should be mandated as required (either Score or Scorer)? 


I agree with you sentiment here (relying on type hints for correct runtime behavior). I believe this originally put in so that we could support scorers that either _returned_ a `Scorer` or were just a plain scorer function, e.g. it was intended that this could work:

```pyhton
@scorer(metrics=[accuracy()])
async def _score(state: TaskState, target: Target):
    return Score(
       value=1,
    )

    return _score
```

And we are using the type annotation to distinguish these forms. That said, we could rather easily use other proxies like the presence of `state` and `target` arguments. Beyond that, while the above form was anticipated in the earliest implementations we've never documented nor encouraged it, so another approach would be to just remove that possibility (since it's unlikely anyone is using it).

We'll clean this up soon (thanks for describing the issue and sorry you hit that sharp edge!).


Explicit error now raised here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2198
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect view extension switching focus to completed tasks makes running experiments while doing analysis hard
Root cause: Algorithmic Error

A workflow which is common for me to do is to continually run experiments in the background, and do analysis  on previous experiments while they are completing. However, this is difficult do do since the extension will switch to the completed tasks, meaning I am consistently switched out of my analysis.

I'll have a look right away- we should be opening the viewer, but ideally not stealing focus. 

This should be fixed v0.3.26 of the extension, rolling out shortly!

We have another extension update which actually completely eliminates auto-opening of the log viewer (it will update the viewer in the background if its open, but it will never open it or activate it automatically). That should mean this buggy behavior is finally behind us.
Repository: UKGovernmentBEIS/inspect_ai
Issue: inspect view: links to sample `n` display sample 1 instead
Root cause: Algorithmic Error

Following a link of the form `/#/logs/<log>/samples/sample/<n>/1` in a new browser tab, the web view will show sample 1, regardless of the value of `n`.

Hi, thanks for reporting this. Could you give it a try with the latest version on main (if you haven't already)? We merged a change earlier this afternoon EST that I believe includes a fix for this.

Just tried it and it is indeed fixed on main. Thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: hf_dataset does not shuffle choices as expected
Root cause: Algorithmic Error

`hf_dataset(..., shuffle_choices=True)` uses [the following logic](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/dataset/_sources/hf.py#L128-L132) to shuffle choices:

```python
    # maybe shuffle the choices
    if isinstance(shuffle_choices, int):
        memory_dataset.shuffle_choices(seed=shuffle_choices)
    elif shuffle_choices is True:
        memory_dataset.shuffle_choices()
```
but `isinstance(True, int) == True`, so passing `shuffle_choices=True` actually calls `memory_dataset.shuffle_choices(seed=True)`, which is treated as `memory_dataset.shuffle_choices(seed=1)`, and the second clause is unreachable.

Suggest reversing the clauses in the `if`, but it would need an extra clause for `False` because `isinstance(False, int) == True` as well:
```python
    # maybe shuffle the choices
    if shuffle_choices is True:
        memory_dataset.shuffle_choices()
    elif shuffle_choices is False:
        pass
    elif isinstance(shuffle_choices, int):
        memory_dataset.shuffle_choices(seed=shuffle_choices)
```

Could you give us a PR for this?
Repository: UKGovernmentBEIS/inspect_ai
Issue: web_search tool only returns summary of search results when using Tavily provider
Root cause: Algorithmic Error

The `tavily_search_provider` for the `web_search` tool currently only returns a summarised `answer` in response to queries, and does not include the actual search results and urls. The results returned are of the format:

```
Here are your web search results. Please read them carefully as they may be useful later!
Lionel Messi is a renowned Argentine soccer player who won the 2022 FIFA World Cup with Argentina and played for FC Barcelona and Paris Saint-Germain. He holds numerous records, including most Ballon d'Or awards. He now plays for Inter Miami CF in the MLS.
```

Considering the current situation that web searching using headless browsers is being blocked by Google (see [this discussion on the Inspect Community slack](https://inspectcommunity.slack.com/archives/C080K7SQ4SG/p1747743099928299?thread_ts=1747697109.162979&cid=C080K7SQ4SG)), it would be very useful to be able to search with the `web_search` tool and then use the `web_browser` tools to visit the web pages returned in the results, since the `web_browser` tool is not able to search via google.
This issue was identified attempting to implement the [browse_comp eval](https://github.com/UKGovernmentBEIS/inspect_evals/pull/345).

The [`google_search_provider`](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/tool/_tools/_web_search/_google.py#L130) returns a list of `SearchLinks` that contain a `snippet` and a `url`, but the [`tavily_search_provider`](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/tool/_tools/_web_search/_tavily.py#L98) returns just the `answer` field and discards the list of `results` that each have `title`, `url`, `content`, `raw_content`, and `score` fields.
It would also be helpful if the structure of the results returned from the different providers were as similar as possible to be able to change out one for the other without having to adapt consuming code. Possibly they could both use a generalised `SearchLink` model.


Example raw response content from Tavily: [messi_tavily.json](https://github.com/user-attachments/files/20451597/messi_tavily.json)
Example raw response content from Google Custom Search Engine: [messi_google_cse.json](https://github.com/user-attachments/files/20451604/messi_google_cse.json)


Fixed in #1940
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view` crashes with list of null Score metadata
Root cause: Validation Gap

If a scorer returns `metadata` where any of the values are a list that contains a `None` value, it seems like `inspect view` crashes when viewing that sample.  For example, consider a scorer that just returns:

```python
return Score(                                                                                                                                                                    
            value=1.0,                                                                                                                                  
            answer="",                                                                                                                                                                   
            explanation="",                                                                                                                                                              
            metadata={"key": [None, None]},                                                                                                                                              
        )
```

After running `inspect view` and clicking on the sample in question, I initially got a blank screen. Upon refreshing, I got this error:

![Error Image](https://github.com/user-attachments/assets/57f17bb2-b522-4315-a396-3a11fcbec7a3)

I'm running with `0.3.100.dev9+gce6d8ebc` (commit ce6d8ebc).  Log file attached: 

[2025-05-28T18-26-10+00-00_arithmetic-demo_RFgifiiGkjggqaKqN2L5FS.eval.gz](https://github.com/user-attachments/files/20493410/2025-05-28T18-26-10%2B00-00_arithmetic-demo_RFgifiiGkjggqaKqN2L5FS.eval.gz)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Default reducer only works with floats - not str
Root cause: Validation Gap

Without custom reducer, scores are converted to float which breaks str scores
Steps to reproduce

TLDR scorer without a specified score_reducer will not work for string values i.e. value_to_float is called by the reducer - by default inspect evals uses a mean reducer which will use value_to_float (which will not work on string values).

i.e. any scorer that returns dict[str] will have issues. The metric list[SampleScore] will have Score.value that are coerced to floats (they get coerced to 0.0). This is because of this default score_reducer - https://github.com/UKGovernmentBEIS/inspect_ai/blob/63de8517769681d79d7f2cc1dcd95b891295b22d/src/inspect_ai/_eval/task/results.py#L175

[Original slack thread where the issue is reported](https://inspectcommunity.slack.com/archives/C080K7SQ4SG/p1748087588876969)


**Expected behavior** 

Metrics SampleScore should not coerce score values to float 0.0 when they are originally str

**Actual behavior** 

SampleScore values in the metric are actually coerced to float 0.0 values.

**Environment** 

This will impact all {OS, Python version} combinations due to the nature of the bug. The inspect evals version is Version: 0.3.dev1272+g66e0e1f.d20250522

**Additional context**

See [thread](https://inspectcommunity.slack.com/archives/C080K7SQ4SG/p1748087588876969)

cc @dragonstyle
Repository: UKGovernmentBEIS/inspect_ai
Issue: mouse support is broken on some terminals

https://github.com/UKGovernmentBEIS/inspect_ai/blob/ac044d9e5932eff203d9e20d532dab93f0f9b3be/src/inspect_ai/_display/core/textual.py#L8-L21

^ This code causes mouse support to fail on some terminals, including at least ghostty.

The problem is that, contrary to the comment in the code, enabling mouse support is not idempotent.

`_enable_mouse_support`  enables the following modes, in order: `1000`, `1003`, `1015`, and `1006`. Separately, `LinuxDriver` sometimes enables mode `1016`. Importantly, modes `1006`, `1015`, and `1016` _conflict with each other_. They are mutually exclusive. If more than one is enabled, the terminal will honor whichever one was enabled most recently.

This is the sequence of events that happens on ghostty (and presumably some other terminals):
- `_enable_mouse_support` is called for the first time, which enables `1000`, `1003`, `1015`, and `1006`.
Current active mouse event format: `1006`.
- `_enable_mouse_pixels` is called, which enables `1016` _and records that we should expect `1016`-style mouse events_.
Current active mouse event format: `1016`.
- The code in the listing above calls `_enable_mouse_support` again, which enables `1000`, `1003`, `1015`, and `1006`.
Current active mouse event format: **`1006`**. Mouse event format Textual _thinks_ is active: **`1016`**.

At this point, the terminal is sending mouse events in `1006` format, meaning it is reporting the (x, y) coordinates of the _cell_ the mouse is currently over. But Textual is expecting `1016` format, meaning the _screen pixel_ the mouse is currently over. The latter is much larger than the former, so Textual divides the incoming coordinates by the size of each cell. It ends up with a very small number, which leads the app to behave as if the mouse is trapped in a very small corner in the upper-left of the screen.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature Request: A method to track the token usage and cost of a custom agent using the bridge solver.
Root cause: Documentation Deficiency

I have a custom multi-agent system that is integrated into inspect_ai using the ```bridge``` function. 

It is capable of calculating the cost and token usage for each of its own runs. 

However, there does not seem to be a way to pass this cost to the output when using the ```eval()``` function. 

Is there currently a workaround for this issue? 

Best, 

Jack. 

The way I would do this is have your bridged agent write to the store (https://inspect.aisi.org.uk/agent-custom.html#agent-store). The store data is then accessible in the eval log under `sample[x].store`.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Args docstring types

I'm curious if it's required to include the `(int)` bit in the args docstring:

<img width="1199" alt="Image" src="https://github.com/user-attachments/assets/7a239e8e-e758-4071-b3bd-01b29287c726" />

 Because in the example source it includes them but later referencing that section, they are omitted:

```
Args:
    x: First number to add.
    y: Second number to add.
```

good catch! these are in fact not required so should be removed from the docs, done here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/a2dfad1a6e88067c126bfdebf0236864585c2c02
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Doc] Loading theory_of_mind() is missing in intro
Root cause: Documentation Deficiency

Perhaps this is not an issue but...

In https://inspect.aisi.org.uk/

'''
from inspect_ai import eval

eval(theory_of_mind(), model="openai/gpt-4o")
'''

Is missing the loading of theory_of_mind()







Thanks! Fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/89af97adc95ba2d6508f0c6355113ec4ca279a00
Repository: UKGovernmentBEIS/inspect_ai
Issue: inspect view: navigating through samples list does not update navigation history
Root cause: Algorithmic Error

To reproduce, open an eval in `inspect view` and click on a sample. Notice the URL corresponds to the sample you clicked, eg `127.0.0.1:7575/logs/#/<eval>/samples/sample/5/1`. Now navigate to different samples by either pressing the left and right arrow keys or clicking the left and right arrows in the header of the sample view. The URL has not changed. It still shows the sample id of the first sample you clicked on.

Thanks for reporting this - I'll have a look asap!