zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature request: Hide inspect_tool_support within sandbox
Root cause: Unimplemented Feature Gap

The `inspect_tool_support` package opens an HTTP port in the container. In cyber evaluations, I've seen models detect this and dedicate significant effort into analyzing this non-standard interface instead of working towards the specified tasks. I've worked around this with some prompting but it doesn't feel like a clean solution.

It would be nice if the process didn't leave open an HTTP port or had any other ways to hide its presence.

This came up in a conversation with @epatey.


@epatey Could we use a Unix domain socket instead? (most libraries should support this pretty seamlessly)
Repository: UKGovernmentBEIS/inspect_ai
Issue: minor typos in dataset docs

1. "A sample has an input, a target, an optional id, and ...". This makes it sound like the target is not optional. Two suggestions:
    - change "target" to "optional target"
    - maybe drop the sentence, because the table already explains what a sample is. The sentence also does not mention `choices` which is in the table.

2. "Note that samples from datasets without and id field will".  "and" should be "an"

3. "If your samples include choices, then the label should be a numeric index into the available choices rather a letter"
    - 'label' should be 'target'? I'm not sure.
    - 'rather a letter' should be 'rather than a letter'
 
4. "you can instead pass a function which takes an index and record".  Looks like you only need to pass a record, not an index too?
5. "See the article on Configuring the AWS CLI for additional details". Missing full-stop at end.

Thank you for these edits! We'll publish a new version of the docs w/ these corrections in the next few days (and will close this issue once we have done so).

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/commit/140c3d63ad42125213bffaa799147135c81857f3
Repository: UKGovernmentBEIS/inspect_ai
Issue: `ChatMessageUser` triggers AttributeError in Pydantic validator for `ChatMessageTool` 
Root cause: Validation Gap

When using `ChatMessageUser` with even just a `ContentText` in a sample, for some reason this ends up calling `ChatMessageTool.convert_tool_error_to_error` even when you aren't using any tools. Then, `values.get("tool_error", None)` raises an AttributeError since the wrong `values` is passed in.

I think this might be some error with Pydantic and inheritance.

Minimal task spec to reproduce:

```python
import inspect_ai
import inspect_ai.dataset
from inspect_ai.model import ChatMessageUser, ContentText
import inspect_ai.scorer
import inspect_ai.solver
from inspect_ai import task

@task
def test_chat_message_user():
    sample_input = ChatMessageUser(content = [
        ContentText(text="What's the capitol of France?"),
    ])
    
    sample = inspect_ai.dataset.Sample(
        input=sample_input,
        target="Paris",
    )

    return inspect_ai.Task(
        dataset=[sample],
        plan=inspect_ai.solver.generate(),
        scorer=inspect_ai.scorer.model_graded_qa(),
    )
```

Never mind, I needed brackets around `[sample_input]` when creating the sample, since `Sample` expects a list of messages. Kinda weird that it errored in this way though--maybe it's still worth looking into why Pydantic validation triggers for `ChatMessageTool` despite not using any tools.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Deterministic subsample
Root cause: Unimplemented Feature Gap

`inspect eval` takes `--limit N` to run eval on a random subsample of the dataset. Given the high cost of evaluating some large datasets listed on [Inspect Evals](https://ukgovernmentbeis.github.io/inspect_evals/), we may want to run model comparisons on just a subset. But I didn't find an easy way to do something like "run evals on 2 different LLMs on the same random subsample of size 500". 

`--sample-id` only takes a list of explicit sample ids. Am I missing something? Thanks!

Typically you do this by passing `shuffle` with a `seed` to the the dataset reading functions  (or using the `.shuffle()` method on the `Dataset` itself.

Thanks for the suggestion, @jjallaire ! To confirm, if I'm working with 70+ evals other people contributed, I'll need to edit the scripts for these evals, right?

Yes, that's correct. Phrasing it that way though makes me think we should make a global version of this available. We will do that.

This has been implemented here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2200
Repository: UKGovernmentBEIS/inspect_ai
Issue: max_connections not set when calling `get_model`
Root cause: Configuration Error

When setting certain config values such as `max_connections` when running an eval (either `inspect eval ... --max-connections 50` or `eval(...., max_connections=50)`) the value does not carry over to calls to models the user creates through `get_model`. This is confusing because in the UI it shows `max_connections: 50` at the top but then at the bottom of the screen it shows `openai: 10/10`.

I think the more intuitive behavior is that these config values set when running the eval are applied to all models, not just those created internally by inspect.

The issue here is that each model you are using might have it's own quotas / rate limits, so if we start applying that everywhere we might "break" (in terms of workload) existing evals. We show the max_connections for each model in the bottom left for clarity. 

I agree this is confusing as `max_connections` at the top seems to imply it will affect all models, but I fear changing this now might be disruptive.

Makes sense that it could mess up some current workloads. However, I still think it could be worth considering making the change in a later version (or making the current behavior more transparent). I only discovered this quirk when two seperate people working on independent projects asked me about why the max_connections for the models at the bottom didn't change when they set max_connections for the eval. I ended up having to dig through the source code to figure it out.

We have made this change here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2181 so `--max-connections` now applies to all models by default.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Doc] Note that the `EvalLog`s returned by `eval_set` are condensed
Root cause: Documentation Deficiency

I ran into a bug expecting the EvalLogs returned by `eval_set` to be complete, but I found them to be missing fields (namely `samples`).  I see from the change log that it was made to be a condensed version of EvalLogs are returned by `eval_set`: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/CHANGELOG.md?plain=1#L147

Prob worth mentioning this in the Eval Set docs though. I couldn't find it. Or better yet, making this functionality toggleable with a param.

Thanks for this. I think we will opt to update the docs as returning the full eval log can be serious footgun (as loading all of those logs into memory can cause the process to OOM depending on the number and size of evals).

Done here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2167
Repository: UKGovernmentBEIS/inspect_ai
Issue: display="plain" explicitly passed to `inspect_ai.eval()` is not honoured
Root cause: External Dependency Breakage

The progress display broke in our environment when upgrading from 0.3.82 to 0.3.105.

The intermediate cause is that our explicit passing of display="plain" to `inspect_ai.eval()` is ignored, and the display style defaults to "full" instead. 

If I set `INSPECT_DISPLAY=plain` as an environment variable, it is honoured. We are using this as a workaround for now. 

### Possible root cause?
I'm not sure what the root cause is, but it may be due to the global variable `_active_display` somehow being set before the call to `inspect_ai.eval` (perhaps during module import). If the global is set, then the value of `display_type()` is ignored here:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/2b2afe3133fff21f4613f7980602850dd3f1f4b6/src/inspect_ai/_display/core/active.py#L17-L33


There was recently a PR adding a "log" display type so it's possible something in those changes affected this behavior.

That said, running on `main` I am not able to repro with this  script:

```python
from inspect_ai import Task, eval, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import exact
from inspect_ai.solver import generate


@task
def hello_world():
    return Task(
        dataset=[
            Sample(
                input="Just reply with Hello World",
                target="Hello World",
            )
        ],
        solver=[
            generate(),
        ],
        scorer=exact(),
    )


if __name__ == "__main__":
    eval(hello_world(), model="mockllm/model", display="plain")

```

So I'd recommend:

(1) See whether you are seeing this behavior when running on `main`

(2) If you are, see if you can built up from this code to an MRE where the problem occurs, then we should be able to fairly easily resolve from there.





Will try to spend more time creating a repro today :). The reason I didn't include one immediately is because it's unusually involved for this issue: if it indeed depends on import-time behaviour, it could be caused by any part of our application that imports Inspect.

(I was only able to even discover that the actual behaviour is `display="full"` by running our production setup with `act` locally.)

I am confident this issue definitely did happen in our environment. But I haven't been able to justify spending further time to create a clean repro, given that we have a workaround. So closing for now.

Insane: I found the source of the issue. 🤯 

Importing `sagemaker.model` causes rich tracebacks to be enabled globally due to a bug in `sagemaker`. (This made me think all of Inspect's rich display is enabled, but it's really just the tracebacks).

https://github.com/aws/sagemaker-python-sdk/issues/5179
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] paired analysis between 2 models on the same benchmark
Root cause: Documentation Deficiency

Really enjoy using the package, thanks for working on it! One use case of `inspect_ai` is to compare 2 models' performance on the same benchmark. This requires paired hypothesis testing. Wondering if this is something on the backlog of inspect_ai?

My current workflow is as follows:

1. `inspect eval` for each of the candidate models
2. `inspect dump` to convert log binary to json files
3. write custom scripts to
    1. parse individual json log files to extract scores
    2. match by sample_id
    3. conduct paired hypothesis testing, optionally account for clustering

Is there's a better workflow for the purpose?

Thanks for the request - there are a couple of things worth checking out which I think might make your workflow easier:

1) You can use `read_eval_log` to read either a json or eval formatted log file to get an already parsed typed interface. More about it here https://inspect.aisi.org.uk/reference/inspect_ai.log.html#read_eval_log

2) Even better, you can read eval logs directly into data frames for analysis, which gets you much closer to where you'd like to be, I think. More here: https://inspect.aisi.org.uk/dataframe.html#overview

We're still working in the area of analysis but are generally waiting to see how people are using it and what types of analysis are common across many use cases before we implement higher level analysis directly in Inspect.
Repository: UKGovernmentBEIS/inspect_ai
Issue: VertexAI.generative_models is deprecative
Root cause: External Dependency Breakage

Noticed [this](https://pypi.org/project/google-cloud-aiplatform/):
> The following Generative AI modules in the Vertex AI SDK are deprecated as of June 24, 2025 and will be removed on June 24, 2026: vertexai.generative_models, vertexai.language_models, vertexai.vision_models, vertexai.tuning, vertexai.caching. Please use the [Google Gen AI SDK](https://pypi.org/project/google-genai/) to access these features. See [the migration guide](https://cloud.google.com/vertex-ai/generative-ai/docs/deprecations/genai-vertexai-sdk) for details. You can continue using all other Vertex AI SDK modules, as they are the recommended way to use the API.

Found because I kept getting this error:
`FailedPrecondition: 400 claude-3-5-haiku is not supported in the generateContent API.  `
when trying to use claude models on vertex.

Thanks for reporting this! We will remove the vertex provider since it's no longer supported for generative_models.

The `anthropic` provider in Inspect now natively supports Anthropic on Vertex: https://inspect.aisi.org.uk/providers.html#anthropic-on-vertex-ai

Removed vertex here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2197
Repository: UKGovernmentBEIS/inspect_ai
Issue: Are `uv` development workflows officially supported?
Root cause: Documentation Deficiency

I recently saw uv introduced in https://github.com/UKGovernmentBEIS/inspect_ai/pull/1895. Is `uv` something that is going to be officially supported going forward? Are there any potential conflicts between the `uv` lockfile and dependencies specified elsewhere?

If so, it might be worth adding some documentation to the `README.md` that `uv` can be used going forward, as well as any potential issues to be aware of when using it.

I know of several devs using `uv` to work on Inspect but I don't personally use it. Would appreciate if any active `uv` users could contribute those bits to the READEME.md as well as flag anything else we might need to do (note that our goal would be to support `uv` but not require it).

I think having a uv lockfile inside a setuptools project is confusing and we should remove it, or at least document that. Someone tried to run `uv add` and got a horrible (and confusing) error message
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Question] How to assign metric to a score defined inside solver?
Root cause: Documentation Deficiency

I follow the example from the docs (https://inspect.aisi.org.uk/solvers.html#sec-scoring-in-solvers) to have my solver also do scoring (in this case it simply records various times):

```
modified_state.scores = {
            "duration_of_generate": Score(
                value=round(total_duration, 2),
                explanation=f"Duration in seconds of the model 'generate' call.\nSeconds: {total_duration}",
            ),
            "first_token_duration": Score(
                value=round(first_token_duration, 2),
                explanation=f"Duration in seconds of the model to generate first token.\nSeconds: {first_token_duration}",
            ),
}
```
By default inspect assigns it an `accuracy` metric , which ultimately computes the same value as mean, but is not ideal for the lack of flexibility. 
How to assign metric(s) to the scores computed inside the solvers?

One way I figured could be when creating `Task` using `metrics` parameter, but I have a list of scorers with multiple metrics and I need to assign metrics only to those scores that were defined inside the solver...
Repository: UKGovernmentBEIS/inspect_ai
Issue: `multiple_choice` solver marks "ANSWER: None of the above" as Options A, B
Root cause: Algorithmic Error

`set_choices_based_on_generated_response()` calls `answer_index()` on every character in the extracted string, treating individual letters from words as choice selections:

Three examples demonstrated by the failing tests in this [branch](https://github.com/UKGovernmentBEIS/inspect_ai/pull/2217):

* "ANSWER: None of the **ab**ove" → marks choices A and B as selected
* "ANSWER: **D**on't know" → marks choice D as selected → scores as CORRECT if D was the correct answer. 
* "ANSWER: **I** **d**on't know" → marks D as selected, and I if it exists in the options


Thanks to Jamie Elsey for calling this to my attention!

Here's a proposed fix, what do you think? https://github.com/UKGovernmentBEIS/inspect_ai/pull/2217

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/2217
Repository: UKGovernmentBEIS/inspect_ai
Issue: File extension typo — `.pyy` instead of `.py`
Root cause: Validation Gap

Thank you for this wonderful tool. While using it, I came across what appears to be a typo: it's displaying `.pyy` where `.py` should be.


https://github.com/UKGovernmentBEIS/inspect_ai/blob/9dacf0cfebb1a0dbc7b798ba98b06aadbf47320c/src/inspect_ai/_eval/loader.py#L491

https://github.com/UKGovernmentBEIS/inspect_ai/blob/9dacf0cfebb1a0dbc7b798ba98b06aadbf47320c/src/inspect_ai/_eval/loader.py#L495

<img width="2908" height="1680" alt="Image" src="https://github.com/user-attachments/assets/c2cf87da-c4ee-4cd4-bde6-a23fe7c1098d" />

Thanks! Fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/4cb007f10f1dcbc308baa0d6a3eb5f69d56ce2f6
Repository: UKGovernmentBEIS/inspect_ai
Issue: `reasoning_effort` seems to be broken in OpenRouter provider
Root cause: Interface Contract Mismatch

I'm a little confused by this one. When I try to use `reasoning_effort` with a model on OpenRouter, I get a crash:

> TypeError: AsyncCompletions.create() got an unexpected keyword argument 'reasoning'

This appears to be caused by `"reasoning_effort": _` being remapped to `"reasoning": {"effort": _}"` in the OpenRouter provider:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/e02d843cbc817a0d00a72da1e5ceaee7c69ee34d/src/inspect_ai/model/_providers/openrouter.py#L116-L128

But the openai completions API does not support this `reasoning` parameter:

https://github.com/openai/openai-python/blob/a8258744cbecf51321587fc870e8920bd2c07809/src/openai/resources/chat/completions/completions.py#L1657-L1698

I'm confused because it seems like this remapping code should have never worked. Maybe it worked against a previous version of the `openai` library? I'm using version `1.99.1`, though I also had the same crash earlier today on `1.88.0`. 

Thanks for reporting this! It looks like those should have gone into "extra_body". This is fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2227
Repository: UKGovernmentBEIS/inspect_ai
Issue: Impossible to use `basic_agent` without a message limit?
Root cause: Validation Gap

It seems like `basic_agent` can't be used with just a `token_limit` and no `message_limit`? 

https://github.com/UKGovernmentBEIS/inspect_ai/blob/2b2afe3133fff21f4613f7980602850dd3f1f4b6/src/inspect_ai/solver/_basic_agent.py#L171-L173 

Since `None` is Falsy, this will always fall back to 50. 

Thanks for reporting this! It does seem like we should both:

a) Respect `message_limit=0` as a way to specify no limit
b) Only do the "backstop" defaulting to 50 when there both no `message_limit` and `token_limit` defined.

Both of these are done here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2245
Repository: UKGovernmentBEIS/inspect_ai
Issue: VSCode Extension Not Loading, Inspect Bin Path Not Found
Root cause: Environment Incompatibility

Hi,

Thanks for the great work on making eval convenient!
 
I'm using uv + VSCode and just installed the editable version of inspect by `uv pip install -e ".[dev]"` in my uv environment.
However, my Inspect VSCode extension is not loading into the workspace. 
I tried to follow the [troubleshooting](https://inspect.aisi.org.uk/vscode.html) page and found that it has found my python interpreter path but not Inspect path.
I can still use `inspect view` to open up a browser window to check the outputs, but it feels a bit sad to be not able to use the extension in VSCode:(
Is there any way to troubleshoot the issue?

<img width="774" height="129" alt="Image" src="https://github.com/user-attachments/assets/7da74947-de3c-498b-841c-b54f3495426a" />

Ah sorry for the stupid mistake. After I restart VSCode the bin path loads.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer crashes with large metadata / large number of sandbox events
Root cause: Resource Mishandling

I have an eval with 6700 sandbox events, and where the `Score` metadata is a list of 6700 items. 

The log viewer is extremely slow, and clicking the "scoring" tab crashes the log viewer. 

I can share a file to reproduce privately. 

Thanks for reporting this - could you DM me (slack or email is fine) a file to repro? We should be virtualizing both events and metadata, so something must be slipping through the cracks or not working as intended. Thx!
Repository: UKGovernmentBEIS/inspect_ai
Issue: ImportError: cannot import name 'ChatCompletionMessageToolCall' from 'openai.types.chat'
Root cause: External Dependency Breakage

It appears that newer versions of OpenAI SDK (1.99.x) do not have the `ChatCompletionMessageToolCall` type anymore

Thanks for reporting this!

This should be fixed here https://github.com/UKGovernmentBEIS/inspect_ai/pull/2239 in version 0.3.120 of Inspect. 

Can you take a look at your Inspect version to confirm you're on the latest version?

(Possible related: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2257)

I think I am also encountering this error despite being on Inspect version 0.3.112 

This should fix it: 
https://github.com/UKGovernmentBEIS/inspect_ai/pull/2257

There were a few changes in this area last week on Thursday, and then again this afternoon (Aug 11) in response to open ai releases. This version hasn't yet made it to PyPi, you may install from main using 

```
pip install git+https://github.com/UKGovernmentBEIS/inspect_ai
```

To get the fix ahead of our release to PyPi.



Great, this worked for me! Thanks very much

Great news! I'm going to go ahead and close this.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Finegrained control over progress bar

I have an eval that is much longer than it is wide (as in it is a very long multi turn conversation and I don't have that many samples to run in parallel). Because of this, this current progress bar will show almost no progress for most of the eval, and jump to the end all at once when all of the samples finish around the same time. It would be a great feature to be able to specify how far through a specific solver or sample we are.

I could see this looking something like this:

```python
from inspect_ai.util import progress

@solver
def solver():
    async def solve():
        progress_bar = progress()
        progress_bar.update(0.5)  # set the progress to a specific fraction done
```
or
```python
from inspect_ai.util import progress

@solver
def solver():
    progress_bar = progress(total=10)  # tells inspect what the total is for this solver before the eval starts (this doesn't work if each sample has a different total)
    async def solve():
        progress_bar.update(5) # alternatively set progress to a number done out of a total for the solver
```

Reach all the way from the "frond end" back into the solvers for progress would be pretty tricky (not impossible, but not trivial!). This is a good suggestion but I don't know if we'll be able to work on it in the short term. cc @dragonstyle