zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Run an infinite loop in the Docker sandbox, it will not terminate
Root cause: Resource Mishandling

If we run an infinite loop in a Docker sandbox environment, the process will not terminate after a timeout.

```python
from inspect_ai import Task, task
from inspect_ai.dataset import MemoryDataset, Sample
from inspect_ai.scorer import (
    CORRECT,
    INCORRECT,
    Score,
    Scorer,
    Target,
    accuracy,
    scorer,
)
from inspect_ai.solver import TaskState
from inspect_ai.util import sandbox
from inspect_ai import eval as ins_eval


@task(name="always_infinite_loop")
def always_infinite_loop() -> Task:

    d = MemoryDataset(samples=[
        Sample(input="", target="", id=1),
        Sample(input="", target="", id=2)
    ])
    return Task(dataset=d,
                epochs=1,
                sandbox="docker",
                time_limit=10,
                scorer=verify())


@scorer(metrics=[accuracy()])
def verify() -> Scorer:

    async def score(state: TaskState, target: Target) -> Score:
        code = "while True: pass"
        explanation = ""

        try:
            result = await sandbox().exec(cmd=["python", "-c", code],
                                          timeout=1)
        except TimeoutError:
            return Score(
                value=INCORRECT,
                answer=code,
                explanation="Code execution timed out after 1 second",
            )
        except Exception as e:
            return Score(
                value=INCORRECT,
                answer=code,
                explanation=f"Error: {e}",
            )

        if result.success:
            explanation += "All test cases passed.\n"
        else:
            explanation += "Code did not pass all test cases.\n"
        return Score(
            value=CORRECT if result.success else INCORRECT,
            answer=code,
            explanation=explanation,
        )

    return score


ins_eval(
    always_infinite_loop(),
    model="openai/gpt-4o-mini",
)
```

After running this Python file, it never terminates on this screen:

<img width="1968" alt="Image" src="https://github.com/user-attachments/assets/21d0cd40-527b-4657-a0b4-003e83be420f" />

The `python -c <code>` process in Docker is also not terminated.

```bash
root           1  0.0  0.0   1136     4 ?        Ss   05:39   0:00 /sbin/docker-init -- tail -f /dev/null
root           7  0.0  0.0   2940   880 ?        S    05:39   0:00 tail -f /dev/null
root          14 99.9  0.0  13208  8292 ?        Rs   05:39   1:22 python -c while True: pass
root          20  0.1  0.0   4612  3736 pts/0    Ss   05:40   0:00 bash -l
root          58  0.0  0.0   8540  4368 pts/0    R+   05:40   0:00 ps -aux
```

Thanks for reporting this! Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1587
Repository: UKGovernmentBEIS/inspect_ai
Issue: bug: commas in type definition stump task args
Root cause: Algorithmic Error

Example: `mode: Literal["honest", "attack"] = "attack",` from https://github.com/UKGovernmentBEIS/control-arena/blob/main/control_arena/settings/infra/infra_task.py

I assume the plugin parses to the comma, expects that to be the end of the line, and then thinks the next argument is `"attack"]`

VSCode:

![Image](https://github.com/user-attachments/assets/f7c764c9-638d-4925-86a4-a6cc99993760)

Command when trying to run:
```shell
inspect eval control_arena/settings/infra/infra_task.py@infra --limit 1 -T protocol=simple -T mode=attack -T "attack"]=test
```

Thanks for reporting this - I'll take a look today!

My recommendation would be to use `control_arena.protocols.all_protocols.ProtocolMode`, it's defined anyway.
Repository: UKGovernmentBEIS/inspect_ai
Issue: expert answer not showing in metadata in view
Root cause: Validation Gap

I tried the basic hello world example and in the metadata tab for all the rows, it is missing the expert's answer `[EXPERT]: {criterion}` in between `[Question]` and `[Submission]`:

<img width="1375" alt="image" src="https://github.com/UKGovernmentBEIS/inspect_ai/assets/25344832/302529c9-9db5-4dea-bbf7-72013d7d8bcd">

The assistant'a output refers to expert answer, which means it must be a bug in how the string is being displayed, rather than bug in evaluation itself.



Thanks for reporting! We can repro this locally and are investigating (it's likely a vagary of the markdown renderer we are using)

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/commit/140c3d63ad42125213bffaa799147135c81857f3
Repository: UKGovernmentBEIS/inspect_ai
Issue: VS Code Inspect extension can't find my logs

VS Code Inspect extension cannot find the logs. There are logs in the "logs" folder. I've selected that folder as the one to load, but the inspect extension just loads without ever identifying any log file. 
Don't know why, I've tried to reinstall the extension. 

Is this a local or remote log directory? Are there any log directories that it does work?

There are a few output channels that could contain helpful clues. Check the `output` panel at the bottom (one of the tabs in the group where the terminal normally is) and check the Inspect channel for anything that looks amiss. If there isn't anything interesting there, I may need to add additional logging to help use understand what is going on.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Possible to export eval logs?
Root cause: Unimplemented Feature Gap

Is there a way to export the eval results to a csv or similar format? Seeing the logs in the inspect view is helpful, but it would be great if I could export the results to a csv file, including both the raw model output for each task and the graded output. 

You can convert either a single log or a directory of logs to JSON with:

```bash
inspect log convert <log-or-log-dir> --to json --output-dir json-logs
```

The log file format is quite hierarchical so isn't trivially convertible into a CSV. That said, most people do reshape the parts of the log they care about into a data frame for the purposes of analysis and visualization. There isn't yet a "standard" way to do this, but this is something we'll be working on in the next few months.

Note that most people don't convert into JSON and then into CSV, they go directly from the strongly typed Python interface  (available via `read_eval_log()`) into the data frame (reading the JSON raw leaves you open to all sorts of errors).
Repository: UKGovernmentBEIS/inspect_ai
Issue: Allow tool choice = 'none' for anthropic API
Root cause: Unimplemented Feature Gap

This is [recently supported](https://docs.anthropic.com/en/release-notes/api#february-27th-2025) by the anthropic API and is sometimes helpful (e.g. giving a model grader context on tools but not allowing any tools to be chosen).

Thanks for pointing this out! Done here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1623/files
Repository: UKGovernmentBEIS/inspect_ai
Issue: `scorer.match(numeric=True)` does not handle `XX%` format
Root cause: Validation Gap

The `match(numeric=True)` scorer mishandles the answer `20%`:
<img width="1713" alt="Image" src="https://github.com/user-attachments/assets/6ffbf505-e426-447d-bdb5-b40e9ebc9491" />

I noticed that `strip_numeric_punctuation` doesn't handle `%`, only currency prefixes and decimal points:
https://github.com/UKGovernmentBEIS/inspect_ai/blob/831b4656b3ff506db6dfb879fcbcbbb41a9593df/src/inspect_ai/_util/text.py#L14-L20

This causes `match_str` to skip `20%` and return the last parseable number, which is `100`:
```python
>>> from inspect_ai._util.text import strip_numeric_punctuation
>>> strip_numeric_punctuation("20%")
'20%'
>>> "20%".replace(".", "").isnumeric()
False
>>> from inspect_ai.scorer._common import first_number_normalized
>>> first_number_normalized(["20%", "100"])
'100'
```

This gap isn't made clear in the documentation: https://inspect.aisi.org.uk/reference/inspect_ai.scorer.html#match
So I'd request changing either the documentation or the behaviour to accomodate `%`.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Docs: Tools Custom refers to both add() and addition() functions

The [Tools Custom page](https://inspect.aisi.org.uk/tools-custom.html) refers sometimes to the add() function and others to the addition() function. Please standardize to use one name or the other.

Thanks for pointing this out! Fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/cbcdc7f858ae45518a9a4b9a6b1e4983a90a6362
Repository: UKGovernmentBEIS/inspect_ai
Issue: Using Eval Set with Multiple Self-Hosted LLM Servers
Root cause: Documentation Deficiency

Thank you for this awesome tool! 

I have two (and soon to be more) OpenAI API compatible LLM servers hosting different models. The servers are dockerized and running on the same host but behind different ports:
- meta/llama-3.2-90b-vision-instruct 
    - `OPENAI_BASE_URL = http://llm-server-host:8002/v1`
- google/gemma-3-27b-it 
    - `OPENAI_BASE_URL = http://llm-server-host:8000/v1`

I have been using the [`openai` provider to interact with one model at a time](https://inspect.aisi.org.uk/providers.html#vllm-server).

So far using `eval_set` works great with just one self-hosted LLM (meta/llama-3.2-90b-vision-instruct) and a few cloud services:
``` python
from inspect_ai import eval_set

eval_set(
   tasks=["MCQ-With-TOP_v2.py"],
   model=[
      "openai/meta/llama-3.2-90b-vision-instruct",     # self hosted
      "google/gemini-2.0-flash",                       # cloud service
      "anthropic/claude-3-7-sonnet-latest"             # cloud service
   ],
   log_dir="logs-compare-llms",
   log_level="http",
   log_format="json"
)
```

But I want to be able to use `eval_set` with more than one self-hosted LLM:
``` python
from inspect_ai import eval_set

eval_set(
   tasks=["MCQ-With-TOP_v2.py"],
   model=[
      "openai/meta/llama-3.2-90b-vision-instruct",      # self hosted
      "openai/google/gemma-3-27b-it ",                  # self hosted
      "google/gemini-2.0-flash",                        # cloud service
      "anthropic/claude-3-7-sonnet-latest"              # cloud service
   ],
   log_dir="logs-compare-llms",
   log_level="http",
   log_format="json"
)
```

But this would seem to require having two different `OPENAI_BASE_URL` values for the two self-hosted models on their separate ports. Is there a way to accomplish this with Inspect AI's `eval_set`?  

The way to do this is to directly instantiate model objects (using `get_model()`, which takes both a `base_url` and `api_key`) then pass those in the `model` argument rather than strings.

That said, within a week or two we are going to introduce a new `openai-api` provider that will let you do this with namespaced environment variables.


@jjallaire thank you so much! This worked perfectly:

``` python
from inspect_ai import eval_set
from inspect_ai.model import get_model


self_hosted_gemma3_27b = get_model(
   model="openai/google/gemma-3-27b-it",
   base_url="http://llm-server-host:8000/v1",
   api_key="dummy_value"
)

self_hosted_llama3_2_90b = get_model(
   model="openai/meta/llama-3.2-90b-vision-instruct",
   base_url="llm-server-host:8002/v1",
   api_key="dummy_value"
)

eval_set(
   tasks=["theory.py"],
   model=[
      self_hosted_gemma3_27b,
      self_hosted_llama3_2_90b,
      "google/gemini-1.5-pro",
      "google/gemini-2.0-flash",
      "anthropic/claude-3-7-sonnet-latest"
   ],
   log_dir="logs-compare-llms",
   log_level="http",
   log_format="json"
)
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: model_graded_qa() scorer does not support multimodal inputs
Root cause: Unimplemented Feature Gap

The `model_graded_qa()` scorer uses `state.output.completion`, which will only be the text content: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/scorer/_model.py#L171

One way to solve this would be to set `answer` to something like "See attached image/video/audio file." in the prompt template, then construct a `ChatMessageUser` that includes the prompt text and the other inputs.

In the meantime, I've written a custom scorer that is basically doing:
```
local_image_path = state.output.choices[0].message.content[0].image  # or something like that
model = get_model("openai/google/gemma-3-4b-it")  # works with the vLLM provider
grade_message = ChatMessageUser(
    content=[
        ContentText(text=grader_prompt),
        ContentImage(image=local_image_path),
    ]
)
response = await model.generate([grade_message])
```

If the `model_graded_qa()` handled this behaviour internally, I could just use it off-the-shelf, or use a custom template if relevant.

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1717
Repository: UKGovernmentBEIS/inspect_ai
Issue: Retrying an evaluation with Azure OpenAI fails.
Root cause: Configuration Error

Retrying an evaluation with an Azure OpenAI model currently does not work for me.

### Steps to reproduce

#### 1. Prepare your `.env`

```
AZUREAI_OPENAI_API_KEY=...
AZUREAI_OPENAI_BASE_URL=...
INSPECT_EVAL_MODEL=openai/azure/...
```

#### 2. Run the example

I ran `inspect eval example.py` on https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/examples/hello_world.py which worked.

#### 3. Attempt retry

I tried `inspect eval-retry logs/...` on the just-produced log file, which yields

```console
❯ inspect eval-retry logs/2025-04-14T16-16-29+02-00_hello-world_hvyb2Gg7tRYJKT8i8rXKUW.eval  

ERROR: Unable to initialise OpenAI client

No OPENAI_API_KEY or AZUREAI_OPENAI_API_KEY defined in the environment.
```

I believe the error happens because the model stored in the log file in the info tab reads `openai/...` instead of `openai/azure/...`. Thus, [this line](https://github.com/UKGovernmentBEIS/inspect_ai/blob/865f17926281cc785a7c26b486938cc8533fb2e8/src/inspect_ai/model/_providers/openai.py#L93) evaluates to false. Replacing the condition with `if True`, seems to fix the issue.



That model should still retain the openai/azure/ prefix in the log (check the log JSON under the `eval` key at the top level). While it might not show it in the Info tab it should be correct in the log (if its not then that is the bug and we have to figure out how).

Unfortunately I'm not longer able to provision quota for OpenAI models on Azure (they must be having capacity issues and my account isn't high tier enough). @dragonstyle Do you have an Azure setup we could use to debug/test?

Okay, we have now reproduced this! Thank you for reporting, we will have a fix soon!


This is resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1669
Repository: UKGovernmentBEIS/inspect_ai
Issue: Gemini 2.5 raises RuntimeError when Gemini returns no candidates or prompt feedback
Root cause: Validation Gap

### The bug

The `GoogleGenAIAPI.generate()` method raises a `RuntimeError` when the Gemini API returns a response with no `candidates` and no `prompt_feedback`.

This can happen when running a benchmark like `gaia`, where each `web_browser_go` call is fetching a GitHub issue list page with filters. That page consistently returns “No results,” then:
- There’s no content for Gemini to generate from.
- If the config uses `temperature: 0.0` and instructs the model to only return very specific formats, Gemini will just return nothing.

Because of that, the runtime hits this error:
`RuntimeError: Google response includes no completion candidates and no block reason`

I've attached the relevant logs below.
[2025-04-15T13-19-27-07-00_gaia_VxibdYtXyDamLZBmBH7Gvc.json](https://github.com/user-attachments/files/19766384/2025-04-15T13-19-27-07-00_gaia_VxibdYtXyDamLZBmBH7Gvc.json)

### Suggested fix

Update `completion_choices_from_candidates()` in `GoogleGenAIAPI` to handle the case where **both** `candidates` and `prompt_feedback` are `None`, and return a fallback like:

```python
ChatCompletionChoice(
    message=ChatMessageAssistant(
        content="No output was generated.",
        model=model,
        source="generate",
    ),
    stop_reason="no_output", //or something else, couldn't find any list of stop reasons in inspect.
)

Thanks for reporting this! Implemented your suggested fix here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1676
Repository: UKGovernmentBEIS/inspect_ai
Issue: Unbound variable on tool timeout
Root cause: Validation Gap

In _call_tools.py, the `results_exception` variable is set within 2 nested `async with` blocks

https://github.com/UKGovernmentBEIS/inspect_ai/blob/e7af455c471c413f978b1df36e4ce213b9d2683f/src/inspect_ai/model/_call_tools.py#L267-L274

Later, outside the `async with` blocks, the variable is referenced

https://github.com/UKGovernmentBEIS/inspect_ai/blob/e7af455c471c413f978b1df36e4ce213b9d2683f/src/inspect_ai/model/_call_tools.py#L317

However, if the initial `async with` blocks fails (e.g, it is canceled), the `results_exception` variable can be unset and lead to `UnboundLocalError: cannot access local variable 'result_exception' where it is not associated with a value`.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Metadata is no longer passed to bridged agent.
Root cause: Interface Contract Mismatch

My team runs tasks that require sample-specific metadata in our bridged agents.

Until `v0.3.82` inside `inspect_ai.solver.bridge` the metadata was passed from `TaskState` to `BridgeSample` and then model_dumped into the agent. Since `v0.3.83` only the messages are passed from the `TaskState` to the `AgentState`, so subsequent solvers don't get the metadata.

We can work around this by adding a solver before the agent solver that stores the metadata to the `store()`. Is that the intended way to do this or is there a better way?



Subsequent solvers not getting the metadata is definitely NOT the intended behavior -- that we will fix ASAP.

Not passing the metadata to the bridged agent is indeed on purpose. This is because bridged agents are now using the `Agent` protocol which purposely has a more narrow interface (to reduce coupling to the task so they can be used in more contexts).

For normal agents, the typical way to address this would be to explicitly use the store to forward/read sample metadata. However, this isn't really a good solution for bridged agents (as it requires that they directly access the store, not the end of the world but it kind of breaks the abstraction). Let me consider whether we should (a) Add `metadata` to the `Agent` interface proper or (b) Add `metadata` specifically to bridged agents if they are running in a sample.

This is resolved here for bridged agents: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1689. This is on `main` now and will go to PyPI later today or tomorrow.

Still considering whether to add `metadata` to `AgentState` (as there are other ways to pass sideband data to agents e.g. the store)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Display metadata as column in `inspect view` table
Root cause: Unimplemented Feature Gap

Is there an easy way to modify the `inspect view` interface's main table to show a specific metadata value for each sample? 

Each of my dataset's questions has a "difficulty" metadata tag (easy, medium, or hard) and I would love to be able to show the difficulty of each question in the main table of `inspect view`. But im not sure that this interface was designed to provide this flexibility? If not, do you have any other suggestions on how to accomplish this (i.e. a way to represent the difficulty of each sample at a glance)?

Thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: GCS support noted in docs but file keys hardcoded for S3 causing inability to launch inspect view
Root cause: Interface Contract Mismatch

I am attempting to write logs remotely to a google cloud storage instance with service account credentials. Ultimately the write happens as expected but when attempting to retrieve the logs using `inspect view` or `inspect list logs` the following error is emitted:

`File "/Users/<username>/<repo_name>/.venv/lib/python3.10/site-packages/inspect_ai/_util/file.py", line 275, in _file_info
    file["mtime"] = file["mtime"] * 1000
TypeError: unsupported operand type(s) for *: 'datetime.datetime' and 'int'` 

GCS file returns with the following keys:

`dict_keys(['kind', 'id', 'selfLink', 'mediaLink', 'name', 'bucket', 'generation', 'metageneration', 'contentType', 'storageClass', 'size', 'md5Hash', 'crc32c', 'etag', 'temporaryHold', 'eventBasedHold', 'timeCreated', 'updated', 'timeStorageClassUpdated', 'timeFinalized', 'type', 'mtime', 'ctime'])`

As you can see the keys from GCS already have `mtime` which invalidates the logic on line 265 of `inspect_ai/_util/file.py` which read as follows:

```
        if "LastModified" in file.keys():
            file["mtime"] = cast(
                datetime.datetime, cast(Any, file)["LastModified"]
            ).timestamp()
        # if we don't yet have an mtime key then fetch created explicitly
        # note: S3 doesn't give you a directory modification time
        if "mtime" not in file.keys() and file["type"] == "file":
            file["mtime"] = self.fs.created(file).timestamp()
        if "mtime" in file.keys():
            file["mtime"] = file["mtime"] * 1000
        else:
            file["mtime"] = None
```

Because GCS contains `mtime` we do not convert to a timestamp and therefore experience the error explained above.
when `mtime` is returned from GCS it is of type `datetime.datetime` and has an example value of `2025-04-16 12:08:13.630000+00:00`

Is this a known issue that is being resolved or can a PR be raised quickly as this seems to exclude developers who do not use S3 buckets and by building ontop of fsspec it would be presumed that inspect can be connected to any remote file store such as GCS or Azure Datalake.

Inspect has been a great tool to use and I look forward to a prompt response!

It's not a known issue but thank you very much for raising it! This is part of fsspec which is not fully specified or portable so we will need some workarounds like this. The quickest way to a resolution is if you can raise a PR (as you have the GCS config and the repro). If that's not practical we will have a look at a repro and fix tomorrow (Friday).

@jjallaire I will raise a PR by the afternoon EST time

Wanted to repro the fix locally anyway + check that it worked with our VS Code extension so went ahead and did this here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1708

@jjallaire when will this be released?
Repository: UKGovernmentBEIS/inspect_ai
Issue: Question: is anyone integrating optimization (PromptWizard, etc) into Inspect?

After running evals (which Inspect makes so easy!), I want to optimize my prompts. I'm experimenting with optimizers, but PromptWizard from Microsoft is a good example of the sort of automated optimizer I'm considering. Has anyone done work towards integrating optimization as a first class citizen of Eval somehow? Or is the expected workflow to Eval, manually optimizer (or optimize externally), and then rerun evals?

You could certainly use intermediate scoring (https://inspect.aisi.org.uk/solvers.html#intermediate-scoring) in a solver to run a `PromptWizard` based optimization loop. Optimizing externally and using Inspect to test and validate is also a good tact.