zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Add 'function' attribute to ChatMessageTool

When looping over messages, it does not seem trivial to find all the ChatMessages for a specific tool.
This can be solved by adding the 'function' attribute to the ChatMessageTool class.

Great idea, thanks for this! Implemented in this PR: https://github.com/UKGovernmentBEIS/inspect_ai/pull/269

thank you!
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Add hydra integration to manage inspect configs
Root cause: Unimplemented Feature Gap

Currently inspect accpet yaml config file, however it would be even better if we can pass a hierarchical, parametrizable configs using `hydra-core` package https://github.com/facebookresearch/hydra/.

For example, this could allow stuff like having a default config, and instead of creating a copy of the file and change the kwargs, you could just inherit from it and make the changes you want
Repository: UKGovernmentBEIS/inspect_ai
Issue: In the Inspect vscode UI, is there a way to specify model_args
Root cause: Unimplemented Feature Gap

Hi, I'm wondering, is there a way to run the debugger while setting custom `model_args` (`-M` in the CLI)?

My understanding is the "debug task" button will draw upon configurations set in the "Inspect" pane in vscode. However, there does not appear to be a place to enter custom `model_args`. What would be a workaround for this?


<img width="319" height="210" alt="Image" src="https://github.com/user-attachments/assets/e2bd9260-d419-4b5d-a0b4-22a8038f9d2e" />


<img width="519" height="706" alt="Image" src="https://github.com/user-attachments/assets/e73ae5fe-a9eb-4e57-8bf3-ad08aef8c3e8" />


Thanks for your help!

I don't think we have this right now, but you could also set the `INSPECT_EVAL_MODEL_ARGS` environment variable (which can take either a YAML or JSON file or a YAML object literal)


Sounds good, thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Compatibility with emerging environment hubs

Have you thought about how Inspect might cooperate with related efforts from Huggingface/Meta and Prime Intellect? Their primary focus may be training, but it doesn't take too much imagination to see how evaluation ties in:
- https://www.primeintellect.ai/blog/environments
- https://huggingface.co/blog/openenv

Hi @jbragg , we have been tracking these but don't yet have plans to do any integration. Would be keen to hear your thoughts on what might make sense.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer: add download option when "JSON too large to display" for a sample
Root cause: Unimplemented Feature Gap

The log viewer, opened at a Sample, opened on the JSON tab, sometimes shows "JSON too large to display". Suggestion: add a button to download the JSON as a file in that case.

This would be a small quality of life improvement. Of course, one can access the data through the `EvalLog` object, but in some cases it is more convenient to just look at the JSON for the currently open sample. 

I'm going to close this issue in favor of:
https://github.com/UKGovernmentBEIS/inspect_ai/issues/2927

which when complete will make the download available whether or not the JSON is too large.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add application header to openrouter queries?

Openrouter allow passing [application headers](https://openrouter.ai/docs/app-attribution) to attribute requests to an app. Inspect could pass a special header, which would then collect how many tokens are used through inspect and to which model.

That could be a cheap way to measure usage, unsure if it's something that would be interesting
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add ability to download .eval file (and/or JSON?) from viewer
Root cause: Unimplemented Feature Gap

~~- [ ] Logs page (select many/all files)~~
- [x] Samples page (downloads entire .eval file)
~~- [ ] Sample page (downloads individual sample JSON?)~~

> edit: Remove `Sample page` TODO in favor of #2927 
> edit: Remove `Logs page` TODO in favor of #2928
Repository: UKGovernmentBEIS/inspect_ai
Issue: How to evaluate OpenAI Realtime API in Inspect AI?
Root cause: Documentation Deficiency

## Background
Inspect AI currently uses a ChatMessage + HTTP request-response pattern, while the OpenAI Realtime API relies on WebSocket connections. I'd like to evaluate realtime models (e.g., gpt-realtime) within Inspect.
## Goal
* **Input**: text or audio files (.wav)
* **Evaluation**: model response quality
* **Objective**: minimal viable integration for realtime models
## Current considerations
1. **Agent Bridge** – integrate an external RealtimeAgent
2. **Model Provider Adapter** – implement a custom ModelAPI that wraps WebSocket connections
## Questions
1. Can I create a custom Solver or ModelAPI for Realtime models using Inspect's existing APIs?
2. Which approach (Agent Bridge vs Model Provider Adapter) would better fit Inspect's roadmap?
---
I recently started exploring the Inspect AI framework and I'm very glad this project exists — it's been extremely helpful to learn from. Thanks to the team for maintaining this clear and practical open-source framework.

You should currently be able to include audio in generation requests. If that is all that is required for this scenario then it might work now? If you actually want realtime (like someone is speaking live in the eval) then I would write that as a either a custom model provider or just a solver or agent that does direct communication.

Thanks for your guidance! I've implemented gpt-realtime as a custom solver. My use case is batch evaluation rather than live interaction, so I:
- Wrapped the WebSocket communication in a solver
- Converted realtime interactions to ChatMessages for logging and visualization in Inspect View
- Processed pre-recorded audio files (.wav) as inputs

Does this approach seem appropriate, or would you recommend any improvements?
Repository: UKGovernmentBEIS/inspect_ai
Issue: eval fails silently when there is a mismatch between model param and environment variable
Root cause: Validation Gap

# Summary 

Calling `eval` with a model parameter that conflicts with environment variables results in the evaluation executing using the environment variable. This means that an accidental mismatch can be hard to notice without careful inspection of the log files. It would be better if this situation resulted in an exception (or at least a warning message) to avoid accidentally using the incorrect model.

For example, calling `eval` with `model=openai/azure/o3` and the environment variable `AZUREAI_OPENAI_BASE_URL="https://.../o4-mini..."` results in `o3` being recorded in the log file as the model used. However, `o4-mini` is the actual model used for the evaluation and recorded in the messages section of the log file, but not the model name.

# Suggested fix location

I believe that the model name is resolved [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_model.py#L1173) and some checks on the model argument validity are done [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/model/_model.py#L998); it would be helpful to add some checks to pick up accidental discrepancies between the model and URL arguments.



I have updated the pull request for this issue. It should be fixed.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Memory ISSUES: Inspect Score causing problems
Root cause: Resource Mishandling

Hi there, I have been using `inspect-ai` with a custom *scorer* which uses a model, other than the one being evaluated, as a judge to calculate the scores. Since both models (the one being evaluated and the one used by my custom *scorer*) are loaded from disk (I am testing a model my organization is building) I have been using 2 GPUs from our cluster because one doesn't have enough memory to hold both models. This is annoying because when we look at the usage of the GPUs in the long run it is about 20% each, since the pipe line goes like this:
```
for-all entry in dataset:
    do generation-with-original-model-(GPU-1)      # GPU-2 idle
    do inference-for-score-calculation-(GPU-2)     # GPU-1 idle
```

In my search for a solution I stumbled into [this](https://inspect.aisi.org.uk/scorers.html#sec-scorer-workflow), that basically explains that one can run the `inspect eval` with `--no-score` option followed by `inspect score`, which would run the **solvers** to the end and then run the **scorers**. I thought that this would most certainly solve my problem,  since the model being evaluated would be loaded first to run the **solvers** and then the second model would be loaded to run my custom **scorer**, and since no generation is necessary from the original model during the scoring phase, certainly the model being evaluated would not be loaded when I run `inspect score`. But it turns out I was very very wrong. It turns out that for whatever reason when you run `inspect score` it loads the model that was evaluated. I went into a rabbit hole with the help of the debugger and I discovered that it happens [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/9cdd59122a99683236f33f6653fee6a08cf78170/src/inspect_ai/_eval/score.py#L330).

I think that there must be a reason why you do this and if it seems to be reasonable for you, I would like to propose that you add some sort of option that allows one to disable this behavior (the original model being loaded). For me that would mean I could use a single GPU to do the evaluation and the scoring while maintaining a good utilization of my organization's resources with the bonus of keeping my co-workers shut because I will no longer be using all the GPUs to run the evaluations.

**Edit:** I am willing to do that myself as long as you let me know how you want  it to be done.

Thanks for digging into this - let me take a closer look to make sure I understand the ways that we depend upon that model being initialized in the task context. I'll revert here!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Agent Bridge doesn't handle plain string output from Pydantic AI agent using Responses API
Root cause: Validation Gap

Hello there, I'm running into an issue when using pydantic-ai agents using the OpenAI Responses API with the agent bridge. Currently, I've monkey patched this, but maybe you want to fix it.

By default, [pydantic-ai agents return plain strings](https://github.com/pydantic/pydantic-ai/blob/main/pydantic_ai_slim/pydantic_ai/agent/__init__.py#L100). When the agent bridge processes the output (`responses_output_items_from_assistant_message`), it assumes `message.content` is always a `list[Content]` and loops through it. When `message.content` is a plain string, it iterates over individual characters and basically skips them. `ChatMessageAssistant.content` is typed as `str | list[Content]`, so strings are valid. But `responses_output_items_from_assistant_message` only handles the list case, so it fails.


Thanks for letting us know! Should we resolve here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2743/commits/e721a49f1201b4f7a18d0201d9e4657bc04b66b1

Thank you @jjallaire!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add default value for message_limit
Root cause: Configuration Error

Right now, the default `message_limit` is set to `None`, which means that certain models like that are likely to refuse (like Opus 4.1) can end up in an infinite loop and burn lots of API credits if one is not monitoring the eval closely. I propose that we set some value that's not none, even if it's something really high like 10,000, so others do not fall prey to this failure mode and use excessive credits unwittingly. 

https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/util/_limit.py#L281
Repository: UKGovernmentBEIS/inspect_ai
Issue: bash_session tool hangs in local sandbox environment
Root cause: Resource Mishandling

I am encountering an issue when running the bash_session tool in a local sandbox environment. The tool hangs and doesn't complete, whether I run the task as a root user or as a non-root user. Even pressing Ctrl+C doesn't stop the process.

Environment:

OS: Ubuntu 20.04.5 LTS

Python version: 3.10.19

Inspect version: 0.3.144

Question: Any ideas on what might be causing this, and how to resolve it?

Thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Create a CONTRIBUTING guide to welcome and orient new people

A CONTRIBUTING guide could welcome new people to the project, help them to get oriented, learn how the process works, and be more confident in their ability to contribute.

Current state: when I view the [Issues](https://github.com/UKGovernmentBEIS/inspect_ai/issues) tab here on GitHub, I see a link to contributing guidelines that takes me to https://opensource.guide/. This is better than nothing, but it's generic and doesn't welcome contributors or explain specific expectations for Inspect.

Ideally, Inspect would have a CONTRIBUTING.md file that answers questions new contributors might have, such as:

1. What's a good first issue to start working on? (I'm aware of the 'good first issue' label, but there don't seem to be any open ones now -- and that's probably a separate concern.)
2. Once I've chosen an issue I'd like to work on, what happens next? Do I say I'd like to work on it, or just start, do my work, and create a PR?
3. Before creating a PR, what should I check or do? Install pre-commit hooks? Run linting, formatting, and tests?
4. Are there code formatting guidelines? If I run the formatter (which appears to be `make check` here), does that handle everything?
5. Do I need to do anything else that's not handled by the formatter and unit tests? Is it OK to submit a PR without running all the tests? Is it OK to submit a PR with failing tests?

These are some of my questions as someone who's just getting started here. I'd love to hear other people's thoughts.

Done!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Re-running eval with a different scorer
Root cause: Architectural Constraint

I am attempting to run the `score` function on a log file that has already been run (contains prompts, generations from a local model, and scores from an initial scorer) with a different scorer function defined in a file. When I run the following command using the Python API:

```
log = read_eval_log(log_path)
scorer= guard_scorer(model=model_name) # this is a custom scorer function --  verified that it works using a standalone call
scored_log = score(log, scorers=scorer, action='append')
```

I notice that the function attempts to start an inference server for the *generator* model -- which I am not sure is necessary, because I want to re-score the generations that already exist in the log file. My original generator used a local vLLM server, and I get the `Starting new server for xxx..` message.

Any ideas on what could be going wrong? Thanks!


We do currently restore the 'model' from the task as a part of setting up the task context before we start scoring. I'll take a closer look if there is anything that we might do to avoid this (though I'm not entirely optimistic we'll be able to avoid it). I'll revert here with more info.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Question] Has anyone implemented Podman or Singularity sandbox environments?
Root cause: Unimplemented Feature Gap

I'm exploring alternative container runtimes for sandboxing, particularly **Podman** and **Singularity/Apptainer** for environments where Docker isn't available or permitted.

I noticed there's a mock Podman implementation in `tests/test_package/inspect_package/sandboxenv/podman.py` that demonstrates the extension API, but it's just a stub for testing.

## Question for the Community

**Has anyone implemented a working Podman or Singularity/Apptainer sandbox environment?**

If so, I'd love to hear about:
- Your implementation (could you share code, even if partial/experimental?)
- Main challenges you faced (rootless mode, compose compatibility, image conversion, etc.)
- Specific issues you encountered
- Whether it's working in production or still experimental

If there's interest but no existing work, I'd be happy to collaborate on building either as an external package (similar to k8s/ec2/proxmox sandboxes).



I'm not aware of any work on Podman or Singularity but we would very much welcome a package for one or both of them! 

@jjallaire Thank you for your response. Will keep you posted. I will close this issue.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature] Support `reasoning_effort="none"` in `GenerateConfig` for GPT-5.1
Root cause: Unimplemented Feature Gap

- GPT-5.1 has a [new “no reasoning” mode on the API](https://openai.com/index/gpt-5-1-for-developers/#:~:text=New%20%E2%80%9Cno%20reasoning%E2%80%9D%20mode)
- "Developers can now use GPT‑5.1 without reasoning by setting reasoning_effort to 'none'."
- `GenerateConfig` currently only supports 'minimal', 'low', 'medium' or 'high'
- I'm not sure if older models support `reasoning_effort="none"`. If not, there may need to be either a fallback to `minimal` or checks that `none` is not used on older models, or Inspect could not worry about it and let the API reject requests with incorrect reasoning_effort (my preferred way).

Working on it now! Agreed that we should let the API handle reasoning="none" rather than trying to map it.

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2767
Repository: UKGovernmentBEIS/inspect_ai
Issue: SampleSummary Truncation
Root cause: Documentation Deficiency

The metadata in samplesummary gets truncated to 1k length. Is there a way to prevent this?
The problem is that the current sample_df function seems to return evalsample summaries, and I have no way to choose if I want EvalSamples instead, and it always gets truncated.

This is intentional - the sample summary is intended to be a small / compact representation of the sample. If you'd like full metadata, you should read the complete sample by passing the `full` parameter to `samples_df` (this will be slower to read but will give you the full EvalSample).

Thank you!
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] Variable shadowing in `call_tools` causes incorrect approver behavior with multiple tool calls
Root cause: Algorithmic Error

There is a variable shadowing bug in the `call_tools` function that breaks tool call approvers when processing multiple tool calls in a single assistant message.

At [line 115](https://github.com/UKGovernmentBEIS/inspect_ai/blob/3de6d99986a2acb052cf0e659788f00507b619ed/src/inspect_ai/model/_call_tools.py#L115), `message` is assigned to `messages[-1]` (the last ChatMessageAssistant):
```
message = messages[-1]
```
This `message` variable is passed to `call_tool` at line 138:
```
result, messages, output, agent = await call_tool(
    tdefs, message.text, call, event, conversation
)
```
The `call_tool` function uses this `message` parameter when invoking tool call approvers, which need access to the assistant's message to make approval decisions.

However, inside the outer loop (line 310), the code reuses `message` as a loop variable:
```
for message in result.messages:
    result_messages.append(message)
    display_conversation_message(message)
```
This shadows the outer `message` variable. After the first tool call completes, `message` is overwritten with the last `ChatMessageTool` from `result.messages`. When the second tool call executes, the approver receives a `ChatMessageTool` instead of the original `ChatMessageAssistant`, breaking the approval logic.

**Suggested Fix:**

Rename the loop variable on line 311 to avoid shadowing:
```
for result_message in result.messages:
    result_messages.append(result_message)
    display_conversation_message(result_message)
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: Tutorial: Various errors in InterCode CTF
Root cause: Documentation Deficiency

The Inspect Tutorial has some errors in [the InterCode CTF section](https://inspect.aisi.org.uk/tutorial.html#sec-intercode-ctf):

1. `from dataset import read_dataset` produces "ModuleNotFoundError: No module named 'dataset'". I believe the correct import is [here in the inspect_evals repo](https://github.com/UKGovernmentBEIS/inspect_evals/blob/f380473e6c46950e8bdce60e75cac2d13a659d28/src/inspect_evals/gdm_capabilities/intercode_ctf/task.py#L14C1-L15C1).
2. `scorer=includes(),` produces "NameError: name 'includes' is not defined"
3. The `@agent` line produces "NameError: name 'agent' is not defined." I believe this requires an updated import: `from inspect_ai.agent import react, agent`
4. the `prompt=SYSTEM_MESSAGE,` line produces "NameError: name 'SYSTEM_MESSAGE' is not defined". This should probably be `prompt=PROMPT,` to use the variable defined above.

I have fixes for these locally, but GitHub is broken now so I'm not able to push. I'll push and submit a PR when GitHub is back.

Update: pushed and opened a PR!

Thanks for this, as noted we aren't trying to make that particular example executable. I think the best way forward might be to preserve all of your changes and then have read_dataset just be a shell function that doesn't have an implementation?

Fixed and done!
Repository: UKGovernmentBEIS/inspect_ai
Issue: eval_set task_identifier should include model_args
Root cause: Algorithmic Error

Currently tasks that differ only in the model args (like betas) have the same identifier and can not be included in the same eval_set.

See: https://inspectcommunity.slack.com/archives/C0805HJTK8U/p1763498325494019

#2704 added model generate config to the task_identifier. This change should be able to follow the same pattern.
Repository: UKGovernmentBEIS/inspect_ai
Issue: eval_set with solver arg causes PrerequisiteError on rerun
Root cause: Algorithmic Error

This was a regression caused by #2704 . That change included the solver in the task_identifier (indirectly through the eval plan). However at the time the task_identifier is computed the solver provided as an arg to eval_set has not be applied, so the task_identifier calculation is incorrect. On rerun, it does not match the task_identifier in the existing log resulting in the PrerequisiteError.

Discussion: https://inspectcommunity.slack.com/archives/C0805HJTK8U/p1763502998361379