zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: GenerateConfig should allow for streaming as a parameter
Root cause: Unimplemented Feature Gap

This makes it difficult to work with Anthropic models with non-Anthropic ANTHROPIC_BASE_URLs

For now we added a `-M streaming=false` model option for anthropic: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/docs/reasoning.qmd#L100-L105. e.g. 

``` bash
inspect eval math.py \
  --model anthropic/claude-3-7-sonnet-latest \
  --reasoning-tokens 4096 \
  -M streaming=false
```

Once we support streaming with more providers we will add it to `GenerateConfig`.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Local server sets "CUDA_VISIBLE_DEVICES" to "None"
Root cause: Validation Gap

@kaifronsdal @jjallaire (tagging you as authors of the relevant file)
[inspect_ai/_util/local_server.py](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/_util/local_server.py), in the [configure_devices()](https://github.com/UKGovernmentBEIS/inspect_ai/blob/76266abac6b84fca380226691d60eccf0fd6e0ca/src/inspect_ai/_util/local_server.py#L345) function, the environment variable `CUDA_VISIBLE_DEVICES` is set to the string `"None"` by default:
```
    devices = None
    if "device" in result and "devices" in result:
        raise ValueError("Cannot specify both device and devices in server args")
    elif "devices" in result:
        devices = result.pop("devices")
    elif "device" in result:
        devices = result.pop("device")

    # Convert device list to comma-separated string if needed
    if isinstance(devices, list):
        device_str = ",".join(map(str, devices))
    else:
        device_str = str(devices)

    # Set CUDA_VISIBLE_DEVICES environment variable
    os.environ["CUDA_VISIBLE_DEVICES"] = device_str
```
Instead, when `"device"` or `"devices"` are not in `result`,  `CUDA_VISIBLE_DEVICES` should not be changed.

The current behavior causes issues when calling `get_model()` on a `vLLM` model without explicitly providing a `device` or `devices` keyword in `model_args`.
```
model_args: Dict[str, Any] = {
        "tensor_parallel_size": 8,
    }
model = get_model("vllm/meta-llama/Llama-3.3-70B-Instruct", max_model_len=1024, **model_args)
```
Fails as CUDA_VISIBLE_DEVICES gets set to "None" (as a string), which causes a failure when vLLM tries to interpret it:
```
INFO:inspect_ai._util.local_server:ValueError: invalid literal for int() with base 10: 'None' 
ERROR:inspect_ai._util.local_server:Server process exited unexpectedly
```

@jordan-taylor-aisi Thanks for reporting this. Could you put up a PR w/ a proposed fix?

@jjallaire Done: 
* #1796

Should be resolved with: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1796
Repository: UKGovernmentBEIS/inspect_ai
Issue: Missing __init__.py in subdirectories
Root cause: Documentation Deficiency

Hello! 

Some subdirectories such as `src/inspect_ai/_eval` should have `__init__.py` in them to help mark the directory as a python module. I noticed this because pyright had some issues with resolving imports from this project. Thank you!

As of python 3.7 this is no longer required: https://peps.python.org/pep-0420/ , though can be used to control namespace exports. 


Our convention is actually to use `_` for "private" modules, so we purposely don't have `__init__.py` in those modules b/c we don't expect you to import from them. In this case, anything public in `inspect_ai/_eval` is actually importable from `inspect_ai` (e.g. `Task`, `eval`, etc.). We further butress this convention by having all public APIs explicitly delineated in `__all__` objects. For example:

- https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/__init__.py
- https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/solver/__init__.py
- etc.

Many (most?) IDEs will only offer auto-imports from `__all__` if is is provided. I know that this is how Pylance works, not sure about Pyright. General guideline would be that if there is an `_` anywhere in the import path it's a private import, so you should fine the right public place to import from (all imports are at most 1 level deep so either in `inspect_ai`, `inspect_ai.model`, `inspect_ai.solver`, etc.

Going to close in hopes this clarification is all that is required. If there are other issues you think need addressing feel free to re-open and/or rename to something more targeted.
Repository: UKGovernmentBEIS/inspect_ai
Issue: `appy_chat_template` TypeError with Qwen3 model when `enable_thinking=True`
Root cause: Interface Contract Mismatch


Hi team,

Qwen recently released a new model (Qwen3) that introduces "thinking" tokens. While experimenting with it on Agent Harm using the local HuggingFace provider, I encountered the following error when applying `appy_chat_template`:

```
TypeError: can only concatenate str (not "list") to str
```

After investigating, I found that the issue comes from the format of the `messages` input to the `hf_chat` function. Specifically, when `enable_thinking=True`, the `ChatMessageAssistant.content` is a list rather than a string, which leads to the concatenation error.

Here is an example of the problematic content structure:

```python
ChatMessageAssistant.content = [
    ContentReasoning(...),
    ContentText(...)
]
```

However, when `enable_thinking=False`, the same field is a plain string and the error does not occur.

I discussed this with @mattmdjaga and have opened a PR proposing a solution. Please check it out for implementation details.

Thanks!


Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1802
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug]Using multiple mcp_server_stdio duplicates the first server tools

- I'm adding two mcp stdio tools to a simple task.
- In the tools block of the view, I see it maps to two tools, but they both have the same description as the first.

```
git_server = mcp_server_stdio(command="python3", args=["-m", "mcp_server_git", "--repository", "."])
fetch_server = mcp_server_stdio(command="python", args=["-m", "mcp_server_fetch"])

return Task(
        dataset=[Sample(
            "What time is it ?"
        )],
        solver=[
            use_tools([git_server, fetch_server]),
            generate()
        ]
    )
```

<img width="1109" alt="Image" src="https://github.com/user-attachments/assets/64313d88-e586-4527-9cc8-11d48e7efaaf" />

This fix seems to work, will send PR later - 
<img width="690" alt="Image" src="https://github.com/user-attachments/assets/423cfbf4-bf73-423c-874a-71dce8202172" />

Okay, thanks for sleuthing this out! I will also take a look at reproing locally and then thinking through the implications of the fix.

Thanks again for reporting this, resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1807

Thank you @jjallaire . That was lightning speed fix !
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Allow for other containerization schemes other than docker
Root cause: Documentation Deficiency

Docker is one of a family of containerization schemes such as podman, lxc, apptainer etc. It would be nice to be able to use inspect without being forced to use docker and instead be able to pass in the variable for which container solution to use.  

For instance we currently cannot use docker on our multi-user system because for SLURM to activate docker needs sudo privileges which then creates a security risk for everyone involved.  So we opted for apptainer which maintains a compatible scheme as docker.

Because docker is hard-coded though it tries to call it directly on the system instead of being able to use a user passed in sandbox environment.  

Sandbox providers are indeed extensible in the system: https://inspect.aisi.org.uk/extensions.html#sec-sandbox-environment-extensions

So far we have Docker, k8s, and Proxmox providers:

- https://github.com/UKGovernmentBEIS/inspect_k8s_sandbox/
- https://github.com/UKGovernmentBEIS/inspect_proxmox_sandbox

You could absolutely create a Podman or any other sort of container solution, a sandbox provider just needs to be written for it.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Still cannot install in Windows
Root cause: Environment Incompatibility

in command as admin run
pip install inspect-ai

Then close and reopen cmd and type 

inspect view

![Image](https://github.com/user-attachments/assets/62c42ffd-cc4c-4e43-bf06-831859b1e735)
I opened an issue on this a week ago and it was closed, but it was not solved. Can you please try running this on a Windows machine and you will see the issue?

@jjallaire 

#1754 previously referenced issue. 

We are able to install successfully on Windows on multiple systems so there is not a general problem here (this is the only report we have of issues on Windows, so it is likely there is some vagary of your system or configuration that is keeping things from working). If we can find a way to reproduce the issue then we will of course resolve it ASAP but we haven't as of yet been able to.

Hi @jjallaire,

Ok can you give me some troubleshooting steps then?  I believe my Windows system is very normal. Microsoft Windows [Version 10.0.26100.3915]

It says
Successfully installed inspect-ai-0.3.94

then I do pip show inspect-ai-0.3.94

WARNING: Ignoring invalid distribution ~ygments (C:\Users\jasmi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages)
WARNING: Package(s) not found: inspect-ai-0.3.94




There are so many variations of Python on Windows and so many ways that the PATH can be configured that it is hard to give general advice. One thing I do recall is that on Windows they were/are moving away from putting the Python bin dir on the PATH. So it may be that you need to figure out what the bin dir is and add that to your path?

Another thing is that on Windows sometimes Python installations are accessible only via the `py` command. So something like `py -m inspect_ai --help` might locate it? Again, there are myriad ways to install and configure Python on Windows (Windows Store, PSF install, conda, etc.) and myriad ways the PATH can be misaligned to any given configuration. The fact that you can pip install successfully and then pip show doesn't work is a mystery to me but I have seen many bizzare failures modes on Windows given the variety of the landscape.


oh ok when I do run python -m inspect_ai view it works. Can you please update the instructions so people know how to manage this? That would be great

There is something specifically wrong with your configuration that is requiring this - typical installs on windows do not have this issue. For example, if you create a clean python environment, activate it, and then install Inspect there does that work? I think without you spending some time debugging the environment, there isn't anything for us to do.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Improve usability of writing a file to a sandbox with a specific owner

Please see @rasmusfaber's PR #1798 for discussion on this and the motivation.

There were 2 concerns with adding an `owner` parameter to `write_file()`:
1. This is a breaking change to the `SandboxEnvironment` interface which is implemented in many sandbox environent providers: SSH, [K8s](https://k8s-sandbox.ai-safety-institute.org.uk/), [Proxmox](https://github.com/UKGovernmentBEIS/inspect_proxmox_sandbox) and likely others which I'm not aware of. These sandboxes would need to be updated to accept the `owner` parameter to use the new version of Inspect.
2. Some sandbox environment providers cannot easily support this functionality. Already, we suffer from this with `exec()`'s `user` parameter ([e.g.](https://k8s-sandbox.aisi.org.uk/design/limitations/#exec-user)). The SSH provider does not support the `user` parameter.

To avoid issue 1., we could add a convenience function such as

```py
def write_file_with_owner(sandbox: SandboxEnvironment, file: str, owner: str, contents: str | bytes):
    # ... ensure path exists
    
    if isinstance(contents, str):
            result = await sandbox.exec(
                [
                    "sh",
                    "-e",
                    "-c",
                    'tee -- "$1" > /dev/null',
                    "write_file_script",
                    file,
                    user = owner,
                ],
                input=contents,
                timeout=TIMEOUT,
            )
     else:
        # ... base64 version of the above.
```

Which leverages the `user` parameter of `exec()`.

However, this still suffers from problem 2. On K8s it requires running the container as root. On SSH (as currently implemented) the user parameter is not supported. It also makes (not unreasonable) assumption that `tee` and `base64` are installed. Finally, for writing large files (we know of some evals writing 100 MiB+), writing the whole file via stdin won't necessarily be the most efficient or reliable way of doing so (e.g. SSH uses scp instead).

We should consider whether adding this feature makes evals less portable between sandboxes, a bit like the `exec(user)` parameter unfortunately did.

Definitely keen on reducing friction when writing evals and providing useful functions and conveniences to users though!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Claude and Gemini web search tools
Root cause: Unimplemented Feature Gap

Great to see addition of the OpenAI web search tool in https://github.com/UKGovernmentBEIS/inspect_ai/pull/1838 @epatey 

Request to consider adding web search tools for other notable model providers as well, eg:
- [Claude](https://www.anthropic.com/news/web-search-api)
- [Google](https://ai.google.dev/gemini-api/docs/grounding?lang=python)

Indeed! This is something we are working on now and planning on landing as either part of this PR or in a follow-on PR.

Doesn't really fall under this ticket's title but
Just wanted to note that Perplexity is another one that would be useful to support @jjallaire @epatey 
https://docs.perplexity.ai/models/model-cards#search-models

It's fantastic Inspect now has support for Claude search tools #1919 

Shall we add support for Gemini's `GoogleSearch` tool? (https://ai.google.dev/gemini-api/docs/grounding?lang=python) and complete this ticket? @epatey @jjallaire 

Together with adding Perplexity (#1975 ) this will round out the search tool options nicely

Now that we have #1997 I'll go ahead and close this ticket since the original ticket was about Claude and Gemini (but it will still be very helpful to complete Perplexity #1975 ). Thanks so much @epatey @jjallaire !
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Add metadata columns/Update viewer table
Root cause: Unimplemented Feature Gap

A feature that would be much appreciated would be the ability to add extra columns to the viewer, such as the sample_id or specific metadata values. I often find I want to look only at samples of a specific type, but where it doesn't make sense to seperate them out into a seperate task. Perhaps the way this would look would be a dropdown menu with checkboxes to select the columns

Additionally, it would be nice if the current sorting implementation was generalized to any of the columns similar to for instance how ActivityMonitor on Macs work. The ability to resize columns would also be a nice-to-have.

The React Tables library might be a good base to build off of. It seems to mostly just implement the logic of the table without and formatting. They have some examples for [sorting](https://tanstack.com/table/latest/docs/framework/react/examples/sorting), [filters](https://tanstack.com/table/v8/docs/framework/react/examples/filters), [column selection](https://tanstack.com/table/latest/docs/framework/react/examples/column-ordering), and a bunch of other features.

@dragonstyle I think we should put this on our short list after we land all of the transcript related stuff.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Alternative text editing formats

Great that Inspect supports the native Claude text editing format and tool.

There is [evidence](https://github.com/Aider-AI/aider/issues/3529#issuecomment-2725011163) that this may not be the ideal editing interface, and it would be useful to experiment with others like:

1. The new OpenAI patch editing format. Reference implementations:
- OpenAI [Cookbook](https://cookbook.openai.com/examples/gpt4-1_prompting_guide#apply-patch) and [Codex](https://github.com/openai/codex/blob/main/codex-rs/apply-patch/src/lib.rs)
- [Aider](https://github.com/Aider-AI/aider/compare/v0.81.0...v0.82.0)

2. The search/replace diff format used in Aider. The OpenAI Cookbook [notes that](https://cookbook.openai.com/examples/gpt4-1_prompting_guide#other-effective-diff-formats) it also worked well for them, and [the Aider folks claim](https://github.com/Aider-AI/aider/issues/3529#issuecomment-2725011163) it out-performs the native Claude format.

Interesting! I would like to see a bit more evidence that just the assertion on the GH issue. 

cc @epatey For your research/consideration.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Outdated docstring for `react` agent

The documentation page for the [ReACT agent](https://inspect.aisi.org.uk/reference/inspect_ai.agent.html#react) points to an outdated version of the code. This is noticeable by comparing the description of the `submit` parameter. 

The doc page points to this:
https://github.com/UKGovernmentBEIS/inspect_ai/blob/cd1fc132a98c04a3f4bafa32e138d96b58bac3e3/src/inspect_ai/agent/_react.py#L77

While the latest release (0.3.96) has updated code that no longer works with the documentation:
https://github.com/UKGovernmentBEIS/inspect_ai/blob/41d0dbba174e804f4ca03158c2f73c3f54d9179c/src/inspect_ai/agent/_react.py#L76

Nevermind, I just noticed the note saying this feature is only available in the `development` version of this package.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [inspect view] [feature request] Filtering the samples by timings
Root cause: Unimplemented Feature Gap

> It would be nice to have a sample filter on timings in the evals view (some few prompts took about 45 minutes to generate, and it is not straightforward to identity them in the inspect view). 

 _Originally posted by @clebreto in [#1869](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1869#issuecomment-2879265190)_
Repository: UKGovernmentBEIS/inspect_ai
Issue: [inspect view] [feature request] Provide tokenizer information

> It would be nice to have the information about which tokenizer was applied within the inspect view. 

 _Originally posted by @clebreto in [#1869](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1869#issuecomment-2879272262)_
Repository: UKGovernmentBEIS/inspect_ai
Issue: [inspect view] [feature request] Provide vllm information
Root cause: Unimplemented Feature Gap

> It would be nice to have which version of vllm was used to execute the benchmark (within the Task section ?). 

 _Originally posted by @clebreto in [#1869](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1869#issuecomment-2879274559)_
Repository: UKGovernmentBEIS/inspect_ai
Issue: [inspect view] [feature request] Provide sampling parameters
Root cause: Unimplemented Feature Gap

> It would be nice to have the sampling parameters used for inference in inspect view. 

 _Originally posted by @clebreto in [#1869](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1869#issuecomment-2883878490)_
Repository: UKGovernmentBEIS/inspect_ai
Issue: [inspect eval] [feature request] vllm logging level to match that of inspect_ai by default
Root cause: Configuration Error

> It would be great if all logs from vllm could pass through inspect_ai in its latest implementation.
> 
> Currently, by default, when using `--log-level debug` only vllm warnings are displayed in the console (no debug, nor info). 

 _Originally posted by @clebreto in [#1869](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1869#issuecomment-2885847876)_
Repository: UKGovernmentBEIS/inspect_ai
Issue: Bug: human_cli solver VS Code Window failing
Root cause: Environment Incompatibility

When running `inspect eval --solver human_cli` and clicking the Login: `VS Code Window` button, I get this popup:

`
There is no application set to open the URL vscode-remote://dev-container+7b22686f737450617468223a222f55736572732f64617669642f6d6574722f70722d7461736b73222c226c6f63616c446f636b6572223a66616c73652c2273657474696e6773223a7b22636f6e74657874223a226f7262737461636b227d2c22636f6e66696746696c65223a7b22246d6964223a312c22667350617468223a222f55736572732f64617669642f6d6574722f70722d7461736b732f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c2265787465726e616c223a2266696c653a2f2f2f55736572732f64617669642f6d6574722f70722d7461736b732f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c2270617468223a222f55736572732f64617669642f6d6574722f70722d7461736b732f2e646576636f6e7461696e65722f646576636f6e7461696e65722e6a736f6e222c22736368656d65223a2266696c65227d7d/home/metr/app/VS%20Code%20Window.
`

Not sure if it's relevant, but I'm running this from a devcontainer.

cc @dragonstyle
Repository: UKGovernmentBEIS/inspect_ai
Issue: investigate issues w/ textual 3.0
Root cause: External Dependency Breakage

Currently we get this with eval-set on textual 3.0:

```
File "/Users/jjallaire/ukinstitute/UKGovernmentBEIS/inspect_ai/.venv/lib/python3.12/site-packages/textual/app.py", line 2154, in run
    event_loop = asyncio.get_event_loop() if loop is None else loop
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/events.py", line 698, in get_event_loop
    raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'MainThread'.
```

hey, it's a bit annoying to have this strict requirement, do you think it will be loosened up sometime soon? my app depends on a more modern version of textual

This had been resolved (latest versions of Inspect can use any version of textual)