zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Make API-dependent tests opt-in
Root cause: Unimplemented Feature Gap

## Current Situation
- Some tests  only run when specific API keys (e.g., OPENAI_API_KEY, GOOGLE_API_KEY, ANTHROPIC_API_KEY, etc) are set in the environment.
- These tests currently run automatically when the corresponding API key is present, potentially spending credits or money without the developer's explicit intention.

## Problem
1. Developers may unknowingly spend their API credits or money when running tests (this happened to me).
2. Developers need to unset environment variables (which may be used by other tools) to prevent these tests from running.

## Proposed Solution
Implement an opt-in mechanism for running API-dependent tests to avoid inadvertently spending developers' resources.

## Suggested Approach
- Introduce a `--runapi` flag in `conftest.py`.
- Only run API-dependent tests when this flag is explicitly set.

## Questions
- How will this new flag interact with existing test decorators? It should supersede them ideally. 

^ @jjallaire-aisi the above comment from `swaggy111` looks like it was posted by a spambot

> ^ @jjallaire-aisi the above comment from `swaggy111` looks like it was posted by a spambot

Thx! Just deleted.

Great idea! Would the `--runapi` flag take provider(s) as an argument or would it imply using the environment variables as a secondary check? It seems like given that we are trying to prevent accidental token usage that the former would be better (i.e. always a very explicit thing to interact w/ providers).

Is this something you want to take a shot at? (don't want to start implementing if you are already planning on doing so!). Otherwise I'm happy to work on this over the next couple of days.

I see your point about being explicit, though it would get very verbose. My intuition says that it should just be binary to simplify the interface. Once users pass --runapi they can also select a subset of the tests to run if they so wish.



But obviously happy to do whatever you think is best. If you did want to add the argument, we could also add an ALL option to reduce the verbosity.

I'm happy to work on this today if you are able to get back to me soon. Rest of the week I'm not sure yet whether I'd have time!

Okay, I can buy `--runapi` being binary (typically you can pretty accurately select for a provider by doing e.g. `pytest -k anthropic`).

If you can work on this great, if not LMK and I'll definitely pick it up.

Thanks JJ. I'll try to get it done today.
Repository: UKGovernmentBEIS/inspect_ai
Issue: OpenAI provider ignores http_proxy/https_proxy environment variables
Root cause: Configuration Error

# Bug: OpenAI provider ignores http(s)_proxy env vars

## Overview
- I want `inspect_ai` models backed by OpenAI to respect the standard `http_proxy`/`https_proxy` environment variables.** Currently the OpenAI HTTP client ignores them and establishes direct connections.
- When the upstream `openai` SDK is used outside of `inspect_ai`, the same environment variables work, so the regression appears in the `inspect_ai` fork.


## Root cause
- `OpenAIAsyncHttpxClient` subclasses `httpx.AsyncClient` but unconditionally injects a raw `httpx.AsyncHTTPTransport` (`inspect_ai/src/inspect_ai/model/_openai.py:736-769`).
- When a custom transport is supplied, `httpx` skips its normal `_get_proxy_map` resolution and never wires environment proxies into `_mounts`.
- Therefore `http_proxy`/`https_proxy` env vars are silently ignored, regardless of whether the async client is used by `inspect_ai` evaluations or directly.

## Fix direction
1. Stop passing a transport in `__init__`. Instead, override `_init_transport` (and `_init_proxy_transport`) to return `AsyncHTTPTransport` instances with the desired socket options.
2. By letting `httpx.AsyncClient.__init__` run without a pre-populated `transport`, `allow_env_proxies` is `True`, so the standard proxy-resolution logic executes and honours env vars.

I already have the above fix sketched locally. If this approach sounds acceptable, I would be happy to open a PR that wires the proxy handling back in.

Yes, that all sounds great and we'd be thrilled to take this PR.

One more note: the only reason we have that custom transport is to port over some the changes Anthropic made to their transport to support longer timeouts for reasoning models. It's possible that OpenAI has already mirrored these changes in their defaults and so the custom transport itself is no longer necessary (worth checking this first b/c that would be a much simpler fix)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Eval-retry only showing tokens used for current samples.
Root cause: Algorithmic Error

Currently if you use the command `eval-retry` to continue and evaluation it only counts the tokens used for the current samples. For example if you used 3M tokens to complete 3/4 samples in an eval and then stopped the eval and restarted it using `eval-retry` and completed the remaining 1 sample using 1M tokens the second log file produced by the retry would only show 1M tokens as being used.

If you don't know this you could use the second log file to estimate the number of tokens used in future runs and underestimate the tokens required. I think it should either not reset the tokens used between runs or show both the tokens used for the current run and the running total or just make it clearer in the log file that this is only the tokens used for the current sample and which samples those are.

interested to see what people think the best fix for this is.

I think maybe eval-retry should save the token usage before calling eval, then just amend the returned token usage which comes back from eval.

okay cool ill have a look at creating a PR for this.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Gemini reasoning models don't work via OpenRouter
Root cause: Unimplemented Feature Gap

When running Inspect tasks on Gemini reasoning models via OpenRouter, I get errors of the following kind:

```
RuntimeError: BadRequestError('Error code: 400 - {\'error\': {\'message\': \'Provider returned error\', \'code\': 400,
\'metadata\': {\'raw\': \'Gemini models require OpenRouter reasoning details to be preserved in each request. Please
refer to our docs: https://openrouter.ai/docs/guides/best-practices/reasoning-tokens#preserving-reasoning-blocks.
Upstream error: {\\n  "error": {\\n    "code": 400,\\n    "message": "Unable to submit request because function call
`default_api:run_programming_task` in the 2. content block is missing a `thought_signature`. Learn more:
https://docs.cloud.google.com/vertex-ai/generative-ai/docs/thought-signatures",\\n    "status": "INVALID_ARGUMENT"\\n
}\\n}\\n\', \'provider_name\': \'Google\'}}, \'user_id\': \'user_2sxOYVmS9AcRqi12cDDGyhs8tTa\'}')
```

I think this is because OpenRouter requires special behavior to [preserve reasoning blocks](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens#preserving-reasoning-blocks) that Inspect doesn't implement.

To reproduce:

```bash
uv run --python=3.13 \
    --with="inspect-ai" \
    --with="git+https://github.com/METR/inspect-metr-task-bridge" \
    --with="git+https://github.com/METR/inspect-agents" \
    --with="openai" \
inspect eval mtb/bridge \
    -T image_tag=registry-1.docker.io/metrevals/public-tasks:sql_injection-0.1.9 \
    --sample-id userdata_flask \
    --solver metr_agents/react \
    --model openrouter/google/gemini-3-pro-preview
```

Since the PR was closed, can this issue also be closed?

I think this is effectively a “won’t fix” for now, so yes.
Repository: UKGovernmentBEIS/inspect_ai
Issue: vscode Inspect AI extension not working to show logs when using an editable install?
Root cause: Environment Incompatibility

Hi, when I use `uv pip install --upgrade inspect`, then the Inspect AI extension on vscode can properly load and view inspect logs. However, when I do `uv pip install -e .`, then the Inspect AI extension does not recognize the log files and prompts me to install with `pip install --upgrade inspect` instead. Is this expected behavior or are there extra steps I need to take to be able to use the vscode inspect extension with an editable install? Thanks!

This is definitely unexpected (it should work just fine and this is typically how I work). A couple of things to try:

- Try running inspect commands from the terminal just to be sure an error of some kind isn't occurring
- There are output channels for Inspect that will show Inspect, Inspect Startup, and Inspect View output. The Inspect output channel is most like to be helpful as it should have some output related to the python environment it is using, etc... Check to see if this all looks sensible / correct
- Sometimes when reinstalling or changing packages, a reload/restart of the vscode window is a cure-all

Let me know what you find and we can keep troubleshooting.
Repository: UKGovernmentBEIS/inspect_ai
Issue: outputting hidden_states with hf provider results in saving None to log
Root cause: Validation Gap

Hi, when I use `-M hidden_states` as a model arg for `eval`, I've tracked that the hidden states are computed and propagated through all the way until https://github.com/UKGovernmentBEIS/inspect_ai/blob/002a14415e29dd9db66077024152072d406ec361/src/inspect_ai/solver/_task_state.py#L439.
However, here they get converted to None because a tensor is not jsonable. Is this expected? If I want to save and analyze hidden states, what is the recommended workflow to do this? Thanks!

I think the provider should get that `Tensor` into some sort of jsonable form. Would be happy to take a PR for this!

Hm, so I tried a bit of a hack in converting tensors to lists of floats. However, this resulted in an error when trying to save to a json because there was too much data apparently (this is with a Qwen3-1.7B model). Might an alternative be to pickle and then save the path to the pickle or something instead?
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] Extremely slow docker sandbox execution
Root cause: Configuration Error

I've just begun using sandboxes to expose files, and have found them to be extremely slow, during an `inspect eval` run.

Here's some details:
- Version: 0.3.146
- Tools: Bash, File Editor
- Dockerfile/compose.yaml: None, using defaults

The observed behavior is that my runtime is in fact very often bottlenecked on Docker, and many containers are running at 100% CPU when I look at my Docker desktop.
When I look at the `inspect eval` TUI, I notice that they're often waiting upto many minutes on simple commands like `ls -F`.

Previously, I was not using a sandbox, and my entire run was much faster, it seems like adding Docker has slowed down my run ~5x or more, despite only running simple built-in tools.

Any ideas what would cause this unusual bottleneck? 

Okay, this is partially user error, partially confusing defaults. For posterity, the root cause is that `grep` is super slow if you're eg: running it over your OS. Most of the slow commands were `grep`s

By default:
- Your working directory is the root dir
- Your files are copied into the root dir
Thus, if the model decides to `grep -r . "foo"` - it's going to search a lot of files.

You can resolve this by:
a) prepending a dir, eg: `mydir` to all your files, 
```
        files = {f"/mydir{path}": content for path, content in task.files.items()}
```
b) add a minimal `compose.yaml` with a new workdir:
```
  default:
    image: "aisiuk/inspect-tool-support"
    command: "tail -f /dev/null"
    init: true
    network_mode: none
    stop_grace_period: 1s
    working_dir: /mydir
```
(Note: this is exactly the default compose.yaml except for the last line)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Are there any good first issues that just aren't labeled?

As someone who's relatively new here and eager to contribute, I'd love to know what issues seem good to start working on, but I'm [not seeing any open](https://github.com/UKGovernmentBEIS/inspect_ai/issues?q=state%3Aopen%20label%3A%22good%20first%20issue%22) with the "good first issue" label. I'm left to scroll through the list of issues, trying to decide which ones might be a good fit. Maybe that's OK and "working as designed" -- I'm just trying to understand it all.

How do issues normally get labeled as "good first issue"? Would it be possible to allow people who create issues to mark them as "good first issue"? Perhaps that could increase the number of them, though I understand that it might lead to issues being marked with the label that shouldn't be.

People can label issues as "good first issue" or "help wanted", or "gfi" when creating new issues. It's done on the right side (on desktop), under Labels. You can assign labels there. Also, there are several repos that are "good first issues" apps. They'll search and find issues for you. HTH.

@QuantumPixelator Are you able to do that? I wasn't able to add labels when I created this issue, and I can't edit to add labels now, either. I'm on desktop and see the Labels section; it's just not editable for me. Same when I create a new one -- not editable.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add LoRA support
Root cause: Unimplemented Feature Gap

vLLM allows you to load one model and then hit it with requests with different LoRAs adapters, such that different completions can use different LoRAs (using LoraRequest). Would be nice if inspect could support that and LoRAs in general.

I guess it could either be an init arg or a arg passed to generate (or both options?)
Repository: UKGovernmentBEIS/inspect_ai
Issue: API Safety Checks with OpenAI's `computer-use-preview`
Root cause: Unimplemented Feature Gap

Hi,

I've been using the `computer-use-preview` model with Inspect's native `computer` tool. Thank you for providing support for that! I am facing an issue that sometimes the model's response includes a safety check which needs to be acknowledged before proceeding, see [here](https://platform.openai.com/docs/guides/tools-computer-use#acknowledge-safety-checks). I am not sure what the optimal strategy is. For my sandboxed application, I believe a malicious action cannot do much harm and would like to acknowledge the checks automatically. However, in general, it may require input from the user. Thank you for any input on this!

cc @epatey 

We do have a whole approval system which we could tap into for this. Also it might be convenient enough (especially for evals) if we just auto-acknowledged them. 

resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/2921

Thank you for looking into this!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature request: Option to score errored samples as incorrect
Root cause: Unimplemented Feature Gap

Hi, I had a feature request and wanted to get a reaction before we went ahead and tried to implement it. I searched for a similar request in the issue tracker but didn't find anything.

When we run an eval, we often will try really hard to eliminate any errors from the server that make it through retries and cause a sample to fail, so that the score reflects only the model's capabilities and not something about the server's flakiness. But sometimes it's just not possible and we'll accept the errors in the "final" run, but we'd rather score the errors as incorrect instead of dropping them from scoring.

For evals with lots of samples this doesn't really make a big difference, but sometimes we're running evals with just a few samples and then even just a couple errors makes a big difference to the scores, so they're hard to compare between models. For example: 8 samples and 2 correct = 20%, versus 8 samples with 2 correct and 2 errors = 33.3%, which looks almost like the model got another sample correct.

We could rescore them post-hoc, but we thought it'd be easier if Inspect had the option to score errors as incorrect instead. Alternatively, if errored samples were given to the `@metric` we could write a custom metric that does this, but right now I think errored samples are completely hidden.

What do you think? If you're on board, any pointers on the best way to do this would be helpful.

I think you could probably implement this by just:

(1) Removing the error state from the sample
(2) Adding the requisite score(s) to the sample
(3) Calling the [recompute_metrics()](https://inspect.aisi.org.uk/reference/inspect_ai.log.html#recompute_metrics) function.
(4) Re-writing the log file.

I'd like to work on this. Will implement the approach outlined - removing error state, adding scores, recomputing metrics, and rewriting the log. Will open a PR.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Streaming support in OpenAI API compatible servers
Root cause: Unimplemented Feature Gap

Is it possible to stream responses from an OpenAI compatible API? Whilst evaluating inference providers, we've notice that it is not possible to pass a stream parameter via `--model-args` and to stream responses. I am aware that inspect collects responses anyway, but some providers timeout after a certain period and depend on streaming to mitigate this.

If this is not possible, is it something that would be useful in the library? 

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/2917
Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature request: support for Gemini Computer Use
Root cause: Unimplemented Feature Gap

Hi,

Gemini now has an [agentic model](https://ai.google.dev/gemini-api/docs/computer-use), called `gemini-2.5-computer-use-preview-10-2025`, that can interact with a browser using a computer use tool with a custom interface. In my understanding, this could be supported using the same `computer` tool that Inspect now employs to abstract away the provider-specific tools from OpenAI and Anthropic. It would be great if Google's API for this was also supported.

As a side note, this model employs a similar mechanism for safety acknowledgements to that of OpenAI's `computer-use-preview`, so issue #2907 is also relevant here.

Thank you!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect View: header based sample sorting in sample list view
Root cause: Unimplemented Feature Gap

Move from the selector / tool based sorting to a clickable header based sort. 

<img width="1035" height="335" alt="Image" src="https://github.com/user-attachments/assets/8e79cfb9-793d-43a1-9d8a-b75822ee432f" />

### Notes

- Support modifier clicking to sort by multiple columns
- Support iterating through asc, desc, and natural sort order
- Will need to deal with dynamic columns (e.g. scorer columns) that are visible
- Will need to address store support for multiple columns
- Maybe have a little trickiness due to descriptors....

### Technical Notes

For direction on visual implementation / experience see Inspect Scout Viewer  scan results header sorting:
https://github.com/meridianlabs-ai/inspect_scout/blob/main/src/inspect_scout/_view/www/src/app/scanner/results/list/ScanHeader.tsx

Relevant source code:
SampleList.tsx/SampleHeader.tsx
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect View: Sample List grid view
GitHub labels: viewer

Allow user to toggle between a list and grid view of samples. The list view is the current fixed height row based virtual list. The grid view is a grid of the samples based upon the sample grid list. Use a segmented control to permit toggling between list and grid view place in the top right tool region (to the right of where the sort selector appears at this time). 

Columns for the grid view should be derived from the sample list, but should omit task, model, or other unnecessary columns (by default) since this view is always showing only a single task and model. Support customizing the columns associated with the sample grid as you can in the full sample list.

### Notes

- Ensure that the footer sample count reflects any filtering applied

### Open Issues
- Rationalize the sorting behavior of grid column headers with the expression based sorting that is currently present. We could become convinced that the header based sorting is sufficient and remove the expression based sorting, or alternatively could merge the two, using the expression syntax to update the column UI (and vice versa).
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect View: Score rendering in sample grid
Root cause: Unimplemented Feature Gap

Currently, we display the raw values of scores within the sample grid (e.g. 'C', 'I', 0.78434, etc...). We have custom score rendering that renders some values with additional visual treatment (green circles, red circles, etc...). Integrate this custom rendering into grid to provide stronger visual cues.

**Grid Text Rendering**
<img width="958" height="165" alt="Image" src="https://github.com/user-attachments/assets/78203ef3-7eb6-49bd-bc71-ec73084ddf91" />

**Visual Rendering**
<img width="522" height="192" alt="Image" src="https://github.com/user-attachments/assets/7dbe04a7-08a5-4bd7-82ca-64d9200b5e6b" />

Will need to experiment with rendering that balances providing stronger visual cue but not overwhelming noise / distraction. Note that current custom rendering shouldn't be take as is, but grid appropriate parallel renderings (or small versions of the current rendering) should be taken.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Download Sample JSON / Transcript
GitHub labels: viewer

### Sample downloads

On the sample detail view, we should just add a 'Download' button adjacent to the copy button. I think that should support either the raw sample JSON or the transcript (as text, similar to the copy button). 

```
[ Download ]
  > Sample JSON
  > Transcript
```

### Technical Notes

- See https://github.com/UKGovernmentBEIS/inspect_ai/pull/2745 comments for notes on conditionalizing download support/button using capabilities.
- This ideally would be accomplishable using BLOB urls in order to make it supportable without coordination from the server
Repository: UKGovernmentBEIS/inspect_ai
Issue: Download multiple log files
GitHub labels: viewer

From views listing log files, allow selection of multiple log files and downloading. To accomplish this, the user should be able to select one or more logs/tasks from the list, then click a 'download' button in the top right side of the view. The download button should appear to the left of the filter text box.

<img width="1081" height="348" alt="Image" src="https://github.com/user-attachments/assets/667748bb-27a5-444a-955b-78d27ebdb4b8" />

### Notes

- Currently the grid of tasks/logs doesn't have good support for selection. We will need to decide whether to better support selection (e.g. click + shift click) using native affordances, add checkboxes as a selection column, or solve through something like a select mode that shows a checkbox column when enabled.
- Will need to package the files as a zip for download. If we do this, be sure to cleanup files once download is complete.
- I would block selection of folders since recursive cases could be very resource intensive.
- This is best accomplished after this https://github.com/UKGovernmentBEIS/inspect_ai/pull/2920 is completed