zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: O1PreviewChatAPIHandler.input_with_tools breaks message context
Root cause: Architectural Constraint

I were debugging unexpected agent behavior and found that for `o1-preview` and `o1-mini` there are extra bits getting added:

```python
    # tool prompt
    tool_prompt = textwrap.dedent(
        f"""
        You are a knowledgable assistant. You can answer questions and perform tasks. ...
        """
```

This may result in split personality and other conflicts with the provided context.

And the extra prompt is added right at the beginning which is not always appropriate.

```python
    # return the tool prompt prepended to the input messages
    return [ChatMessageUser(content=tool_prompt)] + input
```

In effect, giving an agent some tools can result in wildly different behavior.

We have to do something to facilitate tool use (as there is no native tool use capability in these models). i.e. they don't ever receive a list of tools the way normal models do, this is the entirely of what they see. In order to get them to call tools at all we need some prompting. If we do nothing or just dump the JSON schema they probably won't ever call a tool.

We can certainly debate whether this is a good prompt though (it's largely adapted from the one used by Ollama to elicit tool use from models that have no built in API.

These models also have no system message so we have to inject the tools prompt as a user message at the beginning (closest semantically to how proper tool calling models are prompted)

Yeah, I understand. I just would like to have some flexibility around this, but not sure how to make the proper plumbing for this.

I originally put that "handler" in there so people could play with different strategies. I realize it's kind of hard-coded but I don't really like the API so I've hesitated to make it public. One thing you could do is just create your own o1-preview/mini model provider and experiment that way.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support passing file-like objects (StringIO, BytesIO) to `read_eval_log` and similar functions
Root cause: Architectural Constraint

Currently, `log_file` expects a `str` path.

The abstraction supported for `read_eval_log` is a "location" which today could be a local file but could also be a remote filesystem (s3, gs, etc.). Further, we've tried to be careful that "location" is never strictly treated as a file path so we can later support e.g. Prisma-style database connection URLs.  So taking file like doesn't fit with the abstractions and interfaces we've developed around `location` (which don't even take `Path` but rather just `str`). We could of course just take the file like object and dump it into a temp file, but the client could also do that and I think it's better if the client knows that this is happening vs. us doing it behind the scenes while _appearing_ to take the stream directly w/o copying or extra serialization.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Parallel tool calls error for OpenAI o-series models on azure
Root cause: Configuration Error

I'm trying to run inspect with o3-mini from azure (`--model openai/azure/o3-mini-2025-01-31-azr`) but I get an error
```
RuntimeError: BadRequestError('Error code: 400 - {\'error\': {\'message\': "Unsupported parameter: \'parallel_tool_calls\' is not supported with this model.", 
\'type\': \'invalid_request_error\', \'param\': \'parallel_tool_calls\', \'code\': \'unsupported_parameter\'}}')
```

This issue showed up previously and was resolved by special handling for o1 #1025 (and then later o3 #1227) but that doesn't seem to be working for the azure provider. I'll try to track down exactly where this is going wrong and open a PR with a fix
Repository: UKGovernmentBEIS/inspect_ai
Issue: Bug: Erratic VSCode log viewer scroll bar for medium-size .eval files

# Problem Description
When viewing samples in VS Code, the scroll bar acts erratically if I scroll too quickly, changing sizes rapidly in succession and often resetting to the top. It makes it difficult to simply navigate to places lower down in the log file unless I scroll slowly. This wouldn't have been as much of an issue except that the collapsible headers in the viewer have also disappeared, necessitating lots of scrolling.

# Context
This behavior just started today, and persisted through updates and restarts of Cursor, inspect-ai, and the Inspect AI extension.
This happens for both new and week-old old log files. It seems to not be an issue for small log files. I've found it an issue for .eval files as small as 410 KB up to 1.6 MB.
Here are a couple of problematic log files:
- https://github.com/aaron-sandoval/inspect-control-settings/tree/refine-prompts/experiments/logs/apps_baseline/2025-03-11_192833
- https://github.com/aaron-sandoval/inspect-control-settings/tree/refine-prompts/experiments/logs/apps_baseline/2025-03-07_123900

## Stack
WSL Ubuntu
Cursor 0.46.11
VS Code version: 1.96.2
inspect-ai library: 0.3.72
Inspect AI VS Code extension: 0.3.58

Resolved with #1463
Repository: UKGovernmentBEIS/inspect_ai
Issue: Type of `sample_id` in `eval` is incompatible with lists of strings
Root cause: Interface Contract Mismatch

## Problem

The below produces an error by type checkers:

```python
eval(task, sample_id=["A", "B"])
```
The type checker points out that `list` is invariant and hence an argument of type `list[str]` cannot be assigned to a parameter of type `list[str | int]`.

I assume this is unintentional since passing a list of strings seems like an intended use case.

## Solution

I was going to make a PR changing `list[str | int]` to `list[str] | list[int]` in the appropriate places. However, I also saw that the documentation for `eval_async` features the following:

```
sample_id (str | list[str] | None): Evaluate specific sample(s) from the dataset.
```

This made me wonder if `int` and `list[int]` were not intended to be there at all.

Is changing `list[str | int]` to `list[str] | list[int]` the appropriate solution here?

---

I also saw that `limit` has an annotated type of `int | tuple[int, int]`, but the docstring for `limit` says this:

```
limit (str | int | list[str | int] | None): Limit evaluated samples
```

The same issue shows up with the type `Value` [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/ce940854694df442e009770707478069ddc9afac/src/inspect_ai/scorer/_metric.py#L43).

Am I confused about how these types are supposed to work?

Thanks for reporting this, it's unlikely this is all working as intended. Will investigate and come up with a fix ASAP.

A few follow ups here:

1. This code `eval(task, sample_id=["A", "B"])` does not result in any typing errors for me when using MyPy. Are you possibly using a different type-checker or an older version of MyPy?

2. Limit is indeed intended to work with integers as its an index into the dataset rather than an ID.

2. The docstrings you referenced are out of sync with the actual function signatures (and the signatures are correct). We actually no longer depend on docstring for types in our doc generation so those docstrings were actually removed in a commit yesterday (so hopefully this won't cause confusion in the future!)

Sample IDs can indeed be either integer or string and so we do want to preserve all of those types wherever sample ids can be specified.

I was using Pyright instead of mypy, and interestingly the error only shows up with Pyright. The error does show up if you do `mypy --strict .` though.

However, you have to do one other thing to see the error. It turns out I compressed the example a little bit too much: you need to change it to the following to actually see the error:

```python
strings = ["A", "B"]
eval(task, sample_id=strings)
```

For some reason, passing the list of strings directly does not yield the same behavior for the type checker.

Addressed by https://github.com/UKGovernmentBEIS/inspect_ai/pull/1484
Repository: UKGovernmentBEIS/inspect_ai
Issue: "provider caching" is a 404

https://inspect.ai-safety-institute.org.uk/caching.html links to https://inspect.ai-safety-institute.org.uk/sec-provider-caching which is a 404

> # Caching
> ## Overview
>
> Caching enables you to cache model output to reduce the number of API calls made, saving both time and expense. Caching is also often useful during development—for example, when you are iterating on a scorer you may want the model outputs served from a cache to both save time as well as for increased determinism.
>
> There are two types of caching available: Inspect local caching and provider level caching. We’ll first describe local caching (which works for all models) then cover [provider caching](https://inspect.ai-safety-institute.org.uk/sec-provider-caching) which currently works only for Anthropic models.

Thanks! Fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/bf9d846f40aa01d2658cbec7c52b1e998ca09b39
Repository: UKGovernmentBEIS/inspect_ai
Issue: unable to deserialize log if sandbox provider is not available
Root cause: Validation Gap

Seeing this error with a log that has a k8s sandbox provider b/c I am reading it within an environment without the k8s package installed.

```python
  File "~/UKGovernmentBEIS/inspect_ai/.venv/lib/python3.12/site-packages/pydantic/main.py", line 214, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for EvalLog
eval
  Value error, SandboxEnvironment type 'k8s' not recognized. [type=value_error, input_value={'type': 'k8s', 'config':...age_simple/agent.yaml'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/value_error
```

@craigwalton-dsit I think we are going to need a better fallback here.

Thanks for this! Have addressed in #1494
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer filter input error
Root cause: Validation Gap

Version: 0.3.72

Typing in `model_graded_fact == ` into the inspect view filter input threw the following error:

```
index.js:31881 RangeError: Position 23 is out of range for changeset of length 21
    at ChangeSet.mapPos (index.js:26794:17)
    at RangeSet.map (index.js:28824:35)
    at StateField.update [as updateF] (index.js:44317:43)
    at Object.update (index.js:27713:31)
    at EditorState.computeSlot (index.js:28353:89)
    at ensureAddr (index.js:27934:27)
    at new EditorState (index.js:28292:11)
    at EditorState.applyTransaction (index.js:28353:9)
    at get state (index.js:28092:27)
    at EditorView.update (index.js:36524:23)
```

Sorry you've run into this! Is there a specific log file that reproduces the issue and if so would it be possible for you to share it with me? You could always DM it to me to avoid posting it here, if helpful.

No worries! I was just testing stuff out and ended up getting it to work by pasting in the full `model_graded_fact == "C"` into the input. It doesn't seem to matter which log it is, it just freezes after typing in `model_graded_fact ==` and shows that error in the console. But if relevant, this was the task file:

```python
from inspect_ai import Task, eval, task
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import chain_of_thought, generate, self_critique


@task
def theory_of_mind():
    return Task(
        dataset=example_dataset("theory_of_mind"),
        solver=[chain_of_thought(), generate(), self_critique()],
        scorer=model_graded_fact(),
    )


eval(theory_of_mind, limit=5)
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: tools with optional string arguments get `"None"` instead of `None` and nonetheless displays as `null` in `inspect view`
Root cause: Validation Gap

I have a tool like
```python
@tool
def mytool() -> Tool:
    async def go(argument: str | None = None) -> ToolResult:
[...]
```
I often see `argument = null` in `inspect view`, but the python code only ever sees `argument = "None"`, never `argument = None`.  See `tool: edit_proof_tool(iso_source: "plus_0_r_statement", iso_target: null, new_proof: "iso. iso_rel_rewrite Nat.add_zero. iso.")` in  [2025-03-15T02-16-15+00-00_coq-to-lean_io6qeNRRVnd9wYeiUWFMfU.eval.tar.gz](https://github.com/user-attachments/files/19257349/2025-03-15T02-16-15%2B00-00_coq-to-lean_io6qeNRRVnd9wYeiUWFMfU.eval.tar.gz)

Is this an issue with the LLM fine-tuning / response, or with inspect's parsing of it?  Is there an issue with the display?
Repository: UKGovernmentBEIS/inspect_ai
Issue: caching no longer works with inspect 0.3.71
Root cause: External Dependency Breakage

Passing `cache=CachePolicy(expiry=None, per_epoch=False)` works fine with inspect 0.3.70, but is ineffective with inspect 0.3.71

When I run this code:
```python
from inspect_ai import Task, eval, task
from inspect_ai.model import CachePolicy
from inspect_ai.solver import basic_agent, system_message
from inspect_ai.dataset import MemoryDataset, Sample


@task
def mytask(
    cache: CachePolicy | bool = False,
):
    return Task(
        dataset=MemoryDataset([Sample(input="Say hi 100 times")]),
        solver=basic_agent(
            init=system_message("Say hi 100 times"),
            tools=[],
            max_attempts=1,
            message_limit=3,
            cache=cache,
        ),
        token_limit=1_000,
    )


if __name__ == "__main__":
    eval(
        mytask(
            cache=CachePolicy(expiry=None, per_epoch=False),
        ),
        model="openai/gpt-4o-mini-2024-07-18",
        token_limit=1_000,
    )

```
with `for i in 1 2; do time python path/to/file.py ; done` with 0.3.71, I get

real    0m14.041s
user    0m2.266s
sys     0m0.118s

real    0m14.917s
user    0m2.256s
sys     0m0.104s

But with 0.3.70, I get

real    0m14.566s
user    0m2.292s
sys     0m0.135s

real    0m3.581s
user    0m1.909s
sys     0m0.087s

Caching is quite important for being able to debug things, and save on token costs.

Thanks for reporting, will look at this immediately.

This is resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1481/

This fix has been deployed to PyPI as version 0.3.74
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view` crashes with Error: right-hand side of 'in' should be an object, got undefined
Root cause: Validation Gap


An unexpected error occurred.
Error: right-hand side of 'in' should be an object, got undefined

at setPath@http://127.0.0.1:7575/assets/index.js:71797:23
synthesizeComparable@http://127.0.0.1:7575/assets/index.js:71785:20
@http://127.0.0.1:7575/assets/index.js:71605:16
mountMemo@http://127.0.0.1:7575/assets/index.js:6997:25
useMemo@http://127.0.0.1:7575/assets/index.js:18256:20
assets/index.js/exports2.useMemo@http://127.0.0.1:7575/assets/index.js:1138:38
StateEventView@http://127.0.0.1:7575/assets/index.js:71604:44
react-stack-bottom-frame@http://127.0.0.1:7575/assets/index.js:18049:20
renderWithHooks@http://127.0.0.1:7575/assets/index.js:6168:43
updateFunctionComponent@http://127.0.0.1:7575/assets/index.js:7834:21
beginWork@http://127.0.0.1:7575/assets/index.js:8910:20
runWithFiberInDEV@http://127.0.0.1:7575/assets/index.js:2588:18
performUnitOfWork@http://127.0.0.1:7575/assets/index.js:12699:82
workLoopSync@http://127.0.0.1:7575/assets/index.js:12554:60
renderRootSync@http://127.0.0.1:7575/assets/index.js:12537:13
performWorkOnRoot@http://127.0.0.1:7575/assets/index.js:12185:46
performWorkOnRootViaSchedulerTask@http://127.0.0.1:7575/assets/index.js:13297:26
performWorkUntilDeadline@http://127.0.0.1:7575/assets/index.js:1649:60
EventHandlerNonNull*assets/index.js/<@http://127.0.0.1:7575/assets/index.js:1772:11
assets/index.js/<@http://127.0.0.1:7575/assets/index.js:1889:9
assets/index.js@http://127.0.0.1:7575/assets/index.js:1890:7
__require@http://127.0.0.1:7575/assets/index.js:5:50
@http://127.0.0.1:7575/assets/index.js:78707:16



[2025-03-15T03-30-42+00-00_coq-to-lean_nPeo6LDzw27tGUg9Sz4RLN.eval.tar.gz](https://github.com/user-attachments/files/19257688/2025-03-15T03-30-42%2B00-00_coq-to-lean_nPeo6LDzw27tGUg9Sz4RLN.eval.tar.gz)

Seeing this again:

<img width="693" alt="Image" src="https://github.com/user-attachments/assets/2b746f24-363d-4e5d-9670-6e3cf3cc2b5e" />

[2025-03-16T18-51-35+00-00_coq-to-lean_TABNEqsaLRnoGyiSekUyMH.eval.tar.gz](https://github.com/user-attachments/files/19274246/2025-03-16T18-51-35%2B00-00_coq-to-lean_TABNEqsaLRnoGyiSekUyMH.eval.tar.gz)
Repository: UKGovernmentBEIS/inspect_ai
Issue: JS Console error in `inspect view`: Each child in a list should have a unique "key" prop.
Root cause: Validation Gap

I don't see anything visibly broken from this, but there's plausibly a bug lurking somewhere:

hook.js:608 Each child in a list should have a unique "key" prop.

Check the render method of `ANSIDisplay`. See https://react.dev/link/warning-keys for more information. Error Component Stack
    at ANSIDisplay (index.js:23751:15)
    at div (<anonymous>)
    at div (<anonymous>)
    at TabPanel (index.js:58889:7)
    at div (<anonymous>)
    at TabPanels (index.js:58879:26)
    at TabSet (index.js:58772:7)
    at SampleDisplay (index.js:73105:7)
    at div (<anonymous>)
    at div (<anonymous>)
    at InlineSampleDisplay (index.js:73653:7)
    at SamplesTab (index.js:74818:15)
    at div (<anonymous>)
    at TabPanel (index.js:58889:7)
    at div (<anonymous>)
    at TabPanels (index.js:58879:26)
    at TabSet (index.js:58772:7)
    at div (<anonymous>)
    at div (<anonymous>)
    at WorkSpaceView (index.js:76740:7)
    at WorkSpace (index.js:76894:9)
    at div (<anonymous>)
    at AppErrorBoundary (index.js:21835:9)
    at App (index.js:77822:12)

[2025-03-15T18-10-55+00-00_coq-to-lean_cVPifQF25ocKgzim9vbCb5.eval.tar.gz](https://github.com/user-attachments/files/19265064/2025-03-15T18-10-55%2B00-00_coq-to-lean_cVPifQF25ocKgzim9vbCb5.eval.tar.gz)
Repository: UKGovernmentBEIS/inspect_ai
Issue: JS Console Error in `inspect view`: TypeError: Cannot read properties of undefined (reading 'replace')
Root cause: External Dependency Breakage

I don't see any obvious failures from this, but when I click "sample" when viewing [2025-03-15T18-10-55+00-00_coq-to-lean_cVPifQF25ocKgzim9vbCb5.eval.tar.gz](https://github.com/user-attachments/files/19265067/2025-03-15T18-10-55%2B00-00_coq-to-lean_cVPifQF25ocKgzim9vbCb5.eval.tar.gz), I see in console

index.js:65751 TypeError: Cannot read properties of undefined (reading 'replace')
    at htmlEscape (index.js:65988:21)
    at HtmlFormatter.formatValue (index.js:65879:29)
    at HtmlFormatter.format_added (index.js:65934:14)
    at HtmlFormatter.recurse (index.js:65747:26)
    at index.js:65762:16
    at HtmlFormatter.forEachDeltaKey (index.js:65808:11)
    at HtmlFormatter.formatDeltaChildren (index.js:65761:14)
    at HtmlFormatter.format_node (index.js:65929:14)
    at HtmlFormatter.recurse (index.js:65747:26)
    at index.js:65762:16

Thanks for reporting this. I've noticed this as well and it appears it is caused by this issue:

https://github.com/benjamine/jsondiffpatch/issues/367

Specifically in the example eval we are generating a diff of an array that ends with an undefined value (and the value is subsequently removed). So far I have noticed no other side effects aside from the console logging, but will leave open to track.

This has been fixed upstream in jsondiffpatch and we are now running the version with the fix (0.7.3)
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view`: score is improperly formatted?
Root cause: Validation Gap

This is not particularly readable:
<img width="1498" alt="Image" src="https://github.com/user-attachments/assets/f63c8338-587c-4b63-bb4c-9938fc32db23" />

What's going on?

[2025-03-15T18-10-55+00-00_coq-to-lean_cVPifQF25ocKgzim9vbCb5.eval.tar.gz](https://github.com/user-attachments/files/19265070/2025-03-15T18-10-55%2B00-00_coq-to-lean_cVPifQF25ocKgzim9vbCb5.eval.tar.gz)

The issue here is that we're attempting to display an error and not managing the string overflow well at all. Fix incoming.
Repository: UKGovernmentBEIS/inspect_ai
Issue: 401 trying to open the logviewer
Root cause: Validation Gap

The Log Viewer seems to start correctly, but shows a 401 when opened in the browser.

![Image](https://github.com/user-attachments/assets/e62f50e7-da7e-4e1a-93f5-62b2fed62038)

```powershell 
PS C:\Users\TrueM\Repos\AISI> inspect view
Inspect View: file://C:/Users/TrueM/Repos/AISI/logs
======== Running on http://127.0.0.1:7575 ========
(Press CTRL+C to quit)
```

Looks fine in the `Inspect` VSCode plugin:
![Image](https://github.com/user-attachments/assets/971fc3bf-dc55-4f09-a987-e2067cd741d8)

`curl` doesn't get a 401.
```powershell
PS C:\Users\TrueM\Repos\AISI> curl http://127.0.0.1:7575/
<!doctype html>
<html lang="en" data-bs-theme="light">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Inspect View</title>
    <link rel="icon" href="./assets/favicon.svg" />

    <script>
      // Forward the theme (dark or light) onto the root html element
      // this parameter will tend to appear when the view is hosted within
      // an iframe (e.g. in vscode)
      const urlParams = new URLSearchParams(window.location.search);
      const theme = urlParams.get("inspectLogviewThemeCategory");
      if (theme) {
        document.documentElement.setAttribute("data-text-highlight", theme);
        document.documentElement.setAttribute("data-bs-theme", theme);
      }
    </script>
    <script type="module" crossorigin src="./assets/index.js"></script>
    <link rel="stylesheet" crossorigin href="./assets/index.css">
  </head>

  <body style="min-width: 450px">
    <div id="app"></div>
  </body>
</html>
```

Setup: Tutorial tasks (https://inspect.ai-safety-institute.org.uk/tutorial.html)
OS: Windows 11

It looks like the viewer isn't able to access a remote log file (likely because of S3 permissions)?  Are you sure that it is pointed at the same log directory? You could be certain by passing a `--log-dir` explicitly to the `inspect view` command...

Everything (should) happen on my machine.
Still happens with the log dir set.
``` 
PS C:\Users\TrueM\Repos\AISI> inspect view --log-dir ./logs
Inspect View: file://C:/Users/TrueM/Repos/AISI/logs
======== Running on http://127.0.0.1:7575 ========
(Press CTRL+C to quit)
PS C:\Users\TrueM\Repos\AISI> inspect view --log-dir ./logs_temp
Inspect View: file://C:/Users/TrueM/Repos/AISI/logs_temp
======== Running on http://127.0.0.1:7575 ========
(Press CTRL+C to quit)
```
When `logs_temp/` is empty, I get a message that the directory is empty - so that's working. As soon as I add a log I get the error.

We do return this when we think the request is attempting to "escape" the log directory:

```python
 # validate log file requests (must be in the log_dir
 # unless authorization has been provided)
 def validate_log_file_request(log_file: str) -> None:
     if not authorization and (not log_file.startswith(log_dir) or ".." in log_file):
         raise web.HTTPUnauthorized()
```

@dragonstyle I wonder if there is an issue on Windows in terms of the way we store or compute the path prefix?

I am reproducing it on windows - looking now!