zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: In general: how can we integrate inspect-ai into google colab notebooks?
Root cause: Documentation Deficiency

Anybody here know how to integrate inspect-ai into colab notebooks? We at [compassionml.com](http://compassionml.com/)
 are really excited to develop some new evals and want it to be
inspect-ai compatible as well as runnable on a single colab notebook
(for ease of experimentation). But when I tried to integrate it, using
the only implementation I can find on the internet (https://arena-chapter3-llm-evals.streamlit.app/ and their "Running Evals with Inspect" notebook in one of their dropdowns, [https://colab.research.google.com/github/callummcdougall/ARENA_3.0/blob/main/chapter3_llm[…].3_Running_Evals_with_Inspect_solutions.ipynb?t=20250418](https://colab.research.google.com/github/callummcdougall/ARENA_3.0/blob/main/chapter3_llm_evals/exercises/part3_running_evals_with_inspect/3.3_Running_Evals_with_Inspect_solutions.ipynb?t=20250418)), I got an endless series of incompatibilities with actually displaying the eval logs in colab, even with so much help from o3!

I think your best bet for showing logs in colab is to save all of your logs to Amazon S3 (https://inspect.aisi.org.uk/eval-logs.html#sec-amazon-s3) and then just run `inspect view` on the S3 bucket in another browser. You might need to manually refresh the view for new log files but other than that this should work well. I don't think it's realistic to try to get logs to render within colab itself as they really need a fully custom webapp.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Allow users to set the cache path
Root cause: Unimplemented Feature Gap

Controlling the cache path is useful if, for example, we want to ensure the cache is stored in a particular mounted volume.

Unfortunately, it looks like this might require some significant changes to how the caching code is currently set up in `inspect_ai/model/_cache.py`?

A suggestion for an API that would at least be backwards-compatible is that instead of being able to pass a `CachePolicy` to `generate`, we can pass a more general `CacheConfig`.



Or perhaps we could enable setting an environment variable for this? (e.g. `INSPECT_CACHE_PATH`).

@seddy-aisi What do you think?

That certainly would make the change much easier to implement

This is resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/184
Repository: UKGovernmentBEIS/inspect_ai
Issue: Request: log viewer - show actual formatted prompt instead of prompt-template
Root cause: Unimplemented Feature Gap

The log viewer, when viewing a sample, has the 'messages' view. This is nice, but it'd be nicer if it showed the text that actually got sent to the model when calling generate. Instead it shows text from some earlier step before the template has actually been filled in.

Also, it'd be nice if there was line-wrap for the json view. Is this an option and I'm just not finding it?

Also, the formatting for the transcript view gets all messed up if there are multiple backticks like ``` in the message. Screenshot of this attached.

![Image](https://github.com/user-attachments/assets/24eeffb3-8494-44e0-9857-abbdf0fdc048)

The transcript view will have the message actually sent to the model.

@dragonstyle I thought our JSON view should already have been line wrapping -- is that another instance of the issue we resolved yesterday?

See also the note about multiple ``` 

The transcript view will have the message actually sent to the model.

@dragonstyle I thought our JSON view should already have been line wrapping -- is that another instance of the issue we resolved yesterday?

See also the note about multiple ``` 

@nathanneuro I'm not seeing any issues with JSON wrapping (the places I'm checking are wrapping, though perhaps I'm not checking the same places you're seeing the issue). Can you share more info about where you're seeing the JSON not wrapping (screenshot and/or log file are appreciated if you can share).

Definitely a bad rendering issue with code cells that are being rendered in that 'sample summary' at the top. I'll have a look at that now.

![Image](https://github.com/user-attachments/assets/7137cfae-c4d0-4239-9baa-19512bda38ca)

json failing to wrap

Thanks, just reproduced it. Will have a fix here shortly!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Occasional `Page.evaluate: Execution context was destroyed, most likely because of a navigation` when using web browser tool.
Root cause: Resource Mishandling

Occasionally when a page performs a navigation in response to a `submit`, the tool will return `Page.evaluate: Execution context was destroyed, most likely because of a navigation` to the model. This is caused by our code not being mindful of the potential navigation and requesting info on the page during the navigation. I had a nearly identical issue with mouse clicks, and I fixed it with `_click_and_await_navigation` [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/afcedce44792b9236ab8db41099eaa42b50a69e4/src/inspect_tool_support/src/inspect_tool_support/_remote_tools/_web_browser/playwright_page_crawler.py#L251C15-L251C42).

Essentially the same logic must be applied for `submit` operations.

Like all race conditions, this bug does not occur all the time. I've been able to reproduce the issue roughly 1 out of 10 attempts when attempting a search at `www.google.com` when Google's anti-bot code kicks in and they navigate to a URL like [this](https://www.google.com/sorry/index?continue=https://www.google.com/search%3Fq%3Dzappa%26sca_esv%3Da24eb7beebb63ee4%26source%3Dhp%26ei%3Dzq8HaPTVB9nI0PEPhvm5uQ4%26iflsig%3DACkRmUkAAAAAaAe93o8CUnJeq1gnsLxR8hJ6fGBx5rdQ%26ved%3D0ahUKEwj09MvS9OuMAxVZJDQIHYZ8LucQ4dUDCBE%26uact%3D5%26oq%3Dzappa%26gs_lp%3DEgdnd3Mtd2l6IgV6YXBwYTIKEAAYAxjqAhiPATIKEAAYAxjqAhiPATIKEAAYAxjqAhiPATIKEAAYAxjqAhiPATIKEAAYAxjqAhiPATIKEAAYAxjqAhiPATIKEAAYAxjqAhiPATIKEAAYAxjqAhiPATIKEAAYAxjqAhiPATIKEAAYAxjqAhiPAUiBIFDeH1j8H3ACeACQAQCYAQCgAQCqAQC4AQPIAQD4AQGYAgKgAgeoAgqYAwTxBUr8blwW_dLekgcBMqAHALIHALgHAA%26sclient%3Dgws-wiz%26sei%3D168HaO-CLevR1sQPuIiK8Ag&q=EgTQ_D7KGNjfnsAGIjBRbVO9F4SUxBi93AUHxDZavx-UBuNLZzBAl8JZvabGSw2HdSGtg3j71UElxmDDu9gyAnJSWgFD).
Repository: UKGovernmentBEIS/inspect_ai
Issue: max_retries does not seem to default of 5
Root cause: Configuration Error

max_retries is supposed to default to 5:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/645a710870e00a9e413b148deda6f16f6d81c594/src/inspect_ai/model/_generate_config.py#L110-L114

--- 


Easy to reproduce as follows:

Call a nonexistent model.
Override the provider to always retry

```
    def should_retry(self, ex: Exception) -> bool:
        return True
````

Set `log_level="http"`. 

I get retries beyond 5, e.g.:
```
claude-3-7-sonnet-20250219-fakemodel               
retry 6 after waiting for                          
202.3628915051979                    
```

This comment

https://github.com/UKGovernmentBEIS/inspect_ai/blob/645a710870e00a9e413b148deda6f16f6d81c594/src/inspect_ai/model/_model.py#L554-L555

says "try forever", but in fact it sets `stop_after_attempt(config.max_retries)` depending on the `max_retries`, which is confusing

I’m seeing this behaviour in 0.3.82 (the links above are to 645a710, but the same comments exist in 0.3.82 too)

I think the docs are wrong here (they are likely stale from before we made all the change to retry behavior).


Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1756
Repository: UKGovernmentBEIS/inspect_ai
Issue: Exponential backoff waits longer than 30 minutes

This comment says it will wait no longer than 30 minutes

https://github.com/UKGovernmentBEIS/inspect_ai/blob/645a710870e00a9e413b148deda6f16f6d81c594/src/inspect_ai/model/_model.py#L553-L557


But I often see much longer retry periods:

```
                    HTTP     -> gemini-2.5-pro-preview-03-25      _model.py:1237
                             retry 19 after waiting for                         
                             19288.136545209483       
```

i.e. over 5 hours



I’m seeing this behaviour in 0.3.82 (the links above are to https://github.com/UKGovernmentBEIS/inspect_ai/commit/645a710870e00a9e413b148deda6f16f6d81c594, but the same comments exist in 0.3.82 too)

As far as my understanding of tenacity goes, the `max` parameter here should be capping at 1800 seconds: https://github.com/UKGovernmentBEIS/inspect_ai/blob/645a710870e00a9e413b148deda6f16f6d81c594/src/inspect_ai/model/_model.py#L570C58-L570C65


Yes, I'm confused too, but it's clearly retrying somewhere with a longer delay 

That retry message is in the before_sleep callback so definitely coming from tenacity. If you look at the docs it reads to me like this number may be cumulative:

```python
 #: Time spent sleeping in retries
 self.idle_for: float = 0.0
```

I think maybe we want `upcoming_sleep` instead:

```python
#: Next sleep time as decided by the retry manager.
self.upcoming_sleep: float = 0.0
```

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1760
Repository: UKGovernmentBEIS/inspect_ai
Issue: Cannot install in Windows
Root cause: Environment Incompatibility

As an admin open terminal.
Run pip install inspect-ai

![Image](https:/

![Image](https://github.com/user-attachments/assets/590fcef0-583f-456b-aed2-74e9c823f9b6)

/github.com/user-attachments/assets/f650c855-45a5-46cb-9cd1-ff0c0a27f1a3)

Run into circular dependency issues: Could not install packages due to an OS error. It cannot find another package within the inspect-ai folder called xce4screensaver.xml. I have checked and all the folders are there but not the final xml file

@epatey It looks like we might me running afoul of some windows path length issues here? 

@darkness8i8 What happens if you enable long paths as described here: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry

@epatey We should probably see if we can exclude these files from the PyPI package since they are really only needed for building images.


I don't want to go changing registry variables for testing this. I will not be able to run troubleshooting steps on this, but it is a very easy problem to replicate

This eliminates the _resources directory which should get the maximum path well under the limit: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1776

You won't see this fix when installing until it goes to PyPI later on this week.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [CI] `mypy` fails due to Google python-genai update
Root cause: External Dependency Breakage

The [Google python-genai package](https://github.com/googleapis/python-genai) just updated from v11.0 to v1.12.1 ([diff](https://github.com/googleapis/python-genai/compare/v1.11.0...v1.12.1)).

This seems to be the root cause for the various CI mypy failures
- https://github.com/UKGovernmentBEIS/inspect_ai/actions/runs/14652788726/job/41122226313
- https://github.com/UKGovernmentBEIS/inspect_ai/actions/runs/14651973187/job/41119659965
- https://github.com/UKGovernmentBEIS/inspect_ai/actions/runs/14651932415/job/41119524157

Validated this locally by installing the old version of python-genai and seeing the mypy errors go away.

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1758
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect view` does not permit displaying webpages
Root cause: Environment Incompatibility

I tried to run `inspect view` with only [2025-04-26T08-23-31+00-00_coq-to-lean_4mDyvu9ritvSkAyTDLox5o.eval.tar.gz](https://github.com/user-attachments/files/19921307/2025-04-26T08-23-31%2B00-00_coq-to-lean_4mDyvu9ritvSkAyTDLox5o.eval.tar.gz) (unpacked to a `.eval` file) in `logs/`, but when I navigate to http://127.0.0.1:7575/, Chrome and Firefox just show an untitled, empty page loading forever.  This is with inspect ai 0.3.91

I downloaded the log file, put it in a logs directory by itself, and ran:

```python
inspect view --log-dir ./logs
```

That did successfully render for me (Chrome on MacOS). 

Could you provide any other details? One other thing to do is open the browser developer tools and see if there are any JS errors in the console as those may provide additional hints as to what's going wrong.

Hm, it seems to work now.  Previously it was not showing anything in console and the html was just `<html><body></body></html>` or something.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add the ability to ctrl+f to the extension

Currently the web interface allows for searching through contents, but the extension doesn't - this would be a useful feature, particularly for long agent trajectories.

@dragonstyle I think would need to do a bespoke implementation of browser find for this (we did this for the RStudio Help pane, it's not that difficult). Docs for the API here: https://developer.mozilla.org/en-US/docs/Web/API/Window/find

This is now implemented in VS Code: https://github.com/UKGovernmentBEIS/inspect_ai/pull/189

(note this is actually implemented in inspect itself, so no need to update the vscode extension, but updating to latest developer version of inspect will include the new behavior in vscode).
Repository: UKGovernmentBEIS/inspect_ai
Issue: `read_eval_log_sample_summaries` returns empty ModelUsage
Root cause: Documentation Deficiency

I'm trying to use the new `read_eval_log_sample_summaries` method from #1762 but it seems to always return an empty ModelUsage.

I created a test in a draft PR #1769 that fails and demonstrates the issue:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/18adf074aec763d9c50fce692e1335e86c686b63/tests/log/test_log_summaries.py#L34-L43


When running this test, it fails because the model usage in the summary is empty:
```
>               assert(summary.model_usage == full.model_usage)                                      
E               AssertionError: assert {} == {'openai/gpt-..._tokens=None)}                          
E                                                                                                    
E                 Right contains 1 more item:                                                        
E                 {'openai/gpt-4o-mini': ModelUsage(input_tokens=287, output_tokens=202, total_tokens=489, input_tokens_cache_write=None, input_tokens_cache_read=None, reasoning_tokens=None)}
E                 Use -v to get more diff                                                            
                                                                                                     
test_log_summaries.py:56: AssertionError 
```

The problem may be that sample summaries are created at log file write time so no logs created prior to today will have model usage (the only way to do that would be to read the full samples). You can however convert existing logs to logs that have the full summaries by round tripping them.

Ah, I see - I misunderstood how this was implemented and made a mistake when trying to test against a new log. Sorry about that! After regenerating logs and testing it behaves how I'd expect (not in the unit test, just in my actual scripts).

I do think it's a bit surprising that this method returns all the info as expected _except_ ModelUsage and metadata for these older logs. Is there versioning in the log files?  If so it might be nice to raise a warning when these fields are accessed through the summary method.

There is versioning but it's a hard break (i.e. if we label logs as version 3 then any version of Inspect that only knows about version 2 will refuse to read them). I wouldn't break all old versions just for this. 

Hm, perhaps minor version numbers in there could be of some use in the future to indicate non-breaking changes.

Anyway - seems like this is really a non-issue so I'm going to close it. Thanks for talking it through with me!
Repository: UKGovernmentBEIS/inspect_ai
Issue: NaN scores prevent sorting of samples.
Root cause: Validation Gap

Hi!

When I use NaNs to score samples, it breaks the sorting by this metric.

Thanks for taking a look and for your quick response to the previous issue!

![Image](https://github.com/user-attachments/assets/bea788ab-c5cb-4fb0-b697-d840d114c667)

![Image](https://github.com/user-attachments/assets/f56fc7a8-fb28-44d3-aed5-668f999b1e8b)

<details><summary>Reproduction</summary>

```python
import numpy as np
from inspect_ai import Task
from inspect_ai import task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import Score
from inspect_ai.scorer import Scorer
from inspect_ai.scorer import Target
from inspect_ai.scorer import accuracy
from inspect_ai.scorer import scorer
from inspect_ai.solver import TaskState
from inspect_ai.solver import generate


@scorer(metrics=[accuracy()])
def _scorer() -> Scorer:
    """Create a multi-scorer that avoids ordering issues of separate solvers.

    The problem is that the solvers are executed in parallel and the order of the
    solvers is not guaranteed. This can lead to the overlap solver being executed before
    the validation solver. This can lead to a KeyError if the metadata is not set yet.

    """
    i = 0

    async def score(state: TaskState, target: Target) -> Score:  # noqa: ARG001
        nonlocal i
        score = np.nan if i == 0 else 1
        i += 1
        return Score(value=score)

    return score


@task
def hello_world():
    return Task(
        dataset=[
            Sample(
                input="Just reply with Hello World",
                target="Hello World",
            ),
            Sample(
                input="Just reply with Hello World",
                target="Hello World",
            ),
        ],
        solver=[
            generate(),
        ],
        scorer=_scorer(),
    )

```


</details>
Repository: UKGovernmentBEIS/inspect_ai
Issue: Question about parallel scoring.
Root cause: Documentation Deficiency

Hi,

I tried following your advice on running the solving and scoring part of evaluations separately during development.

Perhaps specifically related to my evaluations is that I need to retrieve additional data for each sample, which is required for solving and scoring. Some samples share the same data, allowing me to reduce outgoing requests by caching.

When I run the whole evaluation, solving and scoring, caching works since I am running only 10 samples in parallel. In the solver, I fetch the additional data and store it, allowing me to access it during scoring.

When I run the scoring separately, I need to fetch the data in the scorer. Now, it seems to me like all samples are scored in parallel, which prevents me from using the cache effectively, and the API I am fetching from is overwhelmed with requests, leading to timeout errors.

I see two ways forward at the moment.

1. I need a parameter like `--max-samples` for `inspect score` to limit the number of simultaneously evaluated samples. I suppose this exists for model calls, as otherwise `model_graded_qa` and similar scorers would send too many requests simultaneously.
2. I could save the data while reading the dataset or solving, not in the ephemeral store, but somewhere where it is persisted in the log file, like the sample metadata. I don't know if this is always feasible, but it would work in my case.

How can I best utilize inspect-ai in this situation?

You're right that the score command has no bounding on parallelism - it is relying upon other bounding to provide gates (and if there are none, it will run very parallel). I think your best bet would be to gate access to API directly, which would allow you control of how frequently it is accessed in any context. You would use something like:

https://inspect.aisi.org.uk/reference/inspect_ai.util.html#concurrency

Would this work for you?

Pretty cool utility. I was unaware of semaphores and how they can restrict access to shared resources. Thanks for showing me, @dragonstyle!

I modified my code to this, and it works.

```python
def scorer() -> Scorer:
    async def score(state: TaskState, target: Target) -> Score:
        async with concurrency(name="populate_store", concurrency=10):
            await _populate_store(state)

        ...
```

Now, I also found the [page in the docs](https://inspect.aisi.org.uk/parallelism.html#rest-apis) 😅.

Ahh perfect- sorry that I didn't link to that as well! Glad this solved your issue...
Repository: UKGovernmentBEIS/inspect_ai
Issue: Question: Best practice for registering and looking up answers

Hi,

It seem to be standard practice for agent implementations to copy the final answer (i.e. the argument to the submit tool) to `state.output.completion`, overriding the `message.text` of the first choice in the last chat message.

Likewise scorers usually take the answer from `state.output.completion` instead of looking up the answer argument to the `submit` tool_call.

Is that the API contract? Or just something that have gradually evolved to be common practice?

It seems a bit funny to have to override the `message.text`, but it looks like that is necessary to be compatible with the rest of the eco-system.

The `react()` agent now appends the submit tool answer to the `message.text`: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/agent/_react.py#L197-L200

I think this should be considered the best practice. In this worldview the submission is simply a specially flagged part of the assistant's final message (placed at the end on its own line, where nearly all scorers look for answers). 

Note that the use of a `submit()` tool is a particular choice for some agents so scorers cannot in general rely on this (note that most often scorers need to be written to work with solvers that may or may not be agents or use tools).

Okay, thanks.

Perhaps you could consider adding a set of convenience methods to AgentState to make this explicit going forward (`get_answer()`/`set_answer()` perhaps)?  (Or if you just let me know that you would accept such a PR, I will be happy to do it).

We actually added an `answer` to `AgentState` at one point but removed it -- the problem is that there is already a concept of an `answer` in the system -- scorers extract an `answer` from the completion and tag it as such. The idea is that LLM output is inherently a bit unstructured so scorers must deal with this (honestly, even an `answer` relayed via the `submit()` function can have extra crap in it). We don't want to imply to scorers that they are off the hook for distilling the `answer` by having an explicit `answer` on the task/agent state.
Repository: UKGovernmentBEIS/inspect_ai
Issue: API Error: Function name 'python' is reserved in o4-mini model
Root cause: External Dependency Breakage

When sending a request to the **o4-mini** model with a custom python function in tools, the API returns:

```json
  {
    "error": {
      "message": "The function name 'python' is reserved for use by this model.",
      "code": 400
    }
  }
```

Expected Behavior:

**Revise the default `python` function names**

This should be resolved with this commit from last week: https://github.com/UKGovernmentBEIS/inspect_ai/commit/97cbc8d23f33eb9fbc4d5f2e9741bc521916f478

Could you try updating to the latest version to see if that indeed has resolved it for you?

Thank you for your suggestion! I will try updating to the latest version.

> This should be resolved with this commit from last week: [97cbc8d](https://github.com/UKGovernmentBEIS/inspect_ai/commit/97cbc8d23f33eb9fbc4d5f2e9741bc521916f478)
> 
> Could you try updating to the latest version to see if that indeed has resolved it for you?

I have updated to the latest version, 0.3.93, but unfortunately, the issue persists. I have reviewed the file src/inspect_ai/model/_openai_responses.py as per the changes in commit 97cbc8d, and it appears to have been modified correctly, but it does not seem to resolve the issue.

Could you please suggest any other potential solutions or areas I might need to investigate further? If necessary, I can provide additional error logs or details to help diagnose the problem.

Thank you for your assistance and support.

I think the best course forward is for you to create a minimal reproducible example of the code you are using to get this error (it's possible there is some subtle difference in how you are specifying things). If you could send us that code + the log file then we can get to the bottom of things.