zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] Anthropic providor fails to retry on server overloaded errors sometimes
Root cause: Validation Gap

This is a pretty hard bug to reproduce, as most of the time inspect properly retries when anthropic sends server overload errors.

```
[08/12/25 23:02:13] WARNING  Sample error (id: 42, epoch: 1): {'type': 'error', 'error': {'details': None, 'type': 'api_error', 'message': 'Overloaded'}})           run.py:612
```

I think the source of this bug is this bit of code from the anthropic provider implementation:

```python
 @override
    def should_retry(self, ex: BaseException) -> bool:
        if isinstance(ex, APIStatusError):
            # for unknown reasons, anthropic does not always set status_code == 529
            # for "overloaded_error" so we check for it explicitly
            if isinstance(ex.body, dict):
                if "overloaded_error" in str(ex.body):
                    return True

            # standard http status code checking
            return is_retryable_http_status(ex.status_code)
        elif httpx_should_retry(ex):
            return True
        elif isinstance(ex, APIConnectionError | APITimeoutError):
            return True
        else:
            return False
```

My best guess is that on rare occasion anthropic sends an overload error without "overloaded_error" in the response.

Thanks for reporting! This PR will catch that case: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2280
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] normalize_number bug
Root cause: Validation Gap

Ran into the following error when running the GSM eval. Seems like not all edge cases are handled by the str_to_float.

```
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/eval.py", line 513, in eval_async
  |     await eval_run(
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/run.py", line 249, in eval_run
  |     return await run_single(task_run_options, debug_errors)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/run.py", line 282, in run_single
  |     raise ex.exceptions[0]
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/run.py", line 274, in run_task
  |     result = await task_run(tasks[index])
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py", line 354, in task_run
  |     sample_results = await tg_collect(
  |                      ^^^^^^^^^^^^^^^^^
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_util/_async.py", line 72, in tg_collect
  |     raise ex.exceptions[0] from None
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_util/_async.py", line 60, in run_task
  |     result = await funcs[index]()
  |              ^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py", line 349, in run_sample
  |     async with anyio.create_task_group() as tg:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_util/text.py", line 155, in str_to_float
    |     base_value = float(base_part)
    |                  ^^^^^^^^^^^^^^^^
    | ValueError: could not convert string to float: '31800.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.0'
```

Thanks for reporting....more flexible parsing implemented here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2279
Repository: UKGovernmentBEIS/inspect_ai
Issue: running cheap evals (gsm, ifeval) is bottlenecked by calling inspect.stack() in span function
Root cause: Algorithmic Error

I am using inspect to evaluate small models on some cheap evals like gsm and ifeval. I noticed the evals running slower than expected, and I checked py-spy, which showed that an inordinate amount of time was spent in the `inspect` (not `inspect_ai`) module.

In particular, most of the time was in the line `frame = inspect.stack()[1]` in `src/inspect_ai/util/_span.py`. The following replacement sped up my eval by a factor of 2.

```
- frame = inspect.stack()[1]
- caller = f"{frame.function}() [{frame.filename}:{frame.lineno}]"
+ caller = "optimized"
```

It would be nice if there were an option to disable this caller tracking functionality to improve performance.

Thanks for pointing this out! Resolved here by deferring the check until we need it (which should be very rare): https://github.com/UKGovernmentBEIS/inspect_ai/pull/2278

Thanks for the quick fix!
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Bug] Missing MPC dependency
Root cause: Environment Incompatibility

Hi, we came across this bug whilst working with inspect eval!

From a clean environment, attempting to import the Inspect AI fails because a required dependency, mcp, is not automatically installed.  
(See: https://github.com/UKGovernmentBEIS/inspect_evals/actions/runs/16923484701/job/47955552056?pr=449)

This error can be replicated by pip installing inspect AI (a version past commit 73ebd17 where MCP support is introduced). Trying to import inspect AI returns a` ModuleNotFoundError: No module named 'mcp'` (I think its this line https://github.com/UKGovernmentBEIS/inspect_ai/blob/24356a812cdeaafabdbd32afc739772cb9772e55/src/inspect_ai/tool/_mcp/_local.py#L10C1-L10C25)


Thanks for reporting, resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/1aa4f015bc0aa664e8dcd6d37f0ac4d6770134ae
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add gemini thinkingBudget
Root cause: Configuration Error

Google's Gemini 2.5 models enable generate long reasoning sequences  by default. This feature can be controlled by the thinking_budget parameter (https://ai.google.dev/gemini-api/docs/thinking). Setting thinking_budget to a number limits the tokens, and 0 disables it. Adding this parameter to the fCLI would allow users to manage the chain-of-thought feature for this model, providing more control over response generation and potentially speed.

If there is an existing way to pass this parameter to an endpoint please point me to it!

Indeed you can! Just use `--reasoning-tokens`: https://inspect.aisi.org.uk/reasoning.html#google-gemini

Thanks @jjallaire 

This does not seem to be working for me. For instance, I am running gemini-2.5-pro with 

```
python inspect eval-set <my taskss> ... --max-tokens 4096 --reasoning-tokens 2048
```

And I can see in the logs that in many cases, we log more than 2048 reasoning tokens, e.g.

```
"usage": {
      "input_tokens": 132,
      "output_tokens": 4095,
      "total_tokens": 4227,
      "reasoning_tokens": 3538
},
```

In my setup I am calling this from the CLI using a litellm proxy, which implements the OpenAI API, so my model name is like `openai-api/litellm/gemini-2.5-pro` for instance, in case that could be an issue?
Repository: UKGovernmentBEIS/inspect_ai
Issue: GitHub Pages cannot host Log Viewer
Root cause: Environment Incompatibility

I tried hosting the log viewer on github pages in this [repo](https://github.com/jeqcho/caste-bias-gpt-5) at this [link](https://chojeq.com/caste-bias-gpt-5/#/logs/). I encountered this error.

<img width="1108" height="247" alt="Image" src="https://github.com/user-attachments/assets/9dfa3ee7-db52-4780-ae7f-8ee72a289525" />

I [added logging](https://github.com/jeqcho/caste-bias-gpt-5/commit/33345a1c3a318ef49a5d9b4e6c9891a86ca318b7) to `index.js` and the error I got is
```
Error: Offset is outside the bounds of the DataView
```

I understand that the server must support HTTP range queries, but I note that the github pages server does return 206 so I believe it supports range queries.

## Steps to reproduce

Minimal repo to reproduce bug [here](https://github.com/jeqcho/gh-pages-inspect-bug). Corresponding github pages link [here](https://chojeq.com/gh-pages-inspect-bug/).

Thanks for providing such a great repro case- we did have issues with GH pages specifically, though the change to fix this was made about a month ago. I just tested using your repo and here is my deployment, which appears to be working correctly:

https://dragonstyle.github.io/gh-pages-inspect-bug/#/logs/

Can you double check the version of inspect you're using to be sure you're using at least `0.3.112` and that browser caching of older deployments aren't in play? (Note that if you do need to update your inspect version, you need to re-generate the bundle using `inspect view bundle` to incorporate the rebuilt js file for the updated viewer).

lmk if version might explain this. If not, we likely will need to try to debug what is different between the bundle commands and repro cases...



Thanks for looking into this! I double checked that my inspect version is `0.3.122`. The one thing that is different is that I have it on a custom domain. Do you know if this has caused any errors?

I tried downgrading inspect to `0.3.112` and ran bundle again https://github.com/jeqcho/gh-pages-inspect-bug/commit/c803c148a49623817612a555d1c6a8deee477283, but it still says error after 5 attempts.

Later versions should also work, so version doesn’t seem likely to be the cause. The custom domain is suspicious as it is possible that there is some configuration issue with range requests with the custom domain. 

Could you try disabling the custom domain and see if that resolves it? If it did, we could focus on the domain configuration as a source…

Disabled custom domain and cleared cache but still no luck. (I re-enabled the custom domain now since some of my other links in other projects will otherwise break)

We have tested it locally and it has been working, so not sure what more there is to do here without some additional debugging about the issue. I'm going to go ahead and close this, but if you think there is more we should be doing please re-open and we'll dig in!
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] tool calls dropped when combining consecutive assistant messages
Root cause: Algorithmic Error

I'm running some evals that require prefilling assistant messages where the model might then also generate tool calls. Currently inspect strips away all tool calls when combining these messages.

This seems to be a known [issue](https://github.com/UKGovernmentBEIS/inspect_ai/blame/e8466f8df672663dea4639414dc3bb9328e15c85/src/inspect_ai/model/_model.py#L1410), but the TODO hasn't been updated in 5 months. It would be great if this could be patched.

I'm happy to make a PR with a fix if that would be useful.

cc @epatey do you have any further thoughts on this? would it be okay for @kaifronsdal to make a PR here?

Following up on this
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Question] Passing Azure AD Token and Custom default_headers (e.g projectID) when using openai/azure
Root cause: Unimplemented Feature Gap

I'm trying to use inspect AI to run the Tutorial with an Azure OpenAI deployment, authenticating via Azure AD (not through API Key or Azure Credentials). I get the Azure AD token from an auth URL, and need to pass in my projectID as well.

Whenever i try to pass in the base credentials `AZUREAI_OPENAI_BASE_URL` and `AZUREAI_AUDIENCE` and `AZUREAI_OPENAI_API_VERSION` it defaults to using DefaultAzureCredentials which is not what I need.

Would building my custom ModelAPI be the best choice in this scenario ?



No, I think if there is a better/improved approach in our OpenAI provider then maybe you could give us a PR for it?
Repository: UKGovernmentBEIS/inspect_ai
Issue: Is there an recommended way to check the raw requests send to API (majorly system prompt, tool and messags)
Root cause: Documentation Deficiency

This would be super for debug , e.g. check my own agent implementation  tool setting and Inspect's setting. Sometimes subtle difference in prompt and tool description can make a lot of difference. 

For now I have to either use pdb or monkeypatch some method in inspect. But curious if there is any "blessed" way. 

Yes, within the Inspect View transcript every model call has an "API" tab which you can use to see the raw request and response payloads.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Issues using Agent Bridge with Azure OpenAI endpoint
Root cause: Documentation Deficiency

I'm using an agent bridge followed by a generate solver, but the output of my agent bridge is not showing up in the state used by generate.

The agent bridge uses an AOAI endpoint. I have it set up to connect using `from openai.lib.azure import AsyncAzureOpenAI` and the llm calls are successfully completed and tracked by Inspect AI. When the state is updated with the output from my agent, the model name is switched from "openai/azure/<model>" to "azure/<model>" (as show in the screenshot).

I'm not sure if that is the cause of my issue. I attached a minimal example of using and agent bridge followed by generate, where the bridge output is lost.

[hello_bridge.py](https://github.com/user-attachments/files/21803331/hello_bridge.py)
[hello_inspect.py](https://github.com/user-attachments/files/21803330/hello_inspect.py)

<img width="792" height="187" alt="Image" src="https://github.com/user-attachments/assets/49e8b26f-3b3d-4201-abfc-cca2e491905e" />

The key here is that you need to return both `output` and `messages` in the dict (otherwise the messages are dropped on the floor). So something like this this:

```python
return dict(
    output=output_string,
    messages=sample["input"] + [answer.choices[0].message]
)
```

Thanks, I've attached the updated bridge sample with the messages returned by the agent. It now works as expected!

[hello_bridge.py](https://github.com/user-attachments/files/21825813/hello_bridge.py)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Agent Bridge caching
Root cause: Documentation Deficiency

Is there a way to use Inspect AI's caching for inference taking place in an agent bridge?

The bridge ends up calling the native Inspect model API so AFAIK the cache should work with the bridge as is. That said, if you are bridging to an agent that is calling tools you will very rarely get any benefit from the cache since the trajectory will almost always have subtle variations that invalidate the cache.

I don't see any examples of activating the cache outside of a call to generate. My current workflow starts with the agent bridge, but I'm not sure how to pass a cache parameter to the bridge function or use it since I don't have any generate calls.

In general, I agree that the agent wouldn't be repeating identical inferences, but I'm currently doing a lot of testing, and expect to use the agent bridge to compare variations that won't affect the LLM calls until later in the processing.

I may not be using the agent bridge for its intended purpose. I'm currently using the deprecated bridge() function (and plan to update to the new agent_bridge() function) to connect Inspect AI to an existing system. 

The part I connected for now isn't particularly agentic, but is a proof-of-concept for wrapping a system in Inspect AI to take advantage of scorers/metrics for evaluation.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Using multi_scorer raises AttributeError
Root cause: Documentation Deficiency

```
Traceback (most recent call last)
/opt/anaconda3/envs/<my_env>/lib/python3.10/site-packages/inspect_ai/_eval/task/run.py:182 in task_run
/opt/anaconda3/envs/<my_env>/lib/python3.10/site-packages/inspect_ai/_eval/task/results.py: 27 in eval_results                                                                               
/opt/anaconda3/envs/<my_env>/lib/python3.10/site-packages/inspect_ai/_util/registry.py:179  in registry_info                                                                               

AttributeError: 'function' object has no attribute '__registry_info__'
```
Quick fix: 

Adding:
```
@scorer(metrics=[accuracy(), bootstrap_std()])
```
above line 19 in inspect_ai/scorer/_multi.py:
```
def multi_scorer(scorers: list[Scorer], reducer: ScoreReducer) -> Scorer:
```


The idea behind multi_scorer is that you call it from another scorer (it is itself not intended as a standalone scorer). One reason for this is that we can't know prima facia what the right metrics would be (i.e. is a correct/incorrect score or it is a continuous range which would call for `mean()`. We will definitely add something to do the docs to clarify this.

Thanks for clarifying that. However, when I run something like:
```
@scorer(metrics=[accuracy(), bootstrap_std()])
def custom_multi_scorer(
    scorers=[
    match("end"),
    answer("word"),
    model_graded_fact(),
    ], 
    reducer=majority_vote
  ):
  return multi_scorer(scorers, reducer)
      
@task
def fn_name():
    return Task(
        dataset=example_dataset("theory_of_mind"),
        plan=[
          chain_of_thought(),
          generate(),
        ],
        scorer=custom_multi_scorer,
    )
```
I get:
```
Traceback (most recent call last)
/opt/anaconda3/envs/llm-awareness/lib/python3.10/site-packages/inspect_ai/_eval/task/run.py:165 in task_run                                                                                      
/opt/anaconda3/envs/llm-awareness/lib/python3.10/site-packages/inspect_ai/_eval/task/run.py:266 in task_run_sample                                                                               
TypeError: object function can't be used in 'await' expression
```

I think you just need do to this: `scorer=custom_multi_scorer()` (i.e. call the `custom_multi_scorer()` function)

That does indeed work! Thanks
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] Retrying eval fails when using custom epoch score reducers
Root cause: Validation Gap

## Setup

1. Define a custom score reducer and use it in a task.
2. Run `inspect eval` on the task.
3. Interrupt the task
4. Retry the task

## Expected behavior

Eval run continues without errors


## Actual behavior

```
LookupError: a_custom_score_reducer was not found in the registry
```

## Minimal reproducible repo

See [here](https://github.com/jeqcho/inspect-bug-reducer).

Thanks for reporting and for the excellent repro! This is fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2303
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug][vscode extension]: Extracted answer displayed is unrelated to model output
Root cause: Algorithmic Error

Originally reported by @joschu here: https://github.com/UKGovernmentBEIS/inspect_evals/issues/470

<img width="1318" height="151" alt="Image" src="https://github.com/user-attachments/assets/8cbc8c76-5366-418e-be20-f92dc911b495" />

<img width="1129" height="216" alt="Image" src="https://github.com/user-attachments/assets/8c495eef-32c7-4f43-91c8-56730146f0c1" />

> When I inspect the results from a run using vscode, the model's answer is different from the extracted answer. According to the viewer in the vscode extension, the model's answer was 60. Though reading the text, it appears that the model did indeed answer 20, which is the correct answer...[however,] the model was adding markdown asterisks after the answer, i.e., the output looks like
> 
> ```
> **ANSWER: 20**
> ```
> So it's reasonable that it's marked wrong, since the model added extra formatting. And the bolding makes it hard to tell in vscode.
> 
> But then, confusingly the vscode extension picks some other number to show as the extracted answer.



Could you share the evaluation that causes this? The issue is most likely that the scorer matches the first number in the response (I think the viewer is displaying the answer selected by the scorer).

@joschu originally filed [this issue](https://github.com/UKGovernmentBEIS/inspect_evals/issues/470) when running GSM8K

I'll reply on that original issue - this isn't an issue with the vscode extension or viewer (they merely display the results of whatever was computed in the evaluation). The root issue is that the match scorer is being used with `numeric`, and the markdown around the number makes it unresolvable as a number.
Repository: UKGovernmentBEIS/inspect_ai
Issue: `inspect eval` CLI returns a 0 process return code even when it cannot find the specified task to run
Root cause: Validation Gap

When running `inspect eval <task>`, inspect CLI will emit a 0 return code and log a warning when it cannot find the task. This makes it hard/impossible to detect when something went wrong in a bash script.

It would be nice if the default was to return a non-zero error code or if there was a flag one could pass do get that behavior.

# Repro:

`inspect eval thisevaldoesnotexist`

# Expected behavior:

Ideally, the process should return a non-zero return code, either by default, or by passing some argument such as `--fail-on-missing-tasks`

# Current behavior:

[imperio59@localhost]$ inspect eval thisdoesnotexist && echo $?
[08/18/25 16:40:02] WARNING  No inspect tasks were found at the specified paths.                                                                                              eval.py:530
0



Thanks for reporting this, addressed here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2328