zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Prompt caching for Anthropic when used through OpenRouter
Root cause: Unimplemented Feature Gap

Hi,

I noticed that, although OpenRouter supports the prompt caching feature of Anthropic, it does not seem to be available through Inspect, or at least `cache_prompt="auto"` is not respected. Would it be possible to add the support? 

Thank you!

This is because the OpenRounter API uses OpenAI ChatCompletions as its base and then has a ton of proprietary ways to enable various additional advanced features. We have consistently recommended that people use native model APIs when possible with Inspect, as we go to great lengths to try to support all of the new/advanced features. We don't have the resources to implement each of these twice (once for Anthropic and once for OpenRouter) so we again strongly recommend that if you don't want subtle incompatibilities to bite that you use the `anthropic` provider not OpenRouter. 





I understand, thank you for the response!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Prefilling assistant message with reasoning or partial completion?
Root cause: Documentation Deficiency

Hi, I'm interested in evaluating how a model would behave if you were to prefill the reasoning or its completion partially. Is this behavior currently supported by Inspect?

As a toy example, given the user message `"What's the capital of France?"`, I would be interested in evaluating an assistant's response if you could force its reasoning to contain something like `"The capital of France is London."` And/or, if you forced the assistant model's completion to start with something like `"I think the capital of France is London."`

Thanks for your help!

Pre-filling reasoning doesn't work w/ the main frontier models (they use hashes to ensure that you always play back original model reasoning not synthetic reasoning). 

The vllm and sglang providers do provide some support for assistant prefill (see examples here: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/examples/prefill.py). In terms of other providers generally they error if you give them an assistant message last. If you find a case though where prefill is permitted we could update the providers to look at the prefill metadata value as done w/ vllm.

Thank you for the prompt response--exploring this with non-frontier models via vllm is fine for my purposes at the moment, so this is exactly what I'm looking for!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Killed eval loses JSON output in web viewer
Root cause: Validation Gap

**Bug Description:**
When an eval is killed while running, viewing the eval in the web viewer shows the transcript, but the JSON output page displays "JSON not available". This means the JSON output is lost even when the transcript is preserved.

**Expected behavior:**
If a task was running for a long time, I definitely do not want to lose the JSON output. The JSON should be preserved and accessible in the web viewer, even for killed or interrupted evals.

**Steps to Reproduce:**
1. Start a long-running eval.
2. Kill the eval before completion.
3. Open the eval in the web viewer.
4. Observe that the transcript is visible, but the JSON output page says "JSON not available".

**Impact:**
Losing the JSON output after a long-running eval is problematic, as it could contain valuable results and data. The system should ensure the JSON is saved and available for inspection regardless of how the eval was terminated.



I'm sorry that you ran into this - I'm having trouble reproducing the issue, however. Do you have a sample log file that results in this state?

In most scenarios, the transcript and messages are derived from the sample JSON itself. The only scenario where sample JSON isn't available when transcripts are viewable is when the log is running (and since we're live updating, we're using a stream of events to display the transcript rather than the contents of the log file itself). When I kill a running eval with a live viewer open, it does appear to correctly take the canceled state as expected, so I'm not sure this explains it, but perhaps related.


I'm going to go ahead and close this out - please re-open with additional details if you continue to experience the problem!
Repository: UKGovernmentBEIS/inspect_ai
Issue: multi epoch runs do not report per epoch metrics
Root cause: Unimplemented Feature Gap

The ability to retrieve per-epoch scores for all metrics would be helpful, for eg. to calculate metric variance across epochs. Is there a way to retrieve or easily calculate per-epoch metrics?


There isn't currently a ready made way to do this (but I agree there should be!). In the meantime you could read the samples directly from the log file (https://ukgovernmentbeis.github.io/inspect_ai/eval-logs.html#evallog). The samples have an `epoch` field which you could use to compute manually. 

We have added an epoch reducers feature that enables customisation around how scores from different epochs are combined (by default they are averaged): https://inspect.ai-safety-institute.org.uk/scorers.html#sec-reducing-epochs

This has been by far our biggest feature requests (as opposed to reporting per-epoch metrics as requested here). Since there is a way to compute per-epoch via the log file API I'm going to close this.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Timeouts aren't timeouting
Root cause: Documentation Deficiency

I'm getting started using `inspect`. I'm running an evaluation with Sonnet, and there's a piece of code that takes a long time: 

![Image](https://github.com/user-attachments/assets/84a1fc0b-d053-41dc-984d-5aff910f28ef)

I redacted the code using pink rectangles. The timeout is set to 60 on the CLI, but it's been more than 10 minutes and this step is still in progress.

Why is this happening, and how can I troubleshoot such a situation? I can't share my code with you unfortunately. 

That timeout is for model generations and only applies when there is a retry.

If you want to timeout tool invocations you have a few options:

1. Set a `time_limit` for the sample (can be done in the Task definition or on the CLI) which will result in a hard timeout for the sample.

2. Set a `timeout` on the bash tool itself (e.g. to 5 or 10 minutes).
Repository: UKGovernmentBEIS/inspect_ai
Issue: How to build docs?

Thanks for developing Inspect AI, I really love it! Unfortunately, I couldn't find out how the documentation is built.
Can you give me some pointers?

Sure!

```bash
pip install -e ".[doc]"
cd docs
quarto render # or 'quarto preview'
```

Note that if you intend to work on the docs iteratively you will want to install the Quarto VS Code extension.


Works like a charm! Someone should put it in the README.
Repository: UKGovernmentBEIS/inspect_ai
Issue: KeyError: 'type' when using Responses API with pydantic-ai
Root cause: Validation Gap

Inspect crashes with `KeyError: 'type'` when evaluating agents that use pydantic-ai with the OpenAI Responses API.

In `inspect_ai/model/_openai_responses.py`, several functions directly access `param["type"]` without checking if it exists:

```python
# Line 981
def is_response_input_message(param: ResponseInputItemParam):
    return param["type"] == "message" and ...

# Line 1014
def is_response_output_message(param: ResponseInputItemParam):
    return param["type"] == "message" and ...
```

But according to the OpenAI SDK types, `type` is **optional** (`total=False` means all fields are optional by default unless explicitly marked `Required[...]`):

```python
# From openai/types/responses/easy_input_message_param.py
class EasyInputMessageParam(TypedDict, total=False):
    content: Required[...]
    role: Required[...]
    type: Literal["message"]
```

Pydantic-ai doesn't set `type`, so inspect crashes when it tries to read it. I think just using `.get()` would fix it.


## Reproduction

```python
from inspect_ai import Task, task
from inspect_ai.agent import bridge
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIResponsesModel

@task
def test_responses_api():
    async def agent_fn(sample):
        model = OpenAIResponsesModel(..., provider=...)
        agent = Agent(model)

        # Crashes when inspect intercepts the Responses API call
        result = await agent.run(
            sample["input"][0]["content"],
            message_history=[],
            deps=None
        )
        return {"output": result.output, "messages": []}

    return Task(
        dataset=[{"input": [{"role": "user", "content": "Hi"}]}],
        solver=bridge(agent_fn)
    )
```

## Environment
```
inspect-ai                               0.3.137
pydantic-ai                              1.0.17
openai                                   2.1.0
```

Thanks for reporting this! Here is a fix + a new Pydantic AI example: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2627
Repository: UKGovernmentBEIS/inspect_ai
Issue: What's the preferred way of serving InspectAI as a evaluation service?
Root cause: Unimplemented Feature Gap

Hi,

I would like to serve InspectAI as an evaluation service that my team can use. Then they don't need to install and run the evaluation locally.
What's the best way?

The most obvious way is that I pack InspectAI in a docker container, and put some FastAPI endpoint on top of it. But what could be a more preferred way? Or is there anything on the roadmap to serve InspectAI as an evaluation endpoint?



That sounds like the right approach! 

We don't currently have plans to create this sort of service. 

Cool, cool, thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Do not wait for the entire eval_set to finish before retrying individual tasks
Root cause: Algorithmic Error

The current eval_set retry mechanism waits for all tasks to finish before retrying any failed tasks in the eval_set. This is an issue when the run times of tasks in the eval_set significantly vary. Users are more likely to cancel the eval_set and rerun manually instead of waiting.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] when restarting eval after crash, only new "stats" are saved
Root cause: Algorithmic Error

These stats are being rewritten only with stats of samples that were processed after the eval run crash

<img width="365" height="238" alt="Image" src="https://github.com/user-attachments/assets/97128863-cba1-4e92-bcf1-9afb8b0c30ff" />

```json
"stats": {
    "started_at": "2025-10-16T21:04:14+03:00",
    "completed_at": "2025-10-16T21:11:39+03:00",
    "model_usage": {
      "openai/gpt-5-mini": {
        "input_tokens": 128361,
        "output_tokens": 265038,
        "total_tokens": 393399,
        "input_tokens_cache_read": 0,
        "reasoning_tokens": 210944
      }
    }
  }
```

Same goes for "Duration"

<img width="97" height="42" alt="Image" src="https://github.com/user-attachments/assets/c20c3bf1-959a-465d-869a-7db1f2ed91d4" />
Repository: UKGovernmentBEIS/inspect_ai
Issue: json_dataset loads from S3 are very slow with default fs_options
Root cause: Configuration Error

I've found that using the `json_dataset` util to load jsonl files stored on S3 can be very slow. I believe this is because [DEFAULT_FS_OPTIONS](https://github.com/UKGovernmentBEIS/inspect_ai/blob/fdf4b5a9a2f4cd460ed2bd3718a211d442c645ed/src/inspect_ai/_util/file.py#L440) disables S3 caching and as a result iterating through the lines of the file triggers many redundant get requests.

A quick benchmark loading a 9.5MB jsonl file with 2,158 lines using the default fs options and with setting `fs_options=dict(default_fill_cache=True, default_cache_type="readahead")`:
- defaults mean: 119.49s ± 0.70s over 3 runs
- optimized mean: 1.03s ± 0.50s over 3 runs

I can certainly set these options myself now that I've figured this out. However with such a large performance impact perhaps enabling cache by default would be useful? Was there a reason to disable it by default originally?

Those are disabled because there were members of our team that had major scars from fsspec not properly invalidating its cache, so they strongly recommended we disable caching.

That said, these options you supplied seem like they only affect file content reads, and the issues our colleague had IIRC were more about directory listings. I would be game to do something targeted for just reading dataset content.

Here's a PR: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2632

Does that equivalent to the local fix you applied? 

Makes sense! PR looks great, thank you.

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/2632
Repository: UKGovernmentBEIS/inspect_ai
Issue: Error with VLLM on context length
Root cause: Configuration Error

I ran a MASK eval and for some reason all of my models answers in the eval were 

'max_tokens' or 'max_completion_tokens' is too large: 1000. This model's maximum context length is 1000 tokens and your request has 203 input tokens (1000 > 1000 - 203). None

So I tried reducing it and got this:

This model's maximum context length is 200 tokens. However, your request has 203 input tokens. Please reduce the length of the input messages. None

Model chat fine on VLLM normally, it is not an issue with the model, but with Inspect. Frustratingly Inspect grades every answer using my judge models tokens without telling me there is any error. I have uploaded a log file I got out of this with the configuration shown below

log = eval(
    mask(binary_judge_model="google/gemini-2.5-flash-lite", numeric_judge_model="google/gemini-2.5-flash-lite"),
    max_connections=36,
    model="vllm/meta-llama/Llama-3.1-8B",
    model_args = {"device":"0", "chat_template": LLAMA_31_CHAT_TEMPLATE, "max_model_len":500}
) 


cc @kaifronsdal 


The file you shared does not seem to use inspect at all?

I believe meta-llama/Llama-3.1-8B should have a context length of 128k, what happens if you don't try to set the max context length? You could try controlling max tokens through the max_tokens arg of inspects GenerateConfig as another option to prevent outputs that are too long.

I tried not setting max content length and I still get the error

On Fri, Oct 17, 2025, 17:16 Kai Fronsdal ***@***.***> wrote:

> *kaifronsdal* left a comment (UKGovernmentBEIS/inspect_ai#2634)
> <https://github.com/UKGovernmentBEIS/inspect_ai/issues/2634#issuecomment-3417554188>
>
> The file you shared does not seem to use inspect at all?
>
> I believe meta-llama/Llama-3.1-8B should have a context length of 128k,
> what happens if you don't try to set the max context length? You could try
> controlling max tokens through the max_tokens arg of inspects
> GenerateConfig as another option to prevent outputs that are too long.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/UKGovernmentBEIS/inspect_ai/issues/2634#issuecomment-3417554188>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BLCYVMECU36IKS4FUY5DSW33YGBGRAVCNFSM6AAAAACJNSADPOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIMJXGU2TIMJYHA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Sorry wrong file. I can't upload eval files to Github but it's in the slack #bugs on Inspect slack https://inspectcommunity.slack.com/archives/C0805HJTK8U/p1759876002968799?thread_ts=1759772137.811249&cid=C0805HJTK8U

@kaifronsdal have you been able to reproduce this issue?
Repository: UKGovernmentBEIS/inspect_ai
Issue: Improve display of grouped metrics
Root cause: Unimplemented Feature Gap

Currently, metrics use the number of scorers to determine the display style (either a large labeled value like below or a table grouped by scorer). We should special case grouped metrics and make grouped metrics display in a tabular form grouped into a table by either grouping, with the metric names going across the display. Support the dialog based approach that multiple scorers takes now (if there are too many rows in the table). Need to think through the display of mixed metrics as well as (e.g. add a global `mean()` metric to the below example).

Current display:
<img width="656" height="142" alt="Image" src="https://github.com/user-attachments/assets/674ad684-dc5e-4dd3-adf9-a4e1a9f8eca2" />

Desired display:
|          | Easy | Hard  | Red   | Green | All   |
|----------|------|-------|-------|-------|-------|
| Accuracy | 1.0  | 0.667 | 0.667 | 1.0   | .833  |
| Stderr   | 0.0  | 0.333 | 0.333 | 0.0   | 0.167 |


See `src/app/log-view/title-view/ResultsPanel.tsx` for current implementation of metric results display.

Example evaluation here:

```python
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import accuracy, grouped, match, mean, stderr
from inspect_ai.solver import basic_agent, prompt_template


template_prompt = """
Please answer the following with a single word.

{prompt}
"""

@task
def simple_score():
  return Task(
        dataset=[
            Sample(input="What the most common type of ICE powered transportation in the United States?", target="car", metadata={"level":"easy", "category":"red"}), 
            Sample(input="What has two wheels, pedals, and people ride it?", target=["bike", "bicycle"], metadata={"level":"easy","category":"green"}), 
            Sample(input="What is a small freshwater body of water called?", target="pond", metadata={"level":"easy","category":"green"}),
            Sample(input="What word is spelled incorrectly in every single dictionary?", target="incorrectly", metadata={"level":"hard","category":"green"}),
            Sample(input=" Everyone in the world needs it, but they usually give it without taking it. What is it?", target="advice", metadata={"level":"hard","category":"red"}),
            Sample(input="This question has an intentionally wrong target. What is the coolest word?", target="jambalaya", metadata={"level":"hard","category":"red"}),
        ],
        solver=basic_agent(
            init=prompt_template(template=template_prompt)
        ),
        scorer=match(),
        metrics=[grouped(accuracy(), group_key="level"), grouped(accuracy(), group_key="category", all=False),  grouped(stderr(), group_key="level"), grouped(stderr(), group_key="category", all=False)]
    )

<img width="656" height="142" alt="Image" src="https://github.com/user-attachments/assets/f0c508ad-dd3f-4916-9c3c-2684e3222fc9" />

```

@dragonstyle it looks like this issue may have been fixed with #2668. Is there still more to do on this, or did the issue just not get closed yet?

Yep, this is closed with #2668 thank you!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Typo in "VS Code Extension" section in the docs 

**Under the subheading "Debugging Tasks":**
"Note that when debugging a task, the **Insepct** extension will automatically limit the eval to a single sample (--limit 1 on the command line). "

**Correction:**
"Insepct" should be spelled as "Inspect"

Thanks! This is fixed in this PR which will be merged soon: https://github.com/UKGovernmentBEIS/inspect_ai/pull/262
Repository: UKGovernmentBEIS/inspect_ai
Issue: Find matches invisible content
GitHub labels: viewer

The viewer's find implementation works with virtual lists by 'looking ahead' through items in the list that aren't present in the DOM and using matches there to scroll to / load items into the DOM so they are available to the browser find function. Currently, we look ahead by doing string matching against a JSON representation of the item in the list. This will match a lot of things, including things that aren't visible (e.g. that in hidden tabs). 

We should look at the browser context when matching a result and if the content is in a hidden DOM (e.g. a tab), we should likely activate that DOM.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Missing check for 'NoneType' before applying 'strip'
Root cause: Validation Gap

Currently, the strip() function might be called on a NoneType object in line 809 of the parse_tool_call function in inspect_ai/model/_call_tooly.py, leading to an error. This happens, for example, when running the agentharm_benign eval on llama3.2:3b.

The issue could be fixed by checking if "arguments" is None before applying strip() and assigning an empty string otherwise:
    arguments = arguments.strip() if arguments is not None else ""


I submitted a PR. My only concern is if the main contributors don't want to have so much defensive programming, maybe it's so undesired to allow `arguments` to be None.

@CookiePudding-dev It looks like a PR from @aounleonardo was merged to address this issue. Can this be closed?
Repository: UKGovernmentBEIS/inspect_ai
Issue: create 'cancelled' state for samples that are keyboard interrupted
Root cause: Documentation Deficiency

It would be nice if when I keyboard interrupt during an eval, it doesn't clear the entire sample message history, but preserve it and acknowledge that it was keyboard interrupted at the end. I know there is a manual score override button, but sometimes the scorer takes a long time or I just forget and ^C out of habit.


@jjallaire I'm happy to take this one 😄 

I'm a new contributor though, so any entrypoint would be much appreciated! 🙏 

@jjallaire  Unless I misunderstood it, this is already implemented, right? (details below)

I presume that **you could also mean** that you want it to cancel all remaining tasks and not stop early if I pass multiple tasks to the eval. 
FYI I ran it with two tasks, each sleeping for 3 seconds. If I ^C in the first 3, I get `len(results) == 1`, while if I stop at 4 seconds, I get 2 results (first one has status `success` while the latter has `cancelled`)
_I know this way of testing isn't ideal, but for now it's okay for me and when I submit something I'll do so by mocking a KeyboardInterrupt._
Looking at the commit history, nothing is touching the "cancelled" logic, so I assume this is what you mean. But I'll wait a bit before I sink more time into this.

## Details:
I wrote a test with a sleep_for_solver and ^C during a task, pdb-ed into `src/inspect_ai/_eval/eval.py::_eval_async_inner` right before `cleanup_sample_buffer` and checked the results, the task's status is cancelled

```
(Pdb) results
[{
  "version": 2,
  "status": "cancelled",
  "eval": {
    "eval_id": "BnVvLRvfK3EKm2NREVReew",
    "run_id": "CJegKqNu4K7tDKKJeRfD9p",
    "created": "2025-11-11T18:37:46+00:00",
    "task": "task",
    "task_id": "DDPS9Wse6cGcc6oJbmLTFd",
    "task_version": 0,
    "task_display_name": "task",
    "task_attribs": {},
    "task_args": {},
    "task_args_passed": {},
    "dataset": {
      "samples": 1,
      "sample_ids": [
        1
      ],
      "shuffled": false
    },
    "model": "mockllm/model",
    "model_generate_config": {},
    "model_args": {},
    "config": {
      "epochs": 1,
      "epochs_reducer": [
        "mean"
      ],
      "fail_on_error": true,
      "continue_on_fail": false,
...
```

@jjallaire is this implemented already, as @aounleonardo suggests?
Repository: UKGovernmentBEIS/inspect_ai
Issue: Doc is unclear about what kind of strings are accepted for score values
Root cause: Documentation Deficiency

https://inspect.aisi.org.uk/reference/inspect_ai.scorer.html#value mentions "C" and "I" as an example but doesn't mention the other one available. It shopuld be made clearer that you can put any string as long as you pass a `value_to_float` function that handles them

@Butanium I'm all about better docs, so I'm taking a look at this. I'm not finding "C" and "I" as examples there. Maybe the docs have changed since you reported this? Do you mind looking again and helping me to understand what you'd like to see?

I guess it's here now: https://inspect.aisi.org.uk/scorers.html#score

What I was confused about is what string am I allowed to put. In the source code there are more letters that get converted to floats and they are not mentioned (and the way you can use any strings by having a custom converter is not really mentioned afaik)

Ahhh, yeah, I see that now. I think mentioning the other strings at a minimum would be helpful, and perhaps going further and mentioning the converter would be good too.