zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Azure Anthropic Support
Root cause: Unimplemented Feature Gap

### Problem Description

I am currently unable to run Anthropic models (e.g., Claude) hosted on Azure AI Foundry using Inspect-AI. Although Inspect-AI supports models via the standard Azure AI endpoint, the specific configuration or client required for Anthropic models on Azure seems unsupported, resulting in connection or authentication failures when attempting to use the standard `AZUREAI_API_KEY` and `AZUREAI_BASE_URL` environment variables.

### Reproduction Steps / Context

1.  **Environment Setup:** I set the following environment variables, pointing to the Azure AI endpoint correspective of an Azure deployment:
      * `AZUREAI_API_KEY`
      * `AZUREAI_BASE_URL`
2.  **Inspect-AI Usage:** Attempt to configure and run an Anthropic model (e.g., Claude-Haiku) within an Inspect-AI evaluation.

### Possible Root Cause / Technical Context

The issue may stem from the fact that Anthropic models hosted on Azure AI Foundry require a specific client implementation, `AnthropicFoundry`, from the official Anthropic Python library, as shown in the provided code snippet below.

Unlike standard OpenAI-compatible models on Azure AI, this specialized client may be necessary to correctly handle the API endpoint structure and authentication for Anthropic models on the Azure platform.

```python
from anthropic import AnthropicFoundry

endpoint = "https://<ENDPOINT>.services.ai.azure.com/anthropic/" # Note the /anthropic/ path component
deployment_name = "claude-haiku-4-5"
api_key = "YOUR_API_KEY"

client = AnthropicFoundry(
    api_key=api_key,
    base_url=endpoint # Using the specific AnthropicFoundry client
)

# ... client usage proceeds normally
```

Thanks again for the PR! (now merged)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Converse stopReason handling for Bedrock models
Root cause: Validation Gap

A few `stopReason` strings from the latest Converse client for Amazon Bedrock models aren't handled by the [list of `ConverseStopReason` strings in Inspect](https://github.com/UKGovernmentBEIS/inspect_ai/blob/29e7a4f4bce5580f4cacb60f4db34822a6040217/src/inspect_ai/model/_providers/bedrock.py#L50). From the [latest boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime/client/converse.html), this includes:
- `malformed_model_output` 
- `malformed_tool_use`
- `model_context_window_exceeded`

A fix would need to add these additional `stopReason` cases to `ConverseStopReason` and handle them in [`message_stop_reason`](https://github.com/UKGovernmentBEIS/inspect_ai/blob/29e7a4f4bce5580f4cacb60f4db34822a6040217/src/inspect_ai/model/_providers/bedrock.py#L548).

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/2943
Repository: UKGovernmentBEIS/inspect_ai
Issue: JSONSchema.type should also accept a list of types
Root cause: Validation Gap

In the bridge code, a JSON Schema is constructed from external input https://github.com/UKGovernmentBEIS/inspect_ai/blob/5a534fb176e41afcdd8146270f8443405b7ad73c/src/inspect_ai/agent/_bridge/completions.py#L164

The `type` field [is allowed to be an array of strings](https://json-schema.org/understanding-json-schema/reference/type), i.e. the following is a valid JSON Schema:
```json
{
  "type": "object", 
  "properties": {
    "number_or_string": {"type": ["number", "string"]}
  }
}
```

However, `JSONSchema` invalidates it. Repro:
```python
# main.py
from inspect_ai.util import JSONSchema
from jsonschema import validate
from jsonschema.exceptions import ValidationError


schema = {"type": "object", "properties": {"number_or_string": {"type": ["number", "string"]}}}

# schema is a valid JSON schema
instance1 = {"number_or_string": 42}
instance2 = {"number_or_string": "foo"}
instance3 = {"number_or_string": False}
validate(instance=instance1, schema=schema)
validate(instance=instance2, schema=schema)
try:
    validate(instance=instance3, schema=schema)
except ValidationError:
    pass  # expected error
else:
    raise RuntimeError

JSONSchema.model_validate(schema)
````
Output:
```
Traceback (most recent call last):
  File "/home/calebh/scratch/inspect-scratch/main.py", line 21, in <module>
    JSONSchema.model_validate(schema)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/calebh/scratch/inspect-scratch/.venv/lib/python3.14/site-packages/pydantic/main.py", line 716, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        obj,
        ^^^^
    ...<5 lines>...
        by_name=by_name,
        ^^^^^^^^^^^^^^^^
    )
    ^
pydantic_core._pydantic_core.ValidationError: 1 validation error for JSONSchema
properties.number_or_string.type
  Input should be 'string', 'integer', 'number', 'boolean', 'array', 'object' or 'null' [type=literal_error, input_value=['number', 'string'], input_type=list]
    For further information visit https://errors.pydantic.dev/2.12/v/literal_error
```
Relevant code: https://github.com/UKGovernmentBEIS/inspect_ai/blob/5a534fb176e41afcdd8146270f8443405b7ad73c/src/inspect_ai/util/_json.py#L33

I think it suffices to change the signature for `type` to `JSONType | list[JSONType] | None`

Thanks for this! Should be resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2948
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log files can get corrupted in some situations
Root cause: Validation Gap

Hi,

While in the general case inspect log everything properly so that any interrupted evaluation can be easily respawned and finished with `inspect eval-retry`, I noticed that under some (unusual) circumstances, the log files can get corrupted and unrecoverable. Such an example is when you hit a disk full error, then inspect might fail to write some bits and this can result in a heavily corrupted (unrecoverable) log file (at least in eval format), as it would be an invalid ZIP archive (further attempts to process it would trigger a zip exception).

This might be an issue when running very long evaluation pipelines across hundreds of thousands of files, and one might expect inspect to be able to fully recover from its log file. Not sure if this is a considered use case for inspect though.

Best,



Our code doesn't currently anticipate invalid ZIP archives (although they are in principle possible as you point out). One thing we could do is write to a temporary file and then move it to the final file name after a successful write. Since move is quite a bit more atomic than writing I'd expect this to narrow the possibility of corruption quite substantially 

I'd like to work on this. Will implement the temp file + atomic move pattern as suggested. I'll look at the log writing code and open a PR.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Running a Model and a Judge with Local vLLM
Root cause: Documentation Deficiency

Hello,

I would like to evaluate a model hosted locally using vLLM, and also run a judge model, also served via vLLM.

The issue I’m facing is that when hosting a model with vLLM, I need to specify a single `VLLM_BASE_URL`. Since a vLLM server can only serve one model per instance, this makes it impossible to host both the target model and the judge on the same server.

This seems related to the problem discussed in issue #1238. In that case, the situation was eventually resolved due to changes on Azure’s side, but it doesn’t address the underlying limitation for local vLLM deployments.

Do you have any suggestions for how to run and use these two models together?
I’m also open to using the Python API instead of the HTTP server, if that would allow both models to coexist more cleanly.

Thank you!

You can create multiple instances of the vllm client with different base URLs by creating the models with `get_model()` explicitly (rather than just binding on the string representation). Then each of these can target a different vllm server.

Thank you for your explanation. Indeed that works.

For future reference, the solution is using the model through the python api:

```python
judge: Model = get_model(
    judge_name,
    base_url=judge_base_url,
    api_key=judge_api_key,
    config=generation_config,
)

# Pass the model as an argument to the task.
task = create_task(..., judge, ...)
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: Docker sandbox environment assumes all services/containers have the same `WORKDIR`
Root cause: Architectural Constraint

I think the changes introduced in #293 have exposed a previous false assumption in the Docker sandbox environment. When the environment is initialised, the project's `working_dir` is set to whatever the _last_ service's `WORKDIR` is.

```py
# create sandbox environments
environments: dict[str, SandboxEnvironment] = {}
for service, service_info in services.items():
    # update the project w/ the working directory
    project.working_dir = await container_working_dir(service, project)
```
[link](https://github.com/UKGovernmentBEIS/inspect_ai/blob/7559e0ce372d1d215a66e2c6e28d0f835f287205/src/inspect_ai/util/_sandbox/docker/docker.py#L120)

Now that `exec()` always appends a `--workdir` arg...

```py
final_cwd = Path(self._project.working_dir if cwd is None else cwd)
if not final_cwd.is_absolute():
    final_cwd = self._project.working_dir / final_cwd

args.append("--workdir")
args.append(str(final_cwd))
```
[link](https://github.com/UKGovernmentBEIS/inspect_ai/blob/7559e0ce372d1d215a66e2c6e28d0f835f287205/src/inspect_ai/util/_sandbox/docker/docker.py#L187C9-L192C36)

... commands executed in some containers end up with a `--workdir` parameter to a directory which potentially doesn't exist, resulting in 
```
OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd                                         │
│ ("/user-b") set in config.json failed: no such file or directory: unknown
```

I have something like the following `compose.yaml`:
```yaml
services:
  service-a:
    image: ubuntu
    working_dir: /user-a
  service-b:
    image: ubuntu
    working_dir: /user-b
```

I don't think a docker compose _project_ should have a notion of a working dir. A docker compose _service/container_ may though.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add FrontierScience benchmark evaluation
Root cause: Unimplemented Feature Gap

We(@tommyly201 and @mnarayan)'d love to implement the [FrontierScience benchmark](https://openai.com/index/evaluating-ai-ability-to-perform-scientific-research-tasks/) in the repo. We haven't seen this benchmark in this repo yet, so I'd love to  be implementing it for Inspect.

*About FrontierScience*

FrontierScience was released by OpenAI on December 16, 2025. It evaluates expert-level scientific capabilities across physics, chemistry, and biology.

**Structure:**

- FrontierScience-Olympiad (100 questions): Olympiad-style problems with short answers
- FrontierScience-Research (60 questions): Open-ended research subtasks graded with 10-point rubrics. 700+ total questions written by international olympiad medalists and PhD scientists
- Covers physics, chemistry, and biology subfields

**Implementation Plan**

Core Implementation:

- Dataset loading for both Olympiad and Research splits (HuggingFace directly or download as JSON files)
- Scoring for short-answer format (Olympiad track)
- Rubric-based grading system (Research track)
- Model-graded evaluation support
- Documentation and examples

**Extended Evaluation Capabilities:**

- Chain-of-thought causal mediation analysis - examining how models reason through scientific mechanisms and causal pathways
- Tool calling integration - potentially incorporating computational tools (e.g., calculators, symbolic math, molecular structure tools) to support scientific problem-solving

These extensions could be particularly valuable for the Research track, where multi-step reasoning, mechanistic understanding, and computational support are central to scientific workflows.

**Request tags**: benchmark, evaluation

HI @tommyly201 and @mnarayan, great that you want to implement this. You should put this up on the inspect_evals repo which is where the contribution would go: https://github.com/UKGovernmentBEIS/inspect_evals

Thank you. I just added it in. https://github.com/UKGovernmentBEIS/inspect_evals/issues/813

I'll close this issue
Repository: UKGovernmentBEIS/inspect_ai
Issue: use_tools: Only strict function tools can be auto-parsed
Root cause: Configuration Error

I've been trying to run the tool evaluation example from your documentation

```python
@tool
def add():
    async def execute(x: int, y: int):
        """
        Add two numbers.

        Args:
            x: First number to add.
            y: Second number to add.

        Returns:
            The sum of the two numbers.
        """
        return x + y

    return execute

@task
def addition_problem():
    return Task(
        dataset=[Sample(input="What is 1 + 1?", target=["2"])],
        solver=[
            use_tools(add()), 
            generate()
        ],
        scorer=match(numeric=True),
    )
```

Running this with the following command

```bash
inspect eval src/evals/tool_use.py --model hf-inference-providers/openai/gpt-oss-20b:cheapest 
```

Results in the following error

```
ValueError: `add` is not strict. Only `strict` function tools can be auto-parsed    
```

These are the versions of inspect-ai and openai i have currently installed:

- inspect-ai: 0.3.156
- openai: 2.14.0

Is there a way I can set my tool as "strict" when declaring it or when binding it to a task?

in case it helps, i noticed that i can make the tool test work if I apply the following patch to `inspect_ai/model/_openai.py`.

```diff
def openai_chat_tool_param(tool: ToolInfo) -> ChatCompletionToolParam:
    function = FunctionDefinition(
        name=tool.name,
        description=tool.description,
        parameters=tool.parameters.model_dump(exclude_none=True),
+       strict=True
    )
    return ChatCompletionFunctionToolParam(type="function", function=function)
```

Thanks for reporting this! It looks like from my testing that it's likely only HF Inference Providers that ends up w/ this requirement (normal `openai` or gpt-oss via `ollama` work fine w/o the strict). I therefore went with a targeted fix for only HF inference (as I don't know what the implication of doing this for other backends would be: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2971). 



Thank you so much!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Transcript view breaks when prefilling assistant message
Root cause: Documentation Deficiency

When [prefilling](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prefill-claudes-response) assistant messages, the transcript view breaks and does not display the message history. Instead it only displays the very last assistant message.  The message view works fine. To clarify, the functionally of the prefilling works but not the rendering of it.
 
For example, using the prefill example from https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/examples/prefill.py and the haiku 3 claude model I get the below logs where we see how the message view is complete but not the transcript view.
**Messages view:**
<img width="2858" height="816" alt="Image" src="https://github.com/user-attachments/assets/e4383ec8-b427-49e9-8551-fcea47fdbde2" />
**Transcript view (Missing messages):**
<img width="2858" height="1314" alt="Image" src="https://github.com/user-attachments/assets/fc2f2a9f-b534-4946-af9d-06a1310684a1" />


**To reproduce:**
(I use uv here):
1. `uv init && uv add inspect-ai`
3. `curl --output prefill.py https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_ai/refs/heads/main/examples/prefill.py`
4. `uv run inspect eval prefill.py --model mockllm/model`
5. `uv run inspect view`  and compare the transcript view and the message view

**Expected behavior:**
I would expect the transcript view to show all three messages instead of only the last assistant message as currently is the case.

**Discussion:**
I am not sure how important or not this bug is but I wanted to raise it. I stumbled upon this when implementing a benchmark (The [situational awareness dataset](https://arxiv.org/pdf/2407.04694)) and noticed that they are using this prefilling trick for some models. 

cc @dragonstyle 

The transcript view shows the events associated with a sample (the events may or may not correspond the messages since things like prefilling generate messages without any associated event). In this case, there is only a single model call made (with the  prefilled messages), so there is a single model event in the transcript. The default behavior for model events is to show the trailing assistant message in the `summary` tab (as this is the best summary preview of the event contents). You can use the `all` tab to view the complete message stream associated with the model event.

I'm going to mark this closed as I think it is expected / desired behavior- if you disagree or think it merits some additional discussion feel free to re-open it and keep the conversation going.
Repository: UKGovernmentBEIS/inspect_ai
Issue: HuggingFace dataset streaming support
Root cause: Unimplemented Feature Gap

To run an eval on a very large dataset that can't fit in memory or cached on disk, we have to use HF dataset loader's `streaming` option. Currently Inspect AI's HF dataset wrapper by default will call `dataset.save_to_disk` which is unsupported on the dataset iterator. Additionally, `MemoryDataset` assumes a fully loaded dataset for slicing, shuffling, etc.

I would love to be able to run evals on large datasets so just wanted to put in this as a feature request.

Much <3 to the maintainers of this repo.

If we can do the slicing, shuffling etc. on a streaming dataset then we could create another subclass of `MemoryDataset` that works w/ streaming. That said, there is other downstream code that depends on having all of the samples in memory so this might in require a much  bigger architectural change.

Slicing, shuffling, etc. don't really make sense to me on a stream. I would be okay with not having any of those features and requiring buffering the whole stream via something like `StreamingDataset.to_memory_dataset()` or something to make the conversion to a `MemoryDataset` explicit. We can still get `limit` to work but probably not anything else that requires length/indexing. As for multiple epochs, I think there isn't anything to cache so we'd have to rebuild the dataset (whatever HF decides to store locally for a streaming dataset, unsure of the details here). Something like `StreamingDataset.reload()` to reload the dataset if each epoch needs a fresh iterator.

I understand this would add a fair bit of branching. I need this anyway so I'm happy to hack around it for my use case.
Repository: UKGovernmentBEIS/inspect_ai
Issue: broken link

The link [Best Practices for Tool Definitions](https://docs.anthropic.com/claude/docs/tool-use#best-practices-for-tool-definitions) is out of date


Thanks! Replaced here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/024e32b096d74746e7b16ce29925965c10a7d5f0
Repository: UKGovernmentBEIS/inspect_ai
Issue: Unmatched backtick in call to bash_session breaks future calls
Root cause: Validation Gap

That's my interpretation of https://elaborate-naiad-c50ad7.netlify.app/#/logs/2025-11-10T12-17-15%2B00-00_evaluate-lean-fixing_dEkt4YYr5NPGzgyw2AtAAP.eval/samples/sample/7/1/transcript?event=QUYHCimtvWHpQ6wwBYsXTf. The model was calling bash_session as normal, until it tried 
```
bash_session(action: "type_submit", input: "rg "Spec and proof concerning `backend.serial.u64.constants" -n || true")
```

After that, every call to the bash session just printed `>`, so the model got stuck. This matches the behavior I see if I run the same command interactively in bash, since bash is waiting for a matching backtick

(Inspect 0.3.144, called in https://github.com/Beneficial-AI-Foundation/dalek-lean-ai/commit/95cb7c32)

cc @epatey Is there some special mitigation we can do here to get the session unstuck?
Repository: UKGovernmentBEIS/inspect_ai
Issue: Problem: "pip install inspect-ai" 
Root cause: Environment Incompatibility

HI.

I want to install the inspect-ai framework. but, I have the error as below;

<img width="959" alt="image" src="https://github.com/user-attachments/assets/d310d66c-5b0a-403a-9947-b3d5db3024d8">

My os env is as belows;
- ubuntu-24.04.1-live-server-amd64

How can I solve the program?

Best regards,

Soomyung Park




I think this is a new Python default that is strongly encouraging installation of packages into virtual environments rather than the system environment. Here is some info on creating a virtual environment: https://www.arch.jhu.edu/python-virtual-environments/

It looks like you can also get this to work with the global python package library with the `--break-system-packages` flag but this is heavily discouraged!

Thankyou for reply.

I'll try it.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Anyway of rescoring an evaluation with a new metric?
Root cause: Unimplemented Feature Gap

I have implemented a new metric (e.g., `rate_of_no_answers`) and want to apply it to existing `.eval` files to backfill scores for previous runs without re-running the entire evaluation.

### The Problem

Currently, if I update a task's metrics and try to re-score an existing log using the `--scorer` flag, the new metrics are not included in the output.

```bash
inspect score --scorer scorer.py logs/evaluation-task.eval

```

**Observed Result:**
The command re-scores the conversations and re-computes the **original** metrics, but it ignores any **newly added** metrics in the task definition or scorer.

### Solution?

Is there a solution to this? I would like a way to instruct `inspect score` to recognize and compute metrics that were not present in the original log file, I can't find anything on the documentation.

I guess that this would look like:

* A flag to force a refresh of the metric definitions, something like `--metric`.
* Automatic detection of new metrics when a `--scorer` or `--task` file is provided during re-scoring.


@dragonstyle How straightforward would this be to do?

This wouldn't be too bad I don't think. I will have a look over the weekend or on Monday!

I dug into this a little further- we likely need to add a new entry point to compute one or more metrics for an existing eval log (function and CLI). It is going to take me a little more time than I anticipated so I won't get to this immediately.

In the meantime, in my testing, newly added metrics in the scorer are in fact computed (since we revive scorers from their original source, it really can't use the old metrics). In the case of task metrics, you're right that it uses metrics from the EvalLog itself, so there isn't currently way to add new ones at this level.

Can you use amending the scorer metrics as a workaround for now?
Repository: UKGovernmentBEIS/inspect_ai
Issue: model_info() should not require provider SDKs for metadata lookup
Root cause: Architectural Constraint

Looking up static model metadata (release dates, context lengths) requires having provider packages installed because get_model_info() instantiates the model to get its canonical name.

Needing to install these packages is a minor inconvenience for analysis code that's not actually using the providers. I don't think the SDKs need to be a dependency for looking up this metadata.

Referenced PR resolves