zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: minor code aesthetic improvements

Would you be happy to receive PR's that contain minor improvements to the code, but which do not change functionality.

The two examples I have seen so far:

### Example 1

```python
            isinstance(self.value, str)
            or isinstance(self.value, int)
            or isinstance(self.value, float)
            or isinstance(self.value, bool)
```

could be

```python
            isinstance(self.value, str, int, float, bool)
```
(Or, you could create alias `Scalar = str | int | float | bool` and use that throughout the file)

### Example 2.

Function `example_dataset` can use `Literal` typehint, because you can list precisely what values you are expecting

```python
def example_dataset(
    name: str,
    sample_fields: FieldSpec | RecordToSample | None = None,
) -> Dataset:
...
      name (str): Example dataset name. One of 'security_guide', 'theory_of_mind',
        'popularity', or 'biology_qa'
```

We aren't (yet) setup to take PRs on this repo (we actually do ongoing development on an internal repo and then sync to here every few days). So for now I would post these as issues and then we will resolve them internally and sync to here in due course.

I'd suggest picking up a tool like [Ruff](https://docs.astral.sh/ruff/) to deal with issues like this. That said, I'm not sure about example 2, but example 1 should be taken care of, see [here](https://docs.astral.sh/ruff/rules/non-pep604-isinstance/) and [here](https://docs.astral.sh/ruff/rules/duplicate-isinstance-call/).

@daaain Agreed re: Ruff, we actually are using it now (https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/pyproject.toml#L14-L23) but it's not picking up those cases. Will add the rules you mentioned and cleanup any other cases of this.

Resovled with https://github.com/UKGovernmentBEIS/inspect_ai/commit/140c3d63ad42125213bffaa799147135c81857f3
Repository: UKGovernmentBEIS/inspect_ai
Issue: Structured output functionality breaks for nested objects
Root cause: Algorithmic Error

We've been trying to use Inspect's structured output functionality to produce json-formatted outputs for some tasks where we want models to generate tables. However, we're noticing that inspect's structured output functionality does not handle nesting correctlu. For instance, we have the following output schema definition:
```python
class Cell(BaseModel):
    """
    A Cell Object consists of a paper ID, a column name and 
    the corresponding cell value at that row & column in the table.
    """
    paper_id: str
    column_name: str
    cell_value: str

class Table(BaseModel):
    """
    A Table Object is a List of Cell Objects.
    """
    cell_values: List[Cell]
``` 
When we pass the table schema into the [response_schema](https://inspect.aisi.org.uk/structured.html) variable for the generation config, we always get incorrectly formatted outputs. Looking at the logs, this is what our table schema gets converted into:
```python
"response_schema": {
          "name": "table",
          "json_schema": {
            "type": "object",
            "properties": {
              "cell_values": {
                "type": "array",
                "items": {}
              }
            },
            "additionalProperties": false,
            "required": [
              "cell_values"
            ]
          }
        }
``` 
which shows that the definition of the table object is preserved, but the definition of the cell object is lost. However, passing `Table.schema()` directly to an OpenAIAPI object works correctly (preserves formatting of cell objects too). Looking at the json formatting code, it looks like the definitions of nested objects get [dropped here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/1f6162992c01b0d258377527c2b8473821c56352/src/inspect_ai/util/_json.py#L129). The `model_json_schema()` function in Pydantic called on the Table class will produce the following output:
```python
{'$defs': 
  {'Cell': 
     {'description': 'A Cell Object consists of a paper ID, a column name and \nthe corresponding cell value at that row & column in the table.', 
      'properties': {'paper_id': {'title': 'Paper Id', 'type': 'string'}, 
                             'column_name': {'title': 'Column Name', 'type': 'string'}, 
                             'cell_value': {'title': 'Cell Value', 'type': 'string'}}, 
      'required': ['paper_id', 'column_name', 'cell_value'], 
      'title': 'Cell', 
      'type': 'object'}
   }, 
  'description': 'A Table Object is a List of Cell Objects.', 
  'properties': {'cell_values': {'items': {'$ref': '#/$defs/Cell'}, 'title': 'Cell Values', 'type': 'array'}}, 
  'required': ['cell_values'], 
  'title': 'Table', 
  'type': 'object'}
``` 
Here the `$ref` pointing to `Cell` class definition must first be resolved by looking up its definition from `$defs`, but the code does not do that, which leads to cell schema definition being dropped.

Thanks for reporting this! Should be resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1506
Repository: UKGovernmentBEIS/inspect_ai
Issue: Can I programmatically get the eval log file path when running from python?

I'm probably missing something obvious, but I didn't see a way to get the log file path for a given task eval. 
If I run it on the command line, the best I found was to grep for "Log:".

If you run from Python then the returned `EvalLog` object has a `location` property: https://inspect.aisi.org.uk/eval-logs.html#location
Repository: UKGovernmentBEIS/inspect_ai
Issue: Tutorial uses MATH dataset, which is subject to a DCMA takedown
Root cause: External Dependency Breakage

Currently the "Mathematics" section of the [tutorial](https://inspect.aisi.org.uk/tutorial.html#sec-mathematics) is impossible to complete because Inspect throws the following error when trying to load the dataset:

> `datasets.exceptions.DatasetNotFoundError: Dataset 'hendrycks/competition_math' doesn't exist on the Hub or cannot be accessed.`

It looks like this is because a third party has [filed a DCMA takedown](https://huggingface.co/datasets/hendrycks/competition_math/discussions/5) with Hugging Face. The [takedown notice](https://huggingface.co/datasets/huggingface-legal/takedown-notices/blob/main/2025/2025-01-02-AoPS.md) makes a reasonable case that the dataset infringes the third party's copyright, and the authors don't appear to have submitted a counter-notice in the 2 months since the takedown, so it seems likely the dataset won't return.

duplicate of #1283
Repository: UKGovernmentBEIS/inspect_ai
Issue: Sorting by numerical scores fails in log viewer
Root cause: Validation Gap

Hi, it seems that sorting by numerical scores doesn't work as intended in the log viewer when selecting Sort: "score_desc"/"score_asc". The (seemingly arbitrary) ordering of samples doesn't change between these two selections and a warning "Comparing non-numerics using a numeric score descriptor" is being reported in the console.

## Reproduction steps
1. Run `eval` with a scorer returning numerical values, such as:
```python
from inspect_ai.scorer import Target, Score, mean, scorer
from inspect_ai.solver import TaskState

@scorer(name="Random", metrics=[mean()])
def random_scorer():
    async def score(state: TaskState, target: Target):
        return Score(
            value=random.uniform(0, 1),
            answer=state.output.completion,
        )

    return score
```
2. View the resulting log in Inspect View.
3. Select Sort: "score_asc" or "score_desc" in the user interface.

Thank you for reporting this! Fix is on the way...
Repository: UKGovernmentBEIS/inspect_ai
Issue: hendrycks/competition_math has been removed from huggingface
Root cause: External Dependency Breakage

Following a DMCA takedown notice, [this dataset is not accessible anymore](https://huggingface.co/datasets/hendrycks/competition_math/discussions/5).

Resolved with: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1715
Repository: UKGovernmentBEIS/inspect_ai
Issue: cache misses are nearly impossible to debug
Root cause: Documentation Deficiency

I am trying to debug why there are cache misses in
[2025-03-23T01-27-46+00-00_coq-to-lean_EZMRDo6aNeAtrNVHxVumDJ.eval.tar.gz](https://github.com/user-attachments/files/19406579/2025-03-23T01-27-46%2B00-00_coq-to-lean_EZMRDo6aNeAtrNVHxVumDJ.eval.tar.gz)

I guess I can tell when there is a cache miss because it looks like

![Image](https://github.com/user-attachments/assets/3bb6aab1-8a4e-4e97-886f-74c31c301ed6)
instead of
![Image](https://github.com/user-attachments/assets/98b35e3c-189f-45e2-b7a1-ee5cb78b798d)
(it might be worth including a "(cached)" note and not just indicating a cache hit by not including an API tab)

Moving on, I see
![Image](https://github.com/user-attachments/assets/cd650917-74c8-4153-9662-517da3bbc51f)
which I think indicates that only the last four tokens cause the cache miss?

But I see this every time I run it, there does not seem to be caching going on for this message, so I suspect the issue is on inspect's side.  However, there does not seem to be a way to debug the cache miss on inspect's side.

Also, I see
```
"usage": {
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 13101,
    "input_tokens": 4,
    "output_tokens": 99
  }
```
which suggests that inpsect is not properly telling anthropic to cache the final four tokens?  

Could inspect insert some sort of additional marker indicating the reason for cache miss (maybe it could indicate the longest prefix that matches something in the cache, and where it starts not matching?)

For the Anthropic cache, we are just inserting caching markers in the input exactly according to the Anthropic documentation here: https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching. Anthropic doesn't provide any additional debugging or diagnostics about their cache or why hits/misses occur (and this is all a black box to us from the Inspect side). 

So I don't think there is much else we can do here unless Anthropic makes available additional diagnostics. Note that's its certainly possible that Anthropic is not caching these tokens as a matter of policy (note they have some docs here on not caching shorter prompts, perhaps you have to hit 1024k barriers to end up in the cache? https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching#cache-limitations).


Ah, sorry for the confusion.  Here my complaint is that I am passing `cache=CachePolicy(expiry=None, per_epoch=False)`, and as far as I can tell the messages are the same on every run, but the local cache is being missed.  (I should have uploaded two evals, and my code has since moved on, but if I hit this again I can upload two logs for diffing)

Okay, sounds good. There were issues w/ the cache based on message IDs that we fixed last week so perhaps you were running into that? I'm going to close this issue for now -- feel free to re-open or post a new issue if you see this again and can provide the repro.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [question] How to run `eval_retry` with task parameters
Root cause: Documentation Deficiency

How would I pass task parameters when running failed run?
Example of full run:
```
logs = eval(
        tasks=task(grader_model=grader_model, dataset=dataset_obj, **kwargs),
        model=model,
        log_dir=output_folder,
        limit=samples_limit,
        max_connections=max_connections,
    )
```
When using `eval_rerun` it fails with error:
Code:
```
logs = eval_retry(
            tasks=["logs/2024-07-29T09-43-01-05-00_eval-tool-eval-multi-criteria-grader-list_3kqa33ubZsvviYHt6WQKxT.json"],
            log_dir=output_folder,
            max_connections=max_connections,
        )
```
Error:
```
TypeError: eval_multi_criteria_grader_list() missing 2 required positional arguments: 'grader_model' and 'dataset'
```


`eval-retry` needs to be able to fully re-execute the construction of the task (it can't serialise arbitrary tasks for re-execution as they could have all kinds of Python sauce that can't be saved).

If you do it this way then it should be able to reconstruct the task:

```python
@task
def my_task(grader_model, dataset, **kwargs):
    return Task()

logs = eval(
        tasks=my_task,
        task_args=dict(grader_model=grader_model, dataset=dataset_obj) | kwargs,
        model=model,
        log_dir=output_folder,
        limit=samples_limit,
        max_connections=max_connections,
    )
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: scorers should not take up two lines in the header
Root cause: Unimplemented Feature Gap

[2025-03-23T04-12-55+00-00_coq-to-lean_eYQjuQUbc8i4YeHbuNBDA3.eval.tar.gz](https://github.com/user-attachments/files/19407042/2025-03-23T04-12-55%2B00-00_coq-to-lean_eYQjuQUbc8i4YeHbuNBDA3.eval.tar.gz)

![Image](https://github.com/user-attachments/assets/d7dc87cd-8b59-4c6b-939e-eee731c3c4fe)

I would really rather not have the header take up nearly half the screen.  Also note that the list of scorers is cut off and cannot be scrolled

@dragonstyle I agree that this is not at all what we want. I think we need a way to either show these more compactly or show just one of them and make it obvious how to expand to see the rest.

Totally agree and I've already started making improvements to score display on a branch based upon the logs shared last weekend. Coming soon!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Improve handling of pathlib.Path objects for read_eval_log
Root cause: Validation Gap

Currently there's a bit of a footgun if you pass `pathlib.Path` objects into `read_eval_log` / `write_eval_log`. 

The code expects `str | EvalLogInfo` and then resolves this to a path with:
`log_file = log_file if isinstance(log_file, str) else log_file.name` 

However `Path` objects also have a `.name` attribute which returns the string filename + extension. This means that it will load/save relative to the working directory which is confusing and hard to debug.

I think these functions should either support path objects or fail in a more obvious way.

Good one! Yes, we'll clean this. up and likely just take a `Path` object as well.

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1714
Repository: UKGovernmentBEIS/inspect_ai
Issue: Documentation Typo

Should be `eval-retry` instead of `eval_retry`
![Image](https://github.com/user-attachments/assets/1c1a0e18-44fd-4a10-866d-5aee8b56858f)

Thanks! Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/4d11396dcc8dcd01315501009537c77b4c0c5a93
Repository: UKGovernmentBEIS/inspect_ai
Issue: Dynamically created tasks with a name can't be resumed
Root cause: Validation Gap

It seems that `eval_retry` currently doesn't work for dynamically created tasks with explicitly set `name` (at least, I haven't been able to find a way to make it work). For example, if I define a task as below and interrupt it halfway through:

```python
from inspect_ai import Task, eval, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import exact
from inspect_ai.solver import generate

@task
def repeat_task(phrase: str):
    return Task(
        dataset=[
            Sample(
                input=f"Repeat {phrase} 50 timees.",
                target=(f"{phrase} " * 50).strip(),
                id=i,
            )
            for i in range(50)
        ],
        solver=[generate()],
        scorer=exact(),
        name=f"Repeat-{phrase}",
    )

eval(
    model="vllm/meta-llama/Llama-3.1-8B-Instruct",
    tasks=repeat_task("phrase"),
)
```
any subsequent call to `eval_retry` with the path to the produced partial log file fail with `FileNotFoundError: Task 'Repeat-phrase' not found.`

That's correct, eval-retry needs to be able to locate the task on disk (and it uses the name to do this). 

As far as I understand, this doesn't actually relate to reading any files from the disk but looking up the task in the registry ([eval.py:734](https://github.com/UKGovernmentBEIS/inspect_ai/blob/63d63f286829f70aea517758176e5d92165db2c7/src/inspect_ai/_eval/eval.py#L734)). Calling `eval_retry` works perfectly for the above case when `name` is not set on the task.

In general, it feels slightly counter-intuitive that using a custom name for the task (e.g., to make it a bit easier to identify the correct log file when viewing the results) would prevent it from being resumed. Instead, I would expect that the `EvalLog` data for the task would include both the registry ID from the Task's `RegistryInfo` (`repeat_task` for the example above) and the user-defined custom name (`Repeat-phrase`) so that the task can be correctly looked up when calling `eval_retry`.

The `name` was really put in for "anonymous" tasks (i.e. `Task` instances created without a decorator) so conceptually is a _replacement_ for the registry task name. That said, I do see that in this usage you are just trying to make the log file name more comprehensible. I wonder if one solution would be to use the `name` for the log file only in the case that there is a name available from `@task`. I don't think this would be too hard to pull off, will investigate and report back.

Yes, that would work for my use-case — thanks for looking into this!

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1583
Repository: UKGovernmentBEIS/inspect_ai
Issue: inspect view bundle produces broken webpages
Root cause: Documentation Deficiency

I ran `inspect view bundle --log-dir logs --output-dir logs-www` and then in `logs-www` ran `python -m http.server` (also tried opening `index.html` directly).  The former gives

An error occurred while loading this task.
Error: End of central directory record not found

at openRemoteZipFile@http://localhost:8000/assets/index.js:47294:15

![Image](https://github.com/user-attachments/assets/8e699bbf-f5aa-4296-acfa-e5163631d2f4)

(the latter displays only a white page)

Here are the files:
[bug-bundle.tar.gz](https://github.com/user-attachments/files/19495467/bug-bundle.tar.gz)

I just gave it a spin with those files and it worked for me, so long as I was serving the `log-www` directory. If I served from the root, I did reproduce the error. I'll add a better error message for this case, but I think the root issue may be what directory you're serving. Does that seem right?

(edit: re-read your repro above - I'll update with a little more info in the next comment)

The issue is caused by using `python -m http.server`. The viewer used HTTP Range request to fetch only the portions of the eval file that it needs to display the viewer. The `http.server` doesn't support range requests and is ignoring the header and returning the entire contents of the file. I can also improve this error message, but in the mean time I typically use `rangehttpserver` for local serving:

https://github.com/danvk/RangeHTTPServer

Resolved with improved error message for web servers that don't support range requests. 

Note that the `aiohttp` server does support ranged requests: https://docs.aiohttp.org/en/stable/web_quickstart.html#command-line-interface-cli
Repository: UKGovernmentBEIS/inspect_ai
Issue: GenerateConfig of Model passed to eval is not captured in EvalLog
Root cause: Validation Gap

It seems that the `EvalLog` doesn't currently capture the model's generation config when `eval` is called with a `Model` object. For example, if I call `eval` as follows:
```python
from inspect_ai import Task, eval, task
from inspect_ai.dataset import Sample
from inspect_ai.model import GenerateConfig, get_model
from inspect_ai.scorer import exact
from inspect_ai.solver import generate

@task
def hello_world():
    return Task(
        dataset=[
            Sample(
                input="Just reply with Hello World",
                target="Hello World",
            )
        ],
        solver=[generate()],
        scorer=exact(),
    )


generation_config = GenerateConfig(
    seed=42,
    temperature=0.7,
    top_p=0.95,
    max_connections=1,
)
model = get_model(
    model="vllm/meta-llama/Llama-3.1-8B-Instruct",
    config=generation_config,
)
logs = eval(
    model=model,
    tasks=hello_world(),
)
```
The `EvalLog` returned in `logs` only seems to include the default `GenerateConfig` in the `plan.config` field (incorporating any config keyword arguments passed to `eval`), but not the `generation_config` used when initialising the model.

This also means that when the corresponding task is retried using `eval_retry`, the default config is used instead of the correct model config.

Thank you for pointing this out! We have another branch right now for a "model roles" feature where we have created a log data structure that does capture this info in one place [src/inspect_ai/log/_log.py](https://github.com/UKGovernmentBEIS/inspect_ai/compare/main...feature/model-roles#diff-0225b02c754b96ef68ebcb947c6863d420f8c58fe4a6d890b19540f7025d5864). As part of this PR we'll use the same data structure for the default model and resolve this issue.

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1637
Repository: UKGovernmentBEIS/inspect_ai
Issue: agent bridge should be able to run sandboxed agents
Root cause: Unimplemented Feature Gap

Something like:

```python
bridge("/opt/agents/agent.py", sandbox=TRUE)
```

In this arrangement would be able to provide their own Docker containers and we would setup an HTTP interface to OpenAI that we proxy back into Inspect. We'd also want to capture and proxy the logger.




https://github.com/UKGovernmentBEIS/inspect_ai/pull/2301
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support for running score in Jupyter notebooks
Root cause: Architectural Constraint

While `eval` and `eval_retry` can be run from Jupyter notebooks due to their internal use of `run_coroutine` ([display.py:80](https://github.com/UKGovernmentBEIS/inspect_ai/blob/479f11ba040b1d3e029b4e14680d283abc934d76/src/inspect_ai/_display/rich/display.py#L80)), attempting to call `score` from a Jupyter notebook results in an error.

This seems to be because `score` directly calls `anyio.run` ([score.py:59](https://github.com/UKGovernmentBEIS/inspect_ai/blob/479f11ba040b1d3e029b4e14680d283abc934d76/src/inspect_ai/_eval/score.py#L59)), which raises a `RuntimeError` if the asyncio event loop is already running, even when `nest_asyncio.apply()` has been called before.

It would be very helpful if `score` could also be used within Jupyter notebooks.

Thanks for reporting! Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1580
Repository: UKGovernmentBEIS/inspect_ai
Issue: `bash_session` tool does not support tasks that are long running or require additional input.
Root cause: Unimplemented Feature Gap

As currently implemented, the `bash_session` tool has no affordance for obtaining additional input from the caller when required by the command that was launched.

Add support to return partial command output so that the model can send additional input if it detects that additional input is required.
Repository: UKGovernmentBEIS/inspect_ai
Issue: automatically retry failed samples
Root cause: Unimplemented Feature Gap

There are some evals where transient errors (often involving Docker) cause a sample error. Currently these are retried _eventually_ (after eval-set completes the rest of the first pass) but it would be much better if they could be retried immediately. 

This could be governed by a `--retry-on-error` option (to go along with the existing `--fail-on-error` option. The `--retry-on-error` option could default to a single retry (on the theory that if errors are sparse a single retry will typically resolve things, and the desire to not flog errors that are irrecoverable) but also support `--retry-on-error=3` for multiple retries.




Done here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1712