zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Strange difference between `inspect view` and log viewer obtained through `vite build`
Root cause: Configuration Error

I am using 0.3.57 (e2add88f8079a0a7bc50b77c940c2de6756577d2).

The left is interface obtained from that hash through `vite build`. The right is the interface launched by `inspect view`. Both are looking at exactly the same log file.

The `vite build` (left) UI shows a bunch of `[null]` values. 

Any idea what could be going on here? Are you able to reproduce this?

<img width="2075" alt="Image" src="https://github.com/user-attachments/assets/6df54915-62ac-4257-bd4d-5a67ab788dac" />

In a simple local test, I didn't see this. When I have seen this in the past it has been related to how the log file was written (in particular whether 'None' was excluded from serialization to JSON). Can you share more about the details of your process or perhaps a simplified reproducible case (that should make it very easy to track down).

Also worth noting - `inspect view bundle` or the `bundle_log_dir` is the preferred way to create publishable artifacts, if this would fit in your workflow.

https://inspect.ai-safety-institute.org.uk/log-viewer.html#sec-publishing

> related to how the log file was written (in particular whether 'None' was excluded from serialization to JSON)

Unlikely, as I'm not messing with serialisation at all. The logs are generated by simply calling `inspect_ai.eval()`. It's also the exact same file in both of my examples.

Will work on getting you a more proper repro.

It reproduces when I run `vite build` on e2add88f8079a0a7bc50b77c940c2de6756577d2, then use the resulting built files on any Inspect log file. 

Here are the files if it helps:

[dist.zip](https://github.com/user-attachments/files/18444074/dist.zip)
[cmQZt4RKyY7n6ZCdSacNjZ.json](https://github.com/user-attachments/files/18444079/cmQZt4RKyY7n6ZCdSacNjZ.json)

I'm opening the viewer with `http://127.0.0.1:4000/inspect-viewer/e2add88f/viewer.html?log_file=https%3A%2F%2Fepoch-benchmarks-production-public.s3.us-east-2.amazonaws.com%2Finspect_ai_logs%2FcmQZt4RKyY7n6ZCdSacNjZ.json&log_dir=`

A fully end-to-end repro (with the web server serving the viewer.html) would be a bit more work, let me know if you still can't reproduce it.

> The logs are generated by simply calling inspect_ai.eval()

Whoops, this is wrong! Turns out I'm actually doing `log.model_dump_json()`, which could lead to the problem you mentioned.  Sorry to mess you around here. 

No worries - hopefully that get it cleared up!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature request: global token cap
Root cause: Unimplemented Feature Gap

It would be great when running `inspect eval` or `inspect eval-set` to be able to set a global cap on the `total_tokens` consumed (input tokens + output tokens). This is for budgeting reasons: sometimes an experiment will turn out to be unexpectedly expensive due to a large number of samples or tokens used per sample, and we'd like to be able to stop the expensive experiments automatically.

The ideal behaviour would be to stop all currently running task instances when the global cap is reached, and not to schedule any more, so that the total tokens used stays under budget. Any completed task instances/samples should be recorded as normal so a partial log is still generated (similar to when a run is aborted due to errors).

Being able to set a monetary budget in dollars would be an alternative. I'd consider the features roughly equivalent since cost roughly tracks tokens for a given model (a monetary budget may be more directly useful but harder to implement).
Repository: UKGovernmentBEIS/inspect_ai
Issue: Documentation Request
Root cause: Documentation Deficiency

Please can you provide more details of how to work with the output .eval files? The view web tool is very nice, but it's not clear how to run one's own analysis on the .eval files (even how to pull out the information on the "JSON" tab of the view tool for all evals in a directory would be very helpful. 

We have decent amount of documentation available here:

https://inspect.ai-safety-institute.org.uk/eval-logs.html

There are a number of helpful functions for operating on log files here:

https://inspect.ai-safety-institute.org.uk/eval-logs.html#functions

Hopefully this has enough to get you going - if you have areas that aren't covered there let us know and we'll do our best to get additional documentation created!
Repository: UKGovernmentBEIS/inspect_ai
Issue: human baselining
Root cause: Unimplemented Feature Gap

Solver that enables human baseliners to use identical dataset, sandbox, and scorer configuration that models use when completing tasks

Completed in this PR: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1049

Documentation: https://inspect.ai-safety-institute.org.uk/human-agent.html
Repository: UKGovernmentBEIS/inspect_ai
Issue: Live updating log viewer
Root cause: Unimplemented Feature Gap

Currently the terminal updates in realtime but the log viewer does not (requires an explicit refresh)

To implement this we'll need an efficient side channel to sync transcript events into (as we don't want to push the entire log to S3 just for high granularity event updates). This might take the form of events written alongside logs or alternatively somewhere on the system running the eval if we think that cluttering up the log directory is undesirable.



https://inspect.aisi.org.uk/log-viewer.html#live-view
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log annotation UI

Create a means by which log files / transcript can carry human annotations.

The annotations might be best embedded in the log file or alternatively we may want to make it possible to keep them in a side data store so as to not leak annotations when publishing/sharing logs.


Keeping annotations in a side data store makes sense to me for the reason you described. Otherwise, work needs to be done to avoid leaking annotations, and a leak seems like it could be a serious issue if a bug caused it to happen.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Checkpointing and restarting from checkpoints
Root cause: Unimplemented Feature Gap

For long running samples we may want to save the event stream as a checkpoint, and allow agents/solvers to re-constitute themselves from a checkpoint.

Note that this can't be done in completely general way where any solver can be checkpointed (as there could be in-memory state, etc.) however a cooperative scheme is possible whereby solvers can be offered a checkpoint and restore themselves from it if they can. It seems like `basic_agent()` and other more complex agents could be made to work with the scheme (although sandbox state is another confounder here -- if we assume the sandbox survives after the task exits this could be made to work).
Repository: UKGovernmentBEIS/inspect_ai
Issue: Direct linking to samples and transcript entries in Inspect View

To do this we need to make use of the history API in the viewer. This branch has some of the infrastructure required for this: https://github.com/UKGovernmentBEIS/inspect_ai/tree/feature/log-list

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1636
Repository: UKGovernmentBEIS/inspect_ai
Issue: Improve task scheduler to not require model as an axis
Root cause: Architectural Constraint

The first type of task parallelization we implemented was running a task against `n` models. We later added more arbitrary task parallelization however in doing so remained over-indexed on the model axis. We need to re-write our scheduler to take more arbitrary combinations of task/model.

This will in turn make eval-set scheduling more effecient (as retries won't need work in "passes" based on model permutations). 

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1454
Repository: UKGovernmentBEIS/inspect_ai
Issue: Bridge to convert from compose.yaml to HELM values

We want evals written with compose.yaml to be runnable directly on the k8s sandbox. This is that the community built evals which use Docker compose don't require adaptation and updating for running on k8s (as this is both time consuming and possibly error prone). 

Linking this up with the issue on the `k8s_sandbox` repo which has some more context https://github.com/UKGovernmentBEIS/inspect_k8s_sandbox/issues/10

This has now been implemented, docs here: https://k8s-sandbox.aisi.org.uk/helm/compose-to-helm/
Repository: UKGovernmentBEIS/inspect_ai
Issue: Catch Syntax Error in Eval File
Root cause: Validation Gap

Thank for for the cool library!

When executing `inspect eval eval.py --model xyz` on a file `eval.py` with syntax errors, the current behaviour seems to be throwing `AttributeError: 'NoneType' object has no attribute '__registry_info__'`, which I contend is not very helpful. Is this the intended way of handling syntax errors?

I have seen that error message before and its a nasty one! This also occurs when you address a task in a file and that task doesn't exist. For example: `eval.py@mytask` where there is no "mytask" in the file. I have a PR that improves this error handling here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/117/files

In terms of an `eval.py` with syntax errors, I haven't been able to reproduce what you describe above. Rather, I get this:

```
[07/24/24 11:32:36] WARNING  No inspect tasks were found at the specified paths.    
```

Could you provide a minimal example where you see this beahvior and I'll investigate further based on that?

Going to close this since we couldn't reproduce locally. Pls feel free to re-open if you can give us minimal example that fails in this fashion.

Sorry, I am currently a bit busy. Yes, you are absolutely right, to reproduce it was neccessary to append the `@eval` to the command.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Ability to use agents with no Inspect dependencies
Root cause: Unimplemented Feature Gap

Use cases:

- Using a 3rd party agent framework like CrewAI, LangChain, OpenAgents, etc.
- Using bespoke agents published in the literature 

Note that there are 2 main execution scenarios here:

- Agent running inside the Inspect orchestration process (peer to traditional solvers)
- Agent that is copied into sandboxes. This is useful where the "sandbox" is inherently part of the dataset (e.g. Cybench, SWE-bench).

Resolved with: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1181
Repository: UKGovernmentBEIS/inspect_ai
Issue: Enforcing cost limits
Root cause: Unimplemented Feature Gap

Set a cost limit for a sample or a task and have it automatically terminate when the limit is reached.

Note that there is no general way to determine the pricing of models dynamically (and if we hard code them into the package they are likely to be wrong). Therefore we'll likely want users to let us know what their per-token cost is when using this feature.

LiteLLM handles pricing of various models with a json file they keep and serve here: https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json

You can add to it or change it at runtime.

I'm fine keep this on the user for now if that makes this easier to build. But BOY, I would love to see a token AND price breakdown after the fact!

I wonder if it would be worth switching to use LiteLLM in general
Repository: UKGovernmentBEIS/inspect_ai
Issue: Enable sandboxes to connect to existing Docker or k8s instances
Root cause: Unimplemented Feature Gap

This is not dissimilar in spirit to the SSH provider where a sandbox is provisioned and configured "somehow" and then we connect to it.

Note that while this provides some conveniences, we should approach with caution as sandboxes run this way are not reproducible from source code (as some out of band process created and configured them).


I think we will start with leaning on the SSH provider for this sort of scenario and revisit connecting to existing sandboxes as required.
Repository: UKGovernmentBEIS/inspect_ai
Issue: hf_dataset: None or 0-length values to be mapped to input field cause dataset loading failure
Root cause: Validation Gap

inspect-ai: `v0.3.58`
datasets: `v3.2.0`
python: `3.12.8`

Hi all, not sure what others experiences have been but this has been a common issue for me:

I'm trying to load datasets from huggingface, the included `openai_humaneval` example works great, but the below dataset has trouble loading:

```
dataset = hf_dataset(
    "Hellisotherpeople/one_syllable",
    split="train",
    trust=True, # appears to make no difference
    auto_id=True,
    sample_fields=FieldSpec(
        input="text",
        metadata=["train"],
    )
)
```

One of the rows of the `Hellisotherpeople/one_syllable` dataset has a row of an empty string, length 0 (''). This causes the following exception in [`read_input` of `_util.py`](https://github.com/UKGovernmentBEIS/inspect_ai/blob/33f14d2db430207d9280146b2d293864c289060e/src/inspect_ai/dataset/_util.py#L118):

```
Traceback (most recent call last):
  File ".../base64_attack.py", line 8, in <module>
    dataset = hf_dataset(
              ^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/inspect_ai/dataset/_sources/hf.py", line 121, in hf_dataset
    samples=data_to_samples(dataset.to_list(), data_to_sample, auto_id),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/inspect_ai/dataset/_util.py", line 102, in data_to_samples
    record_samples = as_sample_list(data_to_sample(record))
                                    ^^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/inspect_ai/dataset/_util.py", line 80, in record_to_sample
    input=read_input(record.get(sample_fields.input)),
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../.venv/lib/python3.12/site-packages/inspect_ai/dataset/_util.py", line 121, in read_input
    raise ValueError("No input in dataset")
ValueError: No input in dataset
```

What's the way forward with this? Have I missed an obvious feature? For the meanwhile I can use a CustomReader to do the mapping manually.

If there isn't a way forward, I wonder if there is an appetite for a `skip_empty` option for `hf_dataset`. Will work on a PR for this if wanted. Thanks in advance 🙂.

To me it seems like the problem is that the dataset isn't "shaped" the way we expect. It seems like multiple lines actually constitute a single record and that those blank lines are the separators between records:

https://huggingface.co/datasets/Hellisotherpeople/one_syllable/viewer?row=12

I don't know if there is a clean way to enhance `hf_dataset` to accommodate this as it definitely assumes it is dealing with "rectangular" data. You might just want to create a custom dataset import function that returns `MemoryDataset` based on. your reading of the lower level file.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support DeepSeek `reasoning_content` field
Root cause: Unimplemented Feature Gap

> `reasoning_content`：The content of the CoT, which is at the same level as content in the output structure. 

https://api-docs.deepseek.com/guides/reasoning_model

We'd need to store that in appropriate Inspect data structures.

This could be added to the `ModelOutput` class.

`ChatCompletionChoice`, I think, rather than directly on `ModelOutput`? If you haven't got anyone working on this yet, I can take a crack at it

Sorry, yes. `ChatCompletionChoice` is the right container. I'm wonder if this actually belongs on `ChatMessageAssistant` then? (as then UIs which deal with lists of messages could show `reasoning_content`?)

> I'm wonder if this actually belongs on `ChatMessageAssistant` then?

Yes, it's debatable. Let me play around with the DeepSeek API and think about it some more. Then I'll open a PR. Might not get to it this week though. 

Blocked because the DeepSeek API has been very difficult to access in the last few days

hello, i already have a barebones implementation on deepseek on my fork based off the grok and openai implementations. unfortunately yes, chat API barely works nowadays let alone reasoning for me to test things.
https://github.com/menhguin/inspect_ai/blob/main/src/inspect_ai/model/_providers/deepseek.py

Feel free to PR the change if you think it's ready

PR here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1229

Resolved by https://github.com/UKGovernmentBEIS/inspect_ai/pull/1229
Repository: UKGovernmentBEIS/inspect_ai
Issue: Tutorial: Mathematics dataset not available in HF anymore
Root cause: External Dependency Breakage

The [MATH dataset](https://arxiv.org/abs/2103.03874), which includes 12,500 challenging competition mathematics problems, **is not available in HF anymore**.


https://huggingface.co/datasets/hendrycks/competition_math
<img width="872" alt="Image" src="https://github.com/user-attachments/assets/4664ff78-2fea-4844-8f0f-d93589f59310" />
The discussion: https://huggingface.co/datasets/hendrycks/competition_math/discussions/5

So, in the [tutorial](https://github.com/UKGovernmentBEIS/inspect_ai/blob/302893f50d0844624997f17775423b19effabe73/docs/tutorial.qmd#L327
), this doesn't work:
```
dataset=hf_dataset(
            "hendrycks/competition_math",
            split="test",
            sample_fields=FieldSpec(
                input="problem", 
                target="solution"
            ),
            shuffle=True,
            trust=True,
        ),
```

Closed with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1538

Now resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1715
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer HTTP API does not support `.eval` format
Root cause: Validation Gap

It tries to parse the binary as JSON and raises:

> Error: JSON5: invalid character 'P' at 1:1

The necessary changes are somewhere in [api-http.mjs](https://github.com/UKGovernmentBEIS/inspect_ai/blob/302893f50d0844624997f17775423b19effabe73/src/inspect_ai/_view/www/src/api/api-http.mjs#L37-L37). (Note that I'm about to modify that file with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1184, so it would be best to make this change downstream from that.)

It does support eval files (we use them quite a bit) so there is likely something specific in your scenario that isn't being handled properly by the viewer. Can you say more about how to reproduce this?

Sorry to be unclear. By the HTTP API of the viewer, I mean the stuff defined in [api-http.mjs](https://github.com/UKGovernmentBEIS/inspect_ai/blob/302893f50d0844624997f17775423b19effabe73/src/inspect_ai/_view/www/src/api/api-http.mjs#L37-L37) and [index.mjs](https://github.com/UKGovernmentBEIS/inspect_ai/blob/302893f50d0844624997f17775423b19effabe73/src/inspect_ai/_view/www/src/api/index.mjs#L36-L36)

I'm passing in a URL to an `.eval` file in the `log_file` URL query parameter: 

https://github.com/UKGovernmentBEIS/inspect_ai/blob/302893f50d0844624997f17775423b19effabe73/src/inspect_ai/_view/www/src/api/index.mjs#L36-L36

I'm familiar with that code. 

So you're just forming a url with a log file pointing to an eval file- I'll give that a try later today and report back. 

It's calling `asyncJsonParse` https://github.com/UKGovernmentBEIS/inspect_ai/blob/302893f50d0844624997f17775423b19effabe73/src/inspect_ai/_view/www/src/utils/Json.mjs#L3-L30

and throwing on https://github.com/UKGovernmentBEIS/inspect_ai/blob/302893f50d0844624997f17775423b19effabe73/src/inspect_ai/_view/www/src/utils/Json.mjs#L15

because the contents are not JSON.

The fix might be as simple as using `openRemoteLogFile`

https://github.com/UKGovernmentBEIS/inspect_ai/blob/302893f50d0844624997f17775423b19effabe73/src/inspect_ai/_view/www/src/log/remoteLogFile.mjs#L33

I'm happy to take a crack at it if you'd like. If you could review https://github.com/UKGovernmentBEIS/inspect_ai/pull/1184 first that would be helpful, so that I don't have to deal with potential merge conflicts. 

Yeah I think the question is why it is improperly treating the file as a json file rather than and eval file.

Ah, so it's supposed to already detect `.eval` files when passed in? Can you show me where that happens?

Reasoning about the log file based upon the type happens one layer up in the `client-api`. I've opened a PR which moves the file handling for this case up to that level (see referenced PR).
Repository: UKGovernmentBEIS/inspect_ai
Issue: inspect-ai-vscode adds unnecessary entries to `.gitignore`.

Hi!

When I have the extension running in a project with a `.gitignore` file and create a Python file, I believe the inspect-ai-vscode extension adds entries to the `.gitignore` file, like `logs/`. I think it comes from here.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/d3f6cb47f0b7332528186b57261bd91b9ac44c3b/tools/vscode/src/providers/workspace/workspace-init.ts#L19

I appreciate the helpfulness of the extension, but I would prefer it if it didn't automatically add entries to my `.gitignore` file. It can be disruptive, especially when I don't need these entries.

An opt-out setting, better detection of an inspect-ai workspace or removing the feature would be candidates for solving my issue.


@dragonstyle I'd be okay with our removing this entirely given that the standard GitHub .gitignore file already ignores `__pycache__/` and `.env`: https://github.com/github/gitignore/blob/main/Python.gitignore. I think there could be a case (but not that strong) for `logs/` but only if this code triggered in an actual Inspect project. The fact that it triggers in any project with a .py file to makes it pretty problematic.

So my vote is remove this behavior.....

@dragonstyle Could we prioritise this one?
Repository: UKGovernmentBEIS/inspect_ai
Issue: Not possible to follow the basics Workflow tutorial
Root cause: Documentation Deficiency

I have tried both, installing via pip and cloning the repo, and it is just not possible to follow most of the first tutorial about the basics of Workflows. 

Different problems: 
- Missing functions (e.g. "plot_results()")
- Errors in the execution following literally the tutorial (e.g. "raise RuntimeError(RuntimeError: asyncio.run() cannot be called from a running event loop...")
- Missing (most of the) files (e.g. security.py, security.ipynb (there is actually 0 notebooks as examples in the repository although there is documentation for it), security/ eval suite, ...)


Thank you for this! The **Workflow** article isn't intended a proper tutorial (it's more as you discovered illustrative examples that don't run w/o additional context). We are indeed planning on creating a proper tutorial that you can follow step by step. In the meantime we'll add a note to the workflow article to make it clear that the code is by and large not executable standalone).

Add a clarifying note to the docs on this here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/140c3d63ad42125213bffaa799147135c81857f3