zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Typo

https://ukgovernmentbeis.github.io/inspect_ai/workflow.html

You can pass a notebook to insect eval just the same as a script (including passing task parameters)


Typo "insect eval"

Fixed with https://github.com/UKGovernmentBEIS/inspect_ai/commit/671a239f13fe2706f89ffccaf5ac5eef86958fc3
Repository: UKGovernmentBEIS/inspect_ai
Issue: Dynamically allocating solver to a task gives ValueError in inspect ≥0.3.29 but not in ≤0.3.28
Root cause: External Dependency Breakage

```
@solver
def my_solver(*solver_args) -> Solver
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        # do_something
        return state

    return solve

@task
def my_task(task_solver: callable, *solver_args) -> Task:
    return Task(
        dataset=my_dataset(),
        plan=task_solver(solver_args),
    )

if __name__ == "__main__":
    eval(
        tasks = my_task(task_solver=my_solver, my_arg),
    )
``` 
    
raises ValueError: Object does not have registry info

This is the case even if I use the new solver instead of the plan argument for specifying the solver. 

inspect 0.3.24 did not use to return any such errors and works fine with the above code

Thanks! There is a fix for this merged here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/624

That said, I would recommend moving to using tasks that take actual solvers rather than functions that yield solvers. The reason is mostly support for `eval-retry` (which depends on your task parameters being serialisable, which is now possible for all registry objects). Here is a slightly modified version of the failure case:

```python
from typing import Callable

from inspect_ai import Task, eval, task
from inspect_ai.dataset import Sample
from inspect_ai.solver import Generate, Solver, TaskState, solver

@solver
def my_solver() -> Solver:
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        # do_something
        return state

    return solve

@task
def my_task(task_solver: Callable[[], Solver]) -> Task:
    return Task(
        dataset=[Sample(input="Say hello, world")],
        solver=task_solver(),
    )

if __name__ == "__main__":
    eval(tasks=my_task(my_solver))
```

Here is the diff I recommend:

![Screenshot 2024-10-02 at 09 54 45](https://github.com/user-attachments/assets/ded306d4-3b9b-4db6-bb34-4d94b6ab70c6)

Which as you can see here allows us to capture the solver parameter as a serialisable dict:

![Screenshot 2024-10-02 at 09 55 48](https://github.com/user-attachments/assets/aed312c3-1c50-4a9c-9142-5ed5e6eb20f3)

Contrast that with the original where we just capture the function name (so can't reconstruct it for an `eval-retry`):

![Screenshot 2024-10-02 at 09 59 27](https://github.com/user-attachments/assets/d1c84106-188c-4d20-88ee-f4124c1b8152)




Thanks! That’s super helpful
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inconsistent Sample IDs produced when shuffle=True
Root cause: Algorithmic Error

It appears that https://github.com/UKGovernmentBEIS/inspect_ai/pull/477 introduced a bug where the `sample_id` is dependent on the shuffle order when the sample_id is not explicitly provided (e.g., in MMLU). This makes it difficult to track samples across runs.

I think the answer to this is to introduce an `auto_id` feature that ensures that the samples get the same ID even in the presence of shuffling: https://github.com/UKGovernmentBEIS/inspect_ai/pull/626

The reason we can't just always do that when shuffling is that many datasets will indeed already have an ID field (so our `auto_id` would plow over that).

Also updated evals using `shuffle` that don't already have an ID to use `auto_id=True`.

Should be resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/626

Makes sense, thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Web Browser tool cannot see checkbox state
Root cause: Algorithmic Error

Very minor one, but when using the web browser tools, the state of checkboxes isn't rendered. 

This means when the agent checks a checkbox (e.g. "I agree to terms and services" when signing up to something), it cannot see that it's checked it, and can keep trying in a loop / generally get confused.

I can share an example with devs upon request, but it contains private data so can't post it here. 

This can be used to verify the issue.

```html
<!DOCTYPE html>
<html>
  <body>
    <form>
      <input type="checkbox" id="id" name="id" value="TheValue" />
      <label for="id">Checked? Unchecked? Who's to say?</label><br />
    </form>
  </body>
</html>
```

FWIW, it appears to work properly running code on #1314. I'll go back to `main` and confirm the behavior there.
```
> go file:///Users/ericpatey/Downloads/checkbox.html
[2] checkbox "Checked? Unchecked? Who's to say?"  [checked: false, labelledby]

> click 2
[2] checkbox "Checked? Unchecked? Who's to say?"  [focused: True, checked: true, labelledby]

> click 2
[2] checkbox "Checked? Unchecked? Who's to say?"  [focused: True, checked: false, labelledby]
```

Although I made no explicit fix for this, I'm unable to reproduce the issue. @sdtblckgov, if you can reproduce it with the latest code, reopen this and share the details.
Repository: UKGovernmentBEIS/inspect_ai
Issue: include 'MemoryDataset' in the docs
Root cause: Documentation Deficiency

TLDR: I think it would be useful to add 'MemoryDataset' in the datasets docs page. https://ukgovernmentbeis.github.io/inspect_ai/datasets.html

---
I am trying to parse this dataset:
https://gist.github.com/nix-apollo/fa562b0269d3e1324ef4fae0902bd1f3

I dont think I can parse using the tools provided in https://ukgovernmentbeis.github.io/inspect_ai/datasets.html, because i need a nested for loop to access the individual records from the json object.  But I can do parsing outside of inspect, and then use MemoryDataset to create the Dataset object.

Good catch, will do this in our next sync.

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/commit/2d7561f02213781d603ed4ff1826f8dfe0963da4
Repository: UKGovernmentBEIS/inspect_ai
Issue: docs for `Sample`'s `target` parameter not consistent with code
Root cause: Documentation Deficiency

In docstring and in documentation page, we have `target (str | list[str] | None)`.  But using `target = None` when creating a `Sample` object gives validation error.  Presumbly because in the code, `None` is not a possible type: `target: str | list[str] = Field(default="")`

Thanks! Will be addressed in our next sync.

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/commit/2d7561f02213781d603ed4ff1826f8dfe0963da4
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support task metadata
Root cause: Unimplemented Feature Gap

This feature would be quite useful for myself and I am wondering if folks are still open to supporting it. I don't even think it is particularly necessary to display the metadata field in `inspect view`, it would be useful even just for programmatically loading metadata about an eval run.

> Interestingly we have a `metadata` field in the `EvalSpec` (which is what this is drawing from) but no way to populate it! I wonder if should add a `metadata` field to `Task` and forward it here? (@dragonstyle LMK what you think)

_Originally posted by @aisi-inspect in https://github.com/UKGovernmentBEIS/inspect_ai/issues/50#issuecomment-2175918773_
            

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/656
Repository: UKGovernmentBEIS/inspect_ai
Issue: Incorrect targets in theory of mind dataset
Root cause: Validation Gap

The [theory of mind example dataset](https://github.com/UKGovernmentBEIS/inspect_ai/blob/0846c81cfd9cfec64f30c18682a010016abab40a/src/inspect_ai/dataset/_examples/theory_of_mind.jsonl) contains a few samples that I think have an incorrect target.

I think samples 27, 30, 33 and 99 all have the target of the "true belief" (that is, where the item truly is) whereas the question is asking for the "false belief" that is held by an actor who is not in the room when the item is moved.

For example, in [sample 27](https://github.com/UKGovernmentBEIS/inspect_ai/blob/0846c81cfd9cfec64f30c18682a010016abab40a/src/inspect_ai/dataset/_examples/theory_of_mind.jsonl#L27), because Owen exits the room and then Evelyn moves the shoes from the cupboard to the bucket, Evelyn should think that Owen thinks that the shoes are in the cupboard. However, the target answer of the sample is "bucket".

Interesting! This data was originally taken from OpenAI Evals (https://github.com/openai/evals/blob/main/evals/registry/data/theory_of_mind/LICENSE) which in turn seems to have drawn them from the SocialIQA dataset (https://allenai.org/data/socialiqa).

If there are problems in the targets, maybe best to take them up with the upstream? (just to confirm that there isn't something we are missing).

> If there are problems in the targets, maybe best to take them up with the upstream? (just to confirm that there isn't something we are missing).

Thanks, will do 👍 

Will close this for now, feel free to re-open once we've established what the right thing is.

What is the status of this @jrhender?

I checked the openai dataset (https://github.com/openai/evals/blob/main/evals/registry/data/theory_of_mind/tomi/test.jsonl) and looks like the aisi dataset is just the first 100 samples from that dataset. It includes the same mistake in sample 27.

OpenAI references a facebook repo https://github.com/facebookresearch/ToMi/tree/master which has somewhat intricate code to automatically generate scenarios. Decent chance there was bug from that - but repo has been archived. (The facebook repo credits another repo https://github.com/kayburns/tom-qa-dataset/blob/master/data/tom.zip)

I think we should manually adjust the examples in AISI. What is the downside of this? @jjallaire 

Just wanted to run it doen upstream in case that was a better place for the fix. Given that we've done that, I'd be fine fixing locally
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Doc] Security Guide --No such file or directory: 'security_guide.csv'
Root cause: Documentation Deficiency

TLDR: Cant find the file easily in open AI repo

If I follow the tutorial of 'Security Guide'.

Create a file called sec.py
'''
from inspect_ai import Task, eval, task
from inspect_ai.dataset import csv_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, system_message

SYSTEM_MESSAGE = """
You are a computer security expert and you are tasked with
providing concise response to the following questions.
Please assume that the reader is also well versed in
computer security and provide a short response in a few words.

@task
def security_guide():
    return Task(
        dataset=csv_dataset("security_guide.csv"),
        solver=[system_message(SYSTEM_MESSAGE), generate()],
        scorer=model_graded_fact(),
    )


"""
'''

Execute '$ inspect eval sec.py --model google/gemini-1.0-pro''

And then get 
'No such file or directory: 'security_guide.csv''

Then I proceed to search for the file in the linked Webpage https://github.com/openai/evals

And cant find it directly.
<img width="772" alt="Screenshot 2024-10-07 at 10 45 30" src="https://github.com/user-attachments/assets/1bc6396c-7933-4c9a-b14e-73973e791827">
<img width="1109" alt="Screenshot 2024-10-07 at 10 45 54" src="https://github.com/user-attachments/assets/a2e10a83-c578-43e9-aab0-85ba421546f7">


This is a minor issue and the file can be easily found with a bit of effort. But it creates a minimal user friction that can be solved very easily.

If the authors agree, I will be happy to do a PR solving the issue.

Thanks! I made a change to use `example_dataset()` instead which should resolve this! https://github.com/UKGovernmentBEIS/inspect_ai/commit/d25ba0c99ddc32d2b7d96f33512aed7f96c2d1d4
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] Inspect fails to run in the vs code python 3 devcontainer
Root cause: External Dependency Breakage

When starting up, Inspect sets up a folder to help store temporary files. It uses the https://github.com/tox-dev/platformdirs library (specifically the user_runtime_path function) to find an appropriate place to store these files.

On Linux systems without `$XDG_RUNTIME_DIR` set, this returns a path in `/run/user`. Unfortunately in [VS Code dev containers](https://code.visualstudio.com/docs/devcontainers/containers), there is no `/run/user` folder in the default Python container.

Upstream bug: https://github.com/tox-dev/platformdirs/issues/315

## Proposed next steps

- Wait to see if the [upstream bug](https://github.com/tox-dev/platformdirs/issues/315) is resolved in a reasonable time. It looks like it's already got some traction so I think this should be fixed fine. If it isn't, then we could explore patching this on the Inspect end of things.
- In the mean time, suggest the workaround of setting the `XDG_RUNTIME_DIR` environment variable manually to a folder with read/write permissions.

Another thing we could do is change to `user_state_dir()` ? (which defaults to `~/.local/state`). Our use of this is actually much more in the category of "state" than what is classically associated w/ the runtime dir. If you could confirm that this resolves the issue for you I'll make that change

Apologies for the delay responding! Yes, either the `user_state_dir()` or `user_data_dir()` appear to work fine 👍
Repository: UKGovernmentBEIS/inspect_ai
Issue: ID is inconsistent between list view and sample
Root cause: Algorithmic Error

In my eval dataset I have mapped `FieldSpec(id="number", ...` and subsequently filtered out some of the samples before passing it to the task, so the first one is id=6. But in the sample list view for the eval the first ID is `1`, even though when you click into the sample itself the id is correctly reported at `6`.

![image](https://github.com/user-attachments/assets/888aaa77-4cdb-4513-b6d0-e49203e0d16b)

![image](https://github.com/user-attachments/assets/87b64d9d-902a-49cd-9c05-a575dd117491)


Resolved by renaming the column to `#` to reflect that it is an index number not an ID.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature request] command line flag to capture the values of locals in the printed traceback
Root cause: Unimplemented Feature Gap

In
https://github.com/UKGovernmentBEIS/inspect_ai/blob/d76c54d83e57901a92e5b72e0c2cea83b26773a3/src/inspect_ai/log/_log.py#L430-L437
there should be a command line flag to allow changing
https://github.com/UKGovernmentBEIS/inspect_ai/blob/d76c54d83e57901a92e5b72e0c2cea83b26773a3/src/inspect_ai/log/_log.py#L435
to `show_locals=True`.

This would make debugging significantly easier.

Interestingly, we originally had `show_locals=True` but then multiple people noticed that it was possible for this to leak API keys into the logs. We will add an option to manually re-enble this though as I agree it's very good for debugging.

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1719
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Support live statistics in progress bar
Root cause: Unimplemented Feature Gap

Right now the progress bar that shows when one runs `inspect eval` does not show any running statistics. Sometimes a small fraction of evals can take twice as long or longer than the rest of evals, and it would be convenient in these scenarios to be able to view a running statistic to know how well the run is doing.

Note that we very soon be working on improved live task display (including both statistics as well as drilling down into what's happening w/ individual samples)

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/953

We currently show the first metric of the first scorer in the progress bar, but will very soon be adding support for optionally viewing all of the scorers/metrics/reducers for a given task.

Support for showing all scores here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/982
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect view memory leak
Root cause: Resource Mishandling

I've noticed the inspect view webpage causing a significant memory leak if left open. I believe the source of it is the [worker-based JSON parsing](https://github.com/UKGovernmentBEIS/inspect_ai/blob/6222e4604e954e78171cdac243ef1868c5af0456/src/inspect_ai/_view/www/src/utils/Json.mjs#L3). 

Here's a memory allocation instrumentation over time -

<img width="658" alt="image" src="https://github.com/user-attachments/assets/aa222fd6-e73c-4f42-bdd6-0fd5176867e6">

The blue segments of the bars represent the memory which is retained, relative to the grey which is the memory allocated and then collected.

Admittedly its a small leak, but as it happens every second, it can add up if left to run in the background.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature request] Retry with different `fail-on-error` strategy
Root cause: Unimplemented Feature Gap

Sometimes I run a task and it crashes, then I realise that I should have set `fail-on-error=False`.

In these cases I would like to call `eval-retry` but set `fail-on-error` to a different value so that I can reuse the samples that were already completed.

Thank you, that is an oversight! Addressed here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/682
Repository: UKGovernmentBEIS/inspect_ai
Issue: Error running `inspect score` with a scoring function that uses a task-level sandbox
Root cause: Configuration Error

I am trying to use a task-level sandbox in a scoring function for running generated code and it fails when running `inspect score` command with below error:

`RuntimeError: No sandbox environment has been provided for the current sample or task. Please specify a sandbox for the sample or a global default sandbox for the task`

The error does not appear when running entire evaluation pipeline with `inspect eval`.

Minimum reproduceable code:

```
from inspect_ai.dataset import Sample, MemoryDataset
from inspect_ai import Task, task
from inspect_ai.solver import generate, TaskState
from inspect_ai.scorer import Score, Target, scorer, accuracy, stderr, Scorer
from inspect_ai.util import sandbox


@task
def test_task():
    return Task(
        name=f"test task",
        dataset=MemoryDataset(
            samples=[
                Sample(
                    input="Hello, world!",
                )
            ]
        ),
        solver=generate(),
        scorer=test_scorer(),
        sandbox="docker",
    )


@scorer(metrics=[accuracy(), stderr()])
def test_scorer() -> Scorer:
    async def score(state: TaskState, target: Target):
        env = sandbox()
        return Score(value="C")
    return score

```
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Question] Display dataset record(Sample) metadata in the viewer main page
Root cause: Unimplemented Feature Gap

Is there a way to display additional columns in the Viewer main page? Currently only "Input", "Target", "Answer" and "Score" are displayed, and additional metadata is shown when record is clicked on. 
Is there a way to add additional columns to the main view? 

Not currently (cc @dragonstyle)
Repository: UKGovernmentBEIS/inspect_ai
Issue: support for web browser navigations that open new windows
GitHub labels: web browser

See PR w/ test case here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/680



Test case:
```html
<!DOCTYPE html>
<html lang="en">
<body>
    <a href="https://en.wikipedia.org/wiki/Charlie_Hunter" target="_blank" rel="noopener noreferrer">Click Me!</a>
</body>
</html>
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: `ValueError No model specified` for tutorial example when run in notebook
Root cause: Configuration Error

Error message is: `ValueError: No model specified (and no INSPECT_EVAL_MODEL defined)`.

To reproduce, run following code in a notebook. The error occurs in `self_critique()`

```python
dataset = example_dataset("theory_of_mind")[0:3]
scorer = model_graded_fact(model="openai/gpt-4o")

@task
def theory_of_mind():
    return Task(
        dataset=dataset,
        plan=[
            chain_of_thought(),
            generate(),
            self_critique(),
        ],
        scorer=scorer,
    )

model = "openai/gpt-4o-mini"
logs = eval(
    theory_of_mind(),
    model=model,
)
```

Thanks for this! Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/6a0d18005e766d75db0d860d06b05e662efd683d
Repository: UKGovernmentBEIS/inspect_ai
Issue: Model-based scorers are not able to access full chat histories
Root cause: Unimplemented Feature Gap

Scorers `model_graded_fact` and `model_graded_qa` can be used to assess an answer, given a question. The prompt for scoring is formed using `input_text` from the TaskState, that is, just the first user question ([code here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/6a0d18005e766d75db0d860d06b05e662efd683d/src/inspect_ai/scorer/_model.py#L131)).

This is not suffitient when evaluating later responses in chat, given the whole chat history: e.g. when evaluating the last response, the grading model should be given the transcript of the whole conversation, not just the first question.

I agree we need to make this more flexible. I don't love the approach in the PR of passing a solver though. 

We probably need to add an optional template variable to present the entire conversation. Will work on that soon!

Here is a PR that implements an `include_history` option: https://github.com/UKGovernmentBEIS/inspect_ai/pull/738/files

And a followup #741, which solves the issue perfectly. Thanks!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Make list_eval_logs recursive=True return the log_dir it read the log from 
Root cause: Documentation Deficiency

When plotting logs I make use of list_eval_logs, followed by categorising the logs by attributes in the log file. However, sometimes the only way to categorise logs further by putting them in a different directory, but list_eval_logs does not return this information and neither is this info present in the logs themselves. 

The `EvalLogInfo` object does have a `name` (which I believe is an absolute path to the log file). You could then remove the `log_dir` prefix from that? Would that do you what you need? If not LMK the gist of the core change you'd like here in pseudo-code (i.e. what you want returned) and we can see what we can do!


No, that's perfect, thanks for pointing that out!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Required Docker Version
Root cause: Validation Gap

In the latest version of inspect, docker has a minimum requirement of 24.0.7 and docker compose has a minimum requirement of 2.22.0. However, it seems as though this requirement is solely for the `docker compose pull` '--policy' option. However, this is only ever used if not building/using something local.
```python
if (
    service.get("build", None) is None
    and service.get("x-local", None) is None
):
```
Would it be possible to either override this check, or only check this requirement if using pull is necessary. I am sadly stuck with docker 24.0.6 and do not have permissions to upgrade. I'm happy to make a pull request if needed.

Do you know what version of Docker Compose is included in 24.0.6?

I think what we'd probably do here is ratchet the default versions requirements back and then run another Docker Compose version check right before we pull.

I'm stuck with Docker Compose version 2.21.0

Okay thanks, I'll bump it down to that then re-check if we are going to pull.

That change has been made here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/9cdf8e748d0357bb2b356116c0785d919047bbce