zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Is there a way to send eval logs to an API endpoint?
Root cause: Documentation Deficiency

Hi, I am the core maintainer of https://github.com/Scale3-Labs/langtrace. For context, langtrace is a fully open source and open telemetry based observability platform for LLM powered applications. We are interested in building native support for inspect_ai in a couple of ways:

1. Users can download annotated dataset from langtrace using langtrace's SDK and use it for running evals with inspect_ai.
2. Additionally, users can send the eval logs to langtrace over an API and visualize the report within langtrace.

For 2., I wanted to know if there is a way to send these logs to an API. I noticed that there is support for S3. If this is not in the roadmap, I am happy to contribute and submit a PR. Please let me know. Thanks

Currently, we can write logs to any fsspec compatible filesystem: https://filesystem-spec.readthedocs.io/en/latest/. fsspec has these backends built in (https://filesystem-spec.readthedocs.io/en/latest/api.html#implementations) and these third implementations (https://filesystem-spec.readthedocs.io/en/latest/api.html#external-implementations).  So one path for (2) is to implement an fsspec back-end.

Internally, we have a recorder interface however it's not ready to be externalized (there is too much implicit dependency on it being a traditional filesystem). We would eventually like to make this more generic, but for the time being I'd suggest that the fsspec approach would create much more seamless integration. Note that you can use a setuptools entrypont to make your filesystem available (https://filesystem-spec.readthedocs.io/en/latest/developer.html) so if your package was installed then users could just do e.g. `langtrace://` to access your backend.

Note that this would also work for datasets (so users could also load datasets using `langtrace://`, etc.)



Thanks @jjallaire for the detailed answer. I was unaware of the fsspec compatibility but this is neat and makes a lot of sense. We will go ahead with the fsspec implementation for our backend and keep you all posted on how it goes. This answers my original question and I will close this issue now. Thanks again

Great, glad that works well for you. I am working on adding some more explicit documentation about this (including specifically which fsspec methods we call, as it's definitely a small-ish subset that we rely on).

Here are additional docs on implementing a custom storage provider for Inspect via fsspec: https://ukgovernmentbeis.github.io/inspect_ai/extensions.html#storage

This is very helpful. Thank you again!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Order of samples in output is different from dataset
Root cause: Documentation Deficiency

This happens, even if shuffle parameter is false. 

How do I make sure that order of samples in output json log file matches order in the incoming dataset. 

Thank you!

We sort the samples by sample id, which will match the order of the incoming dataset if the sample IDs are auto-assigned. We also now display the samples in dataset order within the Inspect Viewer.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Bug] Custom metrics can't access any other `Score` attributes except `value`
Root cause: Validation Gap

With the introduction of score reducers for epochs in PR #148, the default reducer is executed even if you didn't specify any epochs, i.e. have only one epoch.

Because all built-in reducers return a reduced `Score` with only the `value` attribute (all use `_reduced_score()` in `scorer/_reducer/reducer.py`), you lose the other attributes (`answer`, `explanation` & `metadata`) when the `Score` object reaches your metrics function.

Don't think this was the intended behavior because `reduce_scores()` in `_eval/task/results.py` references these other attributes when creating `SampleScore` objects _after_ the execution of the reducer (but they are all `None` by then).

Additionally, these other attributes are inconsistently referenced on several places where `SampleScore` objects are created, on one place the `answer` attribute is missing, on three other places the `metadata` attribute is missing.

The workaround is to create your own score reducer that would just return the score without changing it:

```python
@score_reducer(name="first")
def first_score() -> ScoreReducer:

    def reduce(scores: list[Score]) -> Score:
        return scores[0]

    return reduce
```

(But you won't get the `Score.answer` attribute because it's missing when the `SampleScore` is created in the `reduce_scores()` function mentioned above.)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Error when getting logprobs for EOF token from OpenAI API
Root cause: Validation Gap

Sometimes, when I try to get top logprobs via the OpenAI API. I get the error pictured below. The issue seems to be that the OpenAI sometime returns a logprobs object for which the bytes field is `None`, e.g. `{'token': '
', 'bytes': None, 'logprob': -28.681221},`, but pydantic expects the bytes field to be a list. I _think_ this could be fixed by changing the type signature for `LogProbs` in inspect, but I'm not sure. (I tried doing this and for some reason pydantic was still complaining, maybe I made some mistake though.)

Here's the full [log file](https://drive.google.com/file/d/1V_nX9xooOjKS8gfvoG2OEGwHo1dZmEzg/view?usp=sharing)

<img width="1281" alt="image" src="https://github.com/UKGovernmentBEIS/inspect_ai/assets/29761887/01c7e4f9-cc8d-4fc7-943b-e8c9475d6f0c">
<img width="761" alt="image" src="https://github.com/UKGovernmentBEIS/inspect_ai/assets/29761887/dd47f589-cea2-4dde-b0be-ed2cc3fcf4ac">
<img width="680" alt="image" src="https://github.com/UKGovernmentBEIS/inspect_ai/assets/29761887/ba083a50-85f6-444d-bd08-67af847dcdaf">



Yes, this should be a simple schema fix! Do you think it would be legit for us to covert the `None` into an empty list? (just so that readers of the log can depend on having a list vs. having to check for `None`).

This should be resolved with this commit: https://github.com/UKGovernmentBEIS/inspect_ai/commit/8141ef25f2eef2cb23c8ac8a4d2ee02872db6549 (I added `None` to the schema for `bytes`)
Repository: UKGovernmentBEIS/inspect_ai
Issue: log viewer: format target using markdown
Root cause: Unimplemented Feature Gap

In the inspect log viewer, the `Scoring` tab for a sample shows the `Target` and `Answer` in two columns. The Answer is rendered via markdown, but the target isn't. This makes it a little harder to view longer target answers, esp. one that use model-graded-qa and have some code formatting in them.

It looks like this is implemented in `SampleScoreView.mjs` which includes the two columns like this:

```
            <td style=${{ paddingRight: "2em", paddingLeft: "0" }}>
              ${arrayToString(sample?.target || "none")}
            </td>

            <td>
              <${MarkdownDiv} 
                class="no-last-para-padding"
                markdown=${
                  shortenCompletion(answerForSample(sample)
              )} style=${{ paddingLeft: "0" }}/>
            </td>
```

@dragonstyle any reason not to also do marking rendering for target?

There is definitely no reason not to render the target there. I think it also worth considering rendering them in the sample list and header, which I'll tinker with. Should see some changes in an upcoming release!

This should be resolved with our 0.3.15 release (available on PyPI now).
Repository: UKGovernmentBEIS/inspect_ai
Issue: Is there a way to use the Open AI Assistant API?
Root cause: Unimplemented Feature Gap

I would like to compare the same evaluation using an Open AI model and a custom assistant.

https://platform.openai.com/docs/api-reference/assistants

No, we don't currently have support for the assistants API. You could however create a custom model extension to make this work: https://inspect.ai-safety-institute.org.uk/extensions.html#sec-model-api-extensions. I think there would be a bunch of decisions about how to map a simple message/tool interface into the more complex and stateful assistants API.

Closing this as we don't have plans to implement the Assistants API. Note that thread management is already built in to Inspect and execution of Python code via Docker sandboxes is also built in. We've also tracked the community response to assistants and have found that people are quite unhappy with the number of tokens it uses compared to just using chat completions w/ custom tools.

Again though, you can certainly write a custom model extension to experiment with this if you like!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Transcript does not show events for some logs
Root cause: Algorithmic Error

On this log: [2024-09-18T14-11-19+00-00_agent-test-tasks-hello-world-autonomous-systems-agent_5wMS8j9BEqcgSBLKmdHRkB.json](https://github.com/user-attachments/files/17045441/2024-09-18T14-11-19%2B00-00_agent-test-tasks-hello-world-autonomous-systems-agent_5wMS8j9BEqcgSBLKmdHRkB.json) me and Joe have found that the drop down for transcript is broken (here is a screenshot where the "view transrcipt" arrow is down, but no transcript is shown). This is on the version : 0.3.30.dev22+g8a4e5a87
![image](https://github.com/user-attachments/assets/7b3386a6-8d2f-4294-825d-0e16b91369db)





We think that we saw it work and then not work for the same log, hopefully reproducing is relatively easy.
Repository: UKGovernmentBEIS/inspect_ai
Issue: First party support for Inspect on Langtrace

Hi all,

It's me again - core maintainer of Langtrace(https://github.com/Scale3-Labs/langtrace). Wanted to share that, we added first party support for Inspect on Langtrace. What does this mean?
1. Langtrace users can now run evals using Inspect against their traced and curated datasets from Langtrace.
2. The eval results can be reported back to Langtrace for storage and viewing. We have built some nice features for reporting like comparison mode, and have few more things in the hopper.

How did we do this?
As suggested https://github.com/UKGovernmentBEIS/inspect_ai/issues/39#issuecomment-2156812620, we went ahead and implemented a fsspec filesystem implementation which takes care of reading datasets and writing logs to our backend.

You can find our docs here: 
- https://docs.langtrace.ai/features/evaluations
- https://docs.langtrace.ai/features/compare-evaluations

Attaching some screenshots:
![image](https://github.com/UKGovernmentBEIS/inspect_ai/assets/105607645/edd1e24b-26fb-4508-9939-58704ceda13d)
![image](https://github.com/UKGovernmentBEIS/inspect_ai/assets/105607645/9ebaf628-45c3-4c27-bdaa-51c89c2f8edd)


Thanks again for building this tool and also giving us a path forward for integrating it. We would love to hear your feedback/thoughts. 


Thanks fantastic! Thanks so much for sharing.

One thing you might want to add to your docs is using Inspect's support for `.env` files to set both the `LANGTRACE_API_KEY` and `INSPECT_LOG_DIR` (then they don't need to specify it explicitly in the CLI for either `eval` or `view`. More docs on this here: https://ukgovernmentbeis.github.io/inspect_ai/workflow.html#sec-workflow-configuration



Thanks @jjallaire . That's a great feedback. I will update our docs accordingly. 

Thanks again for letting us know about this, will close the issue as I think you've gotten the feedback we had for now.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Error in Inspect View: Cannot read properties of undefined (reading 'length') at generatePreview 
Root cause: Validation Gap

I get the below error trying to read a log (available on request). Inspect version [1913a045](https://github.com/UKGovernmentBEIS/inspect_ai/commit/1913a045957f20bd269352a1a49a18a5674b086b)

```
Error: Cannot read properties of undefined (reading 'length')
at TypeError: Cannot read properties of undefined (reading 'length')
    at generatePreview (http://localhost:7575/assets/index.js:15721:43)
    at b.StateEventView [as constructor] (http://localhost:7575/assets/index.js:15698:25)
    at b.q$1 [as render] (http://localhost:7575/assets/index.js:260:15)
    at O (http://localhost:7575/assets/index.js:182:45)
    at S (http://localhost:7575/assets/index.js:94:214)
    at O (http://localhost:7575/assets/index.js:184:195)
    at S (http://localhost:7575/assets/index.js:94:214)
    at z$1 (http://localhost:7575/assets/index.js:231:41)
    at O (http://localhost:7575/assets/index.js:192:88)
    at S (http://localhost:7575/assets/index.js:94:214)
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add the error type of the sample Error UI
Root cause: Unimplemented Feature Gap

When scrolling through the UI, you can't tell the type of sample errors without clicking into the sample. This is sad when debugging and trying to understand which errors are causing what percentage of issues. Seperately (which might go hand in hand), it might be nice to be able to programatically access the type of these errors.

@dragonstyle I sent you the log pictured below so you can iterate on it.

![image](https://github.com/user-attachments/assets/9c659f5d-0155-4a15-b17d-165a292fe2ce)
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Bug report] VS Code extension does not show `Run` and `Debug` Buttons when task function signature is on multiple lines

## Issue description
In the below snippet, the first task `SimpleTask` has the `Debug Task` and `Run Task` buttons but the second task `SimpleTask2` does not.

<img src="https://github.com/user-attachments/assets/c5e92afc-95a1-4839-b72e-77a8e35330d9" width="500">

The only difference is that `SimpleTask2` has the task function signature across multiple lines.

Everything else seems to work fine (e.g. `cmd+shift+T` and `cmd+shift+U`)

<details> <summary> Source code </summary>

```python
from inspect_ai import Task, task
from inspect_ai.dataset import MemoryDataset, Sample
from inspect_ai.scorer import model_graded_fact


@task
def SimpleTask(foo: str):
    return Task(
        dataset=MemoryDataset([Sample(input="")]),
        plan=[],
        scorer=model_graded_fact(),
    )


@task
def SimpleTask2(
    foo: str,
):
    return Task(
        dataset=MemoryDataset([Sample(input="")]),
        plan=[],
        scorer=model_graded_fact(),
    )
```

</details>

## System info
* Cursor Version: 0.40.4
* Underlying VSCode Version: 1.91.1
* Commit: 63dae81aea2c3fc079420fe72578a8eef26ba870

I'll have a look this AM - you're right and this definitely should be working!

Thank you Charles! 🙏
Repository: UKGovernmentBEIS/inspect_ai
Issue: Non-scrollable content when only one test case
Root cause: Validation Gap

If there's only one one piece of data in the dataset, the inspect web view seems to open in a special mode. Normally there's a table and you can click into elements to view details. However with a single test, the details shows below the row. This makes sense, however the details view seems to not be scrollable with long content (missing some sort of `overflow-y: auto`).

<img width="713" alt="image" src="https://github.com/UKGovernmentBEIS/inspect_ai/assets/2024018/d9746d3e-18ad-4b33-825d-1d115628813a">

Specifically, if I change that to `overflow-y: auto` from `hidden`, I can scroll, otherwise the container is not scrollable and content is clipped

I see there's some code that seems like it should be setting it to scrollable.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/8141ef25f2eef2cb23c8ac8a4d2ee02872db6549/src/inspect_ai/_view/www/src/workspace/WorkSpace.mjs#L81

I did confirm I'm on the latest released version 

```
inspect --version
0.3.14
```

https://github.com/UKGovernmentBEIS/inspect_ai/commit/0c77feab2f11f75980403ce00adb0ca02b4bd463#diff-0b4b6031b5d6f309f5dde139c9e0def5010ebc3a38e4129d3a357d7c89d44670

Looks like this perhaps has just not been released yet. 0.3.14 is before that commit

Yes, we are release 0.3.15 later today. In the meantime you can install with:

```python
pip install git+https://github.com/UKGovernmentBEIS/inspect_ai
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: `hf_dataset(..., shuffle=True)` does nothing
Root cause: Algorithmic Error

hf.py line 86:
```
if shuffle:
        dataset.shuffle(seed=seed)
```
This call expects `Dataset.shuffle` to execute an in-place mutation, which it does not; it returns a new `Dataset` object.
Revised code:
```
if shuffle:
        dataset = dataset.shuffle(seed=seed)
```

inspect-ai version: 0.3.30
datasets version: 3.0.0

Ouch! Thanks for reporting this. Fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/477
Repository: UKGovernmentBEIS/inspect_ai
Issue: Tool calling masks Type Errors coming from the tool (and  likely other errors as well)
Root cause: Validation Gap

The tool calling code catches all type errors from the tool:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/4af809d722146fd549a1937517c24c86b08e7ddc/src/inspect_ai/model/_call_tools.py#L172-L182

This means that any actual type errors from the tool end up being caught, and returned to the model. This leads to errors in the tool code being returned to the model, instead of raised outside of the tool. See here for an example
[2024-09-23T20-05-50+00-00_benchmarks-needle-in-the-haystack_h6HJAgBmfAwAvFNdqVwNoo.json](https://github.com/user-attachments/files/17103639/2024-09-23T20-05-50%2B00-00_benchmarks-needle-in-the-haystack_h6HJAgBmfAwAvFNdqVwNoo.json)

I think this would also be true of the code at, although I haven't ran into it.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/4af809d722146fd549a1937517c24c86b08e7ddc/src/inspect_ai/model/_call_tools.py#L64-L78




I think we can get rid of this. This was originally here to deal with the model calling w/ the wrong types, however we now handle mapping str to first arg and doing JSON schema validation so there is no real reason to handle this separately. 

And should I be worried about the other errors?

https://github.com/UKGovernmentBEIS/inspect_ai/pull/485

The other error should be caught as they are. This is discussed here: https://inspect.ai-safety-institute.org.uk/tools.html#tool-errors

The idea is that people using the default tool loop (`generate()`) should get automatic error reporting to the model rather than their eval blowing up. That said, people writing custom agents may want to handle these errors in more specific fashion. We don't cover this in the docs, but type info and error messages on the tool errors are encoded in the ChatMessageTool, so if you want to handle these and prevent the model from seeing some errors, you can just inspect the ChatMessageTool for specific errors and act accordingly.


Ah great makes sense thank you - apologies, I should have known that.

i.e. in our docs on writing your own tool loop, we show this:

```python
if output.message.tool_calls:
        state.messages.extend(call_tools(output.message, state.tools))
```

Rather than just blindly passing them to the model, you can check them for specific error conditions that make sense for your agent.
Repository: UKGovernmentBEIS/inspect_ai
Issue: typo in docs/tutorial.qmd


@task
def math(shuffle=True):
    return Task(
        dataset=hf_dataset(
            "hendrycks/competition_math",
            split="test",
            sample_fields=sample_fields=FieldSpec( #ERROR: it should be just sample_fields=FieldSpec( 
                input="problem", 
                target="solution"
            ),
            shuffle=True,
            trust=True,
        ),
        plan=[
            prompt_template(PROMPT_TEMPLATE),
            generate(),
        ],
        scorer=expression_equivalence(),
        config=GenerateConfig(temperature=0.5),
    )


sample_fields=sample_fields=FieldSpec( #>>>ERROR: it should be just sample_fields=FieldSpec(


Thanks! Fixed with https://github.com/UKGovernmentBEIS/inspect_ai/commit/52426a1cde5898b5334b5280a92d63d4ecb45bcc
Repository: UKGovernmentBEIS/inspect_ai
Issue: HTML tags stripped out in the sample viewer
Root cause: Validation Gap

I am evaluating a model which is being asked to do chain-of-thought reasoning within a private "scratchpad" which is marked using the `<scratchpad> </scratchpad>` tags and I noticed that while the tags shows up in the preview of a sample they do not show up when I actually click to view a sample.

I understand this is probably done to prevent XSS vulnerabilities but I think escaping them might be a better solution than stripping them out entirely as I can think of plenty of use-cases where you would want to know that a model produced HTML code (for instance, if you are trying to see if it can produce an XSS vulnerability).

@dragonstyle does our new escaping treatment actually show the <thinking> tags? (I believe it does, just want to make sure)

@Suhas-13 If I am correct, our current `main` has this fixed (we'll hopefully be releasing this to PyPI this weekend). In the meantime you can install from GH with:

```python
pip install git+https://github.com/UKGovernmentBEIS/inspect_ai.git
```

This should be resolved with our 0.3.15 release (available on PyPI now).
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] make `subtask` nodes collapsible in the viewer
Root cause: Unimplemented Feature Gap

Currently a task output with subtasks looks something like this:

<img src=https://github.com/user-attachments/assets/6e7ce0ea-8dc3-4898-ad57-40941e6196ba width="500">


For tasks with many subtasks and/or where the summary/transcript is very long, it would be really helpful to be able to collapse the `SUBTASK` node via a toggle in the top left corner like we can do for the parent nodes.

Currently I work around this by adding `transcript.step()` to the subtask, but this creates a lot of nesting and clicking
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] allow configuring tasks from a file
Root cause: Unimplemented Feature Gap

## Summary
For tasks with many parameters we would like the ability to configure the task from a file in addition to CLI.

##  Requirements/Desiderata
* It should still be possible to override any values from the CLI
* Ability to define default sets of arguments for different variants of a task
* Config files should be structured in a way that enables extending configs in a hierarchical way

## Possible solution
* Discussed with JJ the idea of adding `--task-config` param to `inspect eval`

## Notes
* [hydra](https://hydra.cc/docs/intro/) and [hydra-zen](https://mit-ll-responsible-ai.github.io/hydra-zen/) solve this quite nicely. Maybe we could integrate with or take inspiration from them. 

Possibly resolved by https://github.com/UKGovernmentBEIS/inspect_ai/pull/778