zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: csv_dataset is not reading csvs properly
Root cause: Documentation Deficiency

**Issue:** 
When i try to import a csv, the first column name, which is 'input', is being read in with a byte order mark. 

Which is causing the value error 'no input in dataset'.

**Expected behaviour:**

csv_dataset should be flexible enough to handle this.

**Example output:**

>>> from inspect_ai.dataset import csv_dataset, json_dataset
>>> import pandas as pd
>>> dataset1 = csv_dataset("dataset1.csv")
**{'\ufeffinput':** 'which cheese is from switzerland', 'target': 'swiss'}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
 
    raise ValueError("No input in dataset")
ValueError: No input in dataset
>>> pd.read_csv("dataset1.csv")
                              input target
0  which cheese is from switzerland  swiss


nevermind, found the encoding argument!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Clarification on Multiple Choice Question Handling in Inspect
Root cause: Documentation Deficiency

# Issue description
I'm new to the Inspect framework and am experiencing some confusion about the proper way to handle multiple-choice questions (MCQs). I'm unsure if I'm misunderstanding the documentation or if there are aspects of MCQ handling that could be clarified.

I'm preparing my dataset like this:
```python
Sample(
    id=str(row['Record ID']),
    input=row['Question'],
    target=row['Correct Answer'],  # Full text answer
    choices=choices,
    metadata={...}
)
```

where `choices` is a list of string answer options like `["choice 1 text", "choice 2 text", "choice 3 text", "choice 4 text"]` and `target` is the correct choice text string like `"choice 3 text"`.  I'm unsure if this is the correct/simplest approach for MCQs.

When using `multiple_choice()` solver with `choice()` scorer, it seems the scorer expects a letter (A, B, C, D) as the target, but receives the full text answer instead. This leads to incorrect scoring.
```python
@task
def gpqa_evaluation(n: int = len(dataset)):
    return Task(
        dataset=dataset[:n],
        plan=[
            multiple_choice()
        ],
        scorer=choice()
    )
```

I could pre-compute the target letter based on the `choices` index, but this seems like it would break the shuffling of choices during MCQ scoring.

# Documentation confusion
* The [Solvers documentation](https://inspect.ai-safety-institute.org.uk/solvers.html) states to "always pair [`multiple_choice()`] with the `choice()` scorer."
* However, the [Scorers documentation](https://inspect.ai-safety-institute.org.uk/scorers.html) doesn't mention `choice()` as a built-in scorer.
* The [HellaSwag example](https://inspect.ai-safety-institute.org.uk/examples.html#sec-hellaswag) uses `answer("letter")` as a scorer, which isn't documented in the Scorers section.

# Questions
* What is the recommended way to prepare a dataset for multiple-choice questions in Inspect? Should the target be a letter, an index, or the full text of the correct answer?
* How can I ensure that the shuffling of choices is properly handled in relation to scoring? Is there a built-in way to track the correct answer when choices are shuffled?


Apologise, I agree the docs are not clear or cohesive here. The right way to do it is to always encode the `target` as a letter. For example see these benchmarks:

- https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/benchmarks/arc.py
- https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/benchmarks/hellaswag.py
- https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/benchmarks/piqa.py

We will work to improve the docs ASAP!



Here are the improved docs: https://inspect.ai-safety-institute.org.uk/solvers.html#sec-multiple-choice

Hopefully this makes things much more clear! (don't hesitate to ping us back or open up a new issue if not)
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Question] Recommended approach to have execution time of generate for each sample added to eval log scores
Root cause: Documentation Deficiency

I created a custom model that uses streaming API to communicate with my model's endpoint.
I'd like to record execution times inside eval logs as scores, there will be at least a couple of different times I'm interested in:
1. total time of executing model's `generate()` 
2. time to first token while receiving streamed tokens.

In case of 1. we developed custom solver that extends `generate()` solver by recording and adding execution time to `TaskState`'s metadata. Subsequently, a custom scorer reads execution time from metadata. (I wish something of that sort were built-in but it's a different topic).

In 2d case I can't think of anything but the custom model taking care of collecting time to first token: the question is how to store and pass that down the plan to a scorer that would read it? Is it possible to extend and use `ModelUsage` somehow or something of that sort?

If there is a completely different approach to do this, please explain.

The "store" is something we recently introduced for sharing state across solvers and scorers: https://inspect.ai-safety-institute.org.uk/agents-api.html#sharing-state. Note that you can access the store from anywhere via the `store()` function so you could have your custom model write to it.

thanks, that should work. Just to confirm that by importing
```
from inspect_ai.solver import store
```
inside my custom model I will gain full access to the `Store` for the current sample. 

Another idea was using `ModelUsage` to be able to add more info but that is not supported right now.

Yes, correct, the the `store()` function will return the store for the currently executing sample, which you can also access within solvers and scorers.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Differing input token count for image samples
Root cause: External Dependency Breakage

I noticed an odd behaviour I haven't been able to fully explain:

When running a task containing image samples, the displayed input token count seems to differ wildly between models. When testing this with a single sample (a 600kb JPEG of a dog and the input "What animal is this?"), Inspect shows ~1000 input tokens for Claude 3.5 Sonnet, Claude 3 Haiku, or GPT-4o, but ~26,000 input tokens for GPT-4o-mini. 
All models receive the image and answer the question correctly and all logs include the base64 image string in samples > transcript > content. 

Is this a display error on Inspects side or a difference in how the various model APIs handle image inputs? I'd like to understand the behaviour better so I can calculate the expected API costs before running our large image-based eval on various models. 

I am not sure what is causing that difference! Those errors are very unlikely to be display errors. One way to check on this is that we now log all of the raw model API calls (you can see this in the Transcript view) which include token counts. These are the "raw" counts being reported by the model provider. It's possible that we somehow mishandle those but I can't imagine why this would differ by model. It could also be incorrect reporting by GPT-4o-mini?

One possible other thing to investigate is that OpenAI makes available a `detail` parameter: https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/_util/content.py#L14-L26. Perhaps GPT-4o-mini has a different default for "auto"?

I'm going to close this assuming its a provider reporting issue.

I was equally puzzled by this and probably you already figured out why that is. Anyways, here is a reference

> The dollar price per image is the same for GPT-4o and GPT-4o mini. To maintain this, GPT-4o mini uses more tokens per image. Thank you for your observation!

https://community.openai.com/t/gpt-4-o-mini-vision-token-cost-issue/989143/2
Repository: UKGovernmentBEIS/inspect_ai
Issue: Is there a recommended way to run a plan outside an eval?

Let's say I've built a suite of evals for my task, and I've gotten it working well enough that I'd like to deploy a plan to production.

Is there a path today to go from a working plan/config in the evaluation suite to running that plan/config in production?

It seems like one way to do this would be to replicate this loop in my own code: https://github.com/UKGovernmentBEIS/inspect_ai/blob/b6b75a369eb702fca518c03d893f5067301151ec/src/inspect_ai/_eval/task/run.py#L258

... and then extract what I need from the TaskState structure to do whatever I want to with the output.

Any reason that would be inadvisable?

While this isn't really the design center of Inspect (which is fundamentally a harness of testing/evaluation) there is no reason you can't do something like what you are proposing. You would really just need to construct a `TaskState`, run the loop over the plan, and then inspect the `TaskState` at the end for output.

Thanks that's helpful!

What can folks expect in terms of stability of things like the solver interface? Or how to go about instantiating generators?

The sovler interface is stable.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Too large task metadata causes sample_init to crash on launch
Root cause: Architectural Constraint

The sample metadata is passed in the CLI to the call to docker compose , as in https://github.com/UKGovernmentBEIS/inspect_ai/blob/62ce52c663d7c353edd2a916241ace8eca45dc0b/src/inspect_ai/util/_sandbox/docker/docker.py#L89-L97

This means you can't have too large of a metadata, as you run into the limits on the size of arguments you can pass through the CLI. This happens naturally in the case of swebench (for #268 ), where if we pass all of the swe-bench dataset metadata into the sample (which includes the golden agent patch, and the list of tests to run), its too large. 

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/commit/f687864a54b00b46251a794aef8e0ef4f4120746
Repository: UKGovernmentBEIS/inspect_ai
Issue: Is there any way to run this with model from Azure OpenAI
Root cause: Documentation Deficiency

We are using Azure OpenAI model that endpoint is deployed into Azure. Is there any way evaluate the LLM model/service? Thanks!

Yes, definitely! Here are the docs on that: https://ukgovernmentbeis.github.io/inspect_ai/models.html#azure-ai
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support bedrock/meta.llama3-70b-instruct-v1
Root cause: Unimplemented Feature Gap

Currently errors with
```
  File "/Users/tekumara/code3/inspect_ai/src/inspect_ai/model/_providers/bedrock.py", line 66, in __init__
    raise ValueError(f"Unsupported Bedrock model: {model_name}")
ValueError: Unsupported Bedrock model: meta.llama3-70b-instruct-v1
```

Thanks for reporting this! Should be resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/aab1d693882ed7ffd2b6582d141b2e7db8621403
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Support GenerateConfig options in default generate() solver
Root cause: Unimplemented Feature Gap

When putting together a plan, it would be great to be able to change certain `GenerateConfig` options for different `generate()` steps in the plan.
For example, I have a task with two generate calls, where I'd like only the second one to be limited to 30 tokens. My current solution is this:

```python
@solver
def generate_with_config(
    **kwargs: Unpack[GenerateConfigArgs],
) -> Solver:
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        return await generate(state, **kwargs)

    return solve

# ...

@task
def my_task():
    return Task(
        plan=[
            system_message(SYSTEM_MESSAGE),
            generate(),
            # ... other plan steps ...
            generate_with_config(max_tokens=30),
        ],
    )

```

But it would be great to have this with the default `generate()` function, either via kwargs or some sort of `config` parameter. Happy to open a PR for this if you think it makes sense!

Was easy to add! Now merged to `main`: https://github.com/UKGovernmentBEIS/inspect_ai/pull/362
Repository: UKGovernmentBEIS/inspect_ai
Issue: [BUG] Inspect silently collapses system and user messages with no warning for HF models
Root cause: Algorithmic Error

Inspect currently assumes that HF models don't allow for system messages, which is not necessarily true. 

When chat history with a system prompt is passed into a HF model, the function "simple_input_messages" concatenates them into one string which results in unexpected behaviour when the model would otherwise support a system prompt.

I would like this to be an optional feature that's off by default, or at least a warning message of this happening and a way of turning this off.
I am running queries on a fine-tuned model that has been finetuned with a specific system message, and the unexpected change in the tokenisation is breaking the expected model behaviour.

[line with "simple_input_messages"](https://github.com/UKGovernmentBEIS/inspect_ai/blob/c1ede04cbca1b1b6c8ac223265069323a0a1587d/src/inspect_ai/model/_providers/hf.py#L198)

For example, Llama3 8B Instruct supports system messages, so the following input:
`[{"role": "system", "content": "You are a helpful assistant designed to answer questions briefly."}, {"role": "user", "content": "What is the capital of France?"}]`

with the model's normal tokeniser should end up tokenised as 
```


system
. 

You are a helpful assistant designed to answer questions briefly.

user
. 

What is the capital of France?

assistant
. 
```
[based on the model card and its tokeniser output](https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/)


But instead, it gets tokenised as 
```


user
.   

You are a helpful assistant designed to answer questions briefly.    

What is the capital of France?

assistant
.   
```







According to these docs chat templates know about system messages: https://huggingface.co/docs/transformers/main/en/chat_templating

So I think the solution is probably just to revert that behavior and assume it will be handled by `apply_chat_template()`. I will give that a shot!

This is now merged to `main`: https://github.com/UKGovernmentBEIS/inspect_ai/pull/363

LMK if this doesn't have the expected behavior....
Repository: UKGovernmentBEIS/inspect_ai
Issue: inspect eval with task name - IndexError: list index out of range
Root cause: Validation Gap

```
❯ inspect eval security_guide --model bedrock/meta.llama2-70b-chat-v1
Traceback (most recent call last):
  File "/Users/tekumara/code3/inspect_ai/.venv/bin/inspect", line 8, in <module>
    sys.exit(main())
  File "/Users/tekumara/code3/inspect_ai/src/inspect_ai/_cli/main.py", line 45, in main
    inspect(auto_envvar_prefix="INSPECT")
  File "/Users/tekumara/code3/inspect_ai/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/tekumara/code3/inspect_ai/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/tekumara/code3/inspect_ai/.venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/tekumara/code3/inspect_ai/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/tekumara/code3/inspect_ai/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/tekumara/code3/inspect_ai/src/inspect_ai/_cli/common.py", line 46, in wrapper
    return cast(click.Context, func(*args, **kwargs))
  File "/Users/tekumara/code3/inspect_ai/src/inspect_ai/_cli/eval.py", line 238, in eval_command
    eval(
  File "/Users/tekumara/code3/inspect_ai/src/inspect_ai/_eval/eval.py", line 102, in eval
    return asyncio.run(
  File "/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/tekumara/code3/inspect_ai/src/inspect_ai/_eval/eval.py", line 219, in eval_async
    sample_source = eval_log_sample_source(eval_log, eval_tasks[0].dataset)
IndexError: list index out of range
```

[These docs](https://ukgovernmentbeis.github.io/inspect_ai/models.html#:~:text=%24-,inspect%20eval%20security_guide,-%2D%2Dmodel%20openai/gpt) suggest this should work.

Works fine with:
```
inspect eval examples/security_guide.py --model bedrock/meta.llama2-70b-chat-v1
```



Those docs don't assume that you are in the inspect_ai repo, they are just generically about selecting models w/ a given (fictional) eval target. Probably would be better if we used something fully fictional though not one of our own examples!

Hi @jjallaire I can reproduce this outside the inspect_ai repo when trying to run an eval using it's task name rather than python file name.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Viewing all tasks in static webpage
Root cause: Unimplemented Feature Gap

In the documentation there's a description of how to host a static webserver to embed the viewer for sharing. I was wondering if there was a way to keep the selection menubar so that visiters can click through different tasks similar to what you see when you run `inspect view` rather than having to specify `/?log_file=` in the url to see a specific task?

There will be soon (we are working on that right now) cc @dragonstyle 

This feature has just been merged to `main`: https://inspect.ai-safety-institute.org.uk/log-viewer.html#sec-publishing
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Bug report] `hf_dataset` crashes when `split` is not given
Root cause: Validation Gap

I mistakenly didn't give `split` argument when I tried to load dataset "umd-zhou-lab/sRecycled_Alpaca". Then dataset object is loaded with class `DatasetDict` instead of `Dataset`.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/8806522e94c5f8c73fc3a943d8974d978e9f6dec/src/inspect_ai/dataset/_sources/hf.py#L76-L83

```
DatasetDict({
    train: Dataset({
        features: ['data'],
        num_rows: 37114
    })
})
```

```
Dataset({
    features: ['question', 'subject', 'choices', 'answer'],
    num_rows: 112
})
```

When dataset is `DatasetDict`, it doesn't have `to_list` member function and the following line crashes. 

https://github.com/UKGovernmentBEIS/inspect_ai/blob/8806522e94c5f8c73fc3a943d8974d978e9f6dec/src/inspect_ai/dataset/_sources/hf.py#L95


This error is too silent and subtle, so there should be more informative guide to users.

Thanks for reporting this and apologize it took so long to resolve! https://github.com/UKGovernmentBEIS/inspect_ai/pull/965
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect view does not show input
Root cause: Validation Gap

After checking out the repo at commit [0a684c6](https://github.com/UKGovernmentBEIS/inspect_ai/commit/0a684c6285b0b1b30ddb0c7d4182d963e7dd00ee) and running:

```
inspect eval examples/security_guide.py --model bedrock/meta.llama2-70b-chat-v1
inspect view
```

Then navigating to http://localhost:7575/ I see a page without the input column:

![CleanShot 2024-06-08 at 16 33 49@2x](https://github.com/UKGovernmentBEIS/inspect_ai/assets/125105/1252e7cd-d8b8-4ab0-a20b-43de900934e9)


Good catch! We will take care of this ASAP. I am assuming this is a duplicate of #38 ? Will close assuming that it is, feel free to re-open if this is another issue.

👍 The behaviour I'm seeing here is different from #38 (although maybe the solution is the same)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect view is missing column headers
Root cause: Algorithmic Error

After checking out the repo at commit [0a684c6](https://github.com/UKGovernmentBEIS/inspect_ai/commit/0a684c6285b0b1b30ddb0c7d4182d963e7dd00ee) and running:

```
inspect eval examples/security_guide.py --model bedrock/meta.llama2-70b-chat-v1
inspect view
```

Then navigating to http://localhost:7575/ I see a page without the column headers:

![CleanShot 2024-06-08 at 16 33 49@2x](https://github.com/UKGovernmentBEIS/inspect_ai/assets/125105/1252e7cd-d8b8-4ab0-a20b-43de900934e9)

The docs has this image with column headers Input, Target, Answer, Score:

![](https://ukgovernmentbeis.github.io/inspect_ai/images/inspect-view-home.png)


cc @dragonstyle 

Thanks for reporting this! We're in the process of moving our sample listing to a model that better supports large number of samples and the column headers disappeared during this transition. They will return in our next release!

Resolved with this commit: https://github.com/UKGovernmentBEIS/inspect_ai/commit/8141ef25f2eef2cb23c8ac8a4d2ee02872db6549
Repository: UKGovernmentBEIS/inspect_ai
Issue: [New Feature Request] Add capability to customize Scorer display name

Today Scorer display name in the Inspect View is populated from either scorer function name or scorer name attribute in the `@scorer(name=..)` . However if `eval` is run from Python API and code location is in the Python module, inspect-ai will add module name to the scorer name attribute. For example, Python module "eval_tool" and scorer name is "first_token_duration" scorer will be reported as "eval_tool/first_token_duration". 
I understand that this behavior is due to the fact that inspect stores and retrieves scorer objects from internal registry, which is also needed for the re-try functionality.
The request is to add "display_name" attribute to the `@scorer(display_name=...)` and use the attribute in the inspect View if present.