zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Multi-processing w/  s3 logs, unless you use spawn to launch your subprocesses
Root cause: External Dependency Breakage

We use s3fs to access s3, which is apparently not safe for multi-processing. i.e. I have data loading code which calls read_eval_log. Using a ProcessPoolExecutor (but not a ThreadPoolExecutor!) hangs when using read_eval_log on an s3 bucket:

```python
with ProcessPoolExecutor(max_workers=num_workers) as executor:
    futures = {executor.submit(read_eval_log, eval_log): eval_log for eval_log in eval_logs}
```

I opened an issue in their repo, and they [replied saying this was a known limitation](https://github.com/fsspec/s3fs/issues/914#issuecomment-2473922221), and suggested to use spawn (new python interpret state) when doing multi-processing, i.e.:

```python
with ProcessPoolExecutor(max_workers=num_workers,mp_context=get_context("spawn")) as executor:
    futures = {executor.submit(process_eval_log_single, eval_log): eval_log for eval_log in eval_logs}
```
This works, but we should have a more robust solution to multi-processing


@max-kaufmann We have made some other changes to back off of s3fs concurrency (as it may have been the cause of other deadlocks we saw stack traces for). I think in the name of not ever hanging we probably won't do more inside of Inspect here, but I'd certainly be interested if there is a documented solution for the "right" way to do this with multi-processing (and would add that to our docs for users looking to get better parallelism).

Another thought would be to find some robust way to download S3 logs in parallel (an entirely different package or alternate use of s3fs) and then once local use multi-processing on them (as then it will just be zip reading which seems like it would be multi-process safe). Again, if we can sort out something robust here we could either included it in Inspect or minimally add it to the docs.

I'm going to close this for now because we don't have plans to address internally, but again happy to take a PR to the library or the docs if the community discovers a good solution for this.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Is there a way to have max-connections go beyond 100?
Root cause: External Dependency Breakage

Is there a way to have max-connections go beyond 100? When I run an eval with --max-connections above 100, the amount of concurrent connections to the API always plateaus at 100.

This is likely to be a limit imposed by the HTTP client library not Inspect (we don't place any cap on connections outside of max-connections)
Repository: UKGovernmentBEIS/inspect_ai
Issue: `normalize_number` fails on exponent / fractional input
Root cause: Validation Gap

With input such as `1²`, this conditional returns true:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/f2fa3b620749956b7c698277c0b4e531ab9198cb/src/inspect_ai/scorer/_common.py#L98

but the next line fails with `ValueError: could not convert string to float: '1²'`

https://github.com/UKGovernmentBEIS/inspect_ai/blob/f2fa3b620749956b7c698277c0b4e531ab9198cb/src/inspect_ai/scorer/_common.py#L99



Thanks for reporting! This PR should handle it: https://github.com/UKGovernmentBEIS/inspect_ai/pull/891

Although after implementing it did occur to me that another resolution would have been to just use a different isnumeric test (i.e. simply don't handle the case of `1²`). My bias would be to handle it (since obviously a model produced it in your testing!) but I at least wanted to ask the question before merging. LMK what you think.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Add user_message solver
Root cause: Unimplemented Feature Gap

I often find I want to add scafolding to an agent/prompt it in multiple stages. Currently this requires writing custom solvers to add the subsequent user messages (prompt_template only works for the first message). This is the usecase I'm thinking of is something like this
```python
chain(
    system_message("You are a math expert."),
    user_message("How would you go about solving 3x^2+2x+1=10?"),
    generate(),
    user_message("Now solve for x."),
    generate(),
    user_message("What is your final answer for x?"),
    generate()
)
```

The implementation would be pretty simlar to `system_message`. Perhaps something like this
```python
@solver
def user_message(template: str, **params: Any) -> Solver:
    """Solver which appends a user message to the conversation.

    User message template containing any number of optional `params`.
    for substitution. All values contained in sample `metadata` are also
    automatically included in the `params`.

    Args:
      template (str): Template for user message.
      **params (dict[str,Any]): Parameters to fill into the template.

    Returns:
      A solver that appends the parameterised user message.
    """
    # read template
    content = resource(template)

    async def solve(state: TaskState, generate: Generate) -> TaskState:
        kwargs = state.metadata | params
        state.messages.append(
            ChatMessageUser(content=content.format(**kwargs))
        )
        return state

    return solve
```
I could also see a usecase for wanting `assistant_message` and `tool_message` solvers as well that simply append the relevent message type. This is somwhat similar to [sglang](https://sgl-project.github.io/frontend/frontend.html)'s model interface.

A minor update to the sugestion. Instead of using `state.metadata` to fill in the template, it would be more useful to be able to use `store()` to fill in the template.

Resolved here (including adding `store` to the parameter substitution): https://github.com/UKGovernmentBEIS/inspect_ai/pull/1154

I didn't add assistant and tool messages b/c they are significantly more complicated and want to think about the best way to represent them in a solver.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add more documentation on CLI
Root cause: Documentation Deficiency

Hey folks!

As far as I can tell, the documentation at https://inspect.ai-safety-institute.org.uk (which is otherwise fantastic, by the way!) doesn't have any detailed documentation on the CLI.

For example, today, I wanted to know how the `--log-dir` and `--trace` options behave - but googling for `aisi inspect "--log-dir"` basically the only result was https://inspect.ai-safety-institute.org.uk/eval-logs.html, which wasn't obviously what I was looking for (I had to read through the page to find that it did actually have a paragraph on `--log-dir`), and googling for `aisi inspect "--trace"` didn't return _any_ results.

So it would be great to have more docs on the CLI - all the different commands that can be run, and what all the arguments to each command does.

Thanks!

Agree 100%, we will take care of this soon! In the meantime (acknowledging that its a poor substitute), try `inspect --help` and then e.g. ` inspect eval --help` for individual commands. Minimally, we'll add an automated step to our doc build to generate this same output. I believe that Click will also give us a way to interrogate the CLI as JSON which will also enable us to create a nicer page for each command.


Done here: https://inspect.ai-safety-institute.org.uk/options.html

Further CLI reference docs available here: https://inspect.ai-safety-institute.org.uk/reference/
Repository: UKGovernmentBEIS/inspect_ai
Issue: Recognize metadata in JSON format
Root cause: Unimplemented Feature Gap

Hello,

I'm working on a dataset that I'd like to distribute in inspect format. There need to be minimal friction for the dataset users but I discovered that even after using conventional field names `metadata` is not recognized and I need to write additional code to capture it. 

Desired behavior:

* given a file `data.jsonl` with inspect-formatted data
```json
{"input": [{"role": "user", "content": "What is the capital of France?"}], "target": "Paris", "id": "geography-001", "metadata": {"difficulty": "easy", "category": "European capitals"}}

```
* creating a dataset should be as simple as:
```python
dataset = json_dataset("data.jsonl")
```

Current behavior:
* `metadata` is not recognized and I need to write custom code to attach it
```python
def record_to_sample(record):
    return Sample(input=record["input"], target=record["target"], id=record["id"], metadata=record["metadata"])
dataset=json_dataset("data.jsonl", record_to_sample)
```

Let me know if this seems like a valuable addition. I'll be available to work on it and submit a pull request within the next week.
I'd appreciate any feedback on whether this aligns with the project's goals and if there are any design considerations I should be aware of.

I do think what you suggest would be great! We hadn't considered that the dataset would be shaped specifically for ingestion by inspect (more thinking that we are adapting to existing datasets) but it makes complete sense that once you know you are creating a dataset for Inspect its natural to include a JSON `metadata` field. 

Would welcome a PR. I think we will want the `metadata` field in the `FieldSpec` to take either a `str` or `list[str]`, and in the case of `str` interpret it as JSON formatted data (https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/dataset/_dataset.py#L146)
Repository: UKGovernmentBEIS/inspect_ai
Issue: `ContentImage.detail` is not passed down to the `OpenAIAPI`.
Root cause: Validation Gap

Hi everyone!

Thanks for the fantastic package! I am using it for all my evaluations, which helped me a lot.

I was trying to use inspect with images in my dataset and noticed that the `ContentImage.detail` parameter is not respected in the response generation.

### To reproduce

Use the task from `examples/images` but set the `detail` parameter of the images to `"low"`. Execute it with GPT4o.

Notice that the token consumption is much higher than the expected [85 tokens with low resolution](https://platform.openai.com/docs/guides/vision#low-or-high-fidelity-image-understanding). In the JSONs of each sample, the detail parameter is set to `"auto"`.

### Changes

Three changes need to be made.

First, the `detail` parameter is not passed on in line 78. The `if isinstance(content.image, str):` is always true. This is why mypy will never type check the else block and the type error `content.image.detail` is not detected. Notice the types become `Never` when you hover over them in vscode. Using `warn_unreachable` ([link](https://mypy.readthedocs.io/en/stable/config_file.html#confval-warn_unreachable)) detects these cases.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/0f342f7af6c73b8127cf613e849f55b45f273696/src/inspect_ai/dataset/_sources/util.py#L76-L83

The same issue happens here again.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/0f342f7af6c73b8127cf613e849f55b45f273696/src/inspect_ai/_eval/task/images.py#L96-L103

Thirdly, regardless of the `detail` parameter, line 422 will set it to `auto.`

https://github.com/UKGovernmentBEIS/inspect_ai/blob/0f342f7af6c73b8127cf613e849f55b45f273696/src/inspect_ai/model/_providers/openai.py#L419-L425

I can also push my changes and create a PR if my solution, removing the unnecessary if-else clauses in all three cases, is the preferred solution.

Thanks so for reporting this! Fixed here https://github.com/UKGovernmentBEIS/inspect_ai/pull/886
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support for the new Gemini logprobs feature
Root cause: Unimplemented Feature Gap

New Gemini features including logprobs were shipped recently:
https://x.com/OfficialLoganK/status/1845858650990215500

Thanks for reporting this, should be resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/890/
Repository: UKGovernmentBEIS/inspect_ai
Issue: Difficult to expand individual row elements in inspect view UI
Root cause: Architectural Constraint

When opening up `inspect view` and looking at the table of results for an eval, I find it difficult to expand individual rows that might have a large hidden area.  For example, When I try to click row 1 the UI will expand row 2 (most likely because the hidden view for row1 covers row 2 and the click area gets confused).  This gets worse as the hidden area gets longer and thus it makes it impossible to expand some random number of following rows.

Hello! Thanks for taking the time to report this issue! I'm having some trouble reproducing the problem - a couple of questions:

- Which browser are you using?
- Do you have a specific log file that reproduces the problem? If so, any chance you could share?
- Can you say more about what you mean by 'a large hidden area'?



I can't quite share the dataset I was using that triggered this but it uses a long list of conversation messages as input which I think is making the dropdown area too tall.  Here's a screen grab of what I'm seeing.  

And this is happening on Firefox v125.0.3 on MacOS Sonoma 14.4.1

https://github.com/UKGovernmentBEIS/inspect_ai/assets/398211/5a7625fb-c63d-4bf4-bc9b-241b8d6a49a6



Thanks for sharing this - I see what you're talking about! I'm taking a closer look at the log viewer this week and so will try to get to this soon!

great! other than this everything worked really smoothly!

Note that we are re-working the samples view to handle an arbitrarily large number of samples. Part of this work will resolve this issue (as there will be an entirely different expand implementation). So I'll close this issue (and you should see the fix land soon in 0.3.10 or 0.3.11).

Would it be possible to leave the issue open until the main branch has the fix commited? Otherwise anyone else who encounters this issue would be surprised to find that it's not actually fixed.

Typical practice is to only close an issue when uses/developers can access the update rather than when an update is planned to be worked on.

Yes, good call! Re-opening now.

Ah, thank you! I'll keep tabs on this and be sure to test things out once the fix is in~

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/commit/140c3d63ad42125213bffaa799147135c81857f3 (the sample view is now fullscreen)
Repository: UKGovernmentBEIS/inspect_ai
Issue: For large evals, keyboard interrupts are handled slowly, and no feedback is given to the user
Root cause: Validation Gap

For very long running samples, I will often ctrl + c to cancel them. However, they inspect often appears to ignore the interrupt, sometimes for a long time (>1-2 minutes). This causes me to send more keyboard interrupts, which I believe messes with the teardown process (e.g. see me send one interrupt, and then more behind):

![image](https://github.com/user-attachments/assets/6cd0b32e-9dff-4c7f-ba21-966f7959f0ca)

Potentially sending multiple keyboard interrupts like this in bad practice, and I should just "trust" that inspect is handling it behind the scenes. However, it would be great if inspect gave some feedback after the first keyboard interrupt (e.g. "tearing down inspect, please be patient"). Not sure if possible, as I could imagine that inspect sometimes just hands off to a non-interruptible process every so often.

Yes, the second interrupt will indeed mess with our teardown. I have attempted to fully mask subsequent interrupts however this doesn't appear possible. I have a work item that I'll address very soon to provide UI feedback on ctrl+c in the new fullsceen UI (where we do actually get control immediately on ctrl+c b/c of the way the eval worker is scheduled there).
Repository: UKGovernmentBEIS/inspect_ai
Issue: Change or document `token_limit` behaviour
Root cause: Documentation Deficiency

It seems from experimenting, and from a glance at the source code, that:

1. `token_limit` (passed to a `Task`) limits the total tokens (input and output) used for each sample
2. `max_tokens` (passed to a `GenerateConfig`) limits the new (output) tokens generated in a single call to an LLM
3. `token_limit` has no effect outside of multi-step solvers. e.g. in a call to an OpenAI API, it does not cause `max_tokens` to be set to any value.

The third point was unexpected for me. Can we either document the current behaviour, or change the behaviour such that `token_limit` causes an LLM API call's `max_tokens` to be set to the appropriate number? The latter I would find more intuitive and more helpful.

For background on my use case: 
It would be great if there was a way to limit the number of total (input + output) tokens for a single LLM call, since, if I understand correctly, this number corresponds to the `max_model_len` in vLLM. It is helpful to be able to set `max_model_len` at vLLM model instantiation time to something much lower than the maximum supported by the model, because otherwise the models will require an enormous amount of VRAM to be allocated to support very long completions (much longer than needed for many benchmarks). Let me know if this is unclear or you think I'm misunderstanding vLLM (which is very possible).

Clarified in the docs here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/37d4c358a2c2767bffcf458c3215f54800541f6a (the two options are sufficiently different in intent, scope, and implementation that I don't want one to automatically set the other).

In terms of vLLM memory use that is likely controlled by one of the `LLM` constructor arguments? (https://docs.vllm.ai/en/latest/dev/offline_inference/llm.html#vllm.LLM). These can be set from Inspect using `model_args`. The total memory allocated is unlikely to be affected by `max_tokens` (as that is a `generate()` level parameter so _after_ the model has been loaded). Note that the above assumes you are using the Inspect `vllm` provider which does inference from the same process. 

If you are using the vLLM OpenAI compatible server its likely controlled by a CLI argument to `vllm serve` (https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html). 

Remember also that vLLM does automatic batching so the amount of VRAM will generally want to be some multiple of what you expect to use per-sample.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Specifying scorer model causes CUDA OOM error
Root cause: Resource Mishandling

### Summary
When evaluating a local model using hugging face, I tried to use a separate scorer model defined through a string in my scorer. Unfortunately this led to the scorer model being repeatedly loaded by my machine until a CUDA out of memory error occurred. This occurs even with `max_connections`, `max_subprocesses` and `max_tasks` set to 1.

### Steps to reproduce
Run `sycophancy` from [this PR into inspect_evals](https://github.com/UKGovernmentBEIS/inspect_evals/pull/102), but first manually set `scorer_model` in `sycophancy_scorer()` with a string to be a huggingface model. This bug happens with the following (model, scorer_model) combinations: (hugging face, OpenAI), (hugging face, hugging face), (OpenAI, hugging face). It does not occur when the scorer_model is None.
 
I have tested this with 1B models on:
-  An EC2 instance with an A10G with 24GB VRAM
- My M3 Pro MacBook Pro with 32GB unified memory.

### Proposed solution
Supplement the `active_model_context_var` (retrieved  by `get_model`) with  `active_scorer_model_context_var` to enable a single instance of a scorer model to be persisted throughout the evaluation. A `scorer` bool can be given as input to `get_model` to access it.  I think this provides a balance between supporting what I assume to be a common usage pattern (separate scorer model) without making it hard for users to deliberately use different instances of a model.

I think we have this resolved by creating the model in the initialization of the scorer rather than on demand? Closing this issue (feel free to re-open if there are other things we should pursue)
Repository: UKGovernmentBEIS/inspect_ai
Issue: get_model throws error when used before task initilization
Root cause: Validation Gap

When running evals with a scorer (e.g. the built in `model_graded_fact` scorer) with unspecified model I keep getting `ValueError: No model specified (and no INSPECT_EVAL_MODEL defined)`. It would be nice if there was either a param you could pass to the cli to specify the eval model or to default to the provided model (perhaps with a warning).

Currently the call to `get_model` tries to get the `active_model`, but the active model is only set later in `task_run`.

Command to reproduce error:
```bash
inspect eval examples/theory_of_mind.py --model hf/meta-llama/Meta-Llama-3-8B-Instruct -M device='cuda:0'
```

Thanks for reporting this! Should be resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/99
Repository: UKGovernmentBEIS/inspect_ai
Issue: Connecting to in-house models
Root cause: Documentation Deficiency

Greetings,
I have been trying to connect to a Llama model we are hosting in house. I have received the model name, the API key and our custom base URL, however I haven't been able to properly add it to Inspect. I'm running Inspect within a Python script. For now I've been using the env variables:

os.environ["INSPECT_EVAL_MODEL"] = ""
os.environ["INSPECT_EVAL_MODEL_API_KEY"] = ""
os.environ["INSPECT_EVAL_MODEL_BASE_URL"] = ""

The error brings me to the model/_model.py script. Since the api_name defined in line 546 remains empty I end up in the else branch in if/else in line ~563, since "len(modelapi_types) = 0".

For example, I'm testing our "llama-3.1-8b-instruct" and it doesn't work using this name or "meta-llama/llama-3.1-8b-instruct". The error is:

**File "(...)/inspect_ai/model/_model.py", line 603, in get_model
    raise ValueError(f"Model name {model}{from_api} not recognized.")
ValueError: Model name llama-3.1-8b-instruct not recognized.**

Is there a solution already implemented? Thanks!

I'm not sure what you mean by "in house model". You need to use one of the existing model providers to access your Llama 3.1 model -- this could be HuggingFace, vLLM, vLLM over the OpenAI interface, or llama-cpp-python.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add how close a sample is to hitting limits in the new live viewer
Root cause: Unimplemented Feature Gap

I think one of the main questions you ask yourself when you are looking at a sample in the live viewer, you want to ask yourself "how far is it from completion". Writing our how close you are to reaching various limits answers that question.

I'm thinking this would be time, tokens, and messages. Anything else you can think of we should add?

resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/951
Repository: UKGovernmentBEIS/inspect_ai
Issue: OpenAI model provider doesn't handle `invalid_request_error`
Root cause: Validation Gap

Sample failed due to `invalid_request_error`: 

![image](https://github.com/user-attachments/assets/0448a123-b2d0-4bb1-93a6-b1529d0fe5d6)

I think this should be explicitly handled here: https://github.com/UKGovernmentBEIS/inspect_ai/blob/d10ac9c1c02bc062b0f62f61eaaed4ad8d689771/src/inspect_ai/model/_providers/openai.py#L280

Should be resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1076
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer acts weird if sample ID is non-unique
Root cause: Validation Gap

Hi folks, awesome library. While using it for the first time I had some code set identical id's for samples and that didn't throw an error but caused the viewer to act very weird (partially it was showing multiple samples but during conversion and sample navigation it would break).

Probably best to throw an error when a list of samples is detected to have identical IDs if the rest of the app works on the assumption they are always unique.

Thanks for reporting this -  I will have a look at either better supporting this or erroring / warning when this happens!