zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Specify model configs for models created through model-role cli argument
Root cause: Unimplemented Feature Gap

Is there a way to set different model configs for each model created using `--model-role ...` from the cli? Currently what I am doing is creating a short python script that looks something like

```python
red_team = get_model("anthropic/claude-sonnet-4-20250514", config=GenerateConfig(max_tokens=32_000, reasoning_tokens=10_000))
blue_team = get_model("anthropic/claude-sonnet-4-20250514")

eval(
    task,
    model_roles={
      "red_team": red_team,
      "blue_team": blue_team,
  }
)
```

but it would be nice to be able to set configs like this when running `inspect eval`.

Right now that's not possible, however we do have a way to parse YAML style objects from the CLI so you could potentially do something like this:

```bash
--model-role "{name: 'red_team', model: 'anthropic/claude-sonnet-4-20250514', config: { max_tokens=32_000, reasoning_tokens=10_000}"
```

At least I think that might be possible...

If you want to play with this then check out line 904 of src/inspect_ai/_cli/eval.py. That parses the CLI for simple key-value pairs, but you could instead write a function that returns `dict[str, str | Model]` based on taking any objects and instantiating them into `Model` directly.

The closest I could get to this working was
```
--model-role red_team="{name: 'red_team', model: 'anthropic/claude-sonnet-4-20250514', config: { max_tokens=32_000, reasoning_tokens=10_000}}"
```
But this fails with
```
inspect_ai/_eval/eval.py", line 218, in run_task_app
    return await eval_async(
           ^^^^^^^^^^^^^^^^^
inspect_ai/_eval/eval.py", line 439, in eval_async
    raise inner_exception(ex)
inspect_ai/_eval/eval.py", line 388, in run
    result = await _eval_async_inner(
             ^^^^^^^^^^^^^^^^^^^^^^^^
inspect_ai/_eval/eval.py", line 711, in _eval_async_inner
    raise e
inspect_ai/_eval/eval.py", line 530, in _eval_async_inner
    resolved_tasks, approval = eval_resolve_tasks(
                               ^^^^^^^^^^^^^^^^^^^
inspect_ai/_eval/eval.py", line 1125, in eval_resolve_tasks
    resolved_model_roles = resolve_model_roles(model_roles)

inspect_ai/_eval/task/task.py", line 397, in resolve_model_roles
    k: get_model(v, memoize=False) if isinstance(v, str) else v

inspect_ai/model/_model.py", line 1042, in get_model
    raise ValueError(
ValueError: Model API {'name': 'red_team', 'model': 'anthropic of model "{'name': 'auditor', 'model': 'anthropic/claude-sonnet-4-20250514', 'config': {'max_tokens=32_000': None, 'reasoning_tokens=10_000': None}}" not recognized.
```
It seems it just treats this as a string and never parses it into a Model.

@jjallaire could I submit a PR for this feature?
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add option for retaining failed samples in final log files and fail-and-retry
Root cause: Unimplemented Feature Gap

https://inspectcommunity.slack.com/archives/C080K7SQ4SG/p1755607573486889?thread_ts=1755308046.660229&cid=C080K7SQ4SG
Repository: UKGovernmentBEIS/inspect_ai
Issue: Add support for passing Model Metadata in tasks and accessing it from hooks
Root cause: Unimplemented Feature Gap

### Description
I would like to pass model metadata when launching an evaluation job from a provider. This metadata should be easily accessible from a hook during job execution.

### Motivation
Currently, there is no straightforward way to pass and retrieve model-specific metadata (e.g., model step) in an eval job. 

Since I can't access the metadata field in the EvalSpec from inside a provider (only `GenerateConfig` is available), we are relying on setting environment variables to pass this information along. Maybe we can add a `metadata` field to `GenerateConfig` or expose the eval config inside of the provider.

Adding this feature would improve flexibility and allow hooks to adapt based on model metadata.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support for Multiple vLLM endpoints
Root cause: Unimplemented Feature Gap

Hi, 

Incredible work!!

I am using vLLM to run some evaluations. Is there a support to send multiple vLLM endpoints of the same model, such that I can choose from anyone of the N endpoints I provide for generating a response?

Apologies if this is already documented.

Thanks!

Example:

Current 
```
export VLLM_BASE_URL=http://localhost:8080/v1
```

Proposed
```
export VLLM_BASE_URL=[http://localhost:8080/v1, http://localhost:8081/v1, http://localhost:8082/v1, ...]
```

We don't have a round robin capability built-in, but you could implement this pretty easily by using an nginx reverse proxy, e.g. something like this:

```
events {
    worker_connections 1024;
}

http {
    upstream vllm_backend {
        least_conn;  # or use 'round_robin' (default)
        server localhost:8080;
        server localhost:8081;
        server localhost:8082;
        # Add more servers as needed
    }

    server {
        listen 9090;  # Your proxy port
        
        location / {
            proxy_pass http://vllm_backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_read_timeout 300s;
            proxy_connect_timeout 75s;
        }
    }
}
```

Thank you!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Tool parameter descriptions get dropped for nullable and enum fields when calling Google Model.
Root cause: Validation Gap

Here is an example tool section for `text_editor` tool

<img width="982" height="1050" alt="Image" src="https://github.com/user-attachments/assets/970abc6c-2a0e-4144-8276-6523180867bc" />

For nullalbe param, it's handled in https://github.com/UKGovernmentBEIS/inspect_ai/blob/143be5b38b8d84eac27479bff33fa83c52bfdd4b/src/inspect_ai/model/_providers/google.py#L722-L724

The description is in `param.description` but not assigned to `Schema.description`.

For enum param, looks like we can just add into https://github.com/UKGovernmentBEIS/inspect_ai/blob/143be5b38b8d84eac27479bff33fa83c52bfdd4b/src/inspect_ai/model/_providers/google.py#L727-L728 ? 


Thanks for reporting this! Taking a closer look now.....

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2375
Repository: UKGovernmentBEIS/inspect_ai
Issue: "role" in function response for Gemini call
Root cause: Interface Contract Mismatch

Hi,

I noticed that when Inspect returns a function call response for the Gemini model, it sets role to "function", similar to other models:

<img width="530" height="259" alt="Image" src="https://github.com/user-attachments/assets/5e3690d8-1719-4020-985a-89e99d6dc192" />

However, the Gemini API docs seem to suggest the role should instead be set to `"user"`:

<img width="802" height="370" alt="Image" src="https://github.com/user-attachments/assets/394ea4a4-adcb-48a2-9f68-d6f1a326b69d" />


Notably:
* The doc uses `"role": "user"`, not `"function"`.
* Separately, the `response` map uses `"result"` as the key instead of `"content"`.

👉 My question: Should we use `"user"` here for Gemini responses, or is `"function"` still correct and I’m misinterpreting the docs?

Thanks!

I think it should indeed be `role="user"`, resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2378

Thanks @jjallaire for the quick fix!!

I am wondering if we should also use "result" instead of "content"  when constructing `FunctionResponse` on line 602? 

https://github.com/UKGovernmentBEIS/inspect_ai/blob/2ba2c4b6680f4a97e3e677621277efecd35a4707/src/inspect_ai/model/_providers/google.py#L599-L606

Gemini doc: https://ai.google.dev/gemini-api/docs/function-calling?example=meeting#step-4

<img width="737" height="462" alt="Image" src="https://github.com/user-attachments/assets/784c2789-baa3-49d5-8d44-71b1c8b8cf97" />
Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature Request: Delete log file through Inspect View

Hi! It would be great if I could delete log files for cancelled/failed trials directly through the Inspect View client rather than having to copy paste the file names one by one from the viewer into a terminal to delete them.

Thanks!

Great idea! @dragonstyle Let's put in this in queue to work on with the other view related improvements we are contemplating.

VS Code now has a logs pane that has the ability to delete log files: https://inspect.ai-safety-institute.org.uk/vscode.html#viewing-logs

Not a substitute for doing it from Inspect View but an improvement nonetheless.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Unhandled ValueError in str_to_float
Root cause: Validation Gap

I ran into the following stack trace when evaluating a model on mgsm, where apparently the model tried to answer with chinese characters.

```
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py", line 331, in run_sample
    return await task_run_sample(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py", line 982, in task_run_sample
    raise raise_error
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py", line 849, in task_run_sample
    await scorer(state, Target(sample.target))
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/scorer/_common.py", line 26, in score
    answer, matched = match(state.output.completion, value)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/scorer/_match.py", line 28, in check
    return match_str(
           ^^^^^^^^^^
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/scorer/_common.py", line 70, in match_str
    v = first_number_normalized(words)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/scorer/_common.py", line 93, in first_number_normalized
    return normalize_number(number)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/scorer/_common.py", line 98, in normalize_number
    num = str_to_float(number)
          ^^^^^^^^^^^^^^^^^^^^
  File "/tmp/joschu/uv-venv/lib/python3.12/site-packages/inspect_ai/_util/text.py", line 197, in str_to_float
    raise ValueError(f"Value could not be parsed as a float: {s}")
ValueError: Value could not be parsed as a float: 三三
```

This is on version 0.3.124

Thanks for reporting! This is resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2387

I can push this to PyPI but I'd like to first confirm that there are no other issues lurking w/ scoring mgsm correctly. If you can install from main w/ `pip install git+https://github.com/UKGovernmentBEIS/inspect_ai` and LMK how you fare that would be great.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [feature request] Support for evaluation without solver on pre-computed model outputs
Root cause: Documentation Deficiency

I am using inspect.ai to evaluate pre-computed LLM outputs.  I have synthetically generated data for model distillation that I want to assess with my scorers but I do not want to create a new Solver and waste API calls to regenerate the outputs. It would be ideal if I could provide precomputed model outputs in my dataset and run my scorers directly on these outputs.

A similar behaviour can be achieved with a custom solver that returns a dummy `ModelOutput` without using `generate()` but this isn't very clean.  I did this by passing my `model_output_str` column from my dataset to a dummy ModelOuput through the state.metadata. Here, I still needed to provide an LLM to run the script even though I didn't have to use any calls for my evaluation.

```
@solver
def skip_solver(**params):

    async def solve(state: TaskState, generate: Generate) -> TaskState:

        state.output = ModelOutput(
            model="model_output_template",
            choices=[
                {
                    "message": {
                        "role": "assistant",
                        "content": state.metadata["model_output_str"]
                    },
                    "logprobs": None,
                    "finish_reason": "stop",
                    "index": 0
                }
            ],
            usage= {
                "prompt_tokens": 0,
                "completion_tokens": 0,
                "total_tokens": 0
            },
        )

        return state

    return solve
```

It would be nice if there was a built-in solver to handle this more neatly.

I agree with this! I already built a custom solver for existing input and output, but inspect could be more useful to observe data if this is supported.

We have a `mockllm` provider that you can prime with 1 or more outputs. i.e.

```python
model = get_model("mockllm/model", model_args=dict(custom_outputs=...))
```

Further, `ModelOutput` has a static `from_content` method that lets you easily construct a `ModelOuptut` object from a string.

Closing as I believe the `mockllm` provider meets this requirement.
Repository: UKGovernmentBEIS/inspect_ai
Issue: How to enable thinking via command line

I am running evaluations on a script that makes use of inspect

```
inspect eval scicode.py \
    --model $MODEL \
    --temperature $TEMPERATURE \
    --top-p 0.95 \
    --stop-seqs "</answer>" \
    --reasoning-effort high \
    --log-format json \
    -M 'extra_body={"enable_thinking": true, "thinking": true}' \
    -T split=$SPLIT \
    -T output_dir=$OUTPUT_DIR \
    -T with_background=$WITH_BACKGROUND \
    -T mode=normal
```

I was wondering how to pass `enable_thinking` and `thinking` arguments via the command line?

Depending on the model you can use `--reasoning-effort` or `--reasoning-tokens`: https://inspect.aisi.org.uk/reasoning.html

Thanks for the quick response, @jjallaire . 
I am using `Qwen/Qwen3-235B-A22B` and `deepseek-ai/DeepSeek-V3.1` via vLLM. 
It is not clear from the documentation if reasoning is enabled for them on using `--reasoning-effort`

It's not clear to me either whether vLLM makes use of this. We definitely pass it along when you use `--reasoning-effort`. The docs are silent on this b/c I'm not sure either way. @kaifronsdal  any idea how vllm deals w/ this?

Check out https://docs.vllm.ai/en/latest/features/reasoning_outputs.html. You will need to specify a reasoning parser. I think you can do this with `-M reasoning-parser=qwen3`.

As far as I know, reasoning effort doesn't do anything for qwen and deepseek models since they were only trained to either use or not use reasoning. If you want an open source model that you can specify the reasoning effort you could try openai's new oss model.

@jjallaire @kaifronsdal 
Thanks for your response. In that case, ideally, there should be a way to pass optional arguments such as `enable_thinking`.
This would enable us to flexibly switch between the reasoning modes.

For now, I will modify the chat template to enable this behavior.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug report] Log viewer got an error if `Logprob` bytes is None 
Root cause: Validation Gap

I tried to return logprob by constructing my own logprob output as below.

```python
        logprobs = []
        for prob in response["probs"]:
            top_logprobs = [TopLogprob(token=token, logprob=math.log(prob + 1e-6), bytes=None) for token, prob in sorted(prob.items(), key=lambda x: x[1], reverse=True)]
            logprobs.append(Logprob(token=top_logprobs[0].token, logprob=top_logprobs[0].logprob, bytes=None, top_logprobs=top_logprobs))
        logprobs = Logprobs(content=logprobs)
```

After evaluating my script, I couldn't load my log in the viewer.

<img width="1230" alt="image" src="https://github.com/user-attachments/assets/6e672b48-68ad-4546-a9b0-0faa4adfbb8d">

I found related error messages on the terminal, which were helpful to tackle the problem.
```
samples.1.output.choices.0.logprobs.content.4.top_logprobs.65.bytes                                                                                                        
                               Field required [type=missing, input_value={'token': 'C', 'logprob': -13.815510557964274}, input_type=dict]                                                               
                                 For further information visit https://errors.pydantic.dev/2.8/v/missing                      
```

From the error, I gave empty arrays for bytes input, and then I could load the viewer.
```python
        logprobs = []
        for prob in response["probs"]:
            top_logprobs = [TopLogprob(token=token, logprob=math.log(prob + 1e-6), bytes=[]) for token, prob in sorted(prob.items(), key=lambda x: x[1], reverse=True)]
            logprobs.append(Logprob(token=top_logprobs[0].token, logprob=top_logprobs[0].logprob, bytes=[], top_logprobs=top_logprobs))
        logprobs = Logprobs(content=logprobs)
```

From the source code, bytes array can be `None`, but this was not the case for the viewer. Maybe we should omit None from the default values.


Thanks for the clear report - it is likely that the viewer is incorrectly assuming that if logprobs is present, it isn't None. I'll fix this ASAP!

Thanks for quick feedback. I really enjoy developing on top of your library!

This recent fix may resolve the issue: https://github.com/UKGovernmentBEIS/inspect_ai/commit/6308eb0fad6c7b051223242be82de7c304d89898

If it doesn't, could you try reading the log file with the `read_eval_log()` function and see if you get the same error. In that case could you share the log file with us for further investigation.


Suggested fix was not enough to resolve my issue. You should add a default value to bytes in `TopLogprob` class, too.

I fixed it on this PR: https://github.com/UKGovernmentBEIS/inspect_ai/pull/248

Test completed!
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Support for different `max_connections` across different providers
Root cause: Documentation Deficiency

Currently when running `eval_set`, it is only possible to set a single `max_connections` which will be applied across all providers (#2181). For my evaluation setup, it would be helpful to customize that for individual providers with lower concurrency limits.

You can do this by explicitly creating your model and passing it e.g. into your solver or scorer:

```python
grader_model = get_model("openai/gpt-5", config=GenerateConfig(max_connections=10))`

my_scorer = custom_scorer(grader=grader_model)
```
Repository: UKGovernmentBEIS/inspect_ai
Issue: For big eval sets, filtering task-evals on the homepage/landing page either takes >1 min or doesn't work at all
Root cause: Resource Mishandling


Notes to self if I end up working on this:
* would need repro steps for "big eval sets" because `make test; inspect view` does not reproduce the issue when I freshly cloned the repo, it looks nice and fast with just those test results
* _update 2025-11-26_: got access to some large log files, 9GB indeed sounds like it would cause FE perf issues for sync processing => I should be able to reproduce and investigate some debouncing and/or split into async chunks and/or workers and/or propose BE changes to discuss in slack channel
  * still not reproducible with 8.8GB of 523 files, filters run smoothly for me so far...

I can no longer reproduce this. I tried to, with an eval set containing 865 log files. Filtering and sorting is snappy. Hooray!

I did see that it took 3-4 minutes for the "Syncing data" message in the bottom left hand corner of the screen to go away. And, until it, did search results weren't complete. But that seems like a different issue. It could be related to the fact that I'm testing using eval logs in S3, not on my computer's storage.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Azure Managed Identity not working
Root cause: Validation Gap

When trying to use the azure managed identity (not settings the API key), I’m getting an `AssertionError`.

When digging deeper into this, before creating the Azure Client, it does `assert self.api_key`, but the api key is never set when using managed identity. In the constructor, the token provider is set, but it never actually get’s a token/key.

Could you give us a PR for this? (as we don't have an Azure managed identity configuration to test with)

@jjallaire , by PR, do you mean a Pull Request with a proposed fix?

Yes exactly!
Repository: UKGovernmentBEIS/inspect_ai
Issue: [FR] Cloud based inspect AI (variable based log files)
Root cause: Unimplemented Feature Gap

I run inspect AI purely on cloud based notebooks like Colab. I would like the options to:
1. Store log outputs in a variable rather then a file. This makes it much easier for me to work with and extract values from it once it is done.
2. I would like the option to pass a log name to the variable like {model_Name}_run{x}.
3. I would like a way to read the log files stored in a variable and generated without needing any file/server based infrastructure


Ideally I would like something like this

@task
def sample_task() -> Task:
return Task(
dataset=inspect_dataset,
solver=[generate()],
scorer = [
specialized_scorer(
initial_template=INITIAL_TEMPLATE,
per_dimension_prompt=PER_DIMENSION_PROMPT,
model = judge,
log_name={model_name}{score},
log_file=variable,
) for judge in list_of_judge_models
],
epochs=5
)
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect View: Keyboard navigation in log listing page

- Support selecting items in the grid
- Keyboard shortcuts:
    - Arrows to navigate selection
    - Home / End to navigate selection to front / back of current page
    - Ctrl + Home / End to navigate selection to front back of entire list
    - Ctrl + Left / Right to navigate to prev or next page
    - Enter navigate to selected item
    - back space or alt arrow up go up a directory level
    - f5 refresh view contents
Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug report] I can't set base url with VSCode extension
Root cause: Configuration Error

When I try to set model base url in the VSCode inspect extension, a wrong environment variable is inserted in .env file.

<img width="331" alt="image" src="https://github.com/user-attachments/assets/fa9afa02-e2c8-4e16-b947-a314aa4e6c5c">

```
INSPECT_MODE_BASE_URL=http://testurl.com
```

I believe the environment variable name must be fixed.
```
INSPECT_EVAL_MODEL_BASE_URL=http://testurl.com
```

After I fix the name, I could run my task successfully in the vs code extension.


Thanks for reporting this! Fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/48bca11e6ac63e84964a8d8f3298d783e43ae101

Will be deployed in the next release of the VS Code extension.

cc @dragonstyle 

Hey! 

The correct name should be `INSPECT_EVAL_MODEL_BASE_URL`, not `INSPECT_MODEL_BASE_URL`.

I pushed PR to fix this: https://github.com/UKGovernmentBEIS/inspect_ai/pull/247

@aisi-inspect