zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Problem with two model scorers
Root cause: Algorithmic Error

It seems like when one runs a task with at least two 'model_graded_qa' scorers, Inspect creates additional model_graded_qa with mean accuracy and stderr equal to 'nan'. 

Here is an example of this problem:

```
@task
def autoformalize_proofnet(
    num_samples=1,
    use_gt_scorer=True,
):

    solver = [
        system_message(SYSTEM_MESSAGE),
        prompt_template(PROVER_PROMPT),
        generate(cache=CachePolicy(expiry=None)),
    ]

    scorer = [model_graded_qa(template=VERIFIER_PROMPT, model="openai/gpt-4o"),
              model_graded_qa(template=VERIFIER_PROMPT, model="openai/gpt-4o-mini")]

    dataset = hf_dataset(
        path="hoskinson-center/proofnet",
        split="validation",
        sample_fields=record_to_sample,
        limit=num_samples,
    )

    print(f"Scorer : {scorer}")

    return Task(
        dataset=dataset,
        solver=solver,
        scorer=scorer,
    )
```

The eval runs without any errors or warnings, but produces this output:

<img width="1260" alt="Image" src="https://github.com/user-attachments/assets/8d142a83-4851-47e8-ab4a-54924e913d79" />

As a result, the .eval file in logs cannot be opened 

<img width="1174" alt="Image" src="https://github.com/user-attachments/assets/13d4fe24-c3e9-47c1-81dd-400d3e2288e6" />

Has anyone encountered this problem before?

Hi @dragonstyle I think I managed to find the main issue, that is that there seems to be some unintended behaviour with two scorers of the same name. For example [exact(), exact()] will give the same issue.

I delved a bit further into this and might be able to open a PR to fix this unless you already managed to solve this problem.

Thanks for the followup - agree it is definitely caused by the duplicate names. We added some code to detect scores that are coming from our recently added non-inspect agent support and that code is causing the issue. I'm pretty close to having it fixed, so should be able to get this closed shortly!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Feature request: Batch submission providers like OpenAI and Anthropic?
Root cause: Unimplemented Feature Gap

First, thank you for Inspect!

I would love to be able to use inspect to submit batches to expensive providers and wait for the results. (Batch submissions are 50% cheaper). From my searches, I don't believe inspect supports this. In my understanding of the architecture, I could imagine it being somewhat disruptive to add, but wanted to mention it anyway.

https://platform.openai.com/docs/guides/batch
https://docs.anthropic.com/en/docs/build-with-claude/batch-processing

Yeah, Inspect assumes that there may be 'n' calls to the model so the batch approach doesn't work except for the simplest evals. That said, you could manually implement this by creating a new model provider that can operate in one of two modes: 

1. Recording requests (which you'll then submit in batch offline)
2. Replaying requests from the batch process

You would run the task twice, first in recording mode, then in replaying mode.

I don't think we will implement this but if its a big deal for you I think there is a path.

An update here for others that stumble across this issue. Some fine folks added support for this!
https://inspect.aisi.org.uk/models-batch.html

Thanks @jjallaire and @epatey!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Execute `--runslow` tests in CI
Root cause: Unimplemented Feature Gap

Will require that we run docker in CI.

Prior discussion: https://github.com/UKGovernmentBEIS/inspect_ai/issues/726

I have experience with running Docker in GitHub actions. @art-dsit let me know if it would be helpful to chat about this or the CI/testing setup in general (my work email is on my employer's website). Equally, if you already have a clear plan of action / just want to move ahead on your own, that's absolutely fine. Don't let me block you. :)
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature request] Support for LiteLLM and LiteLLM proxy
Root cause: Unimplemented Feature Gap

LiteLLM (https://github.com/BerriAI/litellm) provides a common interface (using the OpenAI API) for [a lot of models and providers](https://github.com/BerriAI/litellm?tab=readme-ov-file#supported-providers-docs). 

Has support for LiteLLM been considered? 

I can imagine it as a prefix for the model given as input e.g. `litellm/azureai/llama-2-70b-chat-wnsnw` for the `litellm` package and `litellm-proxy/azureai/llama-2-70b-chat-wnsnw` for the proxy using `openai`.

This could help expand the range of models and features beyond what's available in `inspect_ai` without requiring a custom model implementation from the consumer.

I don't think we'll end up supporting LiteLLM directly although litellm-proxy can definitely be used with our [OpenAI Compatible](https://inspect.aisi.org.uk/providers.html#openai-api) provider (e.g. `openai-api/litellm/<model>`).

We do a whole bunch of stuff in the model interface to facilitate various features we have (model context overflow handling, measurement of "working time", various tool calling enhancements we've made, etc.). We've had trouble in the past w/ introducing "generic" providers (e.g. we have an openrouter provider) and then people start switching to it for everything (losing all of the tweaks we've made in the native providers). 

If there is an Inspect provider it's always going to provide better integration and capability than using another generic provider. If there isn't an Inspect provider then our OpenAI compatible interface should work for just about everything else.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Log viewer: scroll bar resets when using the scorer drop-down
Root cause: Algorithmic Error

**Steps to reproduce:**

1. open an eval with multiple scorers in the log viewer
2. scroll down the list of samples
3. choose a different scorer from the scorer drop-down


**Actual behaviour**: scrollbar goes back to the top of the list of samples
**Desired behaviour**: stay at the same scroll position in the list of samples

This has been resolved as part of our move on react/zustand in the client.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Using multiple models from the same provider with different base urls and api keys
Root cause: Documentation Deficiency

I am trying to evaluate a gpt-4o-mini model from azure openai using a gpt-4o scorer also from azure openai. These models have different base urls and api keys.

When trying this I faced the issue that the package expects both these models to share the same environment variables names, AZUREAI_OPENAI_API_KEY and AZUREAI_OPENAI_BASE_URL, even though the values are different for each model.

I tried to solve this by explicitly passing my base url and api key to my scorer model using `get_model`, but it doesn't work.

Like this:
```
        scorer=model_graded_fact(
            model=get_model(
                'openai/gpt-4o',
                config=GenerateConfig(temperature=0.0),
                base_url=judge_base_url,
                api_key=api_key,
            ),
            template=CUSTOM_GRADER_TEMPLATE,
            instructions=CUSTOM_GRADER_INSTRUCTIONS,
            grade_pattern=CUSTOM_GRADER_REGEX,
        )
```

Do you have any suggestions for how I can use the two different azure openai models in the same task?

To use Azure models use the `open/azure` prefix (this was a recent-ish change: https://inspect.ai-safety-institute.org.uk/providers.html#openai-on-azure)

That should get you into the Azure codepath which will hopefully handle the multiple models correctly
Repository: UKGovernmentBEIS/inspect_ai
Issue: record default values in log file
Root cause: Unimplemented Feature Gap

e.g. the default for `fail_on_error` is `True` so we should write that to the log rather than leaving the field out (effectively making it `None`).

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1300
Repository: UKGovernmentBEIS/inspect_ai
Issue: Multiagent Converssation with different models and a Scorer model at the end of each conversation step.
Root cause: Documentation Deficiency

Hi,

I try to get my head into the inspect AI framework and still have issues. Could somebody tell me what would be the best setup, to build up an evaluation step, where two llms (locally hosted llms own base URL with openai client) are talking to each other, and for each output, a third llm from a different provider (openai api from openai) should rate each conversation output. Each of the first two models must have stop_word/sequence to generate the output as long the sequence is. 

I tried to build different solvers for each model. But has issues pathing the conversation history between them and also different configuration objects have always to be set as active by using set_active_generate_config from the class. 

Could somebody point me in the right direction. 

(I'm not sure to ask these question here, but I also didn't find a discord channel or something similar)

BR

You would want to use a single solver to orchestrate all of this, and get access to the various models you need using the `get_model()` function:

```python
model = get_model() # model being evaluated
partner_model = get_model("openai/gpt-4o")
grader_model = get_model("google/gemini-1.5-pro")
```



Thanks. But then I would not put the evaluation into a scorer but rather directly into the solver. So the evaluation script has only this scorer? How could I then push the results of a run into the dashboard/logs?

You could run the grader_model inside your scorer --- if you need intermedate grading (checking on whether things are complete) you can call the `score()` from within your solver.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Create clustered standard error metric
Root cause: Unimplemented Feature Gap

Naive standard error might give inconsistent results on benchmarks with non-independent questions e.g. DROP, QuAC, RACE, SQuAD. Create a cluster adjusted standard error metric that accounts for the question clusters. According to Miller, E. (2024). [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://www.google.com/url?q=https%3A%2F%2Farxiv.org%2Fpdf%2F2411.00640), clustered stderr can be 3X larger than naive stderr.

Along with the clustered stderr, report the number of clusters in the output.

The calculation side should be addressed by https://github.com/UKGovernmentBEIS/inspect_ai/pull/1231 !

Now merged: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1231
Repository: UKGovernmentBEIS/inspect_ai
Issue: Unique ID for each run of each sample
Root cause: Unimplemented Feature Gap

* Each run of each sample has a unique ID in the `.eval` file
    * e.g. running `inspect eval` with 2 tasks, each of which has 3 samples, using 4 different models and 5 epochs of each sample would produce 2 x 3 x 4 x 5 = 120 run IDs
* It would be even better if that information were available at runtime, e.g. to the scorer
    * In our particular case, we'd use this to associate artifacts from the eval back to the unique run ID that produced that artifact
    * e.g. we have a "scorer" that captures some files from the task sandbox and uploads them to S3. Ideally we'd be able to save them to `s3://bucket-name/run_id=${RUN_ID}/`

There is currently a unique selector for each sample, but it's pretty awkward to get at and its not available at runtime. Basically, each call to `eval()` already has a `run_id`, and then each task instance has a `task_id` (sometimes these are re-used in the case of eval-retry or eval-set, but the combination of `run_id` and `task_id` is always unique. There are in the `EvalSpec` (in the log):

```python
class EvalSpec(BaseModel):
    """Eval target and configuration."""

    run_id: str = Field(default_factory=str)
    """Unique run id"""


    task_id: str = Field(default_factory=str)
    """Unique task id."""
```

Then of course the `EvalSample` has its own unique identifier based on the combination of `id` and `epoch`:

```python
class EvalSample(BaseModel):
    """Sample from evaluation task."""

    id: int | str
    """Unique id for sample."""

    epoch: int
    """Epoch number for sample."""
```

So `run_id/task_id/id/epoch` is the unique identifier. Again, not available at runtime and kind of awkward to assemble. I think we have three choices here:

1. Add the `run_id` and `task_id` to the `TaskState` so solvers and scorers can form the globally unique id manually.

2. Add a new unique identifier field to the `TaskState` (and `EvalSample`). We already have `id` on the sample and we already have `run_id` on the `EvalSpec`, so naming this one is a bit tricky.

I think (2) might be cleaner/simpler. But what to name it? Some ideas:

`iid` (instance id)

`gid` (global id)

`uuid` (kind of on the nose but is a good literal description of what this is)

`asset_id` (probably too tied to the use case in the issue)







PR here implementing the `uuid` variation: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1374
Repository: UKGovernmentBEIS/inspect_ai
Issue: Pause events recorded in the transcript
Root cause: Unimplemented Feature Gap

It looks like the human agent implements a bespoke version of pauses, where pauses are logged as info events to the transcript. These should be proper PauseEntry events instead of generic info events.

I think we’re likely going to want this for agent runs as well (e.g. to account for rate limit backoffs), so there are convergent reasons.

For background, Vivaria's schema for pauses is:
```js
export enum RunPauseReason {
  CHECKPOINT_EXCEEDED = 'checkpointExceeded',
  HUMAN_INTERVENTION = 'humanIntervention',
  PAUSE_HOOK = 'pauseHook',
  PYHOOKS_RETRY = 'pyhooksRetry',
  SCORING = 'scoring',
  LEGACY = 'legacy',
}

export const RunPauseReasonZod = z.nativeEnum(RunPauseReason)

export const RunPause = z.object({
  runId: RunId,
  agentBranchNumber: AgentBranchNumber,
  start: z.number().int(),
  end: z.number().int().nullish(),
  reason: RunPauseReasonZod,
})
```

* `checkpointExceeded` and `humanIntervention`: related to interactive mode, e.g. can let the agent go for so many seconds/tokens (or only a single action) between "checkpoints" that require human intervention to continue
* `pauseHook`: the human paused the run
* `pyhooksRetry`: pauses inserted while retrying API calls (e.g. rate limit backoff)
* `scoring`: Vivaria does this thing where tasks with intermediate scoring as scored when they first start, which we decided shouldn't count against the agent, so a pause is inserted to have the agent's clock start at 0 after scoring finishes.
* `legacy`: filler value for old rows before we had reasons


There are also pauses for semaphore waits. Tracking down and logging every semaphore wait (including in custom user code) and every network retry in model libraries might prove quite difficult and intrusive. If the use case is determining actual "running time" I wonder if we could turn things on their head and essentially assume that we are always "paused" (e.g. waiting for a retry or waiting on a semaphore) and instead measure "running" time (e.g. time spent in successful model calls, time spend in subprocess execution). I think this would be much easier and cleaner to determine than having to measure e.g. all coroutine suspensions and lock waits.

I really like the idea of only recording running time!

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1297
Repository: UKGovernmentBEIS/inspect_ai
Issue: Score events recorded in the transcript
Root cause: Unimplemented Feature Gap

Similar to #1263, ScoreEvent should be a first-class event in the transcript (currently implemented as an info event only for human agent).

Vivaria's schema for score entries is:
```js
export const IntermediateScoreEC = strictObj({
  type: z.literal('intermediateScore'),
  score: z.union([z.number(), z.literal('NaN'), z.literal('Infinity'), z.literal('-Infinity')]).nullable(),
  message: JsonObj,
  details: JsonObj,
})
```

For background:
* `message` is arbitrary JSON that is returned to the agent
* `details` is arbitrary JSON that is _not_ returned to the agent, but is stored (e.g. for data analysis)

Inspect's concept of a score is a bit more general than Vivaria's (e.g. non-numeric scores), so this might need to be expanded a bit.

The "final" score (as opposed to intermediate scores) are in fact in the transcript as `SampleScore`: https://github.com/UKGovernmentBEIS/inspect_ai/blob/1c16511130a536c2911ec2e0339bc6b01f9207de/src/inspect_ai/log/_transcript.py#L281. 

I'd want to be careful about adding more instances of this to the transcript as I don't want there to be confusion about what the "real" score is. Given this, are info messages for intermediate scores acceptable or do you want something more structured (we could e.g. create an "IntermediateScoreEvent" or have an `intermediate` or `final` flag on the other event.

Info messages are not enough

Okay, I'll add some variation of score event here.

One thing to remember about `info` events is that they can also take arbitrary JSON -- so it is possible for a solver to encode structured information for analysis using `info` events. We generally want specific solvers to do this rather than proliferating solver specific events.

That said, of course the scoring even is quite general so we will definitely create a new event (or variation of the existing scoring event).  For things like `task start` and `task stop` for the human agent we may be better off with JSON based `info` events (e.g. rather than creating `HumanAgentStart` `HumanAgentStop` events). 

The "pause" event discussed in the other issue is one general way to encode things, but there I worry that we are pausing so frequently (sleeps, semaphores, coroutine waits on model calls that never complete, etc.) that this will be nearly impossible to detect with high fidelity (and we might in fact be better off assuming we are always paused and then recording "activity"). 

All good points.

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1297
Repository: UKGovernmentBEIS/inspect_ai
Issue: Run cleanup and finalizers
Root cause: Unimplemented Feature Gap

I've been wanting a “finalizer”, which is a method that gets called at the end of the run but is not a scorer. We’re currently using a Scorer to e.g. upload file artifacts from the run to S3, but it doesn’t do any scoring. It would be good to have such a thing as a first class concept.

Some options discussed:
* Add a "cleanup" concept to tasks (Inspect have a "setup" concept)
* Create a higher-order solver that has a finalizer as part of Inspect
* Document how to do this yourself if you need to (e.g. a solver that wraps another solver and performs the finalization logic)
* Inspect has the concept of a Plan (and it has a finalizer), however it was deprecated as it was one more concept (plan/solver/agent/etc.). We could bring this back

PR here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1373
Repository: UKGovernmentBEIS/inspect_ai
Issue: Custom Displays
Root cause: Unimplemented Feature Gap

I was wondering if there was any play to update the 'active display' logic to support custom display types (i.e. not one of "full", "plain", "rich", "conversation" or "none"). The functionality for this seems quite easy to implement, so some fork could presumably do it pretty easily. Mostly I see three ways you might go about this:

- Keep the current setup where you have globals for `_display_type` etc, and update the display initialization functions to read from some registry of displays.
- Pass in the display to the evaluation as an argument.
- Otherwise modify the output logic to stream information to a generalized "Output" class, which then decides what to do with it. `Display` would then subclass this.

Thanks

Indeed, it wouldn't be hard to make this externally loadable (although honestly the display API wasn't designed for public consumption so might need some revision / clean up). I'm curious though what the scenario for a custom display is (as perhaps it's something we should support internally anyway)

Going to close this for now since our `Display` interface isn't the one I'd actually want to make public (it has too much inside baseball in it). If there are things about the current displays that you'd like to see improved feel free to open other issues to follow up.
Repository: UKGovernmentBEIS/inspect_ai
Issue: improved retry behavior
Root cause: Documentation Deficiency

Various suggestions from @craigwalton-dsit:

Sometimes when running evals against models which have flaky APIs which trigger their is_rate_limit() override to return True, users get confused as to why their eval seems to be "stuck" indefinitely. A few observations:

1. Some provider implementations like google treat numerous errors as is_rate_limit (e.g. 500, 503 and 504) ([source](https://github.com/UKGovernmentBEIS/inspect_ai/blob/100252c7ecb9b7207d3aeb8da45460a2c02aebe0/src/inspect_ai/model/_providers/google.py#L213)). Maybe this override would be better named is_retryable_error.

2. AFAIK it is not that obvious to users that Inspect is busy retrying/waiting to retry failed HTTP requests. The inspect trace anomalies is helpful. It might be even more user-friendly if the "Running samples" UI somehow indicated that a sample was in a retry loop (a bit like how we show "Generating ...").

3. The "rate limits" counter in the UI only updates for actual HTTP 429 errors (not other errors we treat as rate limits/retryable). This might well be the right behaviour, but just wanted to highlight this was one source of confusion. May be resolved by tackling 1.\

Previous discussion: https://github.com/UKGovernmentBEIS/inspect_ai/issues/1174

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1430
Repository: UKGovernmentBEIS/inspect_ai
Issue: MATH dataset used for mathematics tutorial no longer accessible
Root cause: External Dependency Breakage

The [Mathematics](https://inspect.ai-safety-institute.org.uk/tutorial.html#sec-mathematics) tutorial on the tutorial page relies on the huggingface version of the MATH dataset, which was recently taken down for copyright reasons. The existing code will throw an error:
`Couldn't find a dataset script at LOCALPATH or any data file in the same directory. Couldn't find 'hendrycks/competition_math' on the Hugging Face Hub either` (where LOCALPATH is a path on my computer where the data would have been stored.

There are [copies](https://huggingface.co/datasets/DigitalLearningGmbH/MATH-lighteval) of the dataset on the hub that are still accessible and which work with this tutorial, if a quick fix is necessary. (But that might run into copyright trouble?)

Argh! The point of this tutorial is to demonstrate a custom scorer. I think we really just need another dataset which benefits from the expression equivalence solver we demonstrate. Are you aware of any good candidates? (agree we wouldn't point to any of the copies).  

What do you think about this?

<https://github.com/google-deepmind/mathematics_dataset> (Apache-2.0)
<https://huggingface.co/datasets/deepmind/math_dataset>

``` 
Question: Let x(g) = 9*g + 1. Let q(c) = 2*c + 1. Let f(i) = 3*i - 39. Let w(j) = q(x(j)). Calculate f(w(a)).
Answer: 54*a - 30

Question: Let e(l) = l - 6. Is 2 a factor of both e(9) and 2?
Answer: False

Question: Let u(n) = -n**3 - n**2. Let e(c) = -2*c**3 + c. Let l(j) = -118*e(j) + 54*u(j). What is the derivative of l(a)?
Answer: 546*a**2 - 108*a - 118

Question: Three letters picked without replacement from qqqkkklkqkkk. Give prob of sequence qql.
Answer: 1/110
```

Closed with https://github.com/UKGovernmentBEIS/inspect_ai/pull/1538

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1715