zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Question] Scorer execution order enforcement
Root cause: Documentation Deficiency

Hi,
Is there a way to enforce Scorers order of execution? For example, if I have two scorers (scorer-a and scorer-b) as part of the task, is there a ways to start evaluating  scorer-b only after evaluation for scorer-a completed?
I'm trying to find a simple way to introduce custom scorer that will evaluate output of other scorer by storing/retiring key/values from sample "store". For example, after `model_graded_fact` completes and stores some data into "store" to access the "store" from a subsequent Scorer for additional logic. 
For now I was considering two options:
- write a custom Scorer to replicate built-in `model_graded_fact` and add additional functionality
- introduce "completion" flag into "Store" that will be set by wrapper  to `model_graded_fact`, and the new scorer will wait for the flag to be populate to indicate that `model_graded_fact` is completed.



@dragonstyle can probably comment better on this, but I think for this use case you might want to create a single scorer that produces multiple metrics: https://inspect.ai-safety-institute.org.uk/scorers.html#scorer-with-multiple-values

Are the scorers executed sequentially (for a given Sample)? Is there a facility to exchange information between the scorers other than "Store"? 
I could not find scorer information in the Task-state class.

Scorers are run sequentially for each sample.  The store is the only method of coordination between scorers I believe. 

I agree with @jallaire here that if the scoring gets very complex, it will often make sense to just create your own standalone scorer that just emits a dict[str, float] (or whatever).  Emiting a dictionary as a score value is will handled through the result of the system (you can map metrics onto each score in the dictionary, for example) and it will ultimately resolve into multiple named scores in the result.

This also would allow you to create a pretty readable set of code to deal with the complexity (I think since the coordination between scorers will be through loose data structures and order, it will be inherently fragile and hard to reason about). I don't fully know your use case, so take my thoughts with appropriate grains of salt!

@dragonstyle , thank you! The use case is to use `model_graded_fact` as step one. In step 2, in case that `model_graded_fact` evaluates the answer as incorrect, to use have additional LLM call that will produce reason for incorrect answer. Both "Correct/Incorrect" by `model_graded_fact` and the reason-code should appear as a separate scorer in the `inspect view`. Since I wanted to use built-in `model_graded_fact`, I wrote wrapper that intercepts the response and writes needed information for the next step into "Store". I might combine both steps as you have suggested in a single scorer.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Typo

https://ukgovernmentbeis.github.io/inspect_ai/tools.html

 By default web **searches retrieves** 10 results from a provider,

Grammar issue

Fixed with https://github.com/UKGovernmentBEIS/inspect_ai/commit/671a239f13fe2706f89ffccaf5ac5eef86958fc3
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Documentation] Update documentation to replace GenerationConfig  with GenerateConfig

Hi,
The Helper models documentation needs to be updated to correct:
```
from inspect_ai.model import GenerationConfig, get_model
```

To:
```
from inspect_ai.model import GenerateConfig, get_model
```

Thanks! Fixed here: https://github.com/UKGovernmentBEIS/inspect_ai/commit/53fe15a44d83400ad2847cb2f917ac282cba126d
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] Add standardised way to index samples within task
Root cause: Documentation Deficiency

Often I want to run a single Sample from a Task. e.g.:
* Run just the first sample to sanity test setup
* Run a particular sample

Right now it is possible for a Task developer to enable indexing/filtering particular samples but if this isn't included at the time of development then there is no way to select particular Samples without a future change to each Task.

Would it be possible to add something equivalent to Sample indexing i.e. evaluating `task[0:10]` would evaluate only the first 10 samples.

You can currently do this using `--limit`. For example:

```
inspect eval ctf.py --limit 5      # first 5 samples
inspect eval ctf.py --limit 6-10   # samples 6-10
```

Does this do what you are looking for? (the `eval()` function also has a `limit` argument)



@skinnerjc can I close this? (does `--limit` do what you want?)

Going to assume that this is handled by `--limit` (which you can also pass to `eval()`).
Repository: UKGovernmentBEIS/inspect_ai
Issue: plot_results(): Are there any frameworks that allow summarising and visualising inspect logs? 
Root cause: Unimplemented Feature Gap

Many evaluation tools have frameworks to allow summarising and visualising results. An example is [zeno](https://zenoml.com/docs/integrations/eleuther) for [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness#visualizing-results). I understand that results-summarisation & visualisation needs can be quite diverse and one tool may not work for anyone. Still, I think if inspect ai logs can be easily summarised and visualised, researchers could iterate faster. 
I wrote a very quick and dirty class for visualising a list of EvalLogInfos for [my own experiments](https://github.com/sohaibimran7/SOCR-follow-up-experiments/blob/main/src/inspect_helpers/visualizer.py) and was wondering what other people use and whether there is interest in results summarisation visualisation support for inspect.

This is definitely something we are interested in supporting more deeply! We are soon going to make it possible to run a set of analysis code on top of an eval-set and then display that in the viewer. At the same time, we will hopefully discover some useful common idioms and tools that we can provide. Would love to hear from people on this thread about what the general shape of requirements are!

I personally would value the following in a visualisation framework:

1. Ability to categorise logs by {log_dir, run_id, task, dataset, scorer and model}
2. More finely categorise based on substrings of {model, task, log_dir} 
3. Ability to rename categories and their elements and sort and filter by categories using custom sort and filter functions. 
4. Ability to map each category to a plotting element {x axis, y axis, x offset, y offset, colour, horizontal and vertical faceting in a multi-plot figure}
5. Ability to plot any figure I like (bar charts, box plots, violins etc.)
6. Extensibility

We've done some work recently here:

1) Inspect dataframe functions: https://inspect.aisi.org.uk/dataframe.html
2) Inspect Viz plotting library: https://meridianlabs-ai.github.io/inspect_viz/
Repository: UKGovernmentBEIS/inspect_ai
Issue: Semver Dependency Needs Pin
Root cause: External Dependency Breakage

In the version.py file: https://github.com/UKGovernmentBEIS/inspect_ai/blob/d52b1920391eabc69bb32f3ea83470f3ad2ec760/src/inspect_ai/_util/version.py

you use `semver.Version`. This was introduced from semver 3.0.0 and above. In the requirements.txt (https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/requirements.txt#L23) there is no lower-bound pin to semver. So in my environment, where I had semver 2.13.0, inspect_ai was erroring.

Simply setting a lowerbound pin to >=3.0.0 will fix this issue. Then when someone installs inspect_ai, it is guaranteed to have a compatible version of semver installed.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Make using `AsyncAzureOpenAI` an explicit choice
Root cause: Architectural Constraint

Choosing which OpenAI client to use based on which environment variables are present can be misleading.
One can have multiple env vars set and still want to use a specific client.
Or one wants to use both OpenAI and Azure in a single script.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/53fe15a44d83400ad2847cb2f917ac282cba126d/src/inspect_ai/model/_providers/openai.py#L68

Either choose the client based on the format of a passed in api key (again implicitly).
Or preferably, provide an option to initialize the `AsyncAzureOpenAI` client.

Thanks for this! You are right that we don't quite expose enough knobs here. One thing we could do is a a custom model arg that forces Azure (or not Azure). Another thing we could do is leverage the fact that Azure model/deployment names don't use the same structure as OpenAI model names (which are a fixed enumeration). Will improve this soon, thanks again!

any news on this?

Thanks for your patience. This is resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/123

It still does automatic binding, but you can pass an explicit `azure` model argument to force it to use (or not use) Azure.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect view Bug when copying ID
Root cause: Validation Gap

Withiin the inspect view, one can usually click a copy button, to copy the run ID. 
![Screenshot 2024-10-17 at 12 35 34](https://github.com/user-attachments/assets/765487d0-882d-4c18-bb73-87ebe08c44bd)

However, out of nowhere, something happens within the viewer after viewing some runs and suddenly the button looks like follows: 
![Screenshot 2024-10-17 at 12 36 01](https://github.com/user-attachments/assets/5c8014c6-4ff4-4519-8869-9ca7b1ca7583)
From here on when you click this button it won't actually copy the path! So when you paste things, you paste whatever you copied somewhere else. 

This usually fails quite silently since one assumes that the copy button just works.

Not sure what the bug actually is and what triggers it , but it's likely something within the viewer. 

Is there any update on this? It keeps happening and makes the tool quite painful to use.

I haven't been able to reproduce this issue, unfortunately. A few questions that could help narrow things down:

- What version of inspect are you running? Does it still occur if you update to the latest version?
- What browser are you using (or are you viewing in VS Code)?
- Have you noticed a pattern at all as to what precedes this happening, or specific logs that it always occurs for?

Sorry that we haven't been able to fix this!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Occasional incorrect grading by choice scorer with chain-of-thought prompting
Root cause: Algorithmic Error

When running multiple-choice evaluations using chain-of-thought prompting and the `choice` scorer, there are occasional instances where the scorer appears to incorrectly grade the model's response. This occurs despite the model providing a clear final answer in the specified format.

I've seen
1. The scorer fails to extract any answer at all, despite the model providing a clear final answer in the specified format.
2. The scorer extracts an incorrect answer, interpreting the entire response as the answer or extracting something like "A, B, C, D" despite a clear, single-letter answer.

## My setup
I'm running multiple-choice evals with chain-of-thought prompting using the `choice` scorer and the following prompt template:
```
MULTIPLE_CHOICE_TEMPLATE_COT = r"""
Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of {letters}. Before answering, reason in a step-by-step manner as to get the right answer.

{question}

{choices}
""".strip()
```
I'm using Inspect v0.3.25.

## Expected behavior
The scorer should correctly identify and grade the final answer provided by the model, which is formatted as "ANSWER: $LETTER", even when there is "reasoning text" before the final answer.

## Actual behavior
In rare cases (approximately 1 in 100 samples), the scorer appears to incorrectly grade the response. It may interpret the entire response as the answer or fail to recognize the correct final answer line.

Below are two toy examples illustrating the issue. I'm using the toy example to avoid leaking actual eval data onto Github, but I'm happy to share logs where this is happening privately via email. 

### No answer extracted
```
Model Response: 
Let's approach this step-by-step:
1. [reasoning steps...]
...
Therefore, the most likely scenario is that the grass is green.
ANSWER: D

Scorer Output:
ANSWER: 
EXPLANATION: [full model response repeated]
SCORE: I
```
### Incorrect answer extracted
```
Model Response:
To determine the number of moons that Earth has, I will reason step-by-step...
[reasoning steps...]
Conclusion:
The Earth has one moon according to my most recent knowledge cutoff. 
ANSWER: B

Scorer Output:
ANSWER: A, B, C, D
EXPLANATION: [full model response repeated]
SCORE: I
```

Hello! It would be great to get a look at the log (and ideally a specific sample) where you're seeing this behavior to see if I can form a bit more of a theory as to what about the sample is tripping up the scorer. You can email me directly at either cteague at gmail or cteague at rand.org.

Thanks and sorry for the issue!

Sent you an email!

Thank you! I can see is the issue based upon those logs. I will get a fix put together ASAP!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Support Dockerfile (not just compose file) in call to `sandbox()`
Root cause: Unimplemented Feature Gap

The documentation says:

> By default, any Dockerfile and/or compose.yaml file within the task directory will be automatically discovered and used. If your compose file has a different name then you can provide an override specification as follows:
> ```
> sandbox=("docker", "attacker-compose.yaml")
> ```
> The configuration file added to the sandbox spec should always be a compose file (rather than a Dockerfile, which is always discovered automatically).

https://inspect.ai-safety-institute.org.uk/agents.html#sec-sandbox-environments

I think it would be nice to also support Dockerfiles here, since we otherwise support them.

I am aware that internally, an "automatic" compose file consisting of a single service is created for Dockerfile definitions. But I think we should abstract that away from the user.

I don't have a strong view on whether the type that is passed around internally should take the Dockerfile or always require the (auto-)compose file. The user-facing type however, should allow either. Currently both types are [`SandboxEnvironmentSpec`](https://github.com/UKGovernmentBEIS/inspect_ai/blob/090c5c95bf5a8f22e499158d42641cbbad1499c3/src/inspect_ai/util/_sandbox/environment.py#L197).

The other way to avoid inconsistency would be to drop support for pure Dockerfiles and always require a compose file. I personally would be OK with that too, but it's a breaking change that you probably don't want.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Calling eval() does not load model args from env file
Root cause: Configuration Error

It seems like calling the eval() method loads the `.env` file [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/090c5c95bf5a8f22e499158d42641cbbad1499c3/src/inspect_ai/_eval/eval.py#L261), but never actually uses the `INSPECT_EVAL_MODEL_ARGS` environment variable. This is not an issue when calling `inspect eval` because click handles this. 

I haven't looked closely but there might be other envvars that are also missed when running `eval()`.

Yes there are a quite a few values that aren't read from the environment. Technically that is a CLI feature however `.env` files are so convenient that it probably makes sense to make them all work even for `eval()` (currently the most widely used env file variables including `INSPECT_LOG_DIR`, `INSPECT_EVAL_MODEL`, and API keys _do_ work, which makes it extra frustrating that some don't work. We will try to resolve this soon!
Repository: UKGovernmentBEIS/inspect_ai
Issue: How to add custom Model Provider without creating a separate package?
Root cause: Documentation Deficiency

I've implemented a custom model extending ModelAPI per your [instructions](https://ukgovernmentbeis.github.io/inspect_ai/extensions.html#sec-model-api-extensions)

I've successfully created a python package with an entry point. Everything works fine.

But this setup complicates my deployment and I am looking to simply include my custom model code directly into my application without creating a new package. 

Is it possible and how can I do it?

Yes, you can absolutely do this. Your custom model will be available so long as the file that contains it has been imported into the script that uses it. So this will work with `eval()` but will not work with `inspect eval`.

The reason for doing a separate package / setuptools entry point is so that we can resolve model names at the command line (when no Python code has been imported). If you are just calling `eval()` directly though this is not required.

@jjallaire, thank you so much, it worked like a charm! 

`import eval_tool.models.custom_endpoint # my code with custom model`  into the module that contains call to Inspect `eval()` solved the problem.

Maybe it's worth to add your explanation to the [Extensions](url) documentation page.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Additional completion params for ollama API
Root cause: Unimplemented Feature Gap

Ollama's API supports some parameters that OpenAI does not. Their API documentation [here](https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-completion) references the `options` parameter, which refers to the options listed [here](https://github.com/ollama/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values).

In particular, `num_ctx` would be VERY useful to support, because currently in Inspect AFAIK the only way to use ollama models with context length more than 2048 (the ollama default) is to create a custom modelfile on the ollama host machine with the context length you want, then request that custom model from the client side.

We are using the OpenAI interface for Ollama which in theory should be bridging `num_ctx` from `max_tokens`? (not sure if it does). 

I noted that looking at the native Ollama Python API that `num_ctx` is a load time rather than runtime parameter (https://github.com/ollama/ollama-python/blob/main/ollama/_types.py#L148) so I think even if we supported the native API we couldn't set `max_tokens` per request.



Going to close this as I think this is the province of the Ollama OpenAI layer (as well as possibly a feature request for Ollama to support more dynamic context windows, it seems as if now this can only be set at model load time).

> We are using the OpenAI interface for Ollama which in theory should be bridging `num_ctx` from `max_tokens`? (not sure if it does).

Looks like they're working on it! ollama/ollama#6504
Repository: UKGovernmentBEIS/inspect_ai
Issue: Web browser struggles with large pages
GitHub labels: web browser

The browser tool gets hung up trying to process large text files e.g. https://www.gutenberg.org/cache/epub/100/pg100.txt

Example here https://github.com/UKGovernmentBEIS/inspect_ai/blob/browser_large_text/tests/tools/test_web_browser.py#L151

This also happens with large HTML pages such as https://www.gutenberg.org/cache/epub/100/pg100-images.html although it does actually get through it (it takes about 30 minutes to open the page and scroll down once)

There was some o(n) code in the original web browser implementation which we cleaned up here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1314 . That should take care of this issue.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Approvals for Entropy-labs not found in the registry
Root cause: Configuration Error


Hi @jjallaire, we have created approvers inside our [entropy-labs](https://pypi.org/project/entropy-labs/) python package. We followed instructions for the approvals registration here: https://inspect.ai-safety-institute.org.uk/extensions.html#sec-extensions-approvers. In our `pyproject.toml` file, we specified

```
[project.entry-points.inspect_ai]
entropy_labs = "el.supervision.inspect_ai._registry"
```
and this is our `el.supervision.inspect_ai._registry`:
```
from .supervisors import bash_approver, python_approver, human_approver, llm_approver
```

But when running inspect eval ( `inspect eval run.py --approval approval_human.yaml --model openai/gpt-4o --trace` ) with this example `approval_human.yaml`
```
approvers:
  - name: el/human_approver
    tools: "*"
    approval_api_endpoint: "http://localhost:8080"
    agent_id: "my_sample_1_agent"
    n: 1
```

We receive `ValueError: el/human_approver was not found in the registry`. Our approvals are registered only if we manually import our registry to the run.py script where the inspect tasks are defined. 

```
from el.supervision.inspect_ai import _registry
```

But this is just a hack for now and is not good for evals already integrated to Inspect examples. Do you know if we are missing some configuration or we configured something wrong? Is it possible that there is an issue with the Approver Registration code itself?

**To replicate**

Install entropy-labs: 
```
pip install entropy-labs --upgrade
```
And use the .yaml above with inspect eval, e.g.:
```
inspect eval inspect_evals/gdm_in_house_ctf --model openai/gpt-4o --limit 1 --approval approval_human.yaml
```

Thank you for your help!

This came down to aligning on the "names" in play (package name, setuptools entry point name, dist name). This change to your `pyproject.toml` will make things work against Inspect on PyPI:

https://github.com/EntropyLabsAI/sentinel/compare/main...jjallaire:sentinel:patch-1

Note we had to change the project name to `ep` (to align with the package name). You might have reasons for not wanting to do this?

With this change to Inspect (which won't hit PyPI for a week or two) you can keep the project name "entropy_labs":

https://github.com/UKGovernmentBEIS/inspect_ai/pull/750/files




Thank you @jjallaire for looking into this. As you mentioned, the problem was with the name discrepancy `el` vs. `entropy_labs`. We couldn't name our package `el` as that name was already taken in PyPI, but we also wanted to give our users the ease of importing our package as `from el import ..`. That being said, to simplify this and move forward, we renamed everything (package, setuptools entry point name, dist name) to `entropy-labs` and the approval registry now works.