zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: Using multiple scorers as list input to the Task gives error
Root cause: Environment Incompatibility

When trying to run example from the documentation that demos use of multiple scorers via list:
```
Task(
    dataset=dataset,
    plan=[
        system_message(SYSTEM_MESSAGE),
        generate()
    ],
    scorer=[
        model_graded_qa(model="openai/gpt-4"), 
        model_graded_qa(model="google/gemini-1.5-pro")
    ],
)

```
Receiving error:
```
TypeError: 'list' object is not callable
```

Here is modified example of `theory_of_mind.py` to reproduce the error:
```
from inspect_ai import Task, task
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact, model_graded_qa
from inspect_ai.solver import chain_of_thought, generate, self_critique


@task
def theory_of_mind(critique=False):
    # use self_critique if requested
    plan = [chain_of_thought(), generate()]
    if critique:
        plan.append(self_critique())

    return Task(
        dataset=example_dataset("theory_of_mind"),
        plan=plan,
        scorer=[model_graded_fact(), model_graded_qa()],
    )
```

I got this error too, but noticed that it runs fine in the VSCode extension.

I believe proper approach is to use `multi_scorer` instead of the list of scorers. 

Could you confirm the version of Inspect that you're using? We only recently introduced the multiple scorer support, and it is not yet available in the version distributed on PyPi (we've updated to docs to reflect this).

It should work if you pip install this repo directly like:

```bash
pip install git+https://github.com/UKGovernmentBEIS/inspect_ai
```

Sorry for the issue and being a bit out of sync.

I was using `inspect_ai==0.3.17` version. Thank you for looking into it.

I checked the modified code at the start of the issue with the version `pip install git+https://github.com/UKGovernmentBEIS/inspect_ai` , and it is working. I can see two tabs, one for each scorer now at the sample level. Could you clarify how the multiple scorers (provided in the list) are "reduced"? I see that documentation was updated, but still not covering the "reduce" part. I assume it is majority vote. 

The new multiscoring features actually have no reduce step - the goal is support creating more than one score for an eval, for cases when that is useful. The scores are treated as independent and not reduced. 

If you'd like to reduce multiple scores, you can continue to use the `multi_scorer` (perhaps in conjunction with `majority_vote`) to generate single score that represents the majority of graders. More here:

https://ukgovernmentbeis.github.io/inspect_ai/scorers.html#multiple-models

edit: (clarity / typos)

Version with multi-corer is now published to PyPI (docs have updated accordingly with a bit clearer delineation for the multi_scorer / reduce scenario).
Repository: UKGovernmentBEIS/inspect_ai
Issue: RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 with gemma-2-9b
Root cause: External Dependency Breakage

I'm trying to run some evaluations with Gemma-2, but I'm getting the following error
```RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 ```

My `Task` is defined as follows:
`Task(dataset=dataset, plan=[generate()], scorer=model_graded_qa())`

Any help is much appreciated. Thanks!

Could you provide a full snippet of code that includes the model? (not sure if you are using HF, a remote API, Ollama, etc.). Thanks!

@aisi-inspect Sure thing!

```
dataset = json_dataset(
    "outputs/succinct.jsonl", FieldSpec(input="question", target="best_answer")
)


@task
def eval_hallucination():
    return Task(dataset=dataset, plan=[generate()], scorer=model_graded_qa())
```
The model is provided at the command line as follows with the inspect eval command
`--model hf/google/gemma-2-9b-it`


My bet would be that this is a precision error from huggingface's `model.generate` which we call [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/8ab261172252c1956da43b04df17cce706e79cbc/src/inspect_ai/model/_providers/hf.py#L333), rather than anything on our side. 

Passing a higher precision dtype for your model through the model args would likely fix the problem, at the cost of higher memory usage. 

Although I vaguely remember there being some setting for `generate` that would catch these precision errors that we could add in...

@sdtblckgov The default generate conditions are already full-precision, aren't they? 
https://huggingface.co/google/gemma-2-9b#running-the-model-on-a-gpu-using-different-precisions 


@shamikbosefj Could you try passing `remove_invalid_values=True` to generate? You should just be able to add it to the kwargs [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/b3af8eb399b4588a85674ceb6cf6cfff61b5f0ec/src/inspect_ai/model/_providers/hf.py#L136).

see the huggingface docs for more info about what this should do https://huggingface.co/docs/transformers/main/en/main_classes/text_generation 

Sadly I can't actually reproduce the error you're referring to - could you please provide enough details that I would be able to? The snippet above alone isn't enough for us to reproduce on our end. 

What would help here is:

1. Full, reproducible code that will cause the error to occur. I.e not just a snippet like the above, but something we can run on our end. For instance, we don't have access to your dataset from the above snippet. Please provide us with a complete script + details regarding any dependencies. 
2. software versions of pytorch, huggingface, and cuda drivers
3. hardware setup (what gpu are you using - you can just provide the output of `nvidia-smi`)


Did you mean to tag me @sdtblckgov?

I think cc was intended for @shamikbosefj 

@jjallaire-aisi I haven't been looking at this for a while. Will check later today and report back. I might not be able to provide the dataset, but might be able to provide a sample

Correct, not sure how that happened, sorry rusheb!
Repository: UKGovernmentBEIS/inspect_ai
Issue: Adding a scorer without rerunning existing scorers.
Root cause: Documentation Deficiency

I sometimes want to score a task with an additional scorer but not have to rerun the scorers that have already been run. Is there a way to do that?

There is a actually a `score()` function you can use for this (but I don't think it's properly documented on the website, we will remedy this!): https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/_eval/score.py

Doesn't the score function simply return a new EvalLog with only the newly specified scorers? Could there perhaps be a flag that it accepts which when checked, appends the scorer and its results to the EvalLog? 

Also, the score functions can only process a single EvalLog at a time. This is quite a minor thing but it'd be nice if it can score multiple Evallogs simultaneously.

Thanks for the quick response btw!

This should be resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1293

Docs here: https://inspect.ai-safety-institute.org.uk/scorers.html#sec-scorer-workflow
Repository: UKGovernmentBEIS/inspect_ai
Issue: ValueError: could not convert string to float: 'C' when using Mean scoring
Root cause: Validation Gap

Following the documentation here: https://ukgovernmentbeis.github.io/inspect_ai/scorers.html#reducing-multiple-scores and https://ukgovernmentbeis.github.io/inspect_ai/scorers.html#built-in-metrics

I basically copied the scorer into my script.

```
@scorer(metrics=[mean()])
def multi_model_graded(MODELS):
    return multi_scorer(
        scorers = [model_graded_qa(model=model) for model in MODELS],
        reducer = majority_vote
    )
```

Produces the error, ValueError: could not convert string to float: 'C'

I have no issues when switching mean out for accuracy and bootstrap_std. I could be wrong, Im a hacker not a true programmer, but it looks like from the source, mean doesn't call the same ValueToFloat function as the others?

You are correct! The `mean()` metric is assuming that your value is already a float. We may need to add the `ValueToFloat` handling here, but in the meantime you should be able to switch to `accuracy()` and get the exact same results as if `mean()` had this built in. 

Appreciate the reply, I have been using accuracy for now. I see what you are saying, that mean could be used in a custom scorer, but the way it's presented in the documentation suggest it should work in the say way as the other metrics. Feel free to close this if the design intent is correct and it was purely my user error.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature request] Debug logging for cache inputs

It would be very helpful to see debug logs containing the inputs to the cache key for the model calling cache. We added this recently via monkeypatch to debug some of our integration tests which use cached model calls.

For example you could do something like this:
```diff
diff --git a/src/inspect_ai/model/_cache.py b/src/inspect_ai/model/_cache.py
index ce5be49a..eb583a33 100644
--- a/src/inspect_ai/model/_cache.py
+++ b/src/inspect_ai/model/_cache.py
@@ -153,8 +153,17 @@ def _cache_key(entry: CacheEntry) -> str:
 
     base_string = "|".join([str(component) for component in components])
 
+    logger.debug(_cache_key_debug_string(components))
+
     return md5(base_string.encode("utf-8")).hexdigest()
 
+def _cache_key_debug_string(components: list[str]) -> str:
+    components_str = "\n".join(
+        f"  - {component}" 
+        for component in components
+    )
+    return f"Computed cache key from components:\n{components_str}"
+
 
 def _cache_expiry(policy: CachePolicy) -> datetime | None:
     if policy.expiry is None:

```

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/791
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature Request] - log viewer export, perhaps static HTML?
Root cause: Unimplemented Feature Gap

I would love to be able to export and share my logs from a run. Either just for sharing, or perhaps for viewing CI/CD results. Seems like a straightforward way to do this might be as a static directory of html and assets. Zipping that directory up up seems like a portable way to share results.

https://github.com/UKGovernmentBEIS/inspect_ai/blob/6bd0a0a06cd3960a75cec4dce9b9a23cbc3880e9/src/inspect_ai/_view/view.py#L84

Is this API the only dynamic part of the front end? Its a small amount of routes, so seems like perhaps a small surface area for the change. Unfortunately 2 of the routes make use of query params, so it's hard to predetermine all the route patterns that they'd be called with

Yes, I think this would be a pretty straightforward thing to do. There is a actually a viewer mode that takes a log URL as a query param and just displays a standalone viewer. Probably this isn't far off from being able to just use an embedded log file? (or even load arbitrary log files using relative references). @dragonstyle let's chat about what it would take to enable this.

This has been implemented here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/91

Thanks for taking a look at this and implementing it so quick!
Repository: UKGovernmentBEIS/inspect_ai
Issue: A scorer to compute costs based on model usages (for eval model, grader model, etc.)?
Root cause: Documentation Deficiency

I'd like to have a scorer that has access to token counts (model usages) used on evaluation of each sample - those usages that are returned by `generate()`.
My goal is to calculate cost for each sample evaluation based on the usage by solvers and scorers.
All I need is access to the token counts produced by `generate()` by all models called during the task - I'll take care of the rest.
What would be proper way of defining a scorer that does cost computations based on model usages?

Thanks for this suggestion! We just added a `token_usage` property to `TaskState` here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/964

You can access this from any scorer (and if we want to include tokens used by other scorers just make sure the one that counts tokens is specified last).
Repository: UKGovernmentBEIS/inspect_ai
Issue: For certain Model API calls, if they error we don't have the API call logged to the transcript.
Root cause: Validation Gap

As discussed in the Internal AISI slack https://aisafetyinstitute.slack.com/archives/C06LSBZNAPP/p1730846312556919 

Seen this with Groq and HTTP rate limits as well.

This should be resolved here (the request body is now included in the payload of any error with a 400 status): https://github.com/UKGovernmentBEIS/inspect_ai/pull/1134
Repository: UKGovernmentBEIS/inspect_ai
Issue: Question about running code on eval results
Root cause: Unimplemented Feature Gap

Hi! Is there an easy way to run code on the output of a Task from within a Task declaration? Currently, I'm doing something along the lines of this:

```
results = eval(my_eval(), model="groq/llama3-8b-8192")
process_eval(results)

```

but I'd like to be able to define some code to run on the output of the eval within the function that defines the task, so that `process_eval` runs even when I call the `my_eval` with the CLI. Is there any native way I can do that?

Thanks in advance!

There isn't currently a hook for something like this. If you could further delineate the specific use case we can consider what might work well. Note that we are likely to soon provide some ability to control task execution (have `Task` include a `run()` function you can override or comparable w/ functional hooks) which would probably fit the bill. We've also discussed introducing a results filter that lets you make arbitrary changes to the log file before its returned/written. 

The goal is to provide more detailed speed metrics from timestamps and token counts available in the EvalLog. The run feature would be awesome - and being able to write the new metrics to the logfile would be awesome as well.

I wonder if some of this you could just do in a solver or scorer? Exactly which data structures are you wanting to access and compute on?

1. `EvalLog.results`:
   - `total_samples`: Total number of samples in the evaluation
   - `completed_samples`: Number of successfully completed samples

2. `EvalLog.samples`: Contains individual sample data, where each sample has:
   - `events`: List of events for each sample, containing:
     - `event`: Type of event (e.g., 'sample_init', 'step', 'model')
     - `timestamp`: When the event occurred
     - For 'model' events specifically (this is Groq specific):
       - `call.response['usage']` contains:
         - `prompt_time`: Time taken for prompt processing
         - `completion_time`: Time taken for completion generation
         - `total_time`: Total processing time
         - `prompt_tokens`: Number of input tokens
         - `completion_tokens`: Number of output tokens
         - `total_tokens`: Total tokens processed

3. `EvalLog.eval`:
   - `model`: Name of the model being evaluated

4. `EvalLog.plan.config`:
   - `seed`: Seed used

hey @jjallaire! just following up - is it possible to access all of the above from a scorer or a solver? All of this data is needed for what I'm computing.

Should be resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/872
Repository: UKGovernmentBEIS/inspect_ai
Issue: Outdated Package in Dependency
Root cause: External Dependency Breakage

**Problem Description**: when installing `inspect_ai` in my env, I have dependency conflict as the following:

> The conflict is caused by:
>     boto3 1.34.140 depends on botocore<1.35.0 and >=1.34.140
>     s3transfer 0.10.0 depends on botocore<2.0a.0 and >=1.33.2
>     aiobotocore 2.4.2 depends on botocore<1.27.60 and >=1.27.59

**Cause**: My boto3 is set to be `1.34.140`, however I believe `inspect_ai` requires [s3fs>=2023](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/requirements.txt#L19), which itself requires [aiobotocore>=2.5.4,<3.0.0](https://github.com/fsspec/s3fs/blob/main/requirements.txt#L1). [fsspec==2024.6.1.*](https://github.com/fsspec/s3fs/blob/main/requirements.txt#L2) is also causing conflicts for me. Is it possible we can relax the dependency requirements here? Thanks!

Argh, I hate when people upper bound their version requirements! :-( 

I need to consult with the person who put that s3fs constraint in (I recall this was done to workaround yet another dependency problem). Will check back in here once I sort that out.

Apologies for the delay in getting back to you.

The latest version of `s3fs` (2024.6.1) requires

```
s3fs 2024.6.1 Convenient Filesystem interface over S3
├── aiobotocore >=2.5.4,<3.0.0
│   ├── botocore >=1.34.70,<1.34.132 
```

Note: `s3fs`'s primary dependency requirements are sensible - the issue is `aiobotocore` imposing an upper bound on `botocore`.
Note: `botocore 1.34.132` (upper limit) is from June 2024.

And the version of `boto3` which you wish to use (`1.34.140`) requires a version of `botocore >= 1.34.140`.
```
boto3 1.34.140 The AWS SDK for Python
├── botocore >=1.34.140,<1.35.0
```

So AFAIK, **the root issue** here is that `aiobotocore` does not support the version of `boto3` which you wish to use. This seems to be [a common issue](https://github.com/aio-libs/aiobotocore/issues?q=label%3Adeps-update+). 

If we remove inspect's `>=2023` requirement on `s3fs`, we can technically resolve the dependencies, but we end up using an ancient version of s3fs (0.4.2 - which does not use `aiobotocore`) which causes issues:
```
UserWarning: Your installed version of s3fs is very old and known to cause
severe performance issues, see also https://github.com/dask/dask/issues/10276

TypeError: Session.__init__() got an unexpected keyword argument 'cache_regions'
```

My suggestion would be to create an issue or PR in the `aiobotocore` repo to bump the upper limit of botocore to match (or exceed) the version of boto3 that you wish to use. [Source](https://github.com/aio-libs/aiobotocore/blob/7154abc9ff395af13b8cfa7b64a07a6bf7d1c07e/setup.py#L10)

Happy to explain further and do please let me know if you can see another solution which we can help with.

I am going to close this as I think the real problem is aiobotocore having such a narrow dependency on botocore. As illustrated in the many issues filed on their repo, this already preventing aiobotocore from being usable in many pretty vanilla scenarios.
Repository: UKGovernmentBEIS/inspect_ai
Issue: [Question] Easy way to use finetuned OpenAI models?
Root cause: Documentation Deficiency

I finetuned GPT-4o model, and I try to figure out how to use it, but can't come up with a straightforward solution. It seems like at the moment I either have to create a custom model API class, or abandon Inspect altogether and use raw OpenAI API. Is there an easier way to use finetuned OpenAI model?

Figured out thati  might be done just by typing the name of the finetuned model while running inspect. Although it was unclear from the documentation.
Repository: UKGovernmentBEIS/inspect_ai
Issue: Inspect process uses large amounts of memory w/ DockerSandbox, likely because large exec() outputs are loaded into memory
Root cause: Resource Mishandling

I've been running evals which have been falling over due to OOM, even after changes introduced by #781. Here is an example, where you see a very expensive agent call  `python3 /root/solve.py` (using about 40GB of memory), and also an inspect process (`/home/ubuntu/agent_framework/more_datasets_sweep_copy.py`) using about 20GB of memory (this is a big machine):

![image](https://github.com/user-attachments/assets/be8cf3cd-c5dd-471d-8357-2258df6cb96d)

The agent process using so much memory isn't a bug in inspect per se (although unfortunate, and should be reduced by container-specific memory limits), but the inspect process doing so doesn't seem good. This is likely due to (as craig mentions in #781) the current DockerSandbox does not stream its outputs but instead writes the full stdout to memory, and only then checks its not too big *after* the command has finished - so if the agent is doing something wild w/ a very large output command, this could fill up the inspect process' memory.

The likely fix here is to have subprocess() stream, and check tool output limits against the streaming output.  For a reproduction I would guess that a while loop printing to stdout would repro the problem.

Another odd thing is that inspect's timeout feature doesn't kill this process even though its quite long running. @jjallaire thinks this just depends on the semantics of cancelling an asyncio task, if the subprocesses get killed.


Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/925
Repository: UKGovernmentBEIS/inspect_ai
Issue: Disabling exponential backoff
Root cause: Documentation Deficiency

Hello!

How can I disable exponential backoff for rate limits and just have the query be immediately retried until it succeeds?

There are two set of backoff that happen for model requests -- the first is whatever is provided by the model interface package (e.g. `openai`, `anthropic`, etc.). Generally these packages implement an internal retry scheme based on HTTP headers (e.g. a header comes back saying "retry at this particular time" along with the 429). This allows services to both optimise load and keep retries optimally proximate to the original request. 

Assuming this is for use with `groq`, it looks like the `groq` package is doing some internal exponential backoff: https://github.com/groq/groq-python/blob/c32a996df186d04e5d97c084c18acd0d43472e31/README.md?plain=1#L145-L175. We set `max_retries` to 5 by default, so before Inspect does anything with backoff `groq` is doing 5 retries with its own exponential backoff. If there is a way to tune the `groq` package to just hammer retries in immediately then that would get you 5 immediate retries (however, I don't know if this is recommended as the odds of another 429 seem quite high just milliseconds after the first 429).

So the way to get snappier retries is have the service and the client package coordinate explicitly. If this isn't happening then you will immediately fall into the Inspect outer retry handler which does exponential backoff. Note that these retries will occur at ~ 3 seconds, 5 seconds, 8 seconds, etc. so not immediate but very close to it. If 429 keeps coming back time and time again the backoff is helpful in almost all cases as the service does need time to recover.

thanks!