Repository: UKGovernmentBEIS/inspect_ai
Issue: Eval function with task_args to support `eval_retry`
Root cause: Validation Gap

Hello,

I could not re-open issue https://github.com/UKGovernmentBEIS/inspect_ai/issues/153. And I have opened a new issue. 

When coding the `eval` with `task_args` (I have excluded sending any other parameters via `task_args` other than model and the dataset.
```
eval_log = eval(
                    #tasks=task(grader_model=grader_model, dataset=dataset_obj, **kwargs), #original call
                    tasks=task,
                    task_args=dict(grader_model=grader_model, dataset=dataset_obj),
                    model=model,
                    log_dir=output_folder,
                    limit=samples_limit,
                    max_connections=max_connections,
                    log_level=settings.log_level,
                )
```
The code fails with the error:
```
root - ERROR - Job 'eval' failed with error: cannot pickle '_thread.RLock' object
```
You can recreate the issue by running the following code, where grader model is `AzureOpenAI` model. It seems that the `AzureOpenAI` cannot be serialized.
```
serializable_kwargs = copy.deepcopy(dict(grader_model=grader_model))
```
Full trace:
```
Traceback (most recent call last):
  File "/usr/local/stage3technical/var/sand/stage3technical/tcom-middle-tier/job_manager/dispatcher.py", line 186, in dispatch
    job_runner(job_name,
  File "/usr/local/stage3technical/var/sand/stage3technical/tcom-middle-tier/job_manager/jobs/eval_runner.py", line 56, in run
    run_eval_task(
  File "/usr/local/stage3technical/var/sand/stage3technical/tcom-middle-tier/eval_tool/helpers/eval_tool_utils.py", line 169, in run_eval_task
    eval_log = eval(
               ^^^^^
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-7-27-24/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 109, in eval
    return asyncio.run(
           ^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-7-27-24/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 229, in eval_async
    resolved_tasks.extend(resolve_tasks(tasks, task_args, m, toolenv))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-7-27-24/lib/python3.11/site-packages/inspect_ai/_eval/loader.py", line 120, in resolve_tasks
    load_tasks(cast(list[str] | None, tasks), model, task_args)
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-7-27-24/lib/python3.11/site-packages/inspect_ai/_eval/loader.py", line 131, in load_tasks
    return [
           ^
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-7-27-24/lib/python3.11/site-packages/inspect_ai/_eval/loader.py", line 134, in <listcomp>
    for spec in load_task_spec(task_spec, model_name, task_args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-7-27-24/lib/python3.11/site-packages/inspect_ai/_eval/loader.py", line 144, in load_task_spec
    return [task_create(task_spec, model, **task_args)]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-7-27-24/lib/python3.11/site-packages/inspect_ai/_eval/registry.py", line 68, in task_create
    kwargs = deepcopy(kwargs)
             ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle '_thread.RLock' object
```


Yes, task args need to be serialisable (so we can record them in the log for replay during the retry). You can work around this by just referring to models by name rather than passing the instance). Same goes for the dataset.

Here's some working code that adapts the built-in security_guide example:

```
from inspect_ai import Task, eval, eval_retry, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate

@task
def security_guide(grader_model, dataset):
    return Task(
        dataset=json_dataset(dataset),
        plan=[generate()],
        scorer=model_graded_fact(grader_model),
    )

logs = eval(
    security_guide,
    task_args=dict(grader_model="openai/gpt-4o", dataset="security_guide.jsonl"),
)

log = eval_retry(logs)
```

One gotcha, you can't have a task arg named `model` (as that's a special parameter we synthesise for passing the current model to the task).

Working through this example, I think the business of having to explicitly use `task_args` to make this work is quite unfortunate. We should be able to automatically capture these via the decorator so you can write `security_guide("openai/gpt-4o","security_guide.jsonl")` or even `security_guide()` and have it work. We will work on this!


Thank you, the work around might work. I will test it later today/tomorrow

I will also try to create an improved workflow as discussed above (will report back here once that lands)

I think I've got a much better version of this working on a branch here (https://github.com/UKGovernmentBEIS/inspect_ai/pull/180) so I'd suggest you try that out. Basically, so long as the task args are serialsable and the task was created by an `@task` decorated function then its retryable.

This will be merged after our next PyPI release (next couple of days)

Hi,
I tried the solution, but it will not work unless our flow is changed. Today, we pass unserializable class to one of the custom scorers (LLM Model for RAGAS), in addition we also pass grader model, but that can be overcome using your suggestion. The only way to overcome the limitation, is to initialize model inside scorer - however it will be in-efficient. 
I believe the full solution, should allow to pass needed parameters to the "eval_retry" same way parameters can be passed to the "eval" function. Yes, the caller would need to make sure resumed job is running with the same parameters. 

What about if you initialised the model inside the scorer's initialization block (not within the score function, but within the code above where the score function is defined and returned). That would be a one-time operation at task creation.

This is now merged to main. I don't love the idea of re-passing the params to eval_retry as that undermines the integrity of "we retry whatever is in the log file"). I am hopeful you can still do all of your expensive initialisation inside the task function (either at the top level or inside solver/scorer/tool initialisation blocks). 

Here is a canonical example of how to do a sweep over hyperparameters / models with eval_retry: https://gist.github.com/jjallaire-aisi/2384cac4ef6de948e72b51dc299438ab

(note this requires the latest main)