Repository: UKGovernmentBEIS/inspect_ai
Issue: `eval_retry` fails to find task in registery
Root cause: Validation Gap

Hi,
When running `eval_retry` it fails to find task in the registry. Running with version 0.3.20.
Is the problem fixed with https://github.com/UKGovernmentBEIS/inspect_ai/pull/180 ? If yes, how can I install from that branch? [Update] I just rerun the flows with the latest install from the git: `pip install git+https://github.com/UKGovernmentBEIS/inspect_ai`, and the problem is still there.

Example code:

```
@task
def eval_model_graded_fact(inspect_model_grader_vendor,
                                     inspect_eval_model_grader,
                                     eval_cache,
                                     eval_name,
                                     dataset) -> Task:
    # Set Task Name if not provided vi CLI args
    if eval_name is None:
        eval_name = "eval_model_graded_fact"
    
    logging.debug(f"Using End-point eval_cache: {eval_cache}")
    settings = ServerSettings()
    # Get LLM and Embed grader models
    grader_model = get_inspect_model(vendor=inspect_model_grader_vendor,
                                     model=inspect_eval_model_grader,
                                     settings=settings,
                                    )
    return Task(
        dataset=dataset,
        plan=[generate()],
        scorer=model_graded_fact(model=grader_model),
        name = eval_name, 
    )

if 1==2:
        eval_log = eval(
                        tasks=eval_model_graded_fact,
                        task_args=task_settings,
                        model=model,
                        log_dir=output_folder,
                        limit=samples_limit,
                        max_connections=max_connections,
                        log_level=settings.log_level,
                    )
    else:
        eval_log = eval_retry(
                        tasks=["logs/2024-08-04T22-24-51-05-00_eval-model-graded-fact_SknXMbjqWTiHve757hqVfP.json"],
                        log_dir=output_folder,
                        max_connections=max_connections,
                        log_level=settings.log_level,
                    )  
```

Error:
```
File "/usr/local/stage3technical/var/sand/stage3technical/tcom-middle-tier/eval_tool/helpers/eval_tool_utils.py", line 153, in run_eval_task
    eval_log = eval_retry(
               ^^^^^^^^^^^
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-8-4-24/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 376, in eval_retry
    return asyncio.run(
           ^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/stage3technical/var/virtualenv/tcom-middle-tier-8-4-24/lib/python3.11/site-packages/inspect_ai/_eval/eval.py", line 481, in eval_retry_async
    raise FileNotFoundError(f"Task '{task_name}' not found.")
FileNotFoundError: Task 'eval_model_graded_fact' not found.
```

When running `inspect list tasks` the task is listed as following:
```
eval_tool/helpers/eval_tool_utils.py@eval_model_graded_fact
```
Even if I modify log json with the task name above, it still cannot find it.
We run evaluation as a job, and "eval_tool" folder is sub-folder of the project.

It might be explicitly setting the task_name (this will happen automatically when you run the task).

Here is a minimal version of retry that uses the same scheme as above that I've confirmed works:

```python
from inspect_ai import Task, eval, eval_retry, task
from inspect_ai.dataset import Sample
from inspect_ai.model import GenerateConfig
from inspect_ai.scorer import match
from inspect_ai.solver import generate


@task
def my_task(temperature):
    return Task(
        dataset=[Sample(input="Say hello", target="hello")],
        plan=[generate()],
        scorer=match(),
        config=GenerateConfig(temperature=temperature),
    )


log = eval(my_task, task_args=dict(temperature=0.5))[0]

eval_retry(log)
```

You might want to start from this and after confirming it works for you start adding more complexity.

Hi @jjallaire , I was able to reproduce the problem. The problem surface when task has `name` argument. For example:
```
@task
def my_task_2(temperature):
    return Task(
        dataset=[Sample(input="Say hello", target="hello")],
        plan=[generate()],
        scorer=match(),
        config=GenerateConfig(temperature=temperature),
        name="my_task_2",
    )
```
In this case, the task is always recorded in the json as `task='my_task_2'` instead of `task='package-location/my_task_2'`.

Folder structure for the project:
```
test-inspect-retry/
  eval_tool/
    helpers/
      eval_tool_unils_v2.py
      registry.py
    tasks/
      eval_tool_tasks_v2.py
  job_manager/
    __main__.py
    eval_runner.py
  logs/
    2024-08-06T10-50-26-05-00_my-task-2_adYTLGcf5FnSz8pntqiSbC.json
```
I'm running the `eval` as following: `python -m eval_tool`. It works as long as task `name` argument is not present. The work-around is to include package name in the `name` argument input.


Okay, that `name` argument is actually intended for situations where you are creating a dynamic task. In the case of creating a task within a package (or really any task with `@task`) the name is automatically assigned (and given the correct package scope). This is required because there is no way "find" the task for retry if it's name isn't discernable from the function declaration.

TLDR is that `name` shouldn't be used with `@task` (you can however do `@task(name="my_task_name")`). I can try to make this automatic (and print a warning message). 


Okay, we now warn for this case and provide a clear instruction for how to customise the task name: https://github.com/UKGovernmentBEIS/inspect_ai/pull/202