Repository: UKGovernmentBEIS/inspect_ai
Issue: CLI `eval-retry` raises asyncio RuntimeError
Root cause: Documentation Deficiency

# Problem Description
I'm trying to refactor my experiments code so that `inspect eval-retry` works. I've looked at #190, #173, and I'm not sure what about my evals code is causing the error on a retry attempt. All task args are serializable and I'm not using the `name` arg.

# Context
The test log file in the example below was produced by running `settings/apps/apps_setting.py` and terminating with Ctrl+C in the middle of the eval. Here is the script calling `eval`:
```python
tasks = [atomic_trusted_monitoring]
model: ModelAPI = TRUSTED_MODEL
num_epochs = 1
run_type: InspectRunType = InspectRunType.DEBUG

result = eval(
        tasks=tasks,
        task_args=dict(to_attack_func=ToAttackFuncs.NeverAttack.__name__, use_exploit_finder=False, use_attack_implementer=False),
        model=model.get_model_name(),
        log_dir=str(run_type.log_dir),
        max_connections=15,
        max_samples=15,
        max_tasks=5,
        fail_on_error=.02,
        epochs=num_epochs,
    )
```
And here is the task:
```python
@task
def atomic_trusted_monitoring(
    to_attack_func: str, 
    monitor_model: str = TRUSTED_MODEL.name, 
    use_exploit_finder: bool = True,
    use_attack_implementer: bool = True,
):
    monitor_model = ModelAPI[monitor_model]
    to_attack_func = ToAttackFuncs.NAMES_TO_CLASSES[to_attack_func]()
    logger.info("Running task: Atomic Trusted Monitoring")
    store_type = StoreAtomicTrustedMonitoring
    
    use_attack_implementer = use_attack_implementer or to_attack_func != ToAttackFuncs.NeverAttack()
    attack_fork_solvers: dict[str, list[Solver]] = {
        "attack": [],
        "no_attack": [honest_implementer(UNTRUSTED_MODEL, store_type)],
    }
    blue_team_protocol = inspect.currentframe().f_code.co_name
    return Task(
        dataset=dataset,
        solver=[
            *([u_attack_implementer(UNTRUSTED_MODEL, store_type),
            red_attack_opportunity_evaluation(UNTRUSTED_MODEL, store_type)] if use_attack_implementer else []),
            attack_decision(store_type, attack_fork_solvers, to_attack_func, EPISODE_LENGTH, blue_team_protocol),
            trusted_monitor_implementation(monitor_model, store_type),
            *([exploit_finder_atomic_nonreasoning_models(UNTRUSTED_MODEL, store_type)] if use_exploit_finder else []),
            passes_unit_tests_solver(store_type),
            is_valid_exploit_solver(store_type),
        ],
        scorer=[
            passes_unit_tests(store_type),
            attack_attempted(store_type),
            secret_input_is_valid_exploit(store_type),
            is_successful_backdoor(store_type),
            atomic_suspiciousness(store_type),
            atomic_solution_difficulty(store_type),
            atomic_attack_difficulty(store_type),
            code_syntax_correct(store_type),
            problem_source(),
        ],
    )
```

## Repro Example
`git clone https://github.com/aaron-sandoval/inspect-control-settings.git`
`git checkout 5ab3058d7687afb05d42d606897196ac1eda8a4f`
`uv run inspect eval-retry experiments/logs/debug/2025-05-07_232810/2025-05-07T17-28-11-06-00_atomic-trusted-monitoring_kMBoPzm6qBjpwuvhqsgY6g.eval`
Produces: `RuntimeError: asyncio.run() cannot be called from a running event loop`

## Stack
Python 3.13.1
inspect-ai 0.3.94

Thanks for the repro! The problem here is that when we go to load the task from `apps_setting.py` there is a call to `eval()` at the top level of the file -- so we end up calling `eval()` within another call to `eval()`. You can fix this by protecting that code with a `__main__` check as follows:

```python
if __name__ == "__main__":
    model: ModelAPI = TRUSTED_MODEL
    num_epochs = 1
    run_type: InspectRunType = InspectRunType.DEBUG

    result = eval(
        tasks=tasks,
        task_args=dict(
            to_attack_func=ToAttackFuncs.NeverAttack.__name__,
            use_exploit_finder=False,
            use_attack_implementer=False,
        ),
        model=model.get_model_name(),
        log_dir=str(run_type.log_dir),
        max_connections=15,
        max_samples=15,
        max_tasks=5,
        fail_on_error=0.02,
        epochs=num_epochs,
    )
```

Thank you! This partially resolved my issue. I can now retry tasks run from the same module in which the tasks were defined. 

However, when I import tasks into a separate script apps_baseline.py and call `eval` there, then `inspect eval-retry` seems unable to find the tasks when attempting to rebuild them. It seems to be searching for the wrong task name `'settings/t_u_honest_decomp'` when the task is called `'settings/apps/apps_setting.py@t_u_honest_decomp'` when calling `inspect list tasks`. Is there some other step that is needed to enable finding tasks defined in separate modules?

# Context
[2025-05-08_154935](https://github.com/aaron-sandoval/inspect-control-settings/tree/54a56ebb43cdd8b173ac31c684633731fa46b642/experiments/logs/debug/2025-05-08_154935) logs were created from running apps_setting.py. The latest log in the directory was from a successful `inspect eval-retry`.

[2025-05-08_163814](https://github.com/aaron-sandoval/inspect-control-settings/tree/54a56ebb43cdd8b173ac31c684633731fa46b642/experiments/logs/debug/2025-05-08_163814) logs were created from running apps_baseline.py with nearly identical parameters.

## Repro Example
`git checkout 54a56ebb43cdd8b173ac31c684633731fa46b642`

`uv run inspect eval-retry experiments/logs/debug/2025-05-08_154935/2025-05-08T09-49-35-06-00_t-u-honest-decomp_ntbK22jtLCUtzoXfcXagDW.eval`
Successfully resumes and completes the eval, reusing the completed samples

`uv run inspect eval-retry experiments/logs/debug/2025-05-08_163814/2025-05-08T10-38-14-06-00_t-u-honest-decomp_SfqCPhFEvBspYoQsx6UgF6.eval`
Raises `FileNotFoundError: Task 'settings/t_u_honest_decomp' not found.`

Okay, it may be that your task is getting detected as living in a package which is causing the lookup to be package-rooted not directory rooted. Will take a look at those logs early next week.

I have a feeling that the uv project is causing us to mis-detect this as being in a package. It would be incredibly helpful if you could create a minimal version of your repro that start with an `eval` (which I could then interrupt to induce a retry) and then the failing `eval-retry` (just trying to take the guesswork out of my reproing your exact scenario).

Yes, the task is created in the package named settings, so I'm not sure if that causes issues. Here is a minimal repro example.

## Steps
```
git checkout 4043a974f20d025f31c1d5145b9d84d8c8a81ccb
```
Create .env file in the repo root directory.
Add `OPENAI_API_KEY=sk-1234...` in the .env file.
```
uv run python experiments/debug_eval_retry.py
```
Interrupt the eval.
Get the name of the partial log file.
```
uv run inspect eval-retry experiments/logs/debug/eval_retry_1/<NAME OF PARTIAL LOG FILE>
```

## Result
`FileNotFoundError: Task 'settings/u_honest_implementation' not found.`

Let me know if I can clarify anything else.

The issue here is that tasks in package actually are handled specially, and need to be registered in order to be retried. I've updated the error message to explain this and point to this documentation on registering tasks in packages: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1864

I will also note that in your case you were calling `eval_set()`, which also does retries and which does not have this limitation (this is because eval set reads the tasks afresh from disk whereas eval-retry is doing a cold start from a log file so your Python code is not yet loaded and has to be explicitly located).