Repository: UKGovernmentBEIS/inspect_ai
Issue: [bug] Wrong Dockerfile used for sandboxes (uses working directory instead of task directory)
Root cause: Algorithmic Error

The docs explain that:

> By default, any Dockerfile and/or compose.yaml file within the task directory will be automatically discovered and used.

https://inspect.ai-safety-institute.org.uk/agents.html#sec-sandbox-environments

However, in my testing it uses the current working directory, not the task directory. 

----

Here is a repo that reproduces the issue: https://github.com/epoch-research/inspect-dockerfile-repro

Steps to reproduce:

Run `main.py` from root
```shell
./run_from_root.sh
```
Result: **it uses the Dockerfile in the root**.


Run `main.py` from the `task` directory:
```shell
./run_from_task.sh
```
Result: it uses the Dockerfile in `task`

----

Sandbox config files are resolved in `inspect_ai/util/_sandbox/docker/config.py`. We can see e.g. that this function just checks the current directory when `parent` is the empty string:
```python
def has_dockerfile(parent: str = "") -> bool:
    return os.path.isfile(os.path.join(parent, "Dockerfile"))
```

Ideally we would pass the task directory information to a `DockerSandboxEnvironment` and thereafter to the `ComposeProject` constructor.

However, how should this directory be found? There is no way to know from a `Task` instance where it was defined. I believe the registry (see [here](https://github.com/UKGovernmentBEIS/inspect_ai/blob/c1be973c5d27aab69baf600d8c2faac40ef3a489/src/inspect_ai/_eval/registry.py#L28-L51)) could be used for that purpose. 

I am aware that the registry name uses the resolved package name:

```python
def registry_name(o: object, name: str) -> str:
    r"""Compute the registry name of an object.

    This function checks whether the passed object is in a package,
    and if it is, prepends the package name as a namespace
    """
    package = get_package_name(o)
    return f"{package}/{name}" if package else name
```

But parsing the registry name (apart from being an inelegant solution) is not suitable because the name can be customized in the decorator call.

Perhaps a new `RegistryInfo` attribute (e.g. `decorator_path`) is needed?

Note there is also the `__task_file__` attribute. But this only gets set in `loader.py`, not when a task is registered.

```python
TASK_FILE_ATTR = "__task_file__"
```

```python
def create_file_tasks(
    file: Path,
    model: ModelName,
    task_specs: list[str] | list[RegistryInfo] | None = None,
    task_args: dict[str, Any] = {},
) -> list[Task]:
        # ....
        for task_spec in task_specs:
            # create the task from the loaded source file and
            # note that it was loaded from this directory
            # (will be used later to ensure it runs in the directory)
            task = task_create(task_spec, model, **task_args)
            setattr(task, TASK_FILE_ATTR, file.as_posix())
            setattr(task, TASK_RUN_DIR_ATTR, run_dir)
            tasks.append(task)

        # .....

        return tasks
```
[source](https://github.com/UKGovernmentBEIS/inspect_ai/blob/c1be973c5d27aab69baf600d8c2faac40ef3a489/src/inspect_ai/_eval/loader.py#L278-L312)


Thank you for reporting this and providing such a nice repro! Will address this ASAP.

I dug into this and discovered the source of the problem. The binding to Dockerfile inside of the task directory occurs when the task is created by reference to a file, e.g. `inspect eval task/humaneval.py` or `eval("task/humaneval.py")` then the loader is in play and the task run dir is recorded and used.

On the other hand, when a task is imported from a module, the task loader isn't in play (as the task is presented to `eval()` fully loaded). This occurs commonly for tasks within Python packages, and in this case the code ensure that is provides a fully qualified path to the compose file, e.g. https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_capabilities/intercode_ctf/task.py.

So in your example, the import of the task from the module triggers the same codepath as importing from a Python package -- so there is no task run dir set, and the Dockerfile is read from the root.

Note that in the case of only a Dockerfile and no compose file we write a temporary `.compose.yaml` file alongside the `Dockerfile`. In the case of the `Dockerfile` being inside a Python package clearly this would be undesirable though. I think the answer is to to create a `compose.yaml` file and provide a full path to it via `__file__`. Then your code would always work whether invoked via filesystem reference, module, or Python package. This is a "best practice" that we need to improve the documentation on.

Note that its possible that switching to the task file directory for loading/running was a _mistake in the first place_ (b/c it can be surprising and encourages code that doesn't play well in a module/package). It was intended to simplify the code for referring to resources like datasets, prompt templates, docker config, etc. We now face a bit of a choice as if we yank that behaviour out from under people we will break a lot of tasks in the wild (and we are very committed to not breaking people on updates except in vary narrow cases). We are in general pretty unhappy about this state of affairs but the commitment to not breaking people ties our hands somewhat.

I think the best resolution here might to change the documentation to emphasize that full paths to `config` should be used rather than relying on implicit directory switching (which won't be in play for modules and packages).

Another possible resolution would be to see if we can distinguish between your case (import of a module from a directory) and the case of importing from a Python package. If we can do that then I think your code could work as-is. I will investigate that now. 

Certainly also interested in your take on whether there are other paths we haven't considered.


Okay, I figured out how we could correctly handle the local module imports. PR is here (and your test case now works as expected w/ this change): https://github.com/UKGovernmentBEIS/inspect_ai/pull/715


Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/715/