Repository: UKGovernmentBEIS/inspect_ai
Issue: Specifying a custom Dockerfile not named "Dockerfile" is broken
Root cause: Configuration Error

[This commit](https://github.com/UKGovernmentBEIS/inspect_ai/commit/8c7ca3eb75084b709dc6927a2c218430e9f77935) attempts to fix an issue where `is_dockerfile` does not recognize Dockerfiles not named `"Dockerfile"`. However, specifying such a named Dockerfile via `sandbox=("docker", {name}.Dockerfile)` still fails because Inspect still looks for a file named `Dockerfile` instead of the specified file. I believe this is because [the `.compose.yaml` file which is generated](https://github.com/UKGovernmentBEIS/inspect_ai/blob/a2f3dfba3809332abe98a09e988672dfd44df809/src/inspect_ai/util/_sandbox/docker/config.py#L90) does nothing to specify a non-default Dockerfile in the `build` attributes.

This makes it so that specifying sample-specific Dockerfiles (which are not named `"Dockerfile"`) fails in a confusing way where Inspect will actually (attempt to) build an image from the default `Dockerfile` rather than the specified file.

A potential word-around for any users experiencing this error is to just specify a compose configuration that explicitly lists the named Dockerfile in its `build.dockerfile` attribute.

---

As an additional note, [this commit](https://github.com/UKGovernmentBEIS/inspect_ai/commit/8c7ca3eb75084b709dc6927a2c218430e9f77935) fails to identify files named `Dockerfile.{name}` as Dockerfiles, which I understand is also an accepted convention. Should support for this be added to `is_dockerfile`?

Perhaps the if-else structure in [the `create()` function of `ComposeProject`](https://github.com/UKGovernmentBEIS/inspect_ai/blob/bc228855f82ff9b6d6c3e6442e4d0cffd9544eb8/src/inspect_ai/util/_sandbox/docker/util.py#L41) should also check for whether something is properly a Docker compose file so that an error is more explicitly surfaced if someone supplies a path to an invalid file type for the configuration.

Thanks for the report and investigation. 

Looks like https://github.com/UKGovernmentBEIS/inspect_ai/pull/1232 should not have been merged. Does that match your understanding, @jjallaire? Do you want to revert it? Or fix the `.compose.yaml` generation?

For my use case, I realized that storing the Dockerfiles as `{name}/Dockerfile` instead of `{name}.Dockerfile` or `Dockerfile.{name}` works just as well. This also might be more optimal from the perspective of `docker build` based on things like [this](https://stackoverflow.com/questions/26077543/how-to-name-dockerfiles#comment87101040_26084010).

The comment you linked says: 

> Don't have multiple Dockerfiles in the same directory. The containing directory becomes the [context](https://docs.docker.com/engine/reference/builder/#usage) for the docker build command, which means that each image will be needlessly rebuilt if another image's dockerfile is changed

This comment seems confused on several accounts:

* A context must be explicitly passed to the Docker build command https://docs.docker.com/build/concepts/context/
* A `--file` argument may be passed to specify the Dockerfile, which may be stored anywhere inside or outside the build context. If unspecified, a file named `Dockerfile` at the root of the build context is used by default.
* A layer (not entire image) is rebuilt if the files added through `ADD` or `COPY` have changed. If a layer is rebuilt, all layers that depend on it are rebuilt. https://docs.docker.com/get-started/docker-concepts/building-images/using-the-build-cache/. Changes to one Dockerfile do not trigger rebuilds of images defined by other Dockerfiles. If you make a change to `A.Dockerfile`, this will not cause rebuilding of any layers of the image defined by `B.Dockerfile` (unless you're doing `ADD B.Dockerfile` inside `A.Dockerfile`, which would be a very odd thing to do). 

In my experience with Docker in general, it is a common pattern to:
* have multiple Dockerfiles with different names in the same directory, for different purposes
* have Dockerfiles stored somewhere other than the root of the build context

When using Inspect, so far I also haven't personally had a use case that strictly requires a named Dockerfile, but I think it's useful and should be supported eventually. 

> As an additional note, [this commit](https://github.com/UKGovernmentBEIS/inspect_ai/commit/8c7ca3eb75084b709dc6927a2c218430e9f77935) fails to identify files named Dockerfile.{name} as Dockerfiles, which I understand is also an accepted convention.

I don't think I've ever seen this convention. It would go against the general convention for file name extensions: that the extension should tell us the type of file. My IDE does not pick up `Dockerfile.name` as a Dockerfile. 

@jjallaire after thinking about this some more, I think we should fix the `.compose.yaml` generation.

Okay, I just merged the `Dockerfile.name` change (as this convention is referenced in the Docker CLI docs). Could you suggest the appropriate change to `.compose.yaml` generation? (not asking for a PR, just a description of the logic to ensure that I have followed the discussion correctly)

Sure: A change to the `.compose.yaml` file which is generated, adding (if applicable) a `dockerfile` value (nested under the `build` key).

https://docs.docker.com/reference/compose-file/build/

Also you should check that it actually builds correctly once you make the change, since I clearly didn't do a good job of that in https://github.com/UKGovernmentBEIS/inspect_ai/pull/1232 🙈. Sorry for submitting a broken PR. We might be missing something else that is needed.

> fails in a confusing way where Inspect will actually (attempt to) build an image from the default Dockerfile rather than the specified file.

I agree the fallback behaviour is confusing and should ideally be improved if we can. (It has actually bitten me before: https://github.com/UKGovernmentBEIS/inspect_ai/issues/691)

> Perhaps the if-else structure in [the create() function of ComposeProject](https://github.com/UKGovernmentBEIS/inspect_ai/blob/bc228855f82ff9b6d6c3e6442e4d0cffd9544eb8/src/inspect_ai/util/_sandbox/docker/util.py#L41) should also check for whether something is properly a Docker compose file so that an error is more explicitly surfaced if someone supplies a path to an invalid file type for the configuration.

I think to do this, we would need the user to tell us whether they are supplying a Dockerfile or a compose file, which is currently not supported. Doing so would require changing the supported values in the `type`:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/45837a9806ddf0ddad02d962f2b9d5e4f0bc1051/src/inspect_ai/util/_sandbox/environment.py#L283-L287

I think that would be nice, but unsure if worth the additional complexity. Since we'd have to support three values: `docker` (for backward compatibility), `dockerfile` and `docker_compose`. @jjallaire can make a call I guess. (If we could drop `docker`, then this change would _reduce_ complexity instead, because we could do without all the parsing like `is_dockerfile`, `is_compose_file`.)

I think will be pretty easy to determine if its a Dockerfile or compose file by using name heuristics (the existing `is_dockerfile`, presence of `.yml` or `.yaml` in the file extension, etc.)

Regarding [this comment](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1321#issuecomment-2660839024) from @tadamcz:

The default context specified via Inspect is simply `"."` and the `.compose.yaml` file is generated in the same directory as the Dockerfile if I'm not mistaken, so I think the first two bullets are not mitigating for the case of Inspect. For the third bullet, maybe the statement in the comment is just false, which seems to be what you are saying. I am not sure myself, but the 10+ upvotes combined with nobody in the thread contesting it left me thinking there was something about how the Docker build process handles contexts that I didn't understand. I am glad if you are correct though because what you say in your third bullet seems logical.

Regarding [this comment](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1321#issuecomment-2660842537) from @tadamcz:

I also have reservations about the `Dockerfile.{name}` convention. For what it's worth, my IDE (VS Code) does recognize it.

Regarding [this comment](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1321#issuecomment-2660924696) from @jjallaire:

Is it satisfactory to just check `is_yaml_file` and then throw an error if neither that nor `is_dockerfile` is true? As an alternative, one could perhaps check something closer to `likely_is_compose_file` by furthermore reading the YAML file and checking if `services` shows up as a top-level element (as I understand it, this is the one [required](https://docs.docker.com/reference/compose-file/services/) part of a compose file). (I am guessing that checking whether the path stem contains conventional keywords like "compose", as is done with the Dockerfile, might constitute a breaking change if people have been naming their compose files in other ways.)

Regarding [this comment](https://github.com/UKGovernmentBEIS/inspect_ai/issues/1321#issuecomment-2660847671) and whether the generation of `.compose.yaml` should be modified to address this issue:

It occurred to me that only accepting a Docker compose file in `sandbox=` and thus requiring users to specify a custom-named Dockerfile *via* an explicit Docker compose file might be all-things-considered the simplest way to go. The docs could be clarified to say that Docker compose files are required for use of custom-named Dockerfiles. (My initial attempt at specifying a Dockerfile via `sandbox=` which led me to this issue occurred because the wording of the docs led me to think that was supported, though the docs didn't explicitly say that. If they had just explained that I should do this by specifying a Docker compose file, that would have been mostly[^1] fine.) Then this discussion about Dockerfile conventions and such can just be deferred to the user, the question of detecting Dockerfile vs Docker compose is avoided, and there are less avenues for confusing behavior like the underlying bug in this issue to show up. It's also already the case that compose files are somewhat "privileged" since you can do things like sample metadata interpolation in compose files (e.g. `${SAMPLE_METADATA_KEY}`) but not in Dockerfiles. (This was also a point of confusion for me, although... maybe I'm mistaken on this?) Curious what you think here. @tadamcz @jjallaire

[^1]: One issue with this for my original use case is that I wanted to specify sample-specific Dockerfiles by generating Dockerfiles from a template, but without going out of my way I had no "default" Dockerfile that was not sample-specific besides the (non-buildable) template. Simply writing `dockerfile: ${SAMPLE_METADATA_DOCKERFILE_PATH}` did not work because [Inspect seems to require a default for all such interpolations](https://inspect.ai-safety-institute.org.uk/sandboxing.html#sample-metadata) that can be run outside the context of a sample. Specifying Dockerfiles directly via `sandbox=("docker", "docker/{id}/Dockerfile")` or the like seems to avoid this extra step of needing to provide a default file. I imagine this could be resolved some other way though.

This is the minimal change that makes the feature work as intended (specify any Dockerfile as the sandbox config): https://github.com/UKGovernmentBEIS/inspect_ai/pull/1332

I am open to discussing further change/enhancements/etc. based on people's patterns of use, but I think it's best to make a straightforward fix to put the defect behind us.

Resolved here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/1332

Suppose that I have specified the Docker configuration via `sandbox=("docker", dockerfile_path)` and now I want to experiment with allowing internet access for the sandbox. Is the only way to do that by creating a `docker-compose.yaml` file, modifying the `sandbox=` statement to specify a `docker-compose.yaml` file instead, and then having the `docker-compose.yaml` file specify the sample-specific Dockerfile?

The other way you can do it is make that driven by an environment variable and use interpolation in the compose file to pluck the value out of the environment (w/ an appropriate default): https://docs.docker.com/reference/compose-file/interpolation/

I'm not sure if you mean something different from what I had in mind. The compose file I had in mind is like this:

```yaml
services:
  default:
    build:
      context: "."
      dockerfile: ${SAMPLE_METADATA_DOCKERFILE_PATH}
    command: "tail -f /dev/null"
    init: true
    network_mode: ${SAMPLE_METADATA_NETWORK_MODE-none}
    stop_grace_period: 1s
```

(I could use an environment variable that isn't taken from sample metadata for the network mode, but I figure I may as well use the sample metadata in part since I could plausibly want to have it differ between samples.)

Is what you are suggesting different from the above?

If not, what is the best practice to address the requirement that the Dockerfile interpolation needs a default when I only have sample-specific Dockerfiles? (The compose file above breaks as currently written, seemingly because it gets used before any sample is instantiated.) Is there a way to avoid needing to specify a default? If not, should I just put an irrelevant dummy Dockerfile? (I'm not sure what the function of this "pre-sample" default Dockerfile is, hence my confusion on this point.)

When I specified the custom Dockerfile directly via `sandbox=("docker", dockerfile_path)` as in the original post of the issue (instead of specifying a compose file), I avoided having to address this issue of needing to specify a default Dockerfile. But now that I also need to modify the network mode, it seems I can't use the specify-the-Dockerfile-directly approach and need to write a compose file like the above, which raises this question about specifying a Dockerfile default.

This is not handled well right now because the reading of the compose file during task startup doesn't have access to the sample metadata. I think you could just specify a default Dockerfile that is essentially unused by samples, but note you'd need to make sure the sample Dockerfiles are built before running (as they won't get built dynamically during task startup). The solution to this might just be to have a compose file per sample that points at the correct Dockerfile