Repository: UKGovernmentBEIS/inspect_ai
Issue: in_house_ctf: network conflict with multiple epochs 
Root cause: Architectural Constraint

@craigwalton-dsit If we try to run this eval with multiple epochs:

```
inspect eval evals/gdm_dcap/in_house_ctf -T challenges=cmd_injection --epochs 2
```

We get the following error:

```
RuntimeError: Failed to start docker services  Network inspect-in_house_ctf-ik7d25ztcdiw7btjfugwyuc_cmd_injection_network  Creating                                               │
│  Network inspect-in_house_ctf-ik7d25ztcdiw7btjfugwyuc_cmd_injection_network  Created                                                                                              │
│  Container cmd_injection_env  Creating                                                                                                                                            │
│ Error response from daemon: Conflict. The container name "/cmd_injection_env" is already in use by container                                                                      │
│ "9f391feaf2307a91700ec3f40ce2812a14c6e0eccf0783f13cd95890745e89c0". You have to remove (or rename) that container to be able to reuse that name.  
```

Is this a hard constraint of Docker Compose? (you can't run multiple instances of a service?) or is there some other way the compose file should be crafted to enables this. Here is the [file](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/evals/gdm_dcap/in_house_ctf/data/cmd_injection/compose.yaml):

```yaml
services:
  cmd_injection:
    image: marshw/cmd_injection
    container_name: cmd_injection_env
    networks:
      - cmd_injection_network
    ports:
      - "8080:8080"

  attacker:
    image: marshw/attacker
    command: tail -f /dev/null
    container_name: cmd_injection_attacker
    networks:
      - cmd_injection_network
    platform: linux/amd64
    x-default: true
    depends_on:
      - cmd_injection

networks:
  cmd_injection_network:
    driver: bridge
```

I believe this will also affect attempting to run on multiple models? In any case the original paper ran this over 10 epochs so getting this working will make the eval way better b/c you won't need to run 10 separate evals then stitch the final score together from multiple log files.

_Originally posted by @jjallaire in https://github.com/UKGovernmentBEIS/inspect_ai/issues/397#issuecomment-2350118933_
            

I also see that when I attempt to run all of the tasks in parallel there are some port conflicts that prevent this:

```
RuntimeError: Failed to start docker services  Network inspect-in_house_ctf-i5d5qhvo3lqgrpbsmkikqky_pw_reuse_network  Creating                                   │
│  Network inspect-in_house_ctf-i5d5qhvo3lqgrpbsmkikqky_pw_reuse_network  Created                                                                                  │
│  Container pw_reuse_db_env  Creating                                                                                                                             │
│  Container pw_reuse_env  Creating                                                                                                                                │
│  Container pw_reuse_db_env  Created                                                                                                                              │
│  Container pw_reuse_env  Created                                                                                                                                 │
│  Container pw_reuse_attacker  Creating                                                                                                                           │
│  Container pw_reuse_attacker  Created                                                                                                                            │
│  Container pw_reuse_env  Starting                                                                                                                                │
│  Container pw_reuse_db_env  Starting                                                                                                                             │
│  Container pw_reuse_db_env  Started                                                                                                                              │
│ Error response from daemon: driver failed programming external connectivity on endpoint pw_reuse_env                                                             │
│ (aeba8d08a567bc1bce0580e5fd96578260844ca37050c01e8b155de781e3f34e): Error starting userland proxy: listen tcp4 0.0.0.0:22: bind: address                         │
│ already in use                                                                
```

@craigwalton-dsit Do you know of any way we can avoid this?

## Ports
Generally (in the case of using compose for inspect evals), you should avoid using the `ports:` directive - it is for mapping _container_ ports to _host_ ports. The host should not have any of its ports mapped though - this causes clashes and is unnecessary (and makes the eval less sandboxed). If you'd like to "declare" which ports each service exposes (but does not map to the host) then you can use the `expose:` directive.

## Container names
Another source of clashes you might be seeing is assigning containers hard-coded names. There is no need to do this, and it will cause clashes. Instead, let compose name them for you. Each container will be addressable from within the compose project and within inspect's `sandbox()` function by its service name (or `hostname` if you really want to override it).

E.g.

```yaml
  cmd_injection:
    image: marshw/cmd_injection
    networks:
      - cmd_injection_network
    expose:
      - 8080  # or "8080:tcp" if you want to be precise
```

```py
sandbox("cmd_injection")
```

Let me take a look at this today (excited to read the PR!) and see if I've got any other recommendations.

Side note for future kubernetes support:

We've got a kubernetes sandbox provider in the works for a replacement for docker compose. It will have its own yaml schema for defining infrastructure. It won't support mapping container ports to host ports i.e. `ports:` (where host is the machine running the inspect eval).

To simplify migration, we're toying with an adapter to make compose.yaml files usable by kubernetes. In compose the `expose:` directive doesn't really do anything other than document/advertise which ports a service is likely expecting to be using. But in kubernetes, we'll need to explicitly open the ports we want to use, so we'll piggy-back onto this existing schema support for `expose:`

Another thing to note is that by default, a network with `driver: bridge`  is created for compose projects and all services/containers are given access to it. So you could, if you wanted to, update your file to the following. But if you'd rather be explicit, I have no qualms with that.

```yaml
services:
  cmd_injection:
    image: marshw/cmd_injection
    expose:
      - "8080:tcp"

  attacker:
    image: marshw/attacker
    command: tail -f /dev/null
    platform: linux/amd64
    x-default: true
    depends_on:
      - cmd_injection
```

Addressed with https://github.com/UKGovernmentBEIS/inspect_ai/pull/423