Repository: kubernetes
Issue: `volumes[*].hostPath.type: Socket` doesn’t prevent the kubelet from creating a directory instead of waiting for a UNIX socket to be created.
Labels: kind/bug,sig/storage,needs-triage

### What happened?

[`hostPath` volumes have different types](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath-volume-types).
By default, no check is done. Whatever is found is bind-mounted and, if nothing exists, a directory is created on the host.
But, if the type is `Socket`, the kubelet is supposed to check that a UNIX socket exists at the expected location before starting the pod.

Yet, if the container restarts after the UNIX socket has disappeared, then a directory is created instead.
This leads to an unrecoverable issue because the directory will prevent the server from eventually re-creating the UNIX socket.

### What did you expect to happen?

If a container mounting a `Socket` `hostPath` restarts after the socket has been deleted, it should hang until the socket comes back.

### How can we reproduce it (as minimally and precisely as possible)?

Create a pod that mounts a socket:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo
  labels:
    app: foo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: foo
  template:
    metadata:
      labels:
        app: foo
    spec:
      containers:
        - name: sleeper
          image: busybox
          command:
            - sleep
            - infinity
          volumeMounts:
            - name: socket
              mountPath: /tmp/foo.sock
      volumes:
        - name: socket
          hostPath:
            path: /tmp/foo.sock
            type: Socket
```

The pod initially remains pending because there’s no UNIX socket at `/tmp/foo.sock`:

```console
$ kubectl get pods
NAME                  READY   STATUS              RESTARTS   AGE
foo-976c59756-gw2zd   0/1     ContainerCreating   0          97s
$ kubectl describe pod/foo-976c59756-gw2zd 
Name:             foo-976c59756-gw2zd
Namespace:        foo
Priority:         0
Service Account:  default
Node:             gke-gke-lenaic-ubuntu-containerd-38134859-476h/10.0.96.131
Start Time:       Fri, 15 Nov 2024 16:33:09 +0100
Labels:           app=foo
                  pod-template-hash=976c59756
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/foo-976c59756
Containers:
  sleeper:
    Container ID:  
    Image:         busybox
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      infinity
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tmp/foo.sock from socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-52bxl (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  socket:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/foo.sock
    HostPathType:  Socket
  kube-api-access-52bxl:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              cloud.google.com/gke-nodepool=ubuntu-containerd
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    2m20s                default-scheduler  Successfully assigned foo/foo-976c59756-gw2zd to gke-gke-lenaic-ubuntu-containerd-38134859-476h
  Warning  FailedMount  12s (x9 over 2m20s)  kubelet            MountVolume.SetUp failed for volume "socket" : hostPath type check failed: /tmp/foo.sock is not a socket file
```

This is expected.

Then, create the UNIX socket on the node:

```console
# timeout 1 nc -lU /tmp/foo.sock
# ls -l /tmp/foo.sock
srwxr-xr-x 1 root root 0 Nov 15 15:38 /tmp/foo.sock
```

The pod transitions to `Running` state and the containers are started:

```console
$ kubectl describe pod/foo-976c59756-gw2zd 
Name:             foo-976c59756-gw2zd
Namespace:        foo
Priority:         0
Service Account:  default
Node:             gke-gke-lenaic-ubuntu-containerd-38134859-476h/10.0.96.131
Start Time:       Fri, 15 Nov 2024 16:33:09 +0100
Labels:           app=foo
                  pod-template-hash=976c59756
Annotations:      <none>
Status:           Running
IP:               10.4.1.6
IPs:
  IP:           10.4.1.6
Controlled By:  ReplicaSet/foo-976c59756
Containers:
  sleeper:
    Container ID:  containerd://3a6b9dab9a721eba41c8bbb8d5783ec952e1c37e8bafef634d0ece8fa2b8b2a1
    Image:         busybox
    Image ID:      docker.io/library/busybox@sha256:768e5c6f5cb6db0794eec98dc7a967f40631746c32232b78a3105fb946f3ab83
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      infinity
    State:          Running
      Started:      Fri, 15 Nov 2024 16:39:23 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /tmp/foo.sock from socket (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-52bxl (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  socket:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/foo.sock
    HostPathType:  Socket
  kube-api-access-52bxl:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              cloud.google.com/gke-nodepool=ubuntu-containerd
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                     From               Message
  ----     ------       ----                    ----               -------
  Normal   Scheduled    6m46s                   default-scheduler  Successfully assigned foo/foo-976c59756-gw2zd to gke-gke-lenaic-ubuntu-containerd-38134859-476h
  Warning  FailedMount  2m36s (x10 over 6m46s)  kubelet            MountVolume.SetUp failed for volume "socket" : hostPath type check failed: /tmp/foo.sock is not a socket file
  Normal   Pulling      33s                     kubelet            Pulling image "busybox"
  Normal   Pulled       32s                     kubelet            Successfully pulled image "busybox" in 1.273s (1.273s including waiting). Image size: 2166802 bytes.
  Normal   Created      32s                     kubelet            Created container sleeper
  Normal   Started      32s                     kubelet            Started container sleeper
```

This is still fine.

Then, delete the UNIX socket from the node:

```console
# rm /tmp/foo.sock
# ls -l /tmp/foo.sock
ls: cannot access '/tmp/foo.sock': No such file or directory
```

And crash the container that mounts the socket to force its restart:

```console
# pkill -KILL -f "sleep infinity"
```

Then, once the container has restarted, `/tmp/foo.sock` becomes a directory:

```console
# ls -l /tmp/foo.sock
total 0
# ls -ld /tmp/foo.sock
drwxr-xr-x 2 root root 4096 Nov 15 15:46 /tmp/foo.sock
```

### Anything else we need to know?

In real life, this bug affects situations where we have two pods communicating through a UNIX socket shared via a `hostPath` volume:
* A “server” pod has a `hostPath` volume for a directory on the host and creates a UNIX socket inside this directory.
* A “client” pod has a `hostPath` volume for the UNIX socket.

If the container of the “client” pod restarts while the “server” pod is cleanly redeployed (and the “server” process deletes the UNIX socket when shutdown), then a directory is created and it will prevent the “server” pod from re-creating it forever.

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: v1.31.2
Kustomize Version: v5.4.2
Server Version: v1.31.1-gke.1678000
```

</details>


### Cloud provider

<details>
GKE
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux gke-gke-lenaic-ubuntu-containerd-38134859-476h 5.15.0-1067-gke #73-Ubuntu SMP Sat Aug 31 04:29:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>