Repository: kubernetes
Issue: Liveness probe with an error doesn't trigger container restart
Labels: kind/bug,sig/node,triage/needs-information,triage/accepted

### What happened?

We have a specific case when a health check would error out with `500 INTERNAL SERVER ERROR`. In this particular scenario, the readiness probe will correctly mark pod unready but the liveness probe does not trigger a container restart.

Here are some more details:
1. Kubernetes Server Version: v1.24.16.
2. An application pod is running a single container.
3. Both readiness and liveness probes are `httpGet` and are hitting the same `/healthcheck` endpoint.
4. Healthcheck endpoint is responding with `500 INTERNAL SERVER ERROR`.
5. The readiness probe does mark the container as "unready" as expected.
6. The liveness probe does not trigger a container restart, even after the multiple consecutive failures.

I confirmed that `restartPolicy: Always` and when I exec into the container and run curl, I do get the 500 INTERNAL SERVER ERROR, the same as what I see in the application logs. As per documentation, a failing liveness probe should be triggering a restart but it is not happening.

Here's the log from kubelet:
```
Jan 03 17:34:20 ip-1-1-1-1 kubelet[5845]: E0103 17:34:20.352585    5845 prober.go:118] "Probe errored" err="unexpected EOF" probeType="Liveness" pod="mynamespace/application-abc
```

Also, looking at the code [here](https://github.com/kubernetes/kubernetes/blob/release-1.24/pkg/kubelet/prober/prober.go#L115-L124) the error does seem to be captured correctly by the probe but restart is not initiated.

Somehow the error is not being handled gracefully I assume. If I change it to an invalid port and cause a timeout the container gets restarted as expected. Also, switching from `httpGet` to `exec` type of healthcheck and running `curl http://localhost:8000/healthcheck` does trigger a restart so I suspect there is some bug with the httpGet probe itself.

This is what I see in the events:
```
Liveness probe errored: unexpected EOF
Readiness probe errored: unexpected EOF
```

### What did you expect to happen?

A failing healthcheck in a liveness probe would trigger a container restart.

```
containerMessage="Container abc  failed liveness probe, will be restarted"
```

### How can we reproduce it (as minimally and precisely as possible)?

Create a deployment with an app that is crashing and returns 500

Configure healthceck like so:
```
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthcheck
        port: healthcheck
        scheme: HTTP
      initialDelaySeconds: 3
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 3
    name: abc
    ports:
    - containerPort: 8000
      name: healthcheck
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /healthcheck
        port: healthcheck
        scheme: HTTP
      initialDelaySeconds: 3
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 3
```

### Anything else we need to know?

_No response_

### Kubernetes version

<details>

```console
$ kubectl version
Server Version: v1.24.16
```

</details>


### Cloud provider

<details>
AWS
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
# Linux ip-1-1-1-1 5.15.0-1028-aws #32~20.04.1-Ubuntu SMP Mon Jan 9 18:02:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>