Repository: kubernetes
Issue: liveness prober is more likely to time out when a large number of pods are deployed in one node
Labels: kind/bug,sig/node,needs-triage

### What happened?

After upgrading k8s from 1.19.4 to 1.22.1，we found there are more pods encoutner liveness probe timeout fail in some performance test scenario.
Eventually we are sure that it introduced by the the PR #98376, which modified initial random delay of each prober.

### What did you expect to happen?

Keep result consistency of prober  between versions.

Random delay is necessary when pods  deployed at first time not only kubelet restarted.

### How can we reproduce it (as minimally and precisely as possible)?

Deploy more pod instances belongs to one deployment which use a relatively short liveness probe timeout. 
These pods will do liveness probe within a short period of time, which will bring heavy request pressure to CRI and cause numerous pods' timeout .

### Anything else we need to know?

changed code
```golang
	switch probeType {
@@ -130,7 +134,10 @@ func (w *worker) run() {

	// If kubelet restarted the probes could be started in rapid succession.
	// Let the worker wait for a random portion of tickerPeriod before probing.
-	time.Sleep(time.Duration(rand.Float64() * float64(probeTickerPeriod)))
+	// Do it only if the kubelet has started recently.
+	if probeTickerPeriod > time.Since(w.probeManager.start) {
+		time.Sleep(time.Duration(rand.Float64() * float64(probeTickerPeriod)))
+	}
```

Probe failed log: 
```log
E0709 08:15:02.623150   36694 paas_local_runtime.go:372] "ExecSync from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" co
ntainerID="7ed2a625b8c18b05d75fef89d15844d8411738a13ba2d8030db13e4b9cc58c47" cmds="/bin/bash /opt/myapp/scripts/monitor.sh"
I0709 08:15:02.623207   36694 prober.go:116] "Probe failed" probeType="Liveness" pod="fst-manage/bsu-test-pod1-77208b17-151" podUID=eae0d10d-2c81-4afe-8752-7f7f410c064b con
tainerName="bsu-component1" probeResult=failure output=""
E0709 08:15:03.692490   36694 paas_local_runtime.go:372] "ExecSync from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" co
ntainerID="1ca28e44c27ceb49538318d945059367ea5e9cc4a73cb3fb790592f3ea1ddd71" cmds="/bin/bash /opt/myapp/scripts/monitor.sh"
I0709 08:15:03.692545   36694 prober.go:116] "Probe failed" probeType="Liveness" pod="fst-manage/bsu-test-pod1-77208b17-166" podUID=e991a071-a25e-459b-bf20-9c2e5dc3def3 con
tainerName="bsu-component1" probeResult=failure output=""
I0709 08:15:04.950735   36694 pod_workers.go:840] "Sync pod" pod="fst-manage/bsu-test-pod2-77208b17-21" type=0
I0709 08:15:05.767798   36694 prober.go:130] "Probe succeeded again for container" probeType="Liveness" pod="fst-manage/bsu-test-pod1-77208b17-151" podUID=eae0d10d-2c81-4af
e-8752-7f7f410c064b containerName="bsu-component1"
```


statefulset template:

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: bsu-test-pod1-77208b17
  name: bsu-test-pod1-77208b17
  namespace: xxx
spec:
  podManagementPolicy: Parallel
  replicas: 192
...
    livenessProbe:
      exec:
        command:
        - /bin/bash
        - /opt/myapp/scripts/monitor.sh
      failureThreshold: 6
      initialDelaySeconds: 240
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 2
...

```

### Kubernetes version

<details>

```console
k8s 1.22.1
```

</details>


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

```shell
host-192-168-2-2:~ # docker version
Client:
 Version:           18.09.0
 EulerVersion:      18.09.0.200
 API version:       1.39
 Go version:        go1.15.7
 Git commit:        0aa36e0
 Built:             Wed Apr 13 09:00:34 2022
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.0
  EulerVersion:     18.09.0.200
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.15.7
  Git commit:       0aa36e0
  Built:            Wed Apr 13 09:00:34 2022
  OS/Arch:          linux/arm64
  Experimental:     false
```

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>