Repository: kubernetes
Issue: Missing log entries during massive json based logs (logrotate kubelet stdout)
Labels: kind/bug,sig/node,triage/needs-information,needs-triage

### What happened?

We've been running Load Tests for quite a while using normal text based logs to stdout, when our team has rewritten the log framework to use a lot of details and started using json, we began to receive missing logs, about 50 out of every 50 million log entries, also each log entry size grew from 10s/100s of bytes to 3k-4.5kbytes.
The set up is of several applications, they use normal HPA with CPU for most, and the main 2 applications are scaled beforehand to about 50 replicas, and those 2 applications produce most of the workload and logs, we use AWS EKS.
I have verified with fluentd, datadog agents and kubectl logs and stern that the missing logs are consistent.
I have also created mount volumes for each pod to save logs using "| tee -a file.log" to /var/tmp folder with large storage nodes and all logs were in place while some from stdout were missing.
We have identified that it is probably somewhere between the kubelet taking the logs from stdout and writing them to the log file while logrotate happens.
The logging configuration is default, 10MB per log, single worker, 5 files.
The main 2 applications with 50 replicas, create about 10MB of log per minute per container log.
The problem started when we were running kubernetes version 1.26, but we also tested it with 1.28 and 1.29, also 1.29 provided me the fix for kubectl logs to tail the logs.

### What did you expect to happen?

Logs not to be missing.
Also maybe the ability to update the logrotate method if it is copytruncate, update to create or visaversa

### How can we reproduce it (as minimally and precisely as possible)?

I think that if POD creates large log entries close to 5KB each, with high rate of about 10MB per minute it can be reproduced within 3 to 3.5 hours.

### Anything else we need to know?

I am doing another test with very large nodes and will instruct the logmaxsize to 20Gi which gives me about 3 hours of Load test when spanned across 6 large nodes.
We will also upgrade to 1.30 and test.
The automation team created Log verification process with Datadog (dd api) and Cloudwatch (fluentbit) using correlationid entry for every generated flow, that way we can say that logs are missing, with that info we can actually go to the captured logs by kubectl logs cli or DD or Cloudwatch and actually see that.
We didn't see missing logs with small no-json log entries ~100bytes

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```
kubectl version --output=yaml
clientVersion:
  buildDate: "2024-06-11T20:29:44Z"
  compiler: gc
  gitCommit: 39683505b630ff2121012f3c5b16215a1449d5ed
  gitTreeState: clean
  gitVersion: v1.30.2
  goVersion: go1.22.4
  major: "1"
  minor: "30"
  platform: darwin/arm64
kustomizeVersion: v5.0.4-0.20230601165947-6ce0bf390ce3
serverVersion:
  buildDate: "2024-04-30T23:53:58Z"
  compiler: gc
  gitCommit: 9c0e57823b31865d0ee095997d9e7e721ffdc77f
  gitTreeState: clean
  gitVersion: v1.29.4-eks-036c24b
  goVersion: go1.21.9
  major: "1"
  minor: 29+
  platform: linux/amd64
</details>


### Cloud provider

<details>
es
</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```
Amazon AL2_x86_64
</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

containerd | 1.7.11-1.amzn2.0.1
-- | --

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details> 
    addon_name        = "coredns"
    addon_version     = "v1.11.1-eksbuild.4"
    addon_name        = "kube-proxy"
    addon_version     = "v1.29.0-eksbuild.1"
    addon_name        = "vpc-cni"
    addon_version     = "v1.16.0-eksbuild.1"
    addon_name         = "aws-ebs-csi-driver"
    version            = "v1.31.0-eksbuild.1"
</details>