Repository: kubernetes
Issue: Ability to declare pods to be "recreated" on "failure" (instead of restarts)
Labels: kind/feature,sig/apps,lifecycle/frozen,needs-triage

The following is to track the issue raised originally in : https://github.com/kubernetes/kubernetes/issues/24725
That is specific to only the desired behaviour of "If Pod-XYZABC exits for any reason, do not restart it, provide me a new one" pattern.

#### What would you like to be added:

It has been suggested to either
- Implement a "restartPolicy: Recreate" or "restartPolicy: Never" (though this might be confusing behaviour, as this is assumed to work together with replica, which may not be existing intended behaviour)
- A new type of ReplicaSet ( SingleUseReplicaSet?, im probably bad at naming this )

#### Why is this needed:

> The following was taken/edited from: https://github.com/kubernetes/kubernetes/issues/24725#issuecomment-838607507

Compilling the various known use cases for "If Pod-XYZABC exits for any reason, do not restart it, provide me a new one", either mentioned in other issues, or from my own personal experience (in providing infrasturcture consulting to startups) - so as to avoid confusing it with issues that would be solved with "restartTogether", or other potential fixes.

- Persistent errors that survive restarts
    - Due to possible volume driver issues : https://github.com/kubernetes/kubernetes/issues/67643
    - Due to mounting of host resources that has failed
        - mounting of a cache SSD, that is now phsically corrupted
        - mounting of GPU issues
    - Other wierd driver issues 
       (I run x11 with selenium in docker, i cant explain why there is a 1 in 50 chance that a new container will persistently have an error with its graphic driver which survive restarts)
    - Applications logic failure, especially of legacy / buggy apps, despite removing all known file changes (just what is it modifying on the kernel level, wth?)

- Security of container appspace
    - For QEMU/KVM (or VMWare, etc), "snapshot" volumes or images are used to gurantee that no changes persist between restarts [[link-to-docs]](https://qemu.readthedocs.io/en/latest/system/images.html#snapshot-mode). This helps gurantee that in event of a server runtime compromise, if files were to be injected into the container, this can be resolved by a simple "restart". Compliance also really love this feature, and this was nearly a major show stopper for one known bank migration from VM to K8S
    - Read-only disks is not the same behaviour, these are writable (cache, etc) - but is resetted on each restart
    - Strictly speaking this is not the same feature, but a pod recreation is effectively a clean restart

Note: For most of the above listed examples, this does not have issues with PVC itself or its data, so those PVC are reused across pods, also PVC is realtively easier to "reset" with an entrypoint script or sidecar. While it may overlap with "mount issues" in the driver, to be clear the issue being solved here is not issues in PVC, to avoid confusing with other existing issues.

---

In general, while most issues could probably be "fixed" by "fixing" the application software - especially applicaiton logic failure. It is important to recognise that a large percentage of people in charge of the company kubernetes infrastructure, may not have the ability to make such changes. They may not even have a say in the applications involved, as many enterprises have started forcing the move to docker/cloud "at all costs".

- Lack of domain expertise to make changes to volumedrivers / linux kernel code
- Legacy applications, may not have any developers, or even the source code left
- Legacy will be legacy
- Lack of "influence" on the dev teams

In all the above cases, implementing healthchecks does not fully "work", while it does help K8S operators notice the issue (properly fail the application container, and enter infinite restart loop), it means they will need manual intervention using vanilla k8s. And really prevented some operators from being able to "sleep in peace" knowing that if enough of their replicas enter a restart loop without their daily/weekly intervention, would mean a downtime (that was until i advised on automating the restarts on a schedule)

---

As a result of the above, currently in the field i have seen the following being done to work around this lack of "Replace pod on error" feature.

- Custom Schedule / descheduler (what the bank did)
- Custom Controller (As suggested)
- Service account integrated into a sidecar with a custom program doing the termination: https://github.com/kubernetes/kubernetes/issues/24725#issuecomment-724834010
- Custom scripts (watchdog container) which detect such issues and do the respective kubectl/service account delete command
- Less then ideal fixes 
    - Oh it enters an infinite failure loop, just ignore it cause we have Y+ redudancy and this will be fixed when
       - We redeploy every 1 hour (or X minutes, number depends on frequency of issue and redundancy involved)
       - We run on GKE pre-emptible nodes, eventually that 24 hours would replace the host
    - Here at secure enterprise X, we do not allow anyone to provision service accounts + other restrictions
       - I know of 2 sysadmin who is running a kubectl with bash scripts to automate a fix for this issue.

---

Finally, in my opinion, its the ultimate expression of "Cattle" vs "Pets", if there is any issue, kill the pod, and get a new one. And if operators wants to live by this ideal to the extreme, so be it =)

In general most of the above is fixed if "restartPolicy: Recreate/Never" is accepted, and new pods are created on failure, or as mentioned a new type of "ReplicaSet" is created to help this behaviour.

(to be clear, this should not be default due to potential increase resource drains, and such usage should be an decision of the operator)