Repository: kubernetes
Issue: Static pod Lifecycle Formalization (was: Decide fate of mirror pods)
Labels: priority/backlog,sig/node,kind/feature,lifecycle/frozen,area/pod-lifecycle

I would like to open a discussion on how and whether we should continue support mirror pods, given that 1) mirror pods need to be treated differently in multiple places in kubelet, making them brittle, and 2) there are some limitations about mirror pods that may cause confusion (e.g., what to expect if user modifies mirror pods)?

_Background_
kubelet watches a few sources for pods: apiserver, http, and files in a local directory. We support non-apiserver sources for several main use cases:
- Standalone kubelets
- Schedule one pod per-node (e.g., fluentd). This will soon be deprecated in favor of DaemonSet of controller.

These pods are sometimes called _static pods_ in the codebase.

For pods from non-apiserver sources, kubelet would create a corresponding _mirror pod_ in the apiserver and continues to update the status. These mirror pods serve two main purposes:
1. Allow users to inspect the pod status through kubectl.
2. Allow scheduler to account for resource usage.

Note that currently the mirror pods may not reflect truthfully what's been running on the node because apiserver can apply default (or cluster-specific) values to certain fields in the mirror pod, but kubelet will adhere to the original pod spec obtained from the source.

**[Option 1]: Deprecate mirror pods completely.**
- The majority use case is covered by DaemonSet.
- For resource accounting, kubelet can report *AllocatableResources" to the scheduler, which already excludes resources occupied by static pods.
- Mirror pods are meaningless for standalone kubelet users.
- The downside would be that if user wants to use static pods, they would not be able to inspect them from the apiserver.

**[Option 2]: Only allow the use of static pods in standalone kubelets.**
- This eliminates the need of mirror pods at the cost of user flexibility.

**[Option 3]: Continue supporting mirror pods, but mark them readonly (non-updatable) in apiserver**
- User won't be allowed to modify the mirror pods arbitrarily.
- Mirror pods could still have values that are different than its static pod.
- We still need to treat mirror pods different in the kubelet codebase.

**[Option 4]: Continue supporting mirror pods, and kubelet should sync to the mirror pods**
- If apiserver is present, kubelet will first create a mirror pod, and then sync to the pod when it receives the mirror pod through watch.
- If apiserver is not present, kubelet will sync the static pods as well.
- Mirror pods will reflect the truth since kubelet syncs to the mirror pod, and apiserver can supply any additional values in the pod spec.
- Mirror pods are read-only.
- Increased code complexity in kubelet.

**[Option 5]: Option 4 + updatable mirror and static pods**
- If a user updates the mirror pod, kubelet would overwrite the file-based static pod.
  - What if the static pod is from an http source or the directory is not writable?
  - What if user updates the static pod at the same time?
    This is my least preferred option since we'll need to resolve conflicts and handle all corner cases. I think it's reasonable to have a single source of truth.

**[Option 6]: Maintain status quo**
- Static pod is the single source of truth. Users can modify the mirror pods arbitrarily, but kubelet would ignore all the modifications as long as the annotation kubelet uses remains unchanged.
- If the mirror annotation changes in the mirror pod, kubelet recreates the mirror pod.
- If the static pod changes, kubelet recreates the mirror pod.
- Mirror pod may not reflect the true status of the static pod due to apiserver settings, as mentioned before.

/cc a couple folks who have expressed opinions about mirror pods: @bgrant0607 @thockin @davidopp @kubernetes/goog-node