Repository: kubernetes
Issue: In place pod resizing should be designed into the kubelet config state loop, not alongside it
Labels: kind/bug,sig/node,priority/important-longterm,triage/accepted

### What happened?

#102884 in its alpha state introduces a new challenge to the kubelet state machine - partial and deferred acceptance of spec changes to a pod in the kubelet, while at the same time we are realizing that the kubelet is not properly feeding actual state to  components #116970.

Last minute 1.27 fix #116702 works around but doesn't resolve a key side effect of this problem - reverting the pod state incorrectly / temporarily.  The pod worker receives new state (in an ordered fashion from admission, and before that podManager) and then invokes SyncPod with the "latest".  In-place resizing then mutates podManager directly, and continues its execution with a different syncPod.  However, once we fix #116970, this will be impossible - other components have to consult the pod worker for the latest state, not podManager.

Visually this is described at https://docs.google.com/presentation/d/1wbPFANXVbk-e-5BvVHaZScqoqJsfcU_FQVmG-N7pv3c/edit#slide=id.g1de8a1ca1a4_0_1673 (shared with kubernetes-sig-node google group)

To graduate to beta in-place resizing needs a new capability added to the kubelet state machine - the ability to decide when the spec change requested is "acted on" by the kubelet.  There are a couple of ways we could design this, but they need to fit harmoniously into the larger changes we're making to kubelet state vs being working around the kubelet state machine.

To do that we need to make sure we understand the rules that any "level driven update" to pod spec must follow and then implement a mechanism in the kubelet.  This issue covers articulating those rules and getting answers as an input to the change we should make to kubelet admission.

Question:

* What future spec changes to pods have been discussed that would have requirements?
* Can spec changes to a pod be rejected permanently (kubelet never accepts them)?
* When a spec change to a pod is rejected temporarily, what other aspects of the pod spec must be updated?
   * Jordan identified the container hash as a potential problem
   * Is the spec generation "locked" at the last accepted spec change?
* Does a temporary pod spec change block other temporary pod spec changes?  What should the generation be reported as in the kubelet?
* If multiple failed spec changes happen in a row, are there any observable side effects?
* Should we just support full conditional inplace pod changes?  What value comes from preventing in-place updates if we have to implement a full "spec admission" mechanism for a single set of fields?  We already have to support full in-place for static pods anyway.
* ??? (not complete list)

In general my preference is for all config spec changes to happen before a change is accepted by the pod worker, and the pod worker's state to reflect "admitted spec changes".  That means that we would remove the code from syncpod that alters pod, and move it up to right before admission.  It *also* means that we may need to abstract the code so that the kubelet resync loop (HandlePodCleanups) can recheck admission decisions.  A final option may be the soft admission handler inside syncPod, but I don't like that because we have to atomically record which version of the spec the kubelet is acting on while within a config loop which is racy.  

Outcome of this will be a KEP update to https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/1287-in-place-update-pod-resources/README.md#kubelet-and-api-server-interaction to cover these design details in greater detail before beta.

/sig-node
/priority important-soon

@bobbypage @vinaykul I'll use this for tracking the inputs we need to come up with a design change

### What did you expect to happen?

Kubelet spec changes and rejection is a fundamental part of the kubelet state machine, not part of syncPod (which is driven by the state machine).

### How can we reproduce it (as minimally and precisely as possible)?

N/A

### Anything else we need to know?

_No response_

### Kubernetes version

1.27+


### Cloud provider

<details>

</details>


### OS version

<details>

```console
# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

<details>

</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>