Repository: kubernetes
Issue: Kubelet should react more quickly to pod lifecycle transitions
Labels: sig/node,triage/accepted

The Kubelet invokes pod setup, maintenance (collectively referred to as 'pod sync'), and teardown as synchronous functions from a per-pod goroutine.  The sync method in particular can run for seconds to minutes to hours if certain lifecycle actions like hooks are in use.  This means state transitions such as deletion (from API) and eviction (from Kubelet) can sometimes be significantly delayed.  Historically we had a number of asynchronous loops that raced against each other and provided sometimes incorrect behavior to pods - as we have tightened and improved the pod worker component to be more correct that has exacerbated the latency issues.

In general, we prefer to simplify the reasoning of adding new code to the pod's lifecycle and prefer to keep the bulk of pod setup, teardown, and cleanup synchronous.  Certain long running operations like pod start and volume attach/detach are asynchronous to the main pod loop, but others like pod lifecycle hooks are not.  In the long run, we should prefer to make those hooks asynchronous as well and use the pod sync methods to orchestrate hook invocation.  Even with perfect asynchronous logic we still must consider that our invocations of CRI functions may still hang or be delayed, and so at the pod worker level we must be able to interrupt the current invocation of SyncPod on transition.  For SyncTerminatingPod, we must be able to shorten (#113883) the wait before the pod is force killed - given the design of the CRI kill method we must interrupt the current call and make a new call with a shorter timeout in addition to any improvements on the CRI side.  Therefore the appropriate mechanism to reduce the wait is to interrupt the current sync method by cancelling a context passed to the worker.

A few PRs block resolving this issue:

* The pod worker is being improved to periodically resync missing pods - moderate refactor that must be completed #113145 
* If we were to propagate context into sync today, certain `wait` loops would return generic errors instead of context cancellation which would cause spurious errors - a refactor of the wait package that introduces new methods to handle this must be merged first #107826

After these two issues merge, we can propagate context into Sync methods and add tests that verify pods with long lifecycle hooks are terminated quickly and long grace period deletions can be reduced, as well as stress tests that verify that we do not generate excessive errors.  This issue tracks this final work.

In addition, we should continue to assess other sources of transition time, like evented PLEG and https://github.com/kubernetes/kubernetes/issues/116617

#113591 took a slightly more scoped approach to plumbing context in Kubelet (in order to get tracing to work) than #107829 (which was trying to ensure sync*Pod would correctly cancel when pods transitioned to avoid very long latency of deletion requests).  As a consequence, the remaining issues that blocked #107829 in `syncPod` cancellation are unresolved, and those failures block #111591 from landing.  We can mitigate those by substituting a TODO context at the top of each sync method and then subsequently propagate those fixes (which also depends on #107826 for completeness).

#116032

This issue tracks adding that propagation.

/sig node