Repository: kubernetes
Issue: Add PermitExtensions in scheduler to control when to admit the Pod binding process
Labels: sig/scheduling,sig/node,kind/feature,needs-triage,wg/serving

### What would you like to be added?

Let me introduce our user story first:

we have a model cache platform build on top of kubernetes. In model inference, when a Pod bind nodeName successfully and not started, we need to hang the pod from starting and syncing the model weights the inference pod needed from other nodes. To achieve this, we have to inject an initContainer to block pod starting and admit the initContainer until the model pulling finished.

This is **invasive** because we're modifying the Pod yaml and if the initContainer requires to watch CRDs, we may need to update the roles of the default SA for corresponding permissions, actually that's what we're faceing right now.

So we first think of an idea to block the sandbox creation by kubelet after Pod scheduled successfully, however, this may lead to pod resources reserved in scheduler but not consumed physically if pod hand forever for some reasons.

Then we came with another idea to extend the exist Permit Plugin in scheduler. So right now, permit plugin can return a `Wait` status in scheduling cycle, then in binding cycle, we'll block in `waitOnPermit` until condition ready. 

One example is coscheduling plugin, scheduling Pod could be blocked in bindingCycle until all its siblings are ready. 

However, based on my understanding, we have limitations here because once Pod is hanging in bindingCycle, we can only activate it in scheduling cycle by iterating over the waiting pods and return `Success` status. What if the Pod could be activated by other crd events rather than a new scheduling process?

So here comes a new proposal, we'll have a new interface `PermitExtensions`: 

```

type PermitFn func(logger klog.Logger, pod *v1.Pod, oldObj, newObj interface{}) error

type ClusterEventForPermission struct {
	Event    ClusterEvent
	PermitFn PermitFn
}

type PermitExtensions interface {
	Plugin
	RegisterPermissionEvents(context.Context) ([]ClusterEventForPermission, error)
}
```

The logic is quite similar to `EnqueueExtension`. First, we'll call the `RegisterPermissionEvents` to register events, and keep watching them in runtime. Once coming events match with the registered ones, the `PermitFn` will iterate over the waitingPods to decide whether Pods are ready or not, once ready, return `Success` status and the bindingCycle will be unblocked.

In our case, we have a CRD named `Torrent` who will record whether the model syncing task is done or not, and once completed, the Permit plugin's PermitFn will be triggered to decide whether we're ready to bind the Pod.

All existing flows will not be broken, no additional changes but offering a knob for users to decide when to admit a pod binding.

~~Then I came up with idea like maybe we can add some gates just like schedulingGates which will block the scheduling process, the new added gates (let's say xGates) will block the kubelet process, like creating the sandbox. Until the Pod holds a nodeName and no xGates exist, the kubelet will do his job.~~

~~The risk is since Pod is scheduled, the resource is reserved, but physically not consumed, so if the postScheduling process blocks, resource is wasted, what we have to do next is recreate the Pod (pod.nodeName is unchangeable now if set, maybe we can unbind the Pod and don't change the nodeName anymore until `nodeName != "" && xGates == []`, this is a big change I think).~~

~~Another approach would be introduce another logic process in scheduler, and hold the bind process until the condition is ready, quite like the DRA, in DRA, we have a lot of interactions between the scheduler and the dra controller, until everything is ready, open fire. DRA is a huge work, I hope not to introduce more CRDs, so maybe we can add new gates like bindingGates here? Extending the schedulingGates is ambiguous I think.~~

Some rough thoughts here, would like to hear advices or suggestions. Thanks in advance!

cc @liggitt @SergeyKanzhelev @kubernetes/sig-scheduling-misc 
/kind feature
/sig node
/sig scheduling

### Why is this needed?

Once Pod is assumed to be bind with a node, or already bind with a node, we may have more work to do before pod starting, here we introduce a mechanism to make this happen before pod binding.