Repository: kubernetes
Issue: promoting use of parallel image pulls
Labels: sig/node,kind/feature,triage/accepted

### What would you like to be added?

/sig node

kubelet flag `--serialize-image-pulls` is by default true. It has a performance issue that one big image pulling would block all pods starting(that need a new image). 

The current solution is changing the flag `--serialize-image-pulls`. (For instance, kubeadm supports the kubelet Extra flags configuration.)

Some discussions in https://github.com/kubernetes/enhancements/pull/2217#discussion_r556918795, #93239 tries to improve the image pull order on kubelet. 
- Pull critical pods' images first.
- By default, pull images for kube-system namespace before other namespaces.
- Further goals would be pods' images pull priority depending on namespace's specified annotation.

/cc @derekwaynecarr 

### Why is this needed?

IMO, parallel image pulls are better and the trend.

**Notice**: contained 1.6.3+ & 1.7+ fixed a performance issue of parallel image pulling in https://github.com/containerd/containerd/pull/6702

https://github.com/kubernetes/kubernetes/blob/cf12a74b18b66efc577ec819c78a0c68f6d49225/cluster/gce/config-test.sh#L232

https://github.com/kubernetes/kubernetes/blob/cf12a74b18b66efc577ec819c78a0c68f6d49225/test/e2e_node/services/kubelet.go#L126

- openshift installer: https://github.com/openshift/installer/blob/fea1cc4438504c4bf25126d047304f762b91544a/data/data/bootstrap/files/usr/local/bin/kubelet.sh.template#L22
- Huawei-ContainerOps https://github.com/search?q=repo%3AHuawei%2Fcontainerops%20serialize-image-pulls&type=code

#### Advantages:
The problems that can be solved if we change the default to enable paralleling image pulling 
- If the one registry is slow (and maybe with a speed limit), other pods using a different registry will be stuck as well. (I**t can be a CVE if I deploy a daemonset with it in a multi-tenant cluster, and the pod will block all new pod creating that needs pulling image.**)
- For AI or windows, the image is big. For some Java or other images, the image is big as well.
- This is a big problem in node disaster recovery. In this scenario, kubelet needs to pull a lot of images.

#### Disdvantages:
- @rphillips  points out that there is a performance issue. (Maybe performance issue to network load or registry if it is a local registry)
- aufs with docker before 1.9 (dockershim is removed, so we can ignore this)
- To be pointed out.


#### Other solutions:
- Or promoting the use of parallel image pulls in kubeadm and other cluster-lifecycle products.
- More notice in kubelet for big image pulling that may block others
- Manually change the default value of --serialize-image-pulls (may not consist with old versions)
- Add warning during kubelet starting with --serialize-image-pulls=true.


### Steps towards making it possible

- [x] https://github.com/kubernetes/kubernetes/issues/111704 
- [x] https://github.com/containerd/containerd/pull/6702 contained 1.6.3+ & 1.7+: performance improve
- [x] https://github.com/containerd/containerd/pull/7313 image pull metrics
- [x] node level limit for image pulling: the current qps/burst is not that intuitive. https://github.com/kubernetes/enhancements/issues/3673
- - [x] https://github.com/kubernetes/enhancements/pull/3713/
- - [x] https://github.com/kubernetes/kubernetes/pull/115220 
- [x] Closed: https://github.com/kubernetes/kubernetes/pull/114552 promoted the `--serialize-image-pulls=false` by default

Some image pull related enhancements/bugfixes
- https://github.com/kubernetes/kubernetes/pull/117935
- https://github.com/canonical/microk8s/pull/4065