Repository: harvester/harvester

Issue #5382: [BUG] A pending fail pod from different hub is generated after upgrade rancher
Created: 2024-03-18T07:02:07Z
Describe the bug
Based on harvester v1.3.0, by performing a Rancher upgrade form  v2.7.11 to v2.8.3-rc4, we found that a new generated pod is stuck in pending fail.
Note that new one comes from a different image rancher/harvester-cloud-provider:v0.2.0 compare to the running one registry.rancher.com/rancher/harvester-cloud-provider:v0.2.0

To Reproduce


Setup RKE2 cluster based on Harvester v1.3.0 + Rancher v2.7.11



 🟢 
Create RKE2 cluster 






 🟢 
Check deployment with pvc






 🟢 
Check LB







Upgrade Rancher to v2.8.3-rc4



 🟢 
  Cluster is Active






 🔴 
Apps are in health 

${\color{red} \text{0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..}}$





 🟢 
Check deployment with pvc






 🟢 
Check LB







Expected behavior
harvester-cloud-provider should be in health after upgrade
Support bundle

Harvester: supportbundle_cloudproviderhub_2024-03-18T06-38-10Z.zip
Rancher: rancher-2024-03-18_06_41_29.tar.gz

Environment

Harvester

Version: v1.3.0
Profile: QEMU/KVM, 3 nodes (8C/16G/500G)
ui-source: Auto


Rancher

Version: v2.7.11 -> v2.8.3-rc4
Profile: Helm (1C/4G)

Duplicate issue #5873: [BUG] Fail to Upgrade `harvester-cloud-provider`  on RKE2 cluster (`rancher-v2.8.4`)
Created: 2024-05-24T10:55:18Z
Describe the bug
Fail to Upgrade harvester-cloud-provider on RKE2 (v1.28) cluster (rancher-v2.8.4)
Note

Similiar to /issues/5382


0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..

To Reproduce
Steps to reproduce the behavior:


Setup harvester-v1.2.2 and rancher-v2.8.4


Import Harvester to Rancher



 Create RKE2 cluster (v1.28.9+rke2r1)


harvester-cloud-provider and harvester-csi-driver




Upgrade harvester-cloud-provider (0.2.300 -> 103.0.2+up0.2.4) ❌

Upgrade


After

0/1 nodes are available: 1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..




Expected behavior
Should be able to upgrade Rancher harvester app
Support bundle
supportbundle_284FailUpgradeCloudProvider-24T10-50-58Z.zip
Rancher Log
myrke2-128-pool1-ef6ffd22-7jtc8-2024-05-24_10_51_16.tar.gz
Environment

Harvester

Version: v1.2.2
Profile: QEMU/KVM, single nodes (16C/32G/500G)
ui-source: Auto


Rancher

Version: v2.8.4
Profile: Helm (v1.28.9+k3s1) in QEMU/KVM (2C/4G)
RKE2 Version: v1.28.9 +rke2r1



Additional context


helm-operation-cfns7_undefined.log
helm upgrade --history-max=5 --install=true --namespace=kube-system --timeout=10m0s --values=/home/shell/helm/values-harvester-cloud-provider-103.0.2-up0.2.4.yaml --version=103.0.2+up0.2.4 --wait=true harvester-cloud-provider /home/shell/helm/harvester-cloud-provider-103.0.2-up0.2.4.tgz
checking 8 resources for changes
Looks like there are no changes for ServiceAccount "kube-vip"
Patch ServiceAccount "harvester-cloud-provider" in namespace kube-system
Looks like there are no changes for ClusterRole "kube-vip"
Looks like there are no changes for ClusterRole "harvester-cloud-provider"
Looks like there are no changes for ClusterRoleBinding "kube-vip"
Patch ClusterRoleBinding "harvester-cloud-provider" in namespace 
Patch DaemonSet "kube-vip" in namespace kube-system
Patch Deployment "harvester-cloud-provider" in namespace kube-system
beginning wait for 8 resources with timeout of 10m0s
Deployment is not ready: kube-system/harvester-cloud-provider. 0 out of 1 expected pods are ready
DaemonSet is not ready: kube-system/kube-vip. 0 out of 1 expected pods have been scheduled
Deployment is not ready: kube-system/harvester-cloud-provider. 0 out of 1 expected pods are ready
...
2024-05-24T10:45:07.219067445Z Deployment is not ready: kube-system/harvester-cloud-provider. 0 out of 1 expected pods are ready
2024-05-24T10:45:09.135615116Z Error: UPGRADE FAILED: context deadline exceeded



rke2-server.service
# journalctl -u rke2-server.service --follow
May 24 10:13:43 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:13:43Z" level=error msg="error syncing 'kube-system/rke2-coredns': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-coredns\" not found, requeuing"
May 24 10:13:44 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:13:44Z" level=error msg="error syncing 'kube-system/rke2-ingress-nginx': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-ingress-nginx\" not found, requeuing"
May 24 10:13:45 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:13:45Z" level=error msg="error syncing 'kube-system/rke2-metrics-server': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-metrics-server\" not found, requeuing"
May 24 10:13:45 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:13:45Z" level=error msg="error syncing 'kube-system/rke2-snapshot-controller-crd': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-snapshot-controller-crd\" not found, requeuing"
May 24 10:13:46 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:13:46Z" level=error msg="error syncing 'kube-system/rke2-snapshot-controller': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-snapshot-controller\" not found, requeuing"
May 24 10:13:46 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:13:46Z" level=error msg="error syncing 'kube-system/rke2-snapshot-validation-webhook': handler helm-controller-chart-registration: helmcharts.helm.cattle.io \"rke2-snapshot-validation-webhook\" not found, requeuing"
May 24 10:13:52 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:13:52Z" level=info msg="Adding node myrke2-128-pool1-ef6ffd22-7jtc8-a99bff44 etcd status condition"
May 24 10:15:45 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:15:45Z" level=info msg="Tunnel authorizer set Kubelet Port 10250"
May 24 10:22:04 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:22:04Z" level=info msg="Updating TLS secret for kube-system/rke2-serving (count: 11): map[field.cattle.io/projectId:c-m-c79fjcww:p-gz6xh listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.0.52:192.168.0.52 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-myrke2-128-pool1-ef6ffd22-7jtc8:myrke2-128-pool1-ef6ffd22-7jtc8 listener.cattle.io/fingerprint:SHA1=83CDD81E2BC9612F9C7E7E06A70E367C36573FC7]"
May 24 10:22:04 myrke2-128-pool1-ef6ffd22-7jtc8 rke2[2322]: time="2024-05-24T10:22:04Z" level=info msg="Active TLS secret kube-system/rke2-serving (ver=4210) (count 11): map[field.cattle.io/projectId:c-m-c79fjcww:p-gz6xh listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.0.52:192.168.0.52 listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-myrke2-128-pool1-ef6ffd22-7jtc8:myrke2-128-pool1-ef6ffd22-7jtc8 listener.cattle.io/fingerprint:SHA1=83CDD81E2BC9612F9C7E7E06A70E367C36573FC7]"



rancher-system-agent.service 
# journalctl -u rancher-system-agent.service --follow
May 24 10:34:11 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:34:11Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240524-103411/1a601d63558d61145f67d1569292e3450c7597bc81499ae31bebbec678d6a964_0"
May 24 10:34:11 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:34:11Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
May 24 10:34:12 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:34:12Z" level=info msg="[1a601d63558d61145f67d1569292e3450c7597bc81499ae31bebbec678d6a964_0:stdout]: Name Location Size Created"
May 24 10:34:12 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:34:12Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
May 24 10:34:12 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:34:12Z" level=info msg="[K8s] updated plan secret fleet-default/myrke2-128-bootstrap-template-45xrz-machine-plan with feedback"
May 24 10:44:13 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:44:13Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240524-104413/1a601d63558d61145f67d1569292e3450c7597bc81499ae31bebbec678d6a964_0"
May 24 10:44:13 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:44:13Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
May 24 10:44:13 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:44:13Z" level=info msg="[1a601d63558d61145f67d1569292e3450c7597bc81499ae31bebbec678d6a964_0:stdout]: Name Location Size Created"
May 24 10:44:13 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:44:13Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
May 24 10:44:14 myrke2-128-pool1-ef6ffd22-7jtc8 rancher-system-agent[16919]: time="2024-05-24T10:44:14Z" level=info msg="[K8s] updated plan secret fleet-default/myrke2-128-bootstrap-template-45xrz-machine-plan with feedback"

Closing comment (2024-06-12T03:11:04Z): The root cause is the same as #5382's. The image repo/tag of harvester-cloud-provider was changed, which triggered the deployment rollout. Since it is a single-node guest cluster, the situation described in #5348 happened.