Repository: siderolabs/talos

Issue #7123: upgrade to 1.4.0 leads to kernel module load issues (virtio, arm64)
Created: 2023-04-22T14:29:10Z
Bug Report
Description
I've wanted to upgrade my single node cluster to Talos 1.4.0 from 1.3.6 (thus, I skipped 1.3.7).
I'm not sure if there was a change in the Oracle Cloud which now blocks access UDP to 8.8.8.8 or similar (couldn't find anything regarding this), or if there was a change in the behaviour of Talos. Unfortunately, the node is not accessible in this state via talosctl and I can't rollback.
Logs
Upgrade
➜  ~ talosctl upgrade --image ghcr.io/siderolabs/installer:v1.4.0 --wait --debug --preserve
◰ ◳ watching nodes: [<IP>]
    * <IP>: unavailable, retrying...
Signal received, aborting, press Ctrl+C once again to abort immediately...
console logs for nodes ["<IP>"]:
<IP>: user: warning: [2023-04-22T13:45:20.11278299Z]: [talos] upgrade sequence: 15 phase(s)
<IP>: user: warning: [2023-04-22T13:45:20.11375399Z]: [talos] phase drain (1/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:45:20.11479899Z]: [talos] task cordonAndDrainNode (1/1): starting
<IP>: user: warning: [2023-04-22T13:45:20.14646499Z]: [talos] skipping DaemonSet pod kube-system/kube-flannel-cl2wk
<IP>: user: warning: [2023-04-22T13:45:20.14961299Z]: [talos] skipping DaemonSet pod kube-system/kube-proxy-t8g8x
<IP>: user: warning: [2023-04-22T13:45:20.15203999Z]: [talos] skipping mirror pod kube-system/kube-scheduler-talos-2jw-ado
<IP>: user: warning: [2023-04-22T13:45:20.15337299Z]: [talos] skipping mirror pod kube-system/kube-apiserver-talos-2jw-ado
<IP>: user: warning: [2023-04-22T13:45:20.15469199Z]: [talos] skipping mirror pod kube-system/kube-controller-manager-talos-2jw-ado
<IP>: kern:    info: [2023-04-22T13:45:21.04746499Z]: cni0: port 4(veth20b856c5) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:21.05042999Z]: device veth20b856c5 left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:45:21.05151299Z]: cni0: port 4(veth20b856c5) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:21.40711399Z]: cni0: port 8(vethaf0c34cc) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:21.40980599Z]: device vethaf0c34cc left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:45:21.41082999Z]: cni0: port 8(vethaf0c34cc) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:22.71490899Z]: cni0: port 9(veth035ab819) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:22.71690499Z]: device veth035ab819 left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:45:22.71821199Z]: cni0: port 9(veth035ab819) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:25.34686699Z]: cni0: port 2(veth6ce3cb80) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:25.35131699Z]: device veth6ce3cb80 left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:45:25.35489199Z]: cni0: port 2(veth6ce3cb80) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:25.63246699Z]: cni0: port 3(vethf687a2ac) entered disabled state
<IP>: kern:    info: [2023-04-22T13:45:25.63415299Z]: device vethf687a2ac left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:45:25.63579999Z]: cni0: port 3(vethf687a2ac) entered disabled state
<IP>: user: warning: [2023-04-22T13:46:20.19369999Z]: [talos] WARNING: failed to evict pod: failed waiting on pod ...-5c644c945c-wx98g to be deleted: 2 error(s) occurred:
<IP>: user: warning: [2023-04-22T13:46:20.19651099Z]:  pod is still running on the node
<IP>: user: warning: [2023-04-22T13:46:20.19732899Z]:  timeout
<IP>: user: warning: [2023-04-22T13:46:20.19795899Z]: [talos] WARNING: failed to evict pod: failed waiting on pod local-path-storage/local-path-provisioner-5646477f4b-fbf6g to be deleted: 2 error(s) occurred:
<IP>: user: warning: [2023-04-22T13:46:20.20014099Z]:  pod is still running on the node
<IP>: user: warning: [2023-04-22T13:46:20.20170599Z]:  failed to get pod local-path-storage/local-path-provisioner-5646477f4b-fbf6g: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
<IP>: user: warning: [2023-04-22T13:46:20.34159899Z]: [talos] WARNING: failed to evict pod: failed waiting on pod ...-manual-2lk-qlnd5 to be deleted: 2 error(s) occurred:
<IP>: user: warning: [2023-04-22T13:46:20.34430299Z]:  pod is still running on the node
<IP>: user: warning: [2023-04-22T13:46:20.34547199Z]:  timeout
<IP>: kern:    info: [2023-04-22T13:46:20.35284299Z]: cni0: port 10(vetha7a5ca12) entered disabled state
<IP>: kern:    info: [2023-04-22T13:46:20.35762599Z]: device vetha7a5ca12 left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:46:20.36111499Z]: cni0: port 10(vetha7a5ca12) entered disabled state
<IP>: kern:    info: [2023-04-22T13:46:20.39162699Z]: cni0: port 6(vethf567f4ba) entered disabled state
<IP>: kern:    info: [2023-04-22T13:46:20.39378199Z]: device vethf567f4ba left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:46:20.39534999Z]: cni0: port 6(vethf567f4ba) entered disabled state
<IP>: kern:    info: [2023-04-22T13:46:20.45604199Z]: cni0: port 1(veth5b954e52) entered disabled state
<IP>: kern:    info: [2023-04-22T13:46:20.45901399Z]: device veth5b954e52 left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:46:20.46022499Z]: cni0: port 1(veth5b954e52) entered disabled state
<IP>: user: warning: [2023-04-22T13:46:20.73373199Z]: [talos] WARNING: failed to evict pod: failed waiting on pod ...-api-6755c58f65-2fzf2 to be deleted: 2 error(s) occurred:
<IP>: user: warning: [2023-04-22T13:46:20.74102799Z]:  pod is still running on the node
<IP>: user: warning: [2023-04-22T13:46:20.74531199Z]:  timeout
<IP>: kern:    info: [2023-04-22T13:46:20.84037499Z]: cni0: port 5(veth8e82f611) entered disabled state
<IP>: kern:    info: [2023-04-22T13:46:20.84225999Z]: device veth8e82f611 left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:46:20.84328299Z]: cni0: port 5(veth8e82f611) entered disabled state
<IP>: user: warning: [2023-04-22T13:46:21.33082199Z]: [talos] WARNING: failed to evict pod: failed waiting on pod persistent/alpine-b5d69fcdb-vhkqr to be deleted: 2 error(s) occurred:
<IP>: user: warning: [2023-04-22T13:46:21.33304199Z]:  pod is still running on the node
<IP>: user: warning: [2023-04-22T13:46:21.33395499Z]:  timeout
<IP>: user: warning: [2023-04-22T13:46:21.33488899Z]: [talos] task cordonAndDrainNode (1/1): done, 1m1.220291604s
<IP>: user: warning: [2023-04-22T13:46:21.33634699Z]: [talos] phase drain (1/15): done, 1m1.222796012s
<IP>: user: warning: [2023-04-22T13:46:21.33792899Z]: [talos] phase cleanup (2/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:21.33896599Z]: [talos] task stopAllPods (1/1): starting
<IP>: user: warning: [2023-04-22T13:46:21.34050399Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
<IP>: user: warning: [2023-04-22T13:46:21.34217199Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
<IP>: kern:    info: [2023-04-22T13:46:22.11231799Z]: cni0: port 7(vethe9560da8) entered disabled state
<IP>: kern:    info: [2023-04-22T13:46:22.11435599Z]: device vethe9560da8 left promiscuous mode
<IP>: kern:    info: [2023-04-22T13:46:22.11544599Z]: cni0: port 7(vethe9560da8) entered disabled state
<IP>: user: warning: [2023-04-22T13:46:41.44788499Z]: [talos] service[kubelet](Stopping): Sending SIGTERM to task kubelet (PID 8338, container kubelet)
<IP>: user: warning: [2023-04-22T13:46:41.66089899Z]: [talos] service[kubelet](Finished): Service finished successfully
<IP>: user: warning: [2023-04-22T13:46:41.66973699Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.67240799Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.67513199Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.67711599Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.67878299Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.68091299Z]: [talos] stopped pod w...
<IP>: user: warning: [2023-04-22T13:46:41.68235799Z]: [talos] stopped pod n...
<IP>: user: warning: [2023-04-22T13:46:41.68382299Z]: [talos] stopped pod k...
<IP>: user: warning: [2023-04-22T13:46:41.68549599Z]: [talos] stopped pod p...
<IP>: user: warning: [2023-04-22T13:46:41.68695099Z]: [talos] stopped pod l...
<IP>: user: warning: [2023-04-22T13:46:41.69091699Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.69241299Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.70016299Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.70562399Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.70756499Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.70940199Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.71084199Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.71211899Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.71349399Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.71619599Z]: [talos] stopping pod ...
<IP>: user: warning: [2023-04-22T13:46:41.71815199Z]: [talos] stopped container kube-system/kube-scheduler-talos-2jw-ado:kube-scheduler
<IP>: user: warning: [2023-04-22T13:46:41.72395999Z]: [talos] stopped pod kube-system/kube-scheduler-talos-2jw-ado
<IP>: user: warning: [2023-04-22T13:46:41.72843099Z]: [talos] stopped pod ...
<IP>: user: warning: [2023-04-22T13:46:41.73115299Z]: [talos] stopped pod ...
<IP>: user: warning: [2023-04-22T13:46:41.73290199Z]: [talos] stopped container kube-system/kube-proxy-t8g8x:kube-proxy
<IP>: user: warning: [2023-04-22T13:46:41.73441399Z]: [talos] stopped container kube-system/kube-apiserver-talos-2jw-ado:kube-apiserver
<IP>: user: warning: [2023-04-22T13:46:41.73588099Z]: [talos] stopped container kube-system/kube-flannel-cl2wk:kube-flannel
<IP>: user: warning: [2023-04-22T13:46:41.74202999Z]: [talos] stopped container kube-system/kube-flannel-cl2wk:install-cni
<IP>: user: warning: [2023-04-22T13:46:41.74751399Z]: [talos] stopped container kube-system/kube-flannel-cl2wk:install-config
<IP>: user: warning: [2023-04-22T13:46:41.75316299Z]: [talos] stopped pod ...
<IP>: user: warning: [2023-04-22T13:46:41.75711199Z]: [talos] stopped container kube-system/kube-controller-manager-talos-2jw-ado:kube-controller-manager
<IP>: user: warning: [2023-04-22T13:46:41.76051099Z]: [talos] stopped pod ...
<IP>: user: warning: [2023-04-22T13:46:41.76269199Z]: [talos] stopped pod ...
<IP>: user: warning: [2023-04-22T13:46:41.76455899Z]: [talos] stopped pod kube-system/kube-proxy-t8g8x
<IP>: user: warning: [2023-04-22T13:46:41.76920199Z]: [talos] stopped pod kube-system/kube-apiserver-talos-2jw-ado
<IP>: user: warning: [2023-04-22T13:46:41.77398899Z]: [talos] stopped pod kube-system/kube-flannel-cl2wk
<IP>: user: warning: [2023-04-22T13:46:41.77815699Z]: [talos] stopped pod kube-system/kube-controller-manager-talos-2jw-ado
<IP>: user: warning: [2023-04-22T13:46:41.78278299Z]: [talos] task stopAllPods (1/1): done, 20.443880862s
<IP>: user: warning: [2023-04-22T13:46:41.78601999Z]: [talos] phase cleanup (2/15): done, 20.448158197s
<IP>: user: warning: [2023-04-22T13:46:41.78764299Z]: [talos] phase dbus (3/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:41.79186799Z]: [talos] task stopDBus (1/1): starting
<IP>: user: warning: [2023-04-22T13:46:41.79627799Z]: [talos] task stopDBus (1/1): done, 4.376094ms
<IP>: user: warning: [2023-04-22T13:46:41.79775699Z]: [talos] phase dbus (3/15): done, 10.113473ms
<IP>: user: warning: [2023-04-22T13:46:41.79927599Z]: [talos] phase stopServices (4/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:41.80081099Z]: [talos] task stopServicesForUpgrade (1/1): starting
<IP>: user: warning: [2023-04-22T13:46:41.80235399Z]: [talos] service[trustd](Stopping): Sending SIGTERM to task trustd (PID 3927, container trustd)
<IP>: user: warning: [2023-04-22T13:46:41.80417199Z]: [talos] service[etcd](Stopping): Sending SIGTERM to task etcd (PID 4011, container etcd)
<IP>: user: warning: [2023-04-22T13:46:41.80599299Z]: [talos] service[udevd](Stopping): Sending SIGTERM to Process(["/sbin/udevd" "--resolve-names=never"])
<IP>: user: warning: [2023-04-22T13:46:41.81211299Z]: [talos] removed static pod {"component": "controller-runtime", "controller": "k8s.StaticPodServerController", "id": "kube-scheduler"}
<IP>: user: warning: [2023-04-22T13:46:41.81500399Z]: [talos] removed static pod {"component": "controller-runtime", "controller": "k8s.StaticPodServerController", "id": "kube-apiserver"}
<IP>: user: warning: [2023-04-22T13:46:41.81776899Z]: [talos] removed static pod {"component": "controller-runtime", "controller": "k8s.StaticPodServerController", "id": "kube-controller-manager"}
<IP>: user: warning: [2023-04-22T13:46:41.82969399Z]: [talos] service[udevd](Finished): Service finished successfully
<IP>: user: warning: [2023-04-22T13:46:41.85388399Z]: [talos] service[trustd](Finished): Service finished successfully
<IP>: user: warning: [2023-04-22T13:46:41.88295699Z]: [talos] service[etcd](Finished): Service finished successfully
<IP>: user: warning: [2023-04-22T13:46:41.88437799Z]: [talos] service[cri](Stopping): Sending SIGTERM to Process(["/bin/containerd" "--address" "/run/containerd/containerd.sock" "--config" "/etc/cri/containerd.toml"])
<IP>: user: warning: [2023-04-22T13:46:41.90433499Z]: [talos] service[cri](Finished): Service finished successfully
<IP>: user: warning: [2023-04-22T13:46:41.90596299Z]: [talos] task stopServicesForUpgrade (1/1): done, 105.150108ms
<IP>: user: warning: [2023-04-22T13:46:41.90778799Z]: [talos] phase stopServices (4/15): done, 108.515319ms
<IP>: user: warning: [2023-04-22T13:46:41.90925699Z]: [talos] phase unmountUser (5/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:41.91084299Z]: [talos] task unmountUserDisks (1/1): starting
<IP>: user: warning: [2023-04-22T13:46:41.91196499Z]: [talos] task unmountUserDisks (1/1): done, 1.121443ms
<IP>: user: warning: [2023-04-22T13:46:41.91302699Z]: [talos] phase unmountUser (5/15): done, 3.775892ms
<IP>: user: warning: [2023-04-22T13:46:41.91404899Z]: [talos] phase unmount (6/15): 2 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:41.91499699Z]: [talos] task unmountOverlayFilesystems (1/2): starting
<IP>: user: warning: [2023-04-22T13:46:41.91648299Z]: [talos] task unmountPodMounts (2/2): starting
<IP>: user: warning: [2023-04-22T13:46:41.92094499Z]: [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/9fce92e4-dddb-4e21-9b80-28b10725e76e/volumes/kubernetes.io~projected/kube-api-access-szbt4
<IP>: user: warning: [2023-04-22T13:46:41.92979099Z]: [talos] task unmountPodMounts (2/2): done, 14.775328ms
<IP>: user: warning: [2023-04-22T13:46:41.94209399Z]: [talos] task unmountOverlayFilesystems (1/2): done, 27.091009ms
<IP>: user: warning: [2023-04-22T13:46:41.94770299Z]: [talos] phase unmount (6/15): done, 33.650111ms
<IP>: user: warning: [2023-04-22T13:46:41.94915699Z]: [talos] phase unmountBind (7/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:41.95065699Z]: [talos] task unmountSystemDiskBindMounts (1/1): starting
<IP>: user: warning: [2023-04-22T13:46:41.95227599Z]: [talos] task unmountSystemDiskBindMounts (1/1): unmounting /system/state
<IP>: kern:  notice: [2023-04-22T13:46:41.95397799Z]: XFS (sda5): Unmounting Filesystem
<IP>: user: warning: [2023-04-22T13:46:41.95814499Z]: [talos] task unmountSystemDiskBindMounts (1/1): unmounting /var
<IP>: kern:  notice: [2023-04-22T13:46:42.63417399Z]: XFS (sda6): Unmounting Filesystem
<IP>: user: warning: [2023-04-22T13:46:42.70283099Z]: [talos] task unmountSystemDiskBindMounts (1/1): done, 752.172004ms
<IP>: user: warning: [2023-04-22T13:46:42.70442299Z]: [talos] phase unmountBind (7/15): done, 755.250814ms
<IP>: user: warning: [2023-04-22T13:46:42.70923699Z]: [talos] phase unmountSystem (8/15): 2 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:42.71493899Z]: [talos] task unmountEphemeralPartition (1/2): starting
<IP>: user: warning: [2023-04-22T13:46:42.72081399Z]: [talos] task unmountStatePartition (2/2): starting
<IP>: user: warning: [2023-04-22T13:46:42.72256399Z]: [talos] task unmountEphemeralPartition (1/2): done, 5.968819ms
<IP>: user: warning: [2023-04-22T13:46:42.72421299Z]: [talos] task unmountStatePartition (2/2): done, 7.661305ms
<IP>: user: warning: [2023-04-22T13:46:42.72550799Z]: [talos] phase unmountSystem (8/15): done, 16.277774ms
<IP>: user: warning: [2023-04-22T13:46:42.73254599Z]: [talos] phase verifyDisk (9/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:42.73418099Z]: [talos] task verifyDiskAvailability (1/1): starting
<IP>: user: warning: [2023-04-22T13:46:42.73599399Z]: [talos] task verifyDiskAvailability (1/1): done, 1.815886ms
<IP>: user: warning: [2023-04-22T13:46:42.73727999Z]: [talos] phase verifyDisk (9/15): done, 4.739775ms
<IP>: user: warning: [2023-04-22T13:46:42.73868999Z]: [talos] phase upgrade (10/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:42.74010899Z]: [talos] task upgrade (1/1): starting
<IP>: user: warning: [2023-04-22T13:46:42.75427899Z]: [talos] task upgrade (1/1): performing upgrade via "ghcr.io/siderolabs/installer:v1.4.0"
<IP>: user: warning: [2023-04-22T13:46:42.83720999Z]: 2023/04/22 13:46:53 running Talos installer v1.4.0
<IP>: kern:  notice: [2023-04-22T13:46:42.84413699Z]: XFS (sda3): Mounting V5 Filesystem
<IP>: kern:    info: [2023-04-22T13:46:42.87810199Z]: XFS (sda3): Ending clean mount
<IP>: kern:  notice: [2023-04-22T13:46:42.88979599Z]: XFS (sda3): Unmounting Filesystem
<IP>: user: warning: [2023-04-22T13:46:42.90160199Z]: 2023/04/22 13:46:53 running pre-flight checks
<IP>: user: warning: [2023-04-22T13:46:42.90579799Z]: 2023/04/22 13:46:53 host Talos version: v1.3.6
<IP>: user: warning: [2023-04-22T13:46:42.90937699Z]: 2023/04/22 13:46:53 host Kubernetes versions: kubelet: 1.26.2, kube-apiserver: 1.26.2, kube-scheduler: 1.26.2, kube-controller-manager: 1.26.2
<IP>: user: warning: [2023-04-22T13:46:42.91984599Z]: 2023/04/22 13:46:53 all pre-flight checks successful
<IP>: user: warning: [2023-04-22T13:46:42.93981599Z]: 2023/04/22 13:46:53 skipped  (EFI) size 204800 blocks
<IP>: user: warning: [2023-04-22T13:46:42.94675199Z]: 2023/04/22 13:46:53 skipped  (BIOS) size 2048 blocks
<IP>: kern:  notice: [2023-04-22T13:46:42.95041699Z]: XFS (sda3): Mounting V5 Filesystem
<IP>: user: warning: [2023-04-22T13:46:42.95349699Z]: 2023/04/22 13:46:53 skipped  (BOOT) size 2048000 blocks
<IP>: user: warning: [2023-04-22T13:46:42.95544599Z]: 2023/04/22 13:46:53 skipped  (META) size 2048 blocks
<IP>: user: warning: [2023-04-22T13:46:42.95699599Z]: 2023/04/22 13:46:53 skipped  (STATE) size 204800 blocks
<IP>: user: warning: [2023-04-22T13:46:42.95847299Z]: 2023/04/22 13:46:53 skipped  (EPHEMERAL) size 416964608 blocks
<IP>: kern:    info: [2023-04-22T13:46:42.98996499Z]: XFS (sda3): Ending clean mount
<IP>: user: warning: [2023-04-22T13:46:42.99976199Z]: 2023/04/22 13:46:53 copying /usr/install/arm64/vmlinuz to /boot/B/vmlinuz
<IP>: user: warning: [2023-04-22T13:46:43.05055799Z]: 2023/04/22 13:46:53 copying /usr/install/arm64/initramfs.xz to /boot/B/initramfs.xz
<IP>: user: warning: [2023-04-22T13:46:43.17761699Z]: 2023/04/22 13:46:53 writing /boot/grub/grub.cfg to disk
<IP>: user: warning: [2023-04-22T13:46:43.37098299Z]: 2023/04/22 13:46:54 executing: grub-install --boot-directory=/boot --efi-directory=/boot/EFI --removable --target=arm64-efi /dev/sda
<IP>: user: warning: [2023-04-22T13:46:43.38096099Z]: Installing for arm64-efi platform.
<IP>: user: warning: [2023-04-22T13:46:44.01632399Z]: Installation finished. No error reported.
<IP>: user: warning: [2023-04-22T13:46:44.03413499Z]: 2023/04/22 13:46:54 META: loaded 0 keys
<IP>: user: warning: [2023-04-22T13:46:44.04608099Z]: 2023/04/22 13:46:54 META: saved 1 keys
<IP>: kern:  notice: [2023-04-22T13:46:44.07171999Z]: XFS (sda3): Unmounting Filesystem
<IP>: user: warning: [2023-04-22T13:46:44.08374699Z]: 2023/04/22 13:46:54 installation of v1.4.0 complete
<IP>: user: warning: [2023-04-22T13:46:44.12396199Z]: [talos] task upgrade (1/1): upgrade successful
<IP>: user: warning: [2023-04-22T13:46:44.12501299Z]: [talos] task upgrade (1/1): done, 1.384907972s
<IP>: user: warning: [2023-04-22T13:46:44.12621399Z]: [talos] phase upgrade (10/15): done, 1.387531381s
<IP>: user: warning: [2023-04-22T13:46:44.12731399Z]: [talos] phase mountBoot (11/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:44.12828899Z]: [talos] task mountBootPartition (1/1): starting
<IP>: kern:  notice: [2023-04-22T13:46:44.15001099Z]: XFS (sda3): Mounting V5 Filesystem
<IP>: kern:    info: [2023-04-22T13:46:44.18671999Z]: XFS (sda3): Ending clean mount
<IP>: user: warning: [2023-04-22T13:46:44.19191499Z]: [talos] task mountBootPartition (1/1): done, 63.62037ms
<IP>: user: warning: [2023-04-22T13:46:44.19303099Z]: [talos] phase mountBoot (11/15): done, 65.721017ms
<IP>: user: warning: [2023-04-22T13:46:44.19401999Z]: [talos] phase kexec (12/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:44.19489299Z]: [talos] task kexecPrepare (1/1): starting
<IP>: user: warning: [2023-04-22T13:46:44.52244399Z]: [talos] prepared kexec environment kernel="/boot/B/vmlinuz" initrd="/boot/B/initramfs.xz" cmdline="talos.platform=nocloud talos.config=none console=tty1 console=ttyS0 init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512"
<IP>: user: warning: [2023-04-22T13:46:44.52957899Z]: [talos] task kexecPrepare (1/1): done, 334.682105ms
<IP>: user: warning: [2023-04-22T13:46:44.53272399Z]: [talos] phase kexec (12/15): done, 338.705359ms
<IP>: user: warning: [2023-04-22T13:46:44.53545299Z]: [talos] phase unmountBoot (13/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:44.53780699Z]: [talos] task unmountBootPartition (1/1): starting
<IP>: kern:  notice: [2023-04-22T13:46:44.55335199Z]: XFS (sda3): Unmounting Filesystem
<IP>: user: warning: [2023-04-22T13:46:44.55892699Z]: [talos] task unmountBootPartition (1/1): done, 21.11491ms
<IP>: user: warning: [2023-04-22T13:46:44.56518799Z]: [talos] phase unmountBoot (13/15): done, 29.739578ms
<IP>: user: warning: [2023-04-22T13:46:44.57168599Z]: [talos] phase stopEverything (14/15): 1 tasks(s)
<IP>: user: warning: [2023-04-22T13:46:44.57351499Z]: [talos] task stopAllServices (1/1): starting

Machine
The debug console just dumps those lines infinitely:
[  586.629204] [talos] failed looking up "pool.ntp.org", ignored {"component": "controller-runtime", "controller": "time.SyncController", "error": "lookup pool.ntp.org on 8.8.8.8:53: dial udp 8.8.8.8:53: connect: network is unreachable"}

Environment

Talos version: [talosctl version --nodes <problematic nodes>]
Kubernetes version: [kubectl version --short]
Platform: Oracle/arm64

Duplicate issue #7119: Talos 1.4.0 breaks on Hetzner Cloud with Ampere servers
Created: 2023-04-21T11:12:18Z
Bug Report
Description
Now that Hetzner released their new Ampere-based servers, I wanted to play around with them, including setting up a small k8s cluster to verify multiplatform image availability.
While Talos 1.3.7 worked just perfectly, upgrading to 1.4.0 or installing it from zero both results in the nodes failing to boot.
Upgrade path

Install Talos 1.3.7 on a cax11 server. I used the Packer file from the documentation except for downloading the hcloud-arm64.tar.xz asset and using the cax11 server for provisioning. var.talos_version is set to 1.3.7.
Run talosctl bootstrap on the node and wait for everything to come up.
Run talosctl upgrade  --image ghcr.io/siderolabs/installer:v1.4.0 --preserve=true
The upgrade fails with the following logs and the server fails to boot from this point on


Installation path

Install Talos 1.4.0 on a cax11 server. I used the Packer file from the documentation except for downloading the hcloud-arm64.tar.xz asset and using the cax11 server for provisioning. var.talos_version is set to 1.4.0.
The server never boots up, there is never any display output to fetch logs.

Environment

Platform: Hetzner Cloud - Ampere

Closing comment (2023-04-25T10:38:27Z): @frezbo I wonder if it's same as #7123 with modules not being loaded for virtio console (?)