Repository: k3s-io/k3s

Issue #7183: IPVS is non-functional in 1.26
Created: 2023-03-30T19:30:26Z
Environmental Info:
K3s Version:
k3s version v1.26.3+k3s1 (01ea3ff)
go version go1.19.7
Node(s) CPU architecture, OS, and Version:
Debian 11, amd64, hetzner cloud VPS
Linux debian-2gb-hil-1 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
Cluster Configuration:
Single node
Describe the bug:
Starting with 1.26 having IPVS enabled completely breaks the cluster's traffic flow. As soon as IPVS is enabled and the firewall rules/ipvsadm policies kick in, the internal cluster communication breaks in addition to external traffic to rather important ports such as port 22 and port 6443
No pods are able to spin up, attempting to install calico results in the pod not being able to communicate with the API server and the cluster is in a largely failed state.
Steps To Reproduce:
Spin up a fresh Debian 11 VPS -- I used hetzner for this test
Install requisites for ipvs:
apt install ipset ipvsadm

Install k3s and enable ipvs all at once
curl -sfL https://get.k3s.io | sh -s - server --kube-proxy-arg proxy-mode=ipvs

Wait for initialization -- as soon as iptables ipvs policies are enacted, the server is now broken an unable to accept new ingress connections.
Expected behavior:
The cluster works.
I can ssh into my machine with ipvs enabled
I can use the kube api externally with ipvs enabled
I can start pods (i.e. Calico) and finish building up my cluster (note, flannel was left enabled for this test to prove how simple the error is to reproduce)
Actual behavior:
No new connections can be created.
Cannot create a new SSH connection.
Cannot connect to the kube api server from an external connection
Cannot ping the external IP from the command line of the server itself or an external IP
Cannot spin up new pods or install calico
Additional context / logs:
I do not have such issues with IPVS in my vanilla k8s cluster running 1.26.1
The issue was not present in 1.25. Downgrading a broken machine to the latest 1.25 branch results in the behavior disappearing.
It appears the difference between 1.25 and 1.26 is coming down to iptables rules
These iptables rules appear new as of 1.26 (they are not present in k3s 1.25.8):
-A KUBE-IPVS-FILTER -m set --match-set KUBE-LOAD-BALANCER dst,dst -j RETURN
-A KUBE-IPVS-FILTER -m set --match-set KUBE-CLUSTER-IP dst,dst -j RETURN
-A KUBE-IPVS-FILTER -m set --match-set KUBE-EXTERNAL-IP dst,dst -j RETURN
-A KUBE-IPVS-FILTER -m conntrack --ctstate NEW -m set --match-set KUBE-IPVS-IPS dst -j REJECT --reject-with icmp-port-unreachable

Removing the rule allows remote connections to once again be initiated:
-A KUBE-IPVS-FILTER -m conntrack --ctstate NEW -m set --match-set KUBE-IPVS-IPS dst -j REJECT --reject-with icmp-port-unreachable

My vanilla 1.26.1 cluster has the same rules and has none of these issues.
Finally, the issue appears to have silently snuck in during my node's automatic update to 1.26 (my mistake). It's curious to note that nginx ingress traffic on port 80 and 443 were unaffected while the server was still running (though ssh/kube api was already dead) -- that is until I rebooted my server and calico was unable to start, which cascaded into a full failure because no services were able to start upon a reboot.

Duplicate issue #8946: Enabling ipvs on k3s resulted in SSH connection failure
Created: 2023-11-23T09:38:52Z
Environmental Info:
K3s Version:

k3s version v1.26.10+k3s1 (a6687b5)
go version go1.20.10
Node(s) CPU architecture, OS, and Version:

Linux k3s01 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
Cluster Configuration:

1 servers
Describe the bug:

Enabling ipvs on k3s resulted in SSH connection failure
After enabling ipvs, a virtual network card kube-ipvs0 will be added with an IP address 192.168.0.231/32, which conflicts with 192.168.0.231/24 on the eth0 network card, resulting in SSH connection failure
Steps To Reproduce:


Installed K3s:

Expected behavior:

Actual behavior:

Additional context / logs:

Closing comment (2023-11-27T23:45:42Z): I believe this is a duplicate of #7183. This was traced back to upstream issues kubernetes/kubernetes#117613 and kubernetes/kubernetes#117621, and fixed in kubernetes/kubernetes#115019.
As per comments on that thread, the fix wasn't backported to Kubernetes 1.26, so you'll need to upgrade to 1.27 or newer.