Repository: k3s-io/k3s

Issue #3117: High load due to ksoftirqd, growing iptables rules
Created: 2021-03-25T08:29:58Z
Environmental Info:
K3s Version: k3s version v1.20.4+k3s1 (838a906)
go version go1.15.8
Node(s) CPU architecture, OS, and Version: Linux 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64 GNU/Linux
Cluster Configuration: 1 master, 2 workers
Describe the bug: After some time we get high loads one the machine due to high soft irqs:

Output of perf report:

Something goes wrong with the iptables rules:
iptables -L produces 7.0 MB of rules (increasing more and more over time):

Steps To Reproduce:


Installed K3s:  We are using the embed etcd.

Duplicate issue #1597: Raspbian 10 fresh install has broken routing (iptables/nf_tables detection)
Created: 2020-03-28T14:59:04Z
Version:
k3s version v1.17.4+k3s1 (3eee8ac) on a raspberry pi 4 running Raspbian 10.
K3s arguments:
curl -sfL https://get.k3s.io | sh -
Describe the bug
On a fresh install, no traffic is routed inside the cluster, even for core services. Resolution is to uninstall the default iptables v1.8.2 (nf_tables), and install nftables.
On a fresh install, no traffic is routed inside the cluster. nodes cannot reach each other or coredns. Core services can't reach each other or the api. Ports are not opened on the physical host.
The host is not listening on port 80. Traefik LB reports that it is listening on port 80, but sudo netstat -tlp |grep 80 disagrees. External hosts cannot access created ingresses.
To Reproduce

Install k3s on Raspbian 10.
Run shell in a dnsutils container:  kubectl run -it --rm --restart=Never dnsutils --image=gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 sh
Inside that container, run wget -O- github.com, or wget -O- kubernetes.default and observe "invalid name" errors. Try pinging any IP you please - the DNS server, external IPs - and observe failures.

Expected behavior
Traffic inside the cluster should be routed.
Actual behavior
No traffic is routed inside the cluster. Services (even kube-system) can't reach each other, nothing can reach the API server, etc.
First symptom I noticed was that ingresses failed to open port 80, and services couldn't reach their pods.
Additional context / logs
Fresh uninstall/reinstall on a raspbian host with IP 192.168.1.41:
$ sudo kubectl get all -n kube-system
NAME                                          READY   STATUS      RESTARTS   AGE
pod/metrics-server-6d684c7b5-w7swt            1/1     Running     0          25m
pod/coredns-6c6bb68b64-tqs4d                  1/1     Running     0          25m
pod/helm-install-traefik-pvsvx                0/1     Completed   0          25m
pod/svclb-traefik-pr5cn                       2/2     Running     0          22m
pod/traefik-7b8b884c8-826lt                   1/1     Running     0          22m
pod/local-path-provisioner-58fb86bdfd-cdjwd   1/1     Running     2          25m

NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
service/kube-dns             ClusterIP      10.43.0.10      <none>         53/UDP,53/TCP,9153/TCP       25m
service/metrics-server       ClusterIP      10.43.106.186   <none>         443/TCP                      25m
service/traefik-prometheus   ClusterIP      10.43.118.104   <none>         9100/TCP                     22m
service/traefik              LoadBalancer   10.43.138.141   192.168.1.41   80:30192/TCP,443:31737/TCP   22m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/svclb-traefik   1         1         1       1            1           <none>          22m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/metrics-server           1/1     1            1           25m
deployment.apps/coredns                  1/1     1            1           25m
deployment.apps/traefik                  1/1     1            1           22m
deployment.apps/local-path-provisioner   1/1     1            1           25m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/metrics-server-6d684c7b5            1         1         1       25m
replicaset.apps/coredns-6c6bb68b64                  1         1         1       25m
replicaset.apps/traefik-7b8b884c8                   1         1         1       22m
replicaset.apps/local-path-provisioner-58fb86bdfd   1         1         1       25m

NAME                             COMPLETIONS   DURATION   AGE
job.batch/helm-install-traefik   1/1           2m24s      25m

$ sudo kubectl describe service traefik -n kube-system
Name:                     traefik
Namespace:                kube-system
Labels:                   app=traefik
                          chart=traefik-1.81.0
                          heritage=Helm
                          release=traefik
Annotations:              <none>
Selector:                 app=traefik,release=traefik
Type:                     LoadBalancer
IP:                       10.43.138.141
LoadBalancer Ingress:     192.168.1.41
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30192/TCP
Endpoints:                10.42.0.6:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31737/TCP
Endpoints:                10.42.0.6:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

$ sudo kubectl describe ingress nginx
Name:             nginx
Namespace:        default
Address:          192.168.1.41
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host                Path  Backends
  ----                ----  --------
  nginx.cluster.vert
                      /   nginx:80 (<none>)
Annotations:
Events:  <none>

$ sudo netstat -tlpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8125          0.0.0.0:*               LISTEN      461/netdata
tcp        0      0 0.0.0.0:19999           0.0.0.0:*               LISTEN      461/netdata
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      2754/k3s
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2754/k3s
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      2754/k3s
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      2754/k3s
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      523/sshd
tcp        0      0 127.0.0.1:10010         0.0.0.0:*               LISTEN      2821/containerd
tcp6       0      0 ::1:8125                :::*                    LISTEN      461/netdata
tcp6       0      0 :::19999                :::*                    LISTEN      461/netdata
tcp6       0      0 :::10250                :::*                    LISTEN      2754/k3s
tcp6       0      0 :::10251                :::*                    LISTEN      2754/k3s
tcp6       0      0 :::6443                 :::*                    LISTEN      2754/k3s
tcp6       0      0 :::10252                :::*                    LISTEN      2754/k3s
tcp6       0      0 :::30192                :::*                    LISTEN      2754/k3s
tcp6       0      0 :::22                   :::*                    LISTEN      523/sshd
tcp6       0      0 :::31737                :::*                    LISTEN      2754/k3s

This all started with a power loss/reboot of my working pi cluster, after an apt update.
See my eventual resolution. Seems to me that it was applying rules to both nftables and iptables-legacy, and there was some conflict.
UPDATE: changed focus now that I know routing is completely borked.
UPDATE 2: rewrite title/description after I discovered/resolved the problem. Left open because I believe it will affect other Raspbian 10 users and could probably use a PR to improve iptables vs nftables behavior.

Closing comment (2021-09-01T17:02:40Z): @virtualstaticvoid possibly related to #3117 (comment)