Repository: k3s-io/k3s

Issue #802: Removing node doesn't remove node password
Created: 2019-09-11T09:13:43Z
I'm not sure if that's the right place for the bug report, because the error message I got has only one google results and it's pointing to commit message that added password validation below, so here it is.
I have a few rpis: 1, 0W. I installed hypriot on them and after installation of k3s changed some of their hostnames. I changed hostname of black-pearl to rpi1, removed black-pearl node from k3s-server, created another black-pearl on RPI 0W and here comes the problem, k3s of rpi0 (black-pearl) couldn't join cluster because password didn't match:
k3s-agent:
level=info msg="Running load balancer 127.0.0.1:41241 ->[k3s.local:6443]"
level=error msg="Node password rejected, contents of '/var/lib/rancher/k3s/agent/node-password.txt' may not match server passwd entry"
level=error msg="Node password rejected, contents of '/var/lib/rancher/k3s/agent/node-password.txt' may not match server passwd entry"
level=error msg="Node password rejected, contents of '/var/lib/rancher/k3s/agent/node-password.txt' may not match server passwd entry"
level=error msg="Node password rejected, contents of '/var/lib/rancher/k3s/agent/node-password.txt' may not match server passwd entry"
level=error msg="Node password rejected, contents of '/var/lib/rancher/k3s/agent/node-password.txt' may not match server passwd entry"

I spent some time trying to fix it and noticed that old password for black-pearl which is now rpi1 is still in
/var/lib/rancher/k3s/server/cred/node-passwd despite running kubectl delete black-pearl.
Seems that removing node should also remove password for that node in case another node with same hostname (OS is reinstalled?) re-joins the cluster.

Duplicate issue #872: k3s master does a not allow re-entry of worker node once deleted.
Created: 2019-10-07T19:03:03Z
Version:
k3s version v0.9.1 (755bd1c)
Describe the bug
Post delete of node in k3s cluster. node unable to re-join.
Same token as used to join previously.
rpi4 4GB version
32 GB microSD
Raspbian Buster Lite 2019-07-10
worker node was joined twice (same ip) just hostname error.
Master allowed both to exist in cluster at same time (same ip, different hostname)
pi@kmaster:~ $ sudo kubectl get nodes
NAME        STATUS     ROLES    AGE   VERSION
worker01    NotReady   worker   88d   v1.15.4-k3s.1
kworker02   Ready      worker   88d   v1.15.4-k3s.1
kmaster     Ready      master   88d   v1.15.4-k3s.1
kworker01   Ready      worker   88d   v1.15.4-k3s.1
pi@kmaster:~ $ sudo kubectl drain kworker01
node/kworker01 cordoned
error: unable to drain node "kworker01", aborting command...
There are pending nodes to be drained:
kworker01
error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/svclb-traefik-9pc65
pi@kmaster:~ $ sudo kubectl drain kworker01
node/kworker01 already cordoned
error: unable to drain node "kworker01", aborting command...
A clear and concise description of what the bug is.
NAME        STATUS     ROLES    AGE   VERSION
worker01    NotReady   worker   88d   v1.15.4-k3s.1
kworker02   Ready      worker   88d   v1.15.4-k3s.1
kmaster     Ready      master   88d   v1.15.4-k3s.1
kworker01   Ready      worker   88d   v1.15.4-k3s.1
pi@kmaster:~ $ sudo kubectl drain kworker01
node/kworker01 cordoned
error: unable to drain node "kworker01", aborting command...
There are pending nodes to be drained:
kworker01
error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/svclb-traefik-9pc65
pi@kmaster:~ $ sudo kubectl drain kworker01
node/kworker01 already cordoned
error: unable to drain node "kworker01", aborting command...
There are pending nodes to be drained:
kworker01
error: cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/svclb-traefik-9pc65
i@kmaster:~ $ sudo kubectl get nodes
NAME        STATUS                        ROLES    AGE   VERSION
worker01    NotReady                      worker   88d   v1.15.4-k3s.1
kworker01   NotReady,SchedulingDisabled   worker   88d   v1.15.4-k3s.1
kworker03   Ready                         worker   67s   v1.15.4-k3s.1
kworker02   Ready                         worker   88d   v1.15.4-k3s.1
kmaster     Ready                         master   88d   v1.15.4-k3s.1
pi@kmaster:~ $ sudo kubectl get nodes
NAME        STATUS                        ROLES    AGE    VERSION
worker01    NotReady                      worker   88d    v1.15.4-k3s.1
kworker01   NotReady,SchedulingDisabled   worker   88d    v1.15.4-k3s.1
kworker02   Ready                         worker   88d    v1.15.4-k3s.1
kmaster     Ready                         master   88d    v1.15.4-k3s.1
kworker03   Ready                         worker   101s   v1.15.4-k3s.1
pi@kmaster:~ $ sudo kubectl delete node worker01
node "worker01" deleted
pi@kmaster:~ $ sudo kubectl get nodes
NAME        STATUS                        ROLES    AGE    VERSION
kworker01   NotReady,SchedulingDisabled   worker   88d    v1.15.4-k3s.1
kworker03   Ready                         worker   2m2s   v1.15.4-k3s.1
kworker02   Ready                         worker   88d    v1.15.4-k3s.1
kmaster     Ready                         master   88d    v1.15.4-k3s.1
To Reproduce
Expected behavior
After running
delete node.
Should you have to manually un-install sh on worker node?
Should it not try to run it by itself?
I ran it manually on worker node1 after deleting it.
pi@kworker01:~ $ sudo /usr/local/bin/k3s-agent-uninstall.sh

id -u
[ 0 -eq 0 ]
/usr/local/bin/k3s-killall.sh
id -u
[ 0 -eq 0 ]
[ -d /var/lib/rancher/k3s/data/43998d048ba0336d929ad6cb3c2308722839fea65bdbedfd818422e2daf2e201/bin/ ]
export PATH=/var/lib/rancher/k3s/data/43998d048ba0336d929ad6cb3c2308722839fea65bdbedfd818422e2daf2e201/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[ -s /etc/systemd/system/k3s-agent.service ]
basename /etc/systemd/system/k3s-agent.service
systemctl stop k3s-agent.service
[ -x /etc/init.d/k3s* ]
lsof
sed -e s/^[^0-9]*//g; s/  */\t/g
sort -n -u
grep -w k3s/data/[^/]*/bin/containerd-shim
cut -f1
killtree
[ 0 -ne 0 ]
do_unmount /run/k3s
cat /proc/self/mounts
awk {print $2}


sort -r
grep ^/run/k3s


MOUNTS=
[ -n  ]
do_unmount /var/lib/rancher/k3s
cat /proc/self/mounts
awk {print $2}
grep ^/var/lib/rancher/k3s
sort -r
MOUNTS=
[ -n  ]
ip link show
grep master cni0


sed -e s|@.*||
awk -F:  {print $2}


nets=
ip link delete cni0
Cannot find device "cni0"
ip link delete flannel.1
Cannot find device "flannel.1"
rm -rf /var/lib/cni/
which systemctl
/bin/systemctl
systemctl disable k3s-agent
Removed /etc/systemd/system/multi-user.target.wants/k3s-agent.service.
systemctl reset-failed k3s-agent
systemctl daemon-reload
which rc-update
rm -f /etc/systemd/system/k3s-agent.service
rm -f /etc/systemd/system/k3s-agent.service.env
trap remove_uninstall EXIT
[ -L /usr/local/bin/kubectl ]
rm -f /usr/local/bin/kubectl
[ -L /usr/local/bin/crictl ]
rm -f /usr/local/bin/crictl
[ -L /usr/local/bin/ctr ]
rm -f /usr/local/bin/ctr
rm -rf /etc/rancher/k3s
rm -rf /var/lib/rancher/k3s
rm -f /usr/local/bin/k3s
rm -f /usr/local/bin/k3s-killall.sh
remove_uninstall
rm -f /usr/local/bin/k3s-agent-uninstall.sh

Actual behavior
rebooted showed node not in cluster.
attempt to rejoin failed.
Same token as previous cluster, because clusterID and token didn't change.
Additional context
Add any other context about the problem here.

Closing comment (2020-10-22T16:15:59Z): Some of the requests here sound like a dupe of #802, which will be fixed soon.
The original ask is to run the uninstall script on the node when deleted, which I am not sure is something we want to do as things currently are, since that depends on having installed with the install script. If install/uninstall was built into the k3s binary itself that would be a more accessible solution, probably worth exploring.