Kubernetes namespaces you should never miss with.
Tags: kubernetes, cloud, devops, sre
Published: 2021-10-19 21:17:15

Kubernetes is shipped with many namespaces some of them are critical for Kubernetes in order to function correctly.
Messing around in one of these namespaces can damage the Kubernetes system.


And these are:
- `default` the home of the homeless
- `kube-system` The namespace for objects created by the Kubernetes system
- `kube-public` This namespace is created automatically and is readable by all users (including those not authenticated). This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. This is useful for exposing any cluster information necessary to bootstrap components. It is primarily managed by Kubernetes itself.
- `kube-node-lease` This namespace holds Lease objects associated with each node. Node leases allow the kubelet to send heartbeats so that the control plane can detect node failure.


All Kubernetes system's namespaces will be regenerated again even when you delete them accidentally that's what Kubernetes components will try to do.

But sometimes if you are unfortunate enough deleting namespaces might be stuck at the terminating stage giving no option to regenerate the namespace again.

So the below cases mention what is the importance of each namespace in order to know what the symptoms might look like.

### What is the `default`?
`default` namespace is used as the default place of any objects you create with no namespace specified.
### What is the `kube-system`?
`Kube-system` is the namespace for objects and service accounts with high-level privileges within Kubernetes. Utilization of the Kubernetes controller stems from this namespace, in other words we will have some issues with the controllers and maybe in deploying new pods/deployment.

not only that this namespace also contains other important objects such as `kube-dns`,`kube-proxy`,

`kube-dns` is the authoritative name server for the cluster domain (cluster.local) and it resolves external names recursively. Short names that are not fully qualified, such as myservice, are completed first with local search paths. more details [here](https://cloud.google.com/kubernetes-engine/docs/how-to/kube-dns) & [here](https://www.digitalocean.com/community/tutorials/an-introduction-to-the-kubernetes-dns-service)

`kube-proxy` manages the forwarding of traffic addressed to the virtual IP addresses (VIPs) of the cluster’s Kubernetes Service objects to the appropriate backend pods more details [here](https://www.tigera.io/blog/comparing-kube-proxy-modes-iptables-or-ipvs/) & [here](https://arthurchiao.art/blog/cracking-k8s-node-proxy/)

This means you will have trouble with resolving external/internal communications.

### How about the `kube-public`?
`kube-public` contains a single ConfigMap object, cluster-info, that aids discovery and security bootstrap.





All the above namespaces if you tried to delete them the server will respond with
```
Error from server (Forbidden): namespaces "kube-public" is forbidden: this namespace may not be deleted
```
Expect the lucky `kube-node-release` which was added in Kubernetes v1.14, would get deleted the same as any normal namespace.


### What is `kube-node-release`?
`kube-node-lease` This namespace holds Lease objects associated with each node. Node leases allow the kubelet to send heartbeats so that the control plane (Node controller) can detect node failure.

So what would happen if we deleted `kube-node-lease`? usually, Kubernetes will create another one with Lease object for each node, but sometimes the namespace removal gets stuck at the `terminating` status.
Which by then we will have a node Lease with an outdated heartbeat which might tell the Node controller that this node is not reachable and impact the overall communication between the nodes.

### How to fix namespace removal stuck at `terminating`?
Of course, you can try to figure out why the namespace is stuck at terminating but sometimes you can't so here we go with the **force delete**




1-  create a temp json file
```
 kubectl get namespace <terminating-namespace> -o json >tmp.json
```

2-  execute the below
```
$ kubectl proxy
Starting to serve on 127.0.0.1:8001
```

3-  Edit your tmp.json file. Remove the Kubernetes value from the finalizers field and save the file.

4-  Make the below command to update the namespace
 ```
 curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/<terminating-namespace>/finalize
 ```



5-  Your output will look something like the one below
  ```
{
   "kind": "Namespace",
   "apiVersion": "v1",
   "metadata": {
     "name": "<terminating-namespace>",
     "selfLink": "/api/v1/namespaces/<terminating-namespace>/finalize",
     "uid": "b50c9ea4-ec2b-11e8-a0be-fa163eeb47a5",
     "resourceVersion": "1602981",
     "creationTimestamp": "2021-10-18T18:48:30Z",
     "deletionTimestamp": "2021-10-18T18:59:36Z"
   },
   "spec": {

   },
   "status": {
     "phase": "Terminating"
   }
}
  ```