TCP packets traffic visualization for kubernetes by k8spacket and Grafana
Tags: devops, kubernetes, observability, tcp
Published: 2022-08-09 11:18:00

Do you know what your k8s cluster does when you are not watching? Who establishes the TCP communication with him? Who is called by him, e.g., from third-party libraries?

With `k8spacket` and `Grafana`, you can visualize TCP traffic in the cluster. See how your workloads communicate together. Inspect how many connections are established, how many bytes are exchanged, and how long these connections are active.
&nbsp;

###Description

`k8spacket` is the tool written in Golang and uses the `gopacket` third-party library to sniff TCP packets on your workloads (incoming and outgoing). It creates TCP listeners on running container network interfaces. When Kubernetes creates a new container, the CNI plugin is responsible for giving the possibility to communicate either with other containers or from cluster to the world. The most common approach uses `linux namespace` to isolate networks and `veth pair` to connect the isolated namespace with the bridge. If you want to know more, please refer to a great article written by Safak ([How Docker Container Networking Works - Mimic It Using Linux Network Namespaces](https://dev.to/polarbit/how-docker-container-networking-works-mimic-it-using-linux-network-namespaces-9mj)). Besides type `bridge`, the CNI plugin can use other types (`vlan`, `ipvlan`, `macvlan`), but all create a network interface for container linux namespace, which is the main handle for `k8spacket` sniffers. 

`k8spacket` is running with the option 
```
hostNetwork: true
``` 
to eliminate the isolation mentioned earlier. It collects TCP streams, processes data, and exposes results through API to be used by `Grafana`.

Additionally, 
* `k8spacket` is a Kubernetes API client and can resolve sniffed workload as a cluster resource name (`Pods` and `Services`) visible on visualization. 
* It is launched as a `DaemonSet` to listen to network interfaces on all nodes. 
* Implemented watcher checks every `10s` (default) if there is a new (or old) network interface to observe (or forget). 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wjw1gxoq4z1j8czr1iao.png)

In addition to showing a graph of your cluster resources, it also exposes `Prometheus` metrics.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8zj4keb5nlxyg2w31yg9.png)
&nbsp;

###Installation

To install `k8spacket`, you need to have Helm and Grafana installed. Additionally, the tool requires Hamad's Node Grafana API plugin ([Node Graph API](https://grafana.com/grafana/plugins/hamedkarbasi93-nodegraphapi-datasource/)). 
See all installation steps [here](https://k8spacket.github.io/k8spacket/#installation).
&nbsp;

###Usage

####Types of stats

* `connection` - helps understand how many connections are established between your workloads and from/to outside clients. It tells you which sockets stay open and could cause problems. It is helpful when you are behind some SNAT gateway with port exhaustion (e.g., `Azure AKS`).
* `bytes` - shows how many bytes are sent or received by workloads.
* `duration` - calculates the lifetime of connections.


![Graph modes](https://raw.githubusercontent.com/k8spacket/k8spacket/master/docs/graphmode.gif)

####Filters

* `by namespace` - select one or more k8s namespaces

![Namespace](https://raw.githubusercontent.com/k8spacket/k8spacket/master/docs/namespace.gif)

* `by names included` - select workload names for visualization
* `by names excluded` - exclude workload names from visualization

![Include & exlude](https://raw.githubusercontent.com/k8spacket/k8spacket/master/docs/includeexclude.gif)

###Summary

If you want to know what is going on with TCP traffic in your cluster, try `k8spacket`. Investigation about cutting the traffic (`network policy`) could be painless when you see a leak of packages outside. Make a review of third-party libraries to check connections with the world.

The source code can be found at [github.com/k8spacket](https://github.com/k8spacket/k8spacket)