Use Tetragon to Limit Network Usage for a set of Binary
Tags: tetragon, security, ebpf, workstations
Published: 2023-08-03 07:39:02

---
title: Use Tetragon to Limit Network Usage for a set of Binary
published: true
description: 
tags: tetragon,security,ebpf,workstation
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ed8oizv8ac880axjs3ye.png
# published_at: 2023-08-03 09:38 +0000
---
## A matter of trust
Many interesting software are coming from the community, many are distributed through the package manager of the operating system. But for the others, you can download them from Github release pages, use snap or homebrew to cite a few. But this last installation method bypasses the security team that tries to improve the security of your operating system. By doing so, you are implicitly trusting the author he is not distributing malware or implementing backdoors. How many tools did you install by hand? Do you really trust all of them? Confidence is very important, yet it would be nice to limit capabilities for a set of binary that you don't fully trust. In this blog post, we will use [Tetragon](https://github.com/cilium/tetragon)  to forbid network usage for tools that don't need to.
## Goal
We will separate tools installed locally into two families. On one side, tools that need network access in a specific directory. Another directory for tools that don't need internet access. 

For example the following tools use network sockets:

* [k9s](https://github.com/derailed/k9s)
* [kubectl](https://kubernetes.io/fr/docs/tasks/tools/install-kubectl/)
* [helm](https://helm.sh/)

while these do not: 

* [jq](https://jqlang.github.io/jq/)
* [yq](https://github.com/mikefarah/yq)
* [jless](https://jless.io/)

We will move the last tools in a specific directory `~/bin-no-network/` and use [Tetragon](https://github.com/cilium/tetragon) to inject a policy in the kernel as a [eBPF](https://ebpf.io/) program to kill any binary located in `~/bin-no-network/` trying to open a network socket.
## Tetragon installation
Tetragon is "Kubernetes-aware" but it can also be used outside Kubernetes on a regular workstation. You can deploy Tetragon as a container:

```bash
$ docker run --name tetragon --rm -d                 \
    --pid=host --cgroupns=host --privileged          \
    -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \
    quay.io/cilium/tetragon:v0.10.0
```

Because Tetragon needs to inject code in the kernel, we need to bypass most of the docker isolation mechanism. We only use the packaging feature of docker to avoid installation of system libraries and the binary. So you will have to trust ;-) this tetragon binary because it will run like a process running as root on your workstation. Note the mount point: `/sys/kernel/btf/vmlinux`, this is a kind of bridge to eBPF kernel features.
## eBPF Principle
When using eBPF, applications are always composed of two parts, one deployed in the Kernel as an eBPF program and another running as a regular program in "user space". The user space program (Tetragon) will inject a small eBPF program in the kernel to intercept some system calls. This means that every application that uses this system call will trigger this eBPF program that can observe or modify the system call result. A specific data structure is then created in the kernel: a ring buffer. This is used by the eBPF program to store some interesting data. Finally, the user space program Tetragon, which has also read-only access to this data structure, will be able to retrieve information gathered by the eBPF program.
One good point with this architecture is that evaluation of rules is done within the kernel and does not require communication with the user space program. The only drawback is that information observed by the eBPF program can be overridden by new incoming information if the user space part does not read information fast enough. This is how ring buffers are designed.
## Writing a Policy
Our goal is to forbid network usage coming from binaries in a specific folder.
For this we will need a CLI to interact with tetragon. We can use the `tetra` binary in the docker container:

```bash
$ alias tetra="docker exec -ti tetragon tetra"
$ tetra version
server version: v0.10.0
cli version: v0.10.0
```

Even if we are not using Kubernetes, we still need to write some yaml as Tetragon only understands TracingPolicy objects. A TracingPolicy is a kubernetes custom resource to install hooks in the kernel and actions.

Let's start with a TracingPolicy from the [Tetragon documentation](https://tetragon.cilium.io/docs/use-cases/network-observability/): 

```yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
```
This Tracing policy will just observe network events:

* socket creation (tcp_connect)
* traffic in the socket (tcp_sendmsg)
* socket close (tcp_close)

Now we need to add: 

* an action when this kind of event is detected
* a filter to only apply this action if this is generated from a binary located in our specific directory `~/bin-no-network/`.

We need to add a `selectors` section in the TracingPolicy and use the `matchBinaries` selector to apply this policy only for binary in the `~/bin-no-network/` folder:

```yaml
    selectors:
    - matchBinaries:
      - operator: "In"
        values:
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"
```
Unfortunately, only 'In' and 'NotIn' operator are [implemented](https://github.com/cilium/tetragon/pull/686/files#diff-0b6c66cf370068fe2e441466d6124582732e8b67dd0daddccf72c7eb2bef16d4R994-R1000) for the `matchBinaries` selector. [We cannot use the 'Prefix' operator](https://github.com/cilium/tetragon/issues/1278) like this:
```yaml
    selectors:
    - matchBinaries:
      - operator: "Prefix"
        values:
          - "/home/jacroute/bin-no-network/"
```

So the TracingPolicy should look like:

```yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: &selector
    - matchBinaries:
      - operator: "In"
        values:
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"
      matchActions:
      - action: Sigkill
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: *selector
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
    selectors: *selector
```
## Deploy the TracingPolicy
We first need to transfer the file to the container. Suppose that you stored the TracingPolicy in `bin-no-network.yaml` file, you can transfer the policy using the `docker cp` command:

```bash
$ docker cp bin-no-network.yaml tetragon:/tmp/bin-no-network.yaml
```

Then we can use the `tetra` cli to deploy this policy:

```bash
tetra tracingpolicy add /tmp/bin-no-network.yaml
```
## Testing the TracingPolicy
Now we can test if the policy is blocking network access for the two binaries listed in the policy.

```bash
$ mkdir ~/bin-no-network/
$ cp /usr/bin/curl ~/bin-no-network/
$ cp /usr/bin/jq ~/bin-no-network/
$ ~/bin-no-network/curl google.fr
[1] 122677 killed ~/bin-no-network/curl google.fr
$ echo "{}" | ~/bin-no-network/jq
{}
```

`curl` tried to open a socket but the process was killed. `jq` does not try this kind of system call and was not killed.
## Automatically populating the Policy
We can build a shell script to maintain the list of binaries synchronized with the content of the `~/bin-no-network/` directory. Using `find` we can list binaries in this folder:

```bash
$ find ~/bin-no-network/ -executable -type f
/home/jacroute/bin-no-network/curl
/home/jacroute/bin-no-network/jq
```


Then with awk with can add surrounding spaces and quotes needed to integrate the yaml:

```bash
find ~/bin-no-network/ -executable -type f | awk '{ print "          - \""$0"\""}'
          - "/home/jacroute/bin-no-network/curl"
          - "/home/jacroute/bin-no-network/jq"
```

Finally, integrate this in the yaml and inject the policy in the kernel with the following script: 

```shell
#!/bin/bash

docker exec tetragon tetra sensors rm connect

policy=$(mktemp -p /dev/shm)

cat << EOF > $policy
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "connect"
spec:
  kprobes:
  - call: "tcp_connect"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: &selector
    - matchBinaries:
      - operator: "In"
        values:
$(find ~/bin-no-network/ -executable -type f | awk '{ print "          - \""$0"\""}')
      matchActions:
      - action: Sigkill
  - call: "tcp_close"
    syscall: false
    args:
    - index: 0
      type: "sock"
    selectors: *selector
  - call: "tcp_sendmsg"
    syscall: false
    args:
    - index: 0
      type: "sock"
    - index: 2
      type: int
    selectors: *selector
EOF

docker cp $policy tetragon:/tmp/policy.yaml
docker exec tetragon tetra tracingpolicy add /tmp/policy.yaml
rm $policy
```
## Building trust in your relationship with your workstation XD
After a first date with Tetragon during the [Kubernetes Community Days](https://www.kcdfrance.fr/) during 2023, I was seduced by the way it's implemented: most of the time new features are implemented on top of others creating a complex multi layered cathedral. Perfect for a wedding, but I prefer simplicity. 

Using eBPF to enforce "security" seems to be very efficient from the performance point of view. Bypassing Tetragon security by denial of service attacks is unlikely. For example, the sigkill action triggered by the Tetragon’s eBPF program is done synchronously, in the kernel and not at user space, which makes it hard to bypass. In other words, the security mechanism is fully and autonomously implemented at kernel level.

This mechanism proves to be effective in blocking network activity and building trust in our beloved workstation even when running potentially malicious binaries.

Using Tetragon in a Kubernetes context is the next step. The plan is to observe the behavior of an application in a development environment (files read/write, command executed) and generate a profile. Then, promote this profile from development to production, and enforce an *allow-only* behavior using Tetragon.