Learning eBPF: Maps, Ring Buffers and Output
Tags: ebpf, tutorial, python, linux
Published: 2023-07-22 16:14:51

I set the stage for learning [eBPF](https://rnemet.dev/posts/ebpf/learn-ebpf). As mentioned in the previous post, eBPF is a technology that allows us to run code in the kernel.
This is a compelling technology, but it comes with a few limitations. One of them is that we can't use the standard output to print messages. At least not directly. Let's explore how we can do this.

## Why can't I use the standard input/output?

Let's look at this picture ![ebpf-workflow](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xfd70qnoi0ptp0yik0bq.jpg)

The eBPF programs are executed in the kernel. The kernel is the core of the operating system. Other processes exist there that ensure the smooth operation of the system. To make this possible, the kernel is running in the privileged mode.

Programs run by the user are executed in the user space. The user space is running in the unprivileged mode. When the user space program wants to communicate with the kernel, it needs to use the system calls. The system calls are the interface between the user space and the kernel. The system calls are the only way to communicate with the kernel. The kernel is not exposing any other interface to the user space.

Because of that, we need to have the correct permissions for injecting, executing, and reading the eBPF programs. You must already notice that when you are running the simple [hello.py](https://github.com/robert-nemet/learn-ebpf/blob/main/intro/hello.py) script, you need to have root permissions. The root permissions have enough privileges
to communicate with the kernel. In this case, load the eBPF program in the kernel, attach it to the events stream, and execute it.

This is an answer to why we can't use the standard output directly. The standard output is a user space concept. But we can communicate with eBPF programs using the system data structures and calls. They are designed to allow communication between the user space and the kernel, and other eBPF programs can use them to exchange data. Let's scratch the surface of this topic.

## Simple output(bpf_trace_printk)

Let's look at this BCC example:

```python
#!/usr/bin/python3
from bcc import BPF

program = r"""
#include <linux/sched.h>

int hello(void *ctx) {
    
    int pid = bpf_get_current_pid_tgid() >> 32;
    int uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
    char command[TASK_COMM_LEN];
    bpf_get_current_comm(command, sizeof(command));

    bpf_trace_printk("uid = %d, pid = %d, comm %s", uid, pid, command);

    return 0;
}
"""

b = BPF(text=program, cflags=["-Wno-macro-redefined"])
syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")

b.trace_print()
```

This simple program prints the uid, pid, and the command name of the process executing the `execve` system call. I use:

* The [bpf_get_current_pid_tgid](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#4-bpf_get_current_pid_tgid) function to get the pid(process id) of the
process
* The [bpf_get_current_uid_gid](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#5-bpf_get_current_uid_gid) function to get the uid(user id) of the
process
* The [bpf_get_current_comm](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#6-bpf_get_current_comm) function to get the command name of the process

The [bpf_trace_printk](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#1-bpf_trace_printk) is a function that allows us to print messages from the eBPF program. The messages are sent to the predefined pseudo-file on location `/sys/kernel/debug/tracing/trace_pipe`. So, `bpf_trace_printk` helper function sends my messages to the `trace_pipe` file. The `trace_pipe` file is a special file that is used by the kernel to send messages to the user space. Then I can read them with the `trace_print()` function or with the `cat` command.

[![asciicast](https://asciinema.org/a/0Z6EbGoyZNf5xUxeqECbm4UCq.svg)](https://asciinema.org/a/0Z6EbGoyZNf5xUxeqECbm4UCq)

In the above recording, you'll notice whenever I execute the `execve` system call, the eBPF program prints the message on the top third of the screen. The messages are in the `trace_pipe` file, too. Notice the middle part of the screen where `cat /sys/kernel/debug/tracing/trace_pipe` is executed. At the bottom, I run simple commands like the regular user and root. Notice the difference in the messages. The messages that are printed by the root user have the uid 0. Also, notice that opening a new shell session create several message output. There are many messages because several actions are done when opening a new shell.

While working `bpf_trace_printk` is easy, it has some limitations. For example, if you have multiple eBPF programs that are printing messages, all the messages will be mixed in the `trace_pipe` file. It is hard to distinguish which message is coming from which program. Also, the `trace_pipe` file is a special file designed for debugging purposes. It is not intended for production use. So, we must find a better way to communicate with the user space.

I find it helpful to use `bpf_trace_printk` for debugging purposes. It is a quick way to print messages from the eBPF program.

## Maps

Maps as data structures are used to store data. When it comes to eBPF, maps are data structures that are used to exchange data between the user space and the kernel.
Also, maps are used to exchange data between eBPF programs. In general, maps are key-value stores. But still, different types of maps exist. The reason is that some of them are optimized for different use cases. At the same time, other eBPF maps hold information about specific object types. For example, there is the `BPF_MAP_TYPE_QUEUE` map, which is optimized as a FIFO(first in, first out) queue, and `BPF_MAP_TYPE_STACK` which provides a LIFO(last in, first out) stack. Check [linux docs](https://docs.kernel.org/bpf/map_queue_stack.html)
on them for more information.

Or maps used to hold information about [network devices](https://docs.kernel.org/bpf/map_devmap.html).

Let's check this example:

```python
#!/usr/bin/python3  
from bcc import BPF
from time import sleep

program = r"""
struct data_t {
   u64 counter;
   int pid;
   char command[16];
};

BPF_HASH(counter_table, u64, struct data_t);

int hello(void *ctx) {
   struct data_t zero = {};
   struct data_t *val;

   
   u64 uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
   int pid = bpf_get_current_pid_tgid() >> 32;

   val = counter_table.lookup_or_try_init(&uid, &zero);
   if (val) {
      val->counter++;
      val->pid = pid;
      bpf_get_current_comm(&val->command, sizeof(val->command));
   }

   return 0;
}
"""

b = BPF(text=program, cflags=["-Wno-macro-redefined"])

syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")


old_s = ""
while True:
   sleep(2)
   s = ""
   for k,v in b["counter_table"].items():
      s += f"ID {k.value}: cnt: {v.counter} pid: {v.pid} comm: {v.command}\t"
   if s != old_s:
      print(s)
   old_s = s
```

The above example is similar to the previous one. But this time, I'm using the map to store information about the processes. I use the `BPF_HASH` map to create the map
`counter_table`. I use the uid of the process for a key, and for value, I use the `struct data_t` structure. The `struct data_t` structure holds the counter for all commands the user executed, pid, and the command name of the last process run by a user.

The `hello` function is attached to the `execve` system call. When the `execve` system call is executed, I check if the user is already in the map. If not, I add the user to the map. Suppose the user is already in the map. In that case, I increment the counter and update the pid and the command name of the last process executed by the user.

Later, in Python script, I read the map and print the information about the users. I use the `items()` function to iterate over the map. The `items()` function returns the key and value for each entry in the map. I use the `value` to get the `struct data_t` structure. Then, I print the information about the user.

[![asciicast](https://asciinema.org/a/LL95jvykc7z5YWklFVv8KulnP.svg)](https://asciinema.org/a/LL95jvykc7z5YWklFVv8KulnP)

## Ring buffers

Like maps, ring buffers are data structures that exchange data between the user space and the kernel. Look at the picture:

![ring-buffer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/l37ouz0rivp0526wnw1m.png)

In short, ring buffers are circular buffers. They have two pointers: one for reading and one for writing. The pointers are moving in the same direction. If the read pointer catches the write pointer, the buffer is empty. If the write pointer catches the read pointer, the buffer is full. Then, the next element to be written will be dropped.

There are two types of ring buffers [BPF_PERF_OUTPUT](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#2-bpf_perf_output) and [BPF_RINGBUF_OUTPUT](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#5-bpf_ringbuf_output). The `BPF_RINGBUF_OUTPUT` is more advanced than `BPF_PERF_OUTPUT`. I will not go into the details about the differences between them [please check the docs](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#5-bpf_ringbuf_output).

Here is a familiar example:

```python
#!/usr/bin/python3  
from bcc import BPF

program = r"""
BPF_PERF_OUTPUT(counter_table); 
 
struct data_t {     
   int pid;
   int uid;
   char command[16];
};


int hello(void *ctx) {
   struct data_t data = {}; 
 
   data.pid = bpf_get_current_pid_tgid() >> 32;
   data.uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
   
   bpf_get_current_comm(&data.command, sizeof(data.command));
   
   counter_table.perf_submit(ctx, &data, sizeof(data)); 
 
   return 0;
}
"""

b = BPF(text=program, cflags=["-Wno-macro-redefined"]) 
syscall = b.get_syscall_fnname("execve")
b.attach_kprobe(event=syscall, fn_name="hello")
 
def print_event(cpu, data, size):  
   data = b["counter_table"].event(data)
   print(f"{data.pid} {data.uid} {data.command.decode()}")
 
b["counter_table"].open_perf_buffer(print_event) 
while True:   
   b.perf_buffer_poll()
```

This time difference is when reading from the ring buffer, I'm passing the callback `print_event`. The `print_event` callback is called when the data is available in the ring buffer. It has to have three arguments: `cpu`, `data`, and `size`. The `cpu` argument is the cpu number on which the event was generated. The `data` argument is the data that is read from the ring buffer. The `size` argument is the size of the data read from the ring buffer.

## Summary

So, even if eBPF has no direct access to the standard output, there are ways to exchange data between the user space and the kernel. Plus, given data structures, maps are usually optimized for the specific use case. That should make your life easier.  

## References

* [linux/bpf.h](https://elixir.bootlin.com/linux/v6.4.3/source/include/uapi/linux/bpf.h#L905)
* [kernel docs: eBPF maps](https://docs.kernel.org/bpf/maps.html)
* [BCC reference guide: Maps](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#maps)
* [BCC reference guide](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md)