Introduction to Netlink with Go
Tags: linux, containers, networking, go
Published: 2024-04-28 11:44:30

A few years back, I got really interested in understanding how Docker and containers work. One area that grabbed my attention was containers and networking. Docker handles networking pretty simply. It mainly relies on virtual interfaces, bridges, NAT routing, and other features provided by the Linux operating system. But I won't be focusing on Docker and networking in this article.

As I delved deeper, I became curious about how Docker do all this networking. If you're like me—a developer who spends almost all their time in user space—you're probably familiar with interacting with the kernel through syscalls or the filesystem (like `/proc`). At first, I had this naive idea that OS had some kind of magic syscall or something in `/proc` that Docker was tapping into. But nope, I was way off. In Docker's code, I discovered another method: Netlink.

So, what is Netlink? It's another way of exchanging information between user space and the kernel. But this time, the socket-like way. Think of Netlink as a direct socket connection into the kernel, allowing you to send and receive messages. This approach is quite interesting, because I can do asynchronous communication with the kernel or simply listening for messages from the kernel in user space.


![netlink](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8mvfknvofvi5lh0zkqup.png)

With Netlink, I can communicate with various kernel subsystems. For example, I can receive events from SELinux, updates about routing or network links, and even modify routing tables and IP addresses.

If you're comfortable with basic socket programming like me, handling Netlink could be easy to understand. All you need to do is open a socket to the kernel, address the subsystem, and send or receive binary messages.

## Bring interface UP 

  Let's get practical here. Let's start with something super basic—like a Netlink `hello world`. One of the simplest examples I could think of is enabling an network interface. On my system, I've got this `veth0` interface sitting there, in the `DOWN` state:

```bash
$ ip link
...
78: veth0@veth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether de:8f:4e:7e:9c:cd brd ff:ff:ff:ff:ff:ff
```

  I'd like to write my own simple Go program to execute `ip link set veth1 up`.

One note before I start: Since I want to use syscalls and related data structures, I'll need to install the Go `golang.org/x/sys` package.

Let's begin by establishing the socket. The Netlink socket is created using `unix.Socket()`:

```go
// open the Netlink socket 
sock, err := unix.Socket(
	unix.AF_NETLINK,
	unix.SOCK_RAW,
	unix.NETLINK_ROUTE,
)

if err != nil {
	fmt.Printf("Error creating socket: %s\n", err)
	return
}

defer unix.Close(sock)
```

Instead of the `AF_INET` domain, which we typically use for TCP/IP communication, I'm using `AF_NETLINK`. What's also interesting here is the last parameter. This value determines which subsystem I want to communicate with. It could be `NETLINK_NETFILTER`, `NETLINK_SELINUX`, and so on. I'm opting for `NETLINK_ROUTE`, which is dedicated to interfaces, links, IP addresses, and such.

However, just having the socket isn't enough. In TCP/IP communication, we associate a socket with a specific network address using `bind()`. Similarly, in Netlink, I'll set what group and port ID (PID) I want to use. For simplicity, I'll use values of `0`.

```go
// bind the socket to group and PID
err = unix.Bind(sock, &unix.SockaddrNetlink{
	Family: unix.AF_NETLINK,
	Groups: 0,
	Pid: 0,
})

if err != nil {
	fmt.Printf("Error in binding socket: %s\n", err)
	return
}
```

## Create and send message

The easy part is done. At this point, I've got everything ready for sending and receiving messages. Now comes the tricky part—building and parsing Netlink messages. Like in many binary protocols, the Netlink message consists of a header and payload.



![Message](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e2ke5n4sguqd7vckshdx.png)

Let's start from the end—with the payload. The payload is all about what I want to do or get from Netlink. In my case, it's about enabling a network interface. Therefore, I'll use `IfInfomsg`, where I'll set the `Change` field to `IFF_UP`. The good news is that the structure is available in the `golang.org/x/sys` package, so I don't need to write it from scratch.

```go
payload := unix.IfInfomsg{
	Family: unix.AF_UNSPEC,
	Change: unix.IFF_UP,
	Flags:  unix.IFF_UP,
	Index:  int32(ethIndex), // index of network interface I would like to enable (in my case it's 79 - veth1)
}
```

Then, I need to build a header. The header carries information like the type of the payload or the total length of the whole message. The structure for the header is `NlMsghdr`. The type I need to set is `RTM_NEWLINK`, which is related to the `IfInfomsg` payload.

```go
// total length of message is size of header + size of payload
length := unix.SizeofNlMsghdr + unix.SizeofIfInfomsg

header := unix.NlMsghdr{
	Len:   uint32(length),
	Type:  uint16(unix.RTM_NEWLINK),
	Flags: uint16(unix.NLM_F_REQUEST) | uint16(unix.NLM_F_ACK),
	Seq:   1,
}
```

Alright, the message should be almost ready. I just need to put the header and payload into one message structure. I'll create an anonymous structure and fill it with the payload and header:

```go
msg := struct {
   header unix.NlMsghdr
   payload unix.IfInfomsg
}{
   header: header,
   payload: payload,
}
```

The message is ready, and I could write the message data into the socket. But before I call `Sendto()`, I need to convert the message structure to an array (or slice) of bytes:

```go
// first I need convert the `msg` to slice of bytes
var asByteSlice []byte = (*(*[unix.SizeofNlMsghdr + unix.SizeofIfInfomsg]byte)(unsafe.Pointer(&msg)))[:]

// write the data to the socket
err = unix.Sendto(sock, asByteSlice, 0, &unix.SockaddrNetlink{Family: unix.AF_NETLINK})
if err != nil {
	fmt.Printf("Could not write message to socket:%s\n", err)
}
```

## Receiving message

At this point, if I compile and run my simple program with root privileges, the code will bring the `veth1` interface up. Mission accomplished. Right? But what about receiving messages?

Receiving messages might be complicated. The messages might be large, or the information might be broken into multiple pieces. There are various factors to consider. But I'll stick with my simple scenario. I just want to know if my `up` operation failed or if it was successful.

To receive the response, I'll use `unix.Recvfrom()`, which will read all remaining data from the socket into the `buf`:

```go
var buf [1024]byte
n, _, err := unix.Recvfrom(sock, buf[:], 0)

if err != nil {
	fmt.Printf("Could not read data from socket: %s\n", err)
	return
}
```

The next step is parsing the received raw data. Here, I'll use `ParseNetlinkMessage()` to do just that.

```go
// parse data to messages
msgs, err := syscall.ParseNetlinkMessage(buf[:n])

if err != nil {
	fmt.Printf("Could not parse the response: %s\n", err)
	return
}
```

The function will return parsed data as an array of `[]NetlinkMessage`. The `NetlinkMessage` is a simple structure with `Header` and `Data`. The `Header` is `NlMsghdr`, and `Data` is an array of bytes. Based on the type in the `Header`, I can cast the `Data` to the proper type. In my case, the first response message will be `NLMSG_ERROR`, so I'll cast `Data` to `NlMsgerr`.

```go
// the first received message must be `NLMSG_ERROR`
if msgs[0].Header.Type != unix.NLMSG_ERROR {
	fmt.Printf("The first received message is not NLMSG_ERROR\n")
	return
}

// cast the data to NlMsgerr payload
errPayload := (*unix.NlMsgerr)(unsafe.Pointer(&resp[0].Data[0]))
if errPayload.Error != 0 {
	fmt.Printf("Error returned by Netlink\n")
}

fmt.Printf("Interface is UP\n")
```

The full code is available on [github.com/sn3d/netlink-example](https://github.com/sn3d/netlink-example)

## Let's try...

It's time to play with my program. As I mentioned above, I have `veth1` present in my system which is `DOWN`. 

```bash
$ ip link
...
78: veth0@veth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether de:8f:4e:7e:9c:cd brd ff:ff:ff:ff:ff:ff

```

If you look closer, you might see the `veth0` have index `78`. I need this index pass to my program. Now when I run my program with this index, I should get information `Interface is UP`:

```bash
$ sudo go run main.go 78
Interface is UP
```

This is not just a message from my program. If I will check the `veth0`, I will notice the interface is `UP`.

```bash
$ ip link
...
78: veth0@veth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether de:8f:4e:7e:9c:cd brd ff:ff:ff:ff:ff:ff
```

## How to debug messages

As you noticed, creating a Netlink connection, reading from, and writing to the socket is the easy part. Maybe reading bigger chunks of data or data that's broken into smaller chunks is more complicated, but it's something we're familiar with from socket programming.

The tricky part for me was creating a proper Netlink message. But there's a pretty useful way to debug and observe Netlink messages - using `strace`. Modern `strace` has a great feature - it can parse and understand Netlink messages.

If you try to execute `ip link set veth0 up` with `strace`, you might see `sendmsg` with a parsed Netlink message:

```bash
$ sudo strace -Tfe trace=sendmsg ip link set veth0 up
endmsg(4, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{nlmsg_len=52, nlmsg_type=RTM_GETLINK, nlmsg_flags=NLM_F_REQUEST, nlmsg_seq=1714051690, nlmsg_pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0}, [[{nla_len=8, nla_type=IFLA_EXT_MASK}, RTEXT_FILTER_VF|RTEXT_FILTER_SKIP_STATS], [{nla_len=10, nla_type=IFLA_IFNAME}, "veth0"]]], iov_len=52}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 52 <0.000094>
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base=[{nlmsg_len=32, nlmsg_type=RTM_NEWLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK, nlmsg_seq=1714051690, nlmsg_pid=0}, {ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=if_nametoindex("veth0"), ifi_flags=IFF_UP, ifi_change=0x1}], iov_len=32}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 32 <0.000064>
```

It's a bit messy output, but you could notice 2 `sendmsg` calls. One is for `RTM_GETLINK`, and the second is for `RTM_NEWLINK`. In this output, you might see the header and payload. For instance, the header of the second `RTM_NEWLINK` is:

```
{nlmsg_len=32, nlmsg_type=RTM_NEWLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK, nlmsg_seq=1714051690, nlmsg_pid=0}
```

And the payload is:

```
{ifi_family=AF_UNSPEC, ifi_type=ARPHRD_NETROM, ifi_index=if_nametoindex("veth0"), ifi_flags=IFF_UP, ifi_change=0x1}
```

With `strace`, we could study requests like creating a bridge, etc., and reproduce the messages from our code.

## Netlink library

Working with sockets, building, and parsing our own messages require quite a lot of work. Thanks to Vish Abrams, we can use in our project the package `github.com/vishvananda/netlink`, which provides a lot of Netlink functionalities without having to build our own message structures from scratch. It's well-maintained and used by many projects like Docker, Cilium, Flannel, Istio, etc.

Thanks to this library, adding a new bridge to the system is a matter of a few lines:

```go
la := netlink.NewLinkAttrs()
la.Name = "docker0"
dockerBridge := &netlink.Bridge{LinkAttrs: la}
err := netlink.LinkAdd(dockerBridge)
```

## A few words in conclusion...

One important aspect I didn't mention earlier is that the byte order in messages depends on the host's CPU architecture. This means we don't need to worry about converting between little and big-endian for integers. Additionally, it's crucial that our messages follow a four-byte padding rule. For example, if a message is 33 bytes long, we'll need to send 36 bytes.

I wrote about Netlink almost three years ago, but it was in Slovak. I decided to write this English version because, even after three years, I still find Netlink interesting and worth studying and trying.