Quick QEMU setup for Linux kernel module debugging
Tags: linux
Published: 2021-01-19 10:46:41

As my first post here on dev.to, I have decided to share my little note on how to quickly setup up an environment for linux kernel module debugging in QEMU.


# Step 1: Building linux kernel and busybox userland

Download and extract linux kernel and busybox sources.

```
$ wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.6.tar.xz
$ tar xvf linux-5.6.tar.xz
$ wget https://busybox.net/downloads/busybox-1.31.1.tar.bz2
$ tar xvf busybox-1.31.1.tar.bz2
```

Configure and build linux kernel as usual. A couple of points:

- You can safely start with `defconfig`, enabling options your module depends on.
- Make sure that all options you will need during debugging are compiled-in and not built as modules, so you don't have to cram them into the initramfs.
- Enable at least these options if you want to use gdb (see "Kernel hacking" section in `menuconfig`):
 - `CONFIG_DEBUG_KERNEL=y` (enables kernel debugging facilities)
 - `CONFIG_DEBUG_INFO=y` (includes debug symbols)
 - `CONFIG_KGDB=y` (enables kernel GDB backend over serial line)
 - `CONFIG_PROVE_LOCKING=y` (enable lock dependency checker; not needed for gdb to work, but comes handy for deadlock debugging).

You can quickly check if your kernel boots at all: 

```
qemu-system-x86_64 -kernel linux-5.6.2/arch/x86_64/boot/bzImage \
-nographic \
-append "console=ttyS0"
``` 

Now, it is time to build our busybox userland. Again, `defconfig` will do just fine, with a couple of points:

- Busybox must be linked statically: set `CONFIG_STATIC=y` (found in "Settings" - "Build Options" section)
- Expect surprises when building busybox on a system with latest glibc versions. In the case of Archlinux, I've got the following:

```
netstat.c:(.text.ip_port_str+0x50): warning: Using 'getservbyport' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
...
rdate.c:(.text.rdate_main+0xf8): undefined reference to `stime'
```

You can get around both problems by building busybox with musl instead if glibc:

```
# Archlinux
sudo pacman -S musl kernel-headers-musl
make CC=musl-gcc

# Ubuntu
sudo apt install musl musl-dev musl-tools
CFLAGS="-I../linux-5.6/usr/include" make CC=musl-gcc
```

If using kernel headers from the kernel tarball, you have to install them first locally (`make headers`).

Alternatively, you can try to proceed with glibc by applying [this patch](https://git.busybox.net/busybox/patch/?id=d3539be8f27b8cbfdfee460fe08299158f08bcd9) to fix stime calls (of course, this wont fix glibc broken static linking warnings).

# Step 2: Building minimal initramfs

Busybox binary and ab init script are pretty much all we need for our simple initramfs. Let's put busybox in to /bin and create a /bin/sh symlink, so kernel can look up the shell for our init script:

```
mkdir bin
cp ../busybox-1.31.1/busybox bin/
ln -s /bin/busybox bin/sh
touch init && chmod 777 init
```

Our init script should at least do the following:
- mount procfs and sysfs;
- populate /dev using information provided by kernel at sysfs (busybox's mdev can nicely do it for us);
- do your initialization (load a module for debugging, run a test program, you name it); the snippet below also contains an example on how to run some code on per-VM instance basis, which is needed when you are debugging some module that has to communicate to its remote counterpart on another machine;
- drop to interactive shell.

Here's the init script template I use:

```bash
#!/bin/sh
# Install busybox applets as symlinks
/bin/busybox --install -s /bin

# Mount procfs & sysfs, populate /dev
mkdir proc sys
mount -t proc none /proc
mount -t sysfs none /sys
mdev -s

# Optional dynamic device creation:
# 1. Old method using mdev as an uevent helper
# echo /bin/mdev > /proc/sys/kernel/hotplug 
# 2. New method using Netlink (requires busybox 1.31 or newer)
# mdev -d

# Get virtual machine instance number passed as a kernel argument
inst=$(cat /proc/cmdline | grep -Eo 'vm_inst=[^ ]+' | cut -d '=' -f 2)

# Do your common initialization here (insert debugged module, run some test script, etc)
echo the init is running

# Do your per-vm instance initialization here (assign an IP address, etc)
[ $inst == 1 ] && echo first instance init
[ $inst == 2 ] && echo second instance init

# Drop to the shell
exec setsid cttyhack sh
```

A couple of notes:
- Automatic device creation will obviously not work with this simplistic setup, so when, say, you create a character device in your driver, you have to start `mdev -s` again manually.
- In order to enable hotplugging, you have to either use busybox 1.31 or later, witch introduced netlink interface support for mdev (`mdev -d` option).
Alternatively, you can enable legacy `CONFIG_UEVENT_HELPER` option in Linux (disabled by default) and set hotplug executable to mdev: `echo /bin/mdev > /proc/sys/kernel/hotplug`.
- It is not very convenient to just drop to shell using `exec /bin/sh` because at startup the `/dev/console` is used by default as a tty.
The problem is `/dev/console` can't be a controlling terminal, thus rendering the job control stuff broken (usually indicated by `can't access tty; job control turned off` message).
This can be easily worked around with `setsid cttyhack sh`. `setsid` will exec a cttyhack as a session leader, which will enable job control, and `cttyhack` will find a real tty device (usually ttyS0) and reopen stdin/stdout.

Finally, let's generate the initramfs image (assuming `initramfs` is your initramfs root containing `/init` script and the `/bin` directory):

```
cd initramfs                                                                                                                            
find . -print0 | cpio --null -ov -H newc | gzip -9 > ../initramfs.tgz
```

# Step 3: Running QEMU

Running QEMU is as easy as:

```
qemu-system-x86_64 \
-enable-kvm -cpu host -smp 1 \
-m 256M \
-nographic \
-kernel linux-5.6.2/arch/x86_64/boot/bzImage \
-initrd initramfs.tgz \
-chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0 \
-device pci-serial,chardev=gdb \
-append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS1,115200"
```
1. `-enable-kvm -cpu host -smp 2` enables KVM acceleration and sets emulated CPU model to the host CPU; `-smp` sets the number of CPU cores the guest can use (1 by default);
2. `-m 256M` specifies the amount of memory guest can use (128M by default);
3. `-nographic` disables graphical output and instead emulates a serial port (`ttyS0`) attached to stdout/stdin;
4. `-kernel` and `-initrd` are quite self explanatory;
5. `-chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0` creates a socket listening for GDB frontend connections on TCP port 1234; this socket is later bound to the virtual serial port, which will appear as `ttyS1` in the VM;
6. `-device pci-serial,chardev=gdb` creates an emulated PCI serial port device, redirecting its IO to the TCP socket;
7. -append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS1,115200" specifies the kernel command line; remove `kgdbwait` to let the kernel boot without waiting for GDB connection.

Attaching the gdb to the running kernel is also easy:

```
$ cd linux-5.6
$ gdb vmlinux

(gdb) target remote :1234
(gdb) c
```

You'll probably need to explicitly allow gdb to load linux kernel debugging helper scripts. GDB will nicely warn about this if auto-loading is declined:

```
echo 'add-auto-load-safe-path /home/dav/Dev/qemu/linux-5.6/scripts/gdb/vmlinux-gdb.py' >> ~/.gdbinit
```

A bit more evolved example: running two machine instances communicating over emulated null modem.

Instance 1:

```
qemu-system-x86_64 \
	-enable-kvm \
	-cpu host \
	-smp 2 \
	-nographic \
	-kernel linux-5.6.2/arch/x86_64/boot/bzImage \
	-initrd initram.tgz \
	-chardev socket,id=comm,port=8909,host=127.0.0.1 \
	-device pci-serial,chardev=comm \
	-chardev socket,id=gdb,port=1234,server,nowait,host=0.0.0.0 \
	-device pci-serial,chardev=gdb \
	-append "console=ttyS0 vm_inst=1 kgdbwait kgdboc=ttyS2,115200"
```

Instance 2:

```
qemu-system-x86_64 \
	-enable-kvm \
	-cpu host \
	-smp 2 \
	-nographic \
	-kernel linux-5.6.2/arch/x86_64/boot/bzImage \
	-initrd initram.tgz \
	-chardev socket,port=8909,server,id=comm,host=0.0.0.0 \
	-device pci-serial,chardev=comm \
	-chardev socket,id=gdb,port=1235,server,nowait,host=0.0.0.0 \
	-device pci-serial,chardev=gdb \
	-append "console=ttyS0 vm_inst=2 kgdbwait kgdboc=ttyS2,115200"
```

And that's all for now :) Hope you'll find this useful.