Following up on my first blog post, I’ve received a few requests to write about setting up a debugging environment; however, since everybody uses different emulators and so on, I’ll mainly focus on which config options are useful!

You must first obtain the kernel sources for your preferred version before you can set up a kernel suitable for debugging. We’ll have to decompress it after that.

# use your preferred version
$ wget https://git.kernel.org/torvalds/t/linux-5.12-rc5.tar.gz 
$ tar xvf linux-5.12-rc5.tar.gz

After you’ve pulled your source, you’ll need to determine which emulator you’ll use and how you’ll use it. Personally, I prefer QEMU + kvm, but I’ve recently learned about virtme, which is essentially a very useful QEMU wrapper.

Depending on which you’ll use you’ll need some requirements.

QEMU:

  • debootstrap
  • qemu-system

Virtme:

  • python3
  • qemu-system

Once you have installed the dependencies we’ll need to create some default configs depending on which option you choose; I really like how virtme has a tool to generate a minimum viable config.

QEMU:

$ make defconfig
$ make kvm_guest.config

Virtme:

$ virtme-configkernel --defconfig

We’re prepared to do some configurations now that we’ve completed this!

One of the essential configuration options is CONFIG_DEBUG_INFO=y, this adds debugging symbols to the kernel and modules.

Allowing the kernel to print symbolic crash information and symbolic stack backtraces could be desired when debugging, for this we can use CONFIG_KALLSYMS=y. Since all symbols must be loaded into the kernel image, this increases the kernel’s size slightly.

While it improves your systems security, KASLR is making debugging more difficult since kernel symbol addresses change with each restart, the debugger will have to take extra steps to map functions and data in memory to the corresponding debug symbols, therefore I like to disable it in debugging VM’s. On a production system, this should not be disabled.

If you want to disable KASLR, make sure that # CONFIG RANDOMIZE BASE is not set is in your .config file.

If you are debugging, you may also like some options that make it easier to detect bugs. There’s a few options for this let me list them below.

  • KASAN is used to detect use-after-free and out-of-bounds; it’s very useful for debugging, but it degrades performance levels significantly.
  • UBSAN is used to detect undefined behaviour. You can find the features of UBSAN that the linux kernel could use at UBSAN Makefile and check them against UBSAN’s features.
  • KCSAN is a dynamic race detector that detects data races as its primary feature.
  • KFENCE is a low overhead memory integrity error detector. It detects heap out-of-bounds acces, use-after-free and invalid free bugs; Less precise in detecting bugs than KASAN.
  • Kmemleak is used to detect potential kernel memory leaks (like Valgrind for your kernel).

All of those tools need their own configuration.

KASAN:

# Enable KASAN
CONFIG_KASAN=y
# Use software mode with inline instrumentation, this supports both SLUB and SLAB memory allocations and is a lot more performant than outline instrumentation.
CONFIG_KASAN_INLINE=y

UBSAN:

# Enable UBSAN
CONFIG_UBSAN=y 

# You can use this line to allow testing of the entire kernel 
# or add a "UBSAN SANITIZE := y" line to the respectable Makefile to instrument specific bits
# depending on what you're debugging.
CONFIG_UBSAN_SANITIZE_ALL=y

More options for UBSAN at UBSAN usage.

KCSAN:

# Probably the easiest config to set :)
CONFIG_KCSAN=y

KFENCE:

CONFIG_KFENCE=y

Kmemleak:

CONFIG_DEBUG_KMEMLEAK=y

There’s also CONFIG_DEBUG_SLAB, which performs some verification on SLAB allocated memory while poisoning memory on free. This implies that when a byte is allocated, it is set to 0xa5 and then set to 0x6b when it is freed. This allows for the detection of use-after-free bugs. Aside from poisoning, there are several guards before and after allocated memory, and if those guards are updated, we know there’s most likely a memory corruption bug.

So, if you want to debug a kernel, you should enable a debugger. We can choose between KDB and KGDB. Because this comes down to preference I’ll direct you to https://www.kernel.org/doc/html/latest/dev-tools/kgdb.html. Next to those tools there’s also crash which I personally like.

This should hopefully suffice for a simple debugging setup; but, if you want to debug more complicated issues such as locking, you should refer to Kconfig.debug for additional config keys.

It’s time to build the kernel after you’re content with your .config file! This can be achieved using the following command (this may take a while).

# Build the kernel with all available processing units
$ make -j $(nproc)

So, now that you’ve built your kernel, what’s next? It’s time to get your emulator set up!

This move will be really simple for virtme; all you have to do is:

$ virtme-run --kdir PATH_TO_YOUR_KERNEL_DIRECTORY

For QEMU this step will be a little more difficult, or at the very least, time-consuming. We’ll need to create an image, and since I’m a slacker, I always use syzkaller’s create-image.sh to do so.

To create an image:

$ wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-image.sh -O create-image.sh
$ chmod +x create-image.sh
$ ./create-image.sh

I run a really hacky bash script to start the VM, which will leave you with a stretch.id_rsa and a stretch.img:

#!/bin/bash

export KERNEL=/root/linux-5.12-rc3 # Change this to the path to your kernel

qemu-system-x86_64 \
        -m 2G \
        -smp 2 \
        -kernel $KERNEL/arch/x86/boot/bzImage \
        -append "console=ttyS0 root=/dev/sda earlyprintk=serial net.ifnames=0" \
        -drive file=./stretch.img,format=raw \
        -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10021-:22 \
        -net nic,model=e1000 \
        -enable-kvm \
        -nographic \
        -pidfile vm.pid \
        2>&1 | tee vm.log

When you’re done you can kill it with kill $(cat vm.pid).

Thanks to Vergard Nossum, I found out about Plan 9 Resource Sharing Support, and it could be used to boot the VM of your host filesystem.

Fun fact: KVM uses it as its native host-guest file sharing protocol. virtme is also using this to boot up a kernel with your host FS.

If you want to use the 9P2000 protocol, make sure that CONFIG_9P_FS=y is set in your .config file, and then update your start-vm.sh to include:

	-append "rootfstype=9p root=/dev/root rootflags=trans=virtio,version=9p2000.L" \ 
	-fsdev local,id=fsdev0,path=/,security_model=none -device virtio-9-pci,fsdev=fsdev0,mount_tag=/dev/root \

If you want to use 9P2000, you should probably use virtme instead because it’s a lot easier to set up. Nonetheless, I felt compelled to mention it.

That’s what there is to it! Please keep in mind that this setup is far from ideal, and you should not rely solely on my solutions; if you prefer VirtualBox, use VirtualBox; if you prefer Vagrant, use Vagrant.

As always, if you have any comments or suggestions, let me know on jordy [a-t] pwning.systems :)

Happy Debugging,

Jordy