How to build and test your OpenWRT packages with Docker
Tags: linux, docker, devops, openwrt
Published: 2024-01-30 12:00:00

Building and testing software packages for OpenWRT is challenging because this Linux distribution often runs on the devices with exotic architecture and uses centralized configuration ([UCI](https://openwrt.org/docs/guide-user/base-system/uci)) with which you often need to integrate your software. In this article we will use Docker and QEMU to test package installation on MIPS architecture and discuss what scripts and other files to include in the package to better integrate your software with UCI and OpenWRT itself.


# Table of contents

- [Why OpenWRT?](#why)
- [How to build OpenWRT package?](#package).
    - [Post-install scripts](#post-install).
    - [Post-delete scripts](#post-delete).
    - [Pre-delete scripts](#pre-delete).
    - [Init scripts](#init).
    - [First-boot scripts](#first-boot).
    - [Persist files across system upgrades](#persist).
    - [Package contents](#contents).
- [How to test OpenWRT package with Docker](#docker).
    - [Testing packages for host architecture](#docker-host).
    - [Testing packages for non-host architecture](#docker-non-host).
- [Conclusion](#conclusion).



# <a name="why"></a>Why OpenWRT?

[OpenWRT](https://openwrt.org/) is a popular Linux distribution for network routers that brings the power of Linux kernel to resource-constrained devices. Companies use it as a base for their own routers' firmware, regular people use it to replace vendor-provided firmware which is often closed-source and lack many features compared to open-source OpenWRT.

Apart from OpenWRT there is FreeBSD-based [pfSense](https://pfsense.org/). This distribution supports only x86 (64 bit) architecture whereas OpenWRT supports x86, ARM and MIPS architectures. FreeBSD and Linux are completely different kernels, and you would need a whole different setup for building FreeBSD packages.


# <a name="package"></a>How to build OpenWRT package?

![Photo by Debby Hudson on Unsplash.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aewic3k9b9yfp3uhg73b.jpg)

OpenWRT uses [opkg](https://openwrt.org/docs/guide-user/additional-software/opkg) package manager that can install [ipk/opk](https://en.wikipedia.org/wiki/Opkg) packages from online repositories and local file system, and to build your own package you need to use [opkg-utils](https://git.yoctoproject.org/opkg-utils/). Ipk/opk package format is similar to [deb](https://en.wikipedia.org/wiki/Deb_(file_format)) but uses [tar](https://man7.org/linux/man-pages/man1/tar.1.html) instead of [ar](https://man7.org/linux/man-pages/man1/ar.1.html) to package files. The simplest way of building a package is to use `opkg-build` command.

```bash
# Flag `-c` replaces `ar` with `tar`. This is mandatory for OpenWRT.
🌊 opkg-build -c input-dir output-dir
```

The `input-dir` contains the files that you want to install plus `CONTROL` directory with metadata (for *deb* format this directory is called `DEBIAN` and you can use this name with `opkg-build` as well). We don't know other major differences between deb and *ipk/opk*. For this reason we ended up converting to *ipk* from *deb* that was generated by [fpm](https://fpm.readthedocs.io/) — a tool that we use to produce packages for other Linux distributions. However, the contents of pre/post install/update/remove scripts is different for OpenWRT and there are other special files that you might want to include in the package.

The package includes scripts that are run prior to or after the installation, update and removal of the package. Often package maintainers include them to start/stop services and update firewall rules.


## <a name="post-install"></a>Post-install scripts

Setting up firewall rules in packages scripts is not typical of Linux distributions other than OpenWRT. This is due to the fact that OpenWRT uses [Unified Configuration Interface (UCI)](https://openwrt.org/docs/guide-user/base-system/uci) — a centralized way of managing system configuration. Through UCI you can setup firewall rules that are inactive by default and can later be enabled via OpenWRT's [web interface](https://openwrt.org/docs/guide-user/luci/webinterface.overview) or via command-line interface.

The following commands set up firewall rule to accept TCP and UDP traffic on port 1234. The rule is disabled by default.

```bash
# post-install script
uci -q batch >/dev/null <<'EOF'
add firewall rule
set firewall.@rule[-1].dest_port='1234'
set firewall.@rule[-1].src='*'
set firewall.@rule[-1].name='Allow-MyApp-any'
set firewall.@rule[-1].proto='udp tcp'
set firewall.@rule[-1].target='ACCEPT'
set firewall.@rule[-1].enabled='0'
commit firewall
EOF
```

Usually you run each line as a separate _uci_ command (e.g. `uci add firewall rule`), but `uci batch` is generally faster for a large number of lines. It is up to you to enable or disable the rule by default: for public-facing services (e.g. VPNs) it is generally safe to open the port by default, for everything else (e.g. DNS resolver like Stubby) I would rather close the port by default.

Later the rule can be enabled with the following commands.

```bash
# NNN is the actual index of the rule
uci set firewall.@rule[NNN].enabled=1
uci commit firewall
fw3 reload
```


## <a name="post-delete"></a>Post-delete scripts

Deleting the rules is more involved as you need to find the rule index and delete all the matching lines. To distinguish between the package update and package deleting you need to check the first argument of the script.

```bash
# post-delete script

delete_rule() {
    name=
    config_get name "$1" name
    if test "$name" = "Allow-MyApp-any"; then
        uci -q delete firewall."$1" || true
    fi
}

case "$1" in
0 | remove)
    . /lib/functions.sh
    config_load firewall
    config_foreach delete_rule rule
    uci -q delete firewall.mcc || true
    ;;
esac
```

## <a name="pre-delete"></a>Pre-delete scripts

This is the right place to stop your services before deleting the package. Again to distinguish between the package update and package deleting you need to check the first argument of the script.

```bash
#!/bin/sh
case "$1" in
0 | remove)
    /etc/init.d/myapp stop || true
    ;;
esac
```


## <a name="init"></a>Init scripts

OpenWRT uses [procd](https://openwrt.org/docs/techref/procd) as the init system — pid 1 process that launches all other processes in the system on boot. Procd is similar to SysV init, systemd, openrc etc., however, the syntax for the scripts is different. Here is an example of init script for a typical application that does not daemonizes itself and writes logs on standard output and standard error streams.

```bash
#!/bin/sh /etc/rc.common
USE_PROCD=1
START=98 # start order
STOP=99 # stop order
start_service() {
    procd_open_instance
    procd_set_param command /usr/bin/myapp # run the command without daemonizing
    procd_set_param respawn 0 7 0 # respawn after 7 seconds delay
    procd_set_param stdout 1 # redirect stdout to syslog
    procd_set_param stderr 1 # redirect stderr to syslog
    procd_close_instance
}
```

Procd has a lot of features including process jails, capabilities, etc. that are documented on [OpenWRT web site](https://openwrt.org/docs/guide-developer/procd-init-scripts).


## <a name="first-boot"></a>First-boot scripts

OpenWRT firmware can come with packages pre-installed, and in this case the right place to generate firewall rules would be a _uci-defaults_ script. Such scripts are placed in `/etc/uci-defaults` directory and are executed on the first system boot. After successful execution they are deleted by the system. These scripts do not have any special arguments, and generally you repeat post-install script contents there.

Beware that distributing your package as part of OpenWRT firmware image means that your clients would not be able to reclaim free disk space by deleting the package: it will be deleted only in the overlay file system, but not in the underlying real file system.


## <a name="persist"></a>Persist files across system upgrades

Usually OpenWRT firmware is updated separately from the packages using `sysupgrade` command. By default all the configuration files (that you specified in `CONTROL/conffiles` file) are retained during the upgrade, however, your application might generate other files that you want to persist during the upgrade. To do so simply add `/lib/upgrade/keep.d/myapp` file to your package that lists all directories and files that need to be persisted during the upgrade.

The following file lists `/etc/myapp` directory as the one that should be persisted during the system upgrade.

```bash
/etc/myapp
```


## <a name="contents"></a>Package contents

The final package contents would look like the following.

```bash
.
├── conffiles # files that are persisted during system upgrade and that are not overwritten by package update
├── control # package metadata
├── etc
│   ├── init.d
│   │   └── myapp # init script
│   ├── myapp
│   │   └── myapp.conf # app configuration
│   └── uci-defaults
│       └── myapp # the script that runs on the first boot
├── lib
│   └── upgrade
│       └── keep.d
│           └── myapp # the list of files that are persisted during system upgrade
├── postinst # post-install script
├── postrm # post-delete script
├── prerm # pre-delete script
└── usr
    └── bin
        └── myapp # application executable binary
```

Most of the contents can be generated with _fpm_ tool using _deb_ format as the output. Then the following script will convert _deb_ package to _ipk_ package.

```bash
#!/bin/sh

cleanup() {
    rm -rf "$workdir"
}

set -ex
trap cleanup EXIT
workdir="$(mktemp -d)"
mkdir -p "$workdir"/deb "$workdir"/ipk/CONTROL
cd "$workdir"/deb
# unpack deb package
ar x "$1"
tar -C "$workdir"/ipk -xf data.tar.gz
tar -C "$workdir"/ipk/CONTROL -xf control.tar.gz
# remove generated files
rm "$workdir"/ipk/CONTROL/md5sums
# patch architecture for OpenWRT
sed -i -e 's/Architecture: amd64/Architecture: x86_64/g' "$workdir"/ipk/CONTROL/control
# write ipk to /tmp
opkg-build -c "$workdir"/ipk /tmp
```

# <a name="docker"></a>How to test OpenWRT package with Docker

![Photo by Nick de Partee on Unsplash.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rxl144yh58ilhw6nidj7.jpg)

Similar to any other major Linux distribution OpenWRT maintains _rootfs_ [Docker images](https://hub.docker.com/r/openwrt/rootfs) of root file systems, but it has a separate image for each target architecture. The documentation is on [Github](https://github.com/openwrt/docker). There is a separate tag for each architecture plus OpenWRT version combination. There are also _sdk_ and _imagebuilder_ images that are useful for building the package and the firmware image with the package pre-installed, but we will not discuss them here.



## <a name="docker-host"></a>Testing packages for host architecture

Rootfs images are useful to test your package installation/removal without investing into a router running OpenWRT (although, you should definitely invest in such a device to feel comfortable with doing system upgrades, working with web UI etc.). To install and remove the package use the following commands.

```bash
🌊 docker run --rm -it -v /tmp:/tmp openwrt/rootfs


BusyBox v1.36.1 (2024-01-22 12:01:31 UTC) built-in shell (ash)

🌊 opkg update
...
🌊 opkg install /tmp/myapp.x86_64.ipk
Installing myapp (1.0.0) to root...
Configuring myapp.
🌊 opkg remove myapp
Removing package myapp from root...
```

Docker by default pulled and ran `openwrt/rootfs` image that matches the host machine's architecture (x86_64). Then we updated the package index and installed our application from the `/tmp` directory mounted from the host.


## <a name="docker-non-host"></a>Testing packages for non-host architecture

Testing for an architecture other than host's is more involved and requires running a virtual machine. Luckily for us [QEMU](https://www.qemu.org/) and Linux's [binfmt_misc](https://docs.kernel.org/admin-guide/binfmt-misc.html) makes it easy to do so (and without Docker even noticing!).

QEMU is a tool that runs virtual machines on the host, and the most useful feature it has for us is the ability to transparently execute binary files compiled for an architecture other than the host's architecture. The following commands show how to do that on Ubuntu 22.04.3 LTS.

```bash
# install QEMUE binaries that were statically compiled for each supported architecture
🌊 apt-get install qemu-user-static
# list all available QEMU binaries
🌊 ls /usr/bin/qemu-*static
...
/usr/bin/qemu-aarch64-static
...
/usr/bin/qemu-mips-static
...
# execute a file compiled for mips
🌊 qemu-mips-static /tmp/file-compiled-for-mips
```

To execute the binary compiled for architecture _X_ you just add `qemu-X-static` before the command and that's it.

Linux's Support for miscellaneous Binary Formats (_binfmt_misc_) allows us to execute binaries without specifying any QEMU command. Upon execution the kernel detects the actual executable format of the file and executed it via the matching QEMU binary. The matching QEMU binaries and the «magic» bytes that distinguish a particular format from all others are specified in `/proc/sys/fs/binfmt_misc` directory. On Ubuntu 22.04.3 LTS this directory is populated automatically after `qemu-user-static` package is installed.

```bash
🌊 ls /proc/sys/fs/binfmt_misc
...
qemu-mips
...
qemu-aarch64
...
🌊 cat /proc/sys/fs/binfmt_misc/qemu-mips
enabled
interpreter /usr/libexec/qemu-binfmt/mips-binfmt-P
flags: POCF
offset 0
magic 7f454c46010201000000000000000000000200080000000000000000000000000000000000000000
mask ffffffffffffff00fefffffffffffffffffeffff0000000000000000000000000000000000000020
```

Now to run a foreign binary you don't need anything special: it runs the same way as a native binary on the system.

```bash
# you need matching entry in /proc/sys/fs/binfmt_misc for this to work
🌊 /tmp/file-compiled-for-mips
```

Since Docker is not a virtualization platform, but a process isolation tool, _binfmt-misc_ and QEMU allows you to transparently run Docker images that were built for other architectures. Again, you don't need anything special to do that. The following command runs OpenWRT _rootfs_ image for MIPS architecture.

```bash
🌊 docker run --rm -it -v /tmp:/tmp openwrt/rootfs:mips_24kc
Unable to find image 'openwrt/rootfs:mips_24kc' locally
mips_24kc: Pulling from openwrt/rootfs
2dd8ebde9a90: Pull complete
Digest: sha256:58d0bf8e15559e0a331e23915ed0221d678c8b2a569c58c0fa25a4f991e4beca
Status: Downloaded newer image for openwrt/rootfs:mips_24kc
WARNING: The requested image platform (linux/mips_24kc) does not match the detected host platform (linux/amd64/v4) and no specific platform was requested


BusyBox v1.36.1 (2024-01-26 09:19:40 UTC) built-in shell (ash)

🌊 grep ARCH /etc/os-release
OPENWRT_ARCH="mips_24kc"
🌊 opkg update
...
🌊 opkg install /tmp/myapp.mips_24kc.ipk
Installing myapp (1.0.0) to root...
Configuring myapp.
🌊 opkg remove myapp
Removing package myapp from root...
```

Docker warns that the image's architecture does not match the host's, but still successfully runs the image. This is where the power of QEMU and _binfmt-misc_ shows itself.

# <a name="conclusion"></a>Conclusion

![Photo by Justin Wilkens on Unsplash.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bpess80j5vfn2run8qsf.jpg)

Building OpenWRT packages is similar to any other Linux distributions, but has unique requirements if you want to integrate your package with the rest of the system.

- Generate firewall rules in post-install and _uci-defaults_ scripts and delete them in post-delete script.
- List files (in addition to configuration files) that need to be preserved during system upgrades in `/lib/upgrade/keep.d/myapp` file.
- Write _procd_-compatible init script.
- Use `opkg-build` to generate ipk/opk package.
- Optionally, use _fpm_ tool to generate _deb_ package and then convert it to _ipk/opk_.

Testing OpenWRT packages is also very similar to other Linux distributions, provided that you installed QEMU and used _binfmt-misc_ kernel feature to transparently run foreign binaries on your host. OpenWRT maintains root file system images for each combination of version and architecture all of which you can directly run on your host and in your CI/CD pipeline.

_We at Staex help our clients make IoT devices first-class citizens in their private networks, protect from common attacks, reduce mobile data usage, and enable audacious use cases that were not possible before._

_[Subscribe to our newsletter](https://staex.io/blog/subscribe) to get more content like this._