Building Secure Foundations: A Practical Guide to Minimizing Linux Services' Attack Surface
Tags: systemd, security, linux, kubernetes
Published: 2023-11-11 22:47:03

Cybersecurity and its awareness have never been more crucial than they are today. Considering the increasing amount of attacks, it has become clear that protecting digital assets plays a significant role in software development and operations. What concrete steps can be taken to enhance the security of our services even further?

### Starting at a Lower Level

While antivirus a well-executed read-only backup strategy are essential for identifying and reducing the impact of threats, it's important to establish a strong foundation of security from the outset. Rather than solely focusing on mitigating consequences after the fact, reducing the attack surface should be a primary goal.

This can be done by limiting access to the underlying system, like running as an arbitrary user and dropping unneeded privileges. In Kubernetes, this would for example typically mean using non-root base images in combination with [`securityContext`](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) definitions.

But in some cases, it's better or even required to deploy directly on virtual machines. So how can a similar strategy be applied there?

## 🔒 Hardening Nginx: Step by Step

Let's examine a real-world example using the Nginx service file provided by Ubuntu 20.04:

### The Defaults

`david@proxy:~$ systemctl cat nginx.service`
```ini
# /lib/systemd/system/nginx.service
[Unit]
Description=A high performance web server and a reverse proxy server
Documentation=man:nginx(8)
After=network.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed

[Install]
WantedBy=multi-user.target
```

By default, the service runs as the `root` user. Therefore, processes spawned by `/usr/sbin/nginx` have all privileges of the `root` user and group, which could allow malicious software to control every part of the system when there is an exploit for Nginx. While Nginx is also able to use arbitrary users by itself, the main process that's started by the service still has root privileges. In many cases, this is not required and can be avoided by using Systemd's already built-in capabilities.

### Breaking it Down

The `systemd-analyze` cli tool can help to get an overview of potential issues of Systemd services:

```
systemd-analyze security                # provides a high-level overview including a
                                        # numeric "exposure" value of Systemd services

systemd-analyze security <service_name> # shows detailed security-related information
                                        # about a single service
```

The output for the `nginx` service looks like this:

`david@proxy:~$ systemd-analyze security nginx.service --no-pager`

{% collapsible Nginx Service Security Summary %}
```
  NAME                                                        DESCRIPTION                                                       EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                               0.5
✗ User=/DynamicUser=                                          Service runs as root user                                              0.4
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                     0.3
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has administrator privileges                                   0.3
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                               0.3
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                  0.3
✗ RestrictNamespaces=~CLONE_NEWUSER                           Service may create user namespaces                                     0.3
✗ RestrictAddressFamilies=~…                                  Service may allocate exotic sockets                                    0.3
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unres…      0.2
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                   0.2
✗ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has network configuration privileges                           0.2
✗ CapabilityBoundingSet=~CAP_RAWIO                            Service has raw I/O access                                             0.2
✗ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service may load kernel modules                                        0.2
✗ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes may change the system clock                          0.2
✗ DeviceAllow=                                                Service has no device ACL                                              0.2
✗ IPAddressDeny=                                              Service does not define an IP address whitelist                        0.2
✓ KeyringMode=                                                Service doesn't share key material with other services
✗ NoNewPrivileges=                                            Service processes may acquire new privileges                           0.2
✓ NotifyAccess=                                               Service child processes cannot alter service state
✗ PrivateDevices=                                             Service potentially has access to hardware devices                     0.2
✗ PrivateMounts=                                              Service may install system mounts                                      0.2
✗ PrivateTmp=                                                 Service has access to other software's temporary files                 0.2
✗ PrivateUsers=                                               Service has access to other users                                      0.2
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                0.2
✗ ProtectControlGroups=                                       Service may modify the control group file system                       0.2
✗ ProtectHome=                                                Service has full access to home directories                            0.2
✗ ProtectKernelLogs=                                          Service may read from or write to the kernel log ring buffer           0.2
✗ ProtectKernelModules=                                       Service may load or read kernel modules                                0.2
✗ ProtectKernelTunables=                                      Service may alter kernel tunables                                      0.2
✗ ProtectSystem=                                              Service has full access to the OS file hierarchy                       0.2
✗ RestrictAddressFamilies=~AF_PACKET                          Service may allocate packet sockets                                    0.2
✗ RestrictSUIDSGID=                                           Service may create SUID/SGID files                                     0.2
✗ SystemCallArchitectures=                                    Service may execute system calls with all ABIs                         0.2
✗ SystemCallFilter=~@clock                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@debug                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@module                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@mount                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@raw-io                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@reboot                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@swap                                     Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@privileged                               Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@resources                                Service does not filter system calls                                   0.2
✓ AmbientCapabilities=                                        Service process does not receive ambient capabilities
✗ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has audit subsystem access                                     0.1
✗ CapabilityBoundingSet=~CAP_KILL                             Service may send UNIX signals to arbitrary processes                   0.1
✗ CapabilityBoundingSet=~CAP_MKNOD                            Service may create device nodes                                        0.1
✗ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has elevated networking privileges                             0.1
✗ CapabilityBoundingSet=~CAP_SYSLOG                           Service has access to kernel logging                                   0.1
✗ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has privileges to change resource use parameters               0.1
✗ RestrictNamespaces=~CLONE_NEWCGROUP                         Service may create cgroup namespaces                                   0.1
✗ RestrictNamespaces=~CLONE_NEWIPC                            Service may create IPC namespaces                                      0.1
✗ RestrictNamespaces=~CLONE_NEWNET                            Service may create network namespaces                                  0.1
✗ RestrictNamespaces=~CLONE_NEWNS                             Service may create file system namespaces                              0.1
✗ RestrictNamespaces=~CLONE_NEWPID                            Service may create process namespaces                                  0.1
✗ RestrictRealtime=                                           Service may acquire realtime scheduling                                0.1
✗ SystemCallFilter=~@cpu-emulation                            Service does not filter system calls                                   0.1
✗ SystemCallFilter=~@obsolete                                 Service does not filter system calls                                   0.1
✗ RestrictAddressFamilies=~AF_NETLINK                         Service may allocate netlink sockets                                   0.1
✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                          0.1
    SupplementaryGroups=                                        Service runs as root, option does not matter
✗ CapabilityBoundingSet=~CAP_MAC_*                            Service may adjust SMACK MAC                                           0.1
✗ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service may issue reboot()                                             0.1
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree
✗ LockPersonality=                                            Service may change ABI personality                                     0.1
✗ MemoryDenyWriteExecute=                                     Service may create writable executable memory mappings                 0.1
    RemoveIPC=                                                  Service runs as root, option does not apply
✗ RestrictNamespaces=~CLONE_NEWUTS                            Service may create hostname namespaces                                 0.1
✗ UMask=                                                      Files created by service are world-readable by default                 0.1
✗ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service may mark files immutable                                       0.1
✗ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service may lock memory into RAM                                       0.1
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service may issue chroot()                                             0.1
✗ ProtectHostname=                                            Service may change system host/domainname                              0.1
✗ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service may establish wake locks                                       0.1
✗ CapabilityBoundingSet=~CAP_LEASE                            Service may create file leases                                         0.1
✗ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service may use acct()                                                 0.1
✗ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service may issue vhangup()                                            0.1
✗ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service may program timers that wake up the system                     0.1
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                     0.1

→ Overall exposure level for nginx.service: 9.6 UNSAFE 😨
```
{% endcollapsible %}

A lot of those capabilities are not required to run a web server, so it's best to limit the service's privileges. As interfacing with the Linux kernel can be very complex and is prone to changes, Systemd services offer a way to define common configurations directly in the service files. Given the multitude of configuration parameters for Systemd services, this example will concentrate on values significantly affecting security. It will use a standard Kubernetes `securityContext` as a foundation.

### The Principle of Least Privilege

Adopting the principle of least privilege is crucial. By restricting access and privileges to the bare essentials, the attack surface diminishes significantly. When using Kubernetes resources, you'd usually use a `securityContext` definition to limit capabilities of a Pod:

```yaml
...
    securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    runAsGroup: 2001
    allowPrivilegeEscalation: false
    privileged: false
    readOnlyRootFilesystem: true
    capabilities:
        drop:
        - all
...
```

In the above example, the process runs without root privileges on a read-only filesystem and all capabilities are dropped. A similar setup can be achieved using a Systemd service:

* `runAsNonRoot: true` ➜ no equivalent, if possible `DynamicUser` can be used
* `runAsUser: 1001` ➜ `User=<username>`
* `runAsGroup: 2001` ➜ `Group=<groupname>`
* `allowPrivilegeEscalation: false` ➜ `NoNewPrivileges=true`
* `privileged: false` ➜ no equivalent, `PrivateDevices=<...>`, `Protect<...>=<...>` etc. can be used
* `readOnlyRootFilesystem: true` ➜ `ProtectSystem=strict` / `TemporaryFileSystem=/:ro` (this also hides all files, needs Systemd >= 238)
* `capabilities.drop: ["all"]` ➜ `CapabilityBoundingSet=<...>`

There are a lot more ways to control the capabilities and permissions of Systemd services which are documented [here](https://www.freedesktop.org/software/systemd/man/systemd.exec.html). After applying some of these parameters to the Nginx service, the Unit File looks as follows:

`david@proxy:~$ systemctl cat nginx`

```ini
# /etc/systemd/system/nginx.service
# Rootless Nginx service based on https://github.com/stephan13360/systemd-services/blob/master/nginx/nginx.service
[Unit]
# This is from the default nginx.service
Description=nginx (hardened rootless)
Documentation=https://nginx.org/en/docs/
Documentation=https://github.com/stephan13360/systemd-services/blob/master/nginx/README.md
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
# forking is not necessary as `daemon` is turned off in the nginx config
Type=exec
User=nginx
Group=nginx
## can be used e.g. for accessing directory containing SSL certs
#SupplementaryGroups=acme
# define runtime directory /run/nginx as rootless services can't access /run
RuntimeDirectory=nginx
# write logs to /var/log/nginx
LogsDirectory=nginx
# write cache to /var/cache/nginx
CacheDirectory=nginx
# configuration is in /etc/nginx
ConfigurationDirectory=nginx

ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
# PID is not necessary here as the service is not forking
ExecReload=/usr/sbin/nginx -s reload

Restart=on-failure
RestartSec=10s

# Hardening
# hide the entire filesystem tree from the service and also make it read only, requires systemd >=238
TemporaryFileSystem=/:ro
# Remount (bind) necessary paths, based on https://gitlab.com/apparmor/apparmor/blob/master/profiles/apparmor.d/abstractions/base,
# https://github.com/jelly/apparmor-profiles/blob/master/usr.bin.nginx,
# https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RootDirectory=
#
# This gives access to (probably) necessary system files, allows journald logging
BindReadOnlyPaths=/lib/ /lib64/ /usr/lib/ /usr/lib64/ /etc/ld.so.cache /etc/ld.so.conf /etc/ld.so.conf.d/ /etc/bindresvport.blacklist /usr/share/zoneinfo/ /usr/share/locale/ /etc/localtime /usr/share/common-licenses/ /etc/ssl/certs/ /etc/resolv.conf
BindReadOnlyPaths=/dev/log /run/systemd/journal/socket /run/systemd/journal/stdout /run/systemd/notify
# Additional access to service-specific directories
BindReadOnlyPaths=/usr/sbin/nginx
BindReadOnlyPaths=/run/ /usr/share/nginx/

PrivateTmp=true
PrivateDevices=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectKernelTunables=true

# Network access
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6

# Miscellaneous
SystemCallArchitectures=native
# also implicit because settings like MemoryDenyWriteExecute are set
NoNewPrivileges=true
MemoryDenyWriteExecute=true
ProtectKernelLogs=true
LockPersonality=true
ProtectHostname=true
RemoveIPC=true
RestrictSUIDSGID=true
ProtectClock=true

# Capabilities to bind low ports (80, 443)
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target
```

Now, not only is the service running as non-root, but the process and sub-processes also only have access to a very limited part of the system. All filesystem access is dropped by default and only necessary system directories are either made available or substituted by temporary paths. Besides that, persistence is only possible where necessary which further limits the attack surface. Running `systemd-analyze` again on the new service, the results are showing effect:

`david@proxy:~$ systemd-analyze security nginx.service --no-pager`
{% collapsible Nginx Service Security Summary %}
```
  NAME                                                        DESCRIPTION                                                       EXPOSURE
✗ PrivateNetwork=                                             Service has access to the host's network                               0.5
✓ User=/DynamicUser=                                          Service runs under a static non-root user identity
✗ CapabilityBoundingSet=~CAP_SET(UID|GID|PCAP)                Service may change UID/GID identities/capabilities                     0.3
✗ CapabilityBoundingSet=~CAP_SYS_ADMIN                        Service has administrator privileges                                   0.3
✗ CapabilityBoundingSet=~CAP_SYS_PTRACE                       Service has ptrace() debugging abilities                               0.3
✗ RestrictAddressFamilies=~AF_(INET|INET6)                    Service may allocate Internet sockets                                  0.3
✗ RestrictNamespaces=~CLONE_NEWUSER                           Service may create user namespaces                                     0.3
✓ RestrictAddressFamilies=~…                                  Service cannot allocate exotic sockets
✗ CapabilityBoundingSet=~CAP_(CHOWN|FSETID|SETFCAP)           Service may change file ownership/access mode/capabilities unres…      0.2
✗ CapabilityBoundingSet=~CAP_(DAC_*|FOWNER|IPC_OWNER)         Service may override UNIX file/IPC permission checks                   0.2
✗ CapabilityBoundingSet=~CAP_NET_ADMIN                        Service has network configuration privileges                           0.2
✓ CapabilityBoundingSet=~CAP_RAWIO                            Service has no raw I/O access
✓ CapabilityBoundingSet=~CAP_SYS_MODULE                       Service cannot load kernel modules
✓ CapabilityBoundingSet=~CAP_SYS_TIME                         Service processes cannot change the system clock
✗ DeviceAllow=                                                Service has a device ACL with some special devices                     0.1
✗ IPAddressDeny=                                              Service does not define an IP address whitelist                        0.2
✓ KeyringMode=                                                Service doesn't share key material with other services
✓ NoNewPrivileges=                                            Service processes cannot acquire new privileges
✓ NotifyAccess=                                               Service child processes cannot alter service state
✓ PrivateDevices=                                             Service has no access to hardware devices
✓ PrivateMounts=                                              Service cannot install system mounts
✓ PrivateTmp=                                                 Service has no access to other software's temporary files
✗ PrivateUsers=                                               Service has access to other users                                      0.2
✗ ProtectClock=                                               Service may write to the hardware clock or system clock                0.2
✓ ProtectControlGroups=                                       Service cannot modify the control group file system
✗ ProtectHome=                                                Service has full access to home directories                            0.2
✓ ProtectKernelLogs=                                          Service cannot read from or write to the kernel log ring buffer
✓ ProtectKernelModules=                                       Service cannot load or read kernel modules
✓ ProtectKernelTunables=                                      Service cannot alter kernel tunables (/proc/sys, …)
✗ ProtectSystem=                                              Service has full access to the OS file hierarchy                       0.2
✓ RestrictAddressFamilies=~AF_PACKET                          Service cannot allocate packet sockets
✓ RestrictSUIDSGID=                                           SUID/SGID file creation by service is restricted
✓ SystemCallArchitectures=                                    Service may execute system calls only with native ABI
✗ SystemCallFilter=~@clock                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@debug                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@module                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@mount                                    Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@raw-io                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@reboot                                   Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@swap                                     Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@privileged                               Service does not filter system calls                                   0.2
✗ SystemCallFilter=~@resources                                Service does not filter system calls                                   0.2
✗ AmbientCapabilities=                                        Service process receives ambient capabilities                          0.1
✗ CapabilityBoundingSet=~CAP_AUDIT_*                          Service has audit subsystem access                                     0.1
✗ CapabilityBoundingSet=~CAP_KILL                             Service may send UNIX signals to arbitrary processes                   0.1
✓ CapabilityBoundingSet=~CAP_MKNOD                            Service cannot create device nodes
✗ CapabilityBoundingSet=~CAP_NET_(BIND_SERVICE|BROADCAST|RAW) Service has elevated networking privileges                             0.1
✓ CapabilityBoundingSet=~CAP_SYSLOG                           Service has no access to kernel logging
✗ CapabilityBoundingSet=~CAP_SYS_(NICE|RESOURCE)              Service has privileges to change resource use parameters               0.1
✗ RestrictNamespaces=~CLONE_NEWCGROUP                         Service may create cgroup namespaces                                   0.1
✗ RestrictNamespaces=~CLONE_NEWIPC                            Service may create IPC namespaces                                      0.1
✗ RestrictNamespaces=~CLONE_NEWNET                            Service may create network namespaces                                  0.1
✗ RestrictNamespaces=~CLONE_NEWNS                             Service may create file system namespaces                              0.1
✗ RestrictNamespaces=~CLONE_NEWPID                            Service may create process namespaces                                  0.1
✗ RestrictRealtime=                                           Service may acquire realtime scheduling                                0.1
✗ SystemCallFilter=~@cpu-emulation                            Service does not filter system calls                                   0.1
✗ SystemCallFilter=~@obsolete                                 Service does not filter system calls                                   0.1
✓ RestrictAddressFamilies=~AF_NETLINK                         Service cannot allocate netlink sockets
✗ RootDirectory=/RootImage=                                   Service runs within the host's root directory                          0.1
✓ SupplementaryGroups=                                        Service has no supplementary groups
✗ CapabilityBoundingSet=~CAP_MAC_*                            Service may adjust SMACK MAC                                           0.1
✗ CapabilityBoundingSet=~CAP_SYS_BOOT                         Service may issue reboot()                                             0.1
✓ Delegate=                                                   Service does not maintain its own delegated control group subtree
✓ LockPersonality=                                            Service cannot change ABI personality
✓ MemoryDenyWriteExecute=                                     Service cannot create writable executable memory mappings
✓ RemoveIPC=                                                  Service user cannot leave SysV IPC objects around
✗ RestrictNamespaces=~CLONE_NEWUTS                            Service may create hostname namespaces                                 0.1
✗ UMask=                                                      Files created by service are world-readable by default                 0.1
✗ CapabilityBoundingSet=~CAP_LINUX_IMMUTABLE                  Service may mark files immutable                                       0.1
✗ CapabilityBoundingSet=~CAP_IPC_LOCK                         Service may lock memory into RAM                                       0.1
✗ CapabilityBoundingSet=~CAP_SYS_CHROOT                       Service may issue chroot()                                             0.1
✓ ProtectHostname=                                            Service cannot change system host/domainname
✗ CapabilityBoundingSet=~CAP_BLOCK_SUSPEND                    Service may establish wake locks                                       0.1
✗ CapabilityBoundingSet=~CAP_LEASE                            Service may create file leases                                         0.1
✗ CapabilityBoundingSet=~CAP_SYS_PACCT                        Service may use acct()                                                 0.1
✗ CapabilityBoundingSet=~CAP_SYS_TTY_CONFIG                   Service may issue vhangup()                                            0.1
✓ CapabilityBoundingSet=~CAP_WAKE_ALARM                       Service cannot program timers that wake up the system
✗ RestrictAddressFamilies=~AF_UNIX                            Service may allocate local sockets                                     0.1

→ Overall exposure level for nginx.service: 6.1 MEDIUM 😐
```
{% endcollapsible %}

The score shows there's still room for improvement, but in the end, a lot of potential attack vectors have been mitigated in comparison to the officially provided Unit file.

## 🚀 Where to Continue

In summary, Systemd offers a straightforward method for constraining a process's capabilities, primarily leveraging Linux namespaces. This approach can significantly enhance security, but it does have its constraints. That is where Mandatory Access Control steps in, with tools such as [AppArmor](https://ubuntu.com/server/docs/security-apparmor) and [SELinux](https://www.redhat.com/en/topics/linux/what-is-selinux) providing fine grained control over system access. These tools enable a more nuanced approach to restricting system access, albeit with a more intricate configuration process. It's worth noting that numerous Linux distributions provide predefined profiles for a wide range of services, simplifying the implementation of these controls.

Ultimately, achieving a balance between security and practical implementation boils down to leveraging Systemd's capabilities alongside predefined Mandatory Access Control profiles. This approach strikes an effective compromise, ensuring both enhanced security and efficient deployment timelines.