Binding privileged port 514 to Logstash 7.10.0
Tags: devops, beginners, webdev, linux
Published: 2020-12-04 12:38:22

Sometimes there comes a scenario where you need to receive syslog traffic in the default `port 514` in your logstash server.  But as we know ports in range 1 to 1024 are privileged ports and only the root user can listen to them. By default, logstash daemon starts with its own user, so to resolve our problem we need to follow any below option. My production logstash server is on CentOS 7.

To bind logstash below 1024 ports we have 3 options.
1. run logstash as root.
2. use iptables to forward port 514 to an unprivileged port. 
3. use `setcap` to grant java permission to use privileged ports.

We could run the Logstash daemon through the root user but this process is not recommended and the process is also more complicated in the newer version of logstash so we gonna escape that option. You may prefer the 2nd option which is through iptables but their is limitation. If our logstash server is receiving millions of packets then every packet going through iptables rule will definitely increase our server CPU usages and we may face a performance bottleneck. So, therefore, we will choose the 3rd option in our case.  

I will install `logstash-7.10.0` through YUM repo which is bundled with JDK 11. That means I don't have to install java manually for my CentOS 7 anymore before logstash installation. Logstash offers architecture-specific downloads that include AdoptOpenJDK 11, the latest long term support (LTS) release of JDK.   

### Install logstash 7.10.0

Download and install the public signing key:
```yaml
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
```
Add the following in your `/etc/yum.repos.d/` directory in a file with a `.repo` suffix, for example `logstash.repo`  

`vim /etc/yum.repos.d/logstash.repo`
```yaml
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
```

And your repository is ready for use. You can install it with:
```yaml
yum install logstash
```
Now check bundled java version.
```yaml
/usr/share/logstash/jdk/bin/java -version
```
```yaml
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.8+10, mixed mode)
```

#### Add capabilities to the java binary  

**What is capabilities?**
(According to it's [man](https://linux.die.net/man/7/capabilities) page)
For the purpose of performing permission checks, traditional UNIX implementations distinguish two categories of processes: privileged processes (whose effective user ID is 0, referred to as superuser or root), and unprivileged processes (whose effective UID is nonzero). Privileged processes bypass all kernel permission checks, while unprivileged processes are subject to full permission checking based on the process's credentials (usually: effective UID, effective GID, and supplementary group list).

Starting with kernel 2.2, Linux divides the privileges traditionally associated with superuser into distinct units, known as capabilities, which can be independently enabled and disabled. Capabilities are a per-thread attribute. 

**CAP_NET_BIND_SERVICE:** Bind a socket to Internet domain privileged ports (port numbers less than 1024).  

The capabilities are added per file. This is why we need to modify the java binary itself. The capability we need to add is `CAP_NET_BIND_SERVICE`, which is explicitly defined as the capacity for an executable to bind to a port less than 1024.  
First, locate the java path in logstash. It's on `/usr/share/logstash/jdk/bin/java`
```yaml
setcap CAP_NET_BIND_SERVICE=+eip /usr/share/logstash/jdk/bin/java
``` 
Now check that the capability is added: 
```yaml
getcap /usr/share/logstash/jdk/bin/java

java = cap_net_bind_service+eip
```
`ldd` will show shared library dependencies of an executable binary `java`  
```yaml
ldd /usr/share/logstash/jdk/bin/java
```
```yaml
linux-vdso.so.1 =>  (0x00007ffd85fcf000)
libz.so.1 => /lib64/libz.so.1 (0x00007faa3d2c5000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faa3d0a9000)
libjli.so => /usr/share/logstash/jdk/bin/../lib/jli/libjli.so (0x00007faa3ce98000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007faa3cc94000)
libc.so.6 => /lib64/libc.so.6 (0x00007faa3c8c6000)
/lib64/ld-linux-x86-64.so.2 (0x00007faa3d6de000)
```
```yaml
systemctl start logstash
```
Now you will see error message in `/var/log/message` saying `libjli.so` missing library. Java couldn't find that library.
```yaml
logstash: /usr/share/logstash/jdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory.
```
Above `setcap` command breaks how java looks for it's library to run. To fix this, we need to symlink the library it’s looking for into `/usr/lib` and then run `ldconfig`
```yaml
ln -s /usr/share/logstash/jdk/lib/jli/libjli.so /usr/lib/
```
Execute command `ldconfig`. 
```yaml
ldconfig
```
Now create a test config file `/etc/logstash/conf.d/test.conf` listening on port `514`.
```yaml
input {
    syslog {
        port => "514"
    }
}

output {
    stdout { codec => rubydebug }
}
```

```yaml
systemctl restart logstash
```
```yaml
netstat -plnu
```
```yaml
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 127.0.0.1:323           0.0.0.0:*                           676/chronyd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           26999/java
udp6       0      0 ::1:323                 :::*                                676/chronyd
```
That's all. We have now successfully bind privileged port `514` to logstash.


In case if you need to remove capability then use  `setcap` command again with the option `-ep`
```yaml
setcap cap_net_bind_service=-ep /usr/share/logstash/jdk/bin/java
```