Deploy Nginx Load Balancer for Rancher
Tags: rancher, loadbalancer, nginx
Published: 2024-04-03 22:53:40

## Introduction
In the dynamic landscape of container orchestration, efficient load balancing is paramount. As organizations adopt Kubernetes and Rancher rke2 for managing their workloads, the need for robust load balancing solutions becomes increasingly critical. Nginx is a versatile web server and reverse proxy that can also serve as a powerful Layer 4 load balancer.

In this article, we delve into the intricacies of leveraging Nginx to distribute incoming requests across control plane nodes in your Kubernetes cluster. By harnessing Nginx’s stream module, we unlock the ability to perform Layer 4 load balancing, ensuring optimal traffic distribution and high availability for our applications and also a fixed registration address for rke2 nodes.
<br/>

## Prerequisites
we've installed Nginx on a Debian 12 with IP address 192.168.100.100. We also have 3 Server(Control plane nodes) and 3 Agent(Worker) nodes. In total, our machines are listed here:

* **Control-Plane-1** => IP: 192.168.100.11 , OS: Debian 12 , Hostname: kuber-master-1

* **Control-Plane-2** => IP: 192.168.100.12 , OS: Debian 12 , Hostname: kuber-master-2

* **Control-Plane-3** => IP: 192.168.100.13 , OS: Debian 12 , Hostname: kuber-master-3

* **Worker-1** => IP: 192.168.100.14 , OS: Debian 12 , Hostname: kuber-worker-1

* **Worker-2** => IP: 192.168.100.15 , OS: Debian 12 , Hostname: kuber-worker-2

* **Worker-3** => IP: 192.168.100.16 , OS: Debian 12 , Hostname: kuber-worker-3

* **L4 Load Balancer** => ip: 192.168.100.100
<br/>

### Install and Enable Stream module

To use Nginx as a layer 4 load balancer, it must have the **stream** module included or be able to dynamically use it. To check if Nginx can use Stream module, check Nginx configured modules:
```bash
nginx -V
```

The output will be something like below:
```bash
configure arguments: --with-cc-opt='-g -O2
-ffile-prefix-map=/build/nginx-AoTv4W/nginx-1.22.1=.
-fstack-protector-strong -Wformat -Werror=format-security
-fPIC -Wdate-time -D_FORTIFY_SOURCE=2'
--with-ld-opt='-Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf
--http-log-path=/var/log/nginx/access.log
--error-log-path=stderr --lock-path=/var/lock/nginx.lock
--pid-path=/run/nginx.pid
--modules-path=/usr/lib/nginx/modules
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat
--with-debug --with-pcre-jit --with-http_ssl_module
--with-http_stub_status_module --with-http_realip_module
--with-http_auth_request_module --with-http_v2_module
--with-http_dav_module --with-http_slice_module
--with-threads --with-http_addition_module
--with-http_flv_module --with-http_gunzip_module
--with-http_gzip_static_module --with-http_mp4_module
--with-http_random_index_module
--with-http_secure_link_module --with-http_sub_module
--with-mail_ssl_module --with-stream_ssl_module
--with-stream_ssl_preread_module --with-stream_realip_module --with-http_geoip_module=dynamic
--with-http_image_filter_module=dynamic
--with-http_perl_module=dynamic
--with-http_xslt_module=dynamic --with-mail=dynamic
--with-stream=dynamic --with-stream_geoip_module=dynamic
```
Check for the option `--with-stream=dynamic`. This argument is necessary as it allows us to use the stream module in Nginx.

**NOTE**: If you can't find the `--with-stream=dynamic` option, you've to recompile and install Nginx with this option.

Now that Nginx supports stream module, we must install this module so as to use it in Nginx. Install the stream module using the command below:
```bash
apt-get install -y libnginx-mod-stream
```

After the module is installed, we should include it in the Nginx config file. Go to `/etc/nginx/nginx.conf` and include it:
```bash
user nginx;
worker_processes  auto;

load_module modules/ngx_stream_module.so;
### Other lines are emitted
```

The modules are usually located in `/usr/lib/nginx/modules`. If you want to verify it, use `nginx -V` and check the output. There is the option `--modules-path` that indicates the modules path.
<br/>

### Configure Stream in Nginx

Now that the stream module has been installed and imported, it's time to use it.
At the end of the Nginx main config file in `/etc/nginx/nginx.conf` add the following lines:
```bash
stream {
        log_format basic '$remote_addr [$time_local] '
                     '$protocol $status $bytes_sent $bytes_received '
                     '$session_time';
        include /etc/nginx/conf.d/*.conf;
}
```
Here we've specified the log format of the stream module. It's also been specified that files in `/etc/nginx/conf.d` ending with **.conf** are operated in layer 4 and not layer 7.

As the Nginx instance is supposed to be used for Rancher and Kubernetes, it should have two upstreams, one for rke2 and one for the kube-api server.

First of all, we add configuration for Rancher. Create `/etc/nginc/conf.d/rke.conf` and add the following lines:
```bash
upstream rancher_nodes {
        server 192.168.100.11:9345 fail_timeout=1s max_fails=2;
        server 192.168.100.12:9345 fail_timeout=1s max_fails=2;
        server 192.168.100.13:9345 fail_timeout=1s max_fails=2;
}

server {
    listen 9345;

    access_log  /var/log/nginx/domains/rke/access.log basic;

    proxy_pass rancher_nodes;

}
```
The rke2 service listens on port 9345 and here Nginx proxy passes to port 9345 on Control plane nodes.

The next part will be the Kubernetes part. Again create `/etc/nginc/conf.d/k8s.conf` and add the below content:
```bash
upstream kubernetes_nodes {
        server 192.168.100.11:6443 fail_timeout=1s max_fails=2;
        server 192.168.100.12:6443 fail_timeout=1s max_fails=2;
        server 192.168.100.13:6443 fail_timeout=1s max_fails=2;
}

server {
    listen 6443;

    access_log  /var/log/nginx/domains/k8s/access.log basic;

    proxy_pass kubernetes_nodes;

}
```
The kube-api server operates on port 6443 So we've used reverse proxy for these ports on Control plane nodes.
<br/>

### Run Nginx

Test Nginx configuration:
```bash
nginx -t
```

If everything is ok, Restart the Nginx service:
```bash
systemctl restart nginx
```

Now if you check open ports, there must be 6443 and 9345 present.
```bash
ss -ntlp
```
<br/>

Congratulations! Nginx from now on, should listen and load balance your requests to Control planes.
<br/>

## Summary
In this article we made Nginx to operate as a layer 4 load balancer with the help of Stream module.
We also covered how to use Nginx as a fixed registration address for Rancher rke2 and Kubernetes plus load balancing the requests to the Kubernetes API server.
To get more info about using Nginx and integrating it with Rancher rke2, please refer to the official documentation of Nginx and Rancher:
* https://nginx.org/en/docs/stream/ngx_stream_core_module.html
* https://docs.rke2.io/
<br/>


You could also visit my other article about deploying Rancher rke2 with Cilium and Metallb if you're interested:
https://dev.to/arman-shafiei/install-rke2-with-cilium-and-metallb-48a4
<br/>


Thank you for reading this article and please leave a comment if you have any suggestions or figured out an issue with this post.