Impact of Hosting Large Numbers of Nginx SSL VHosts
Tags: nginx, tls, networking
Published: 2023-11-07 18:22:41

Originally posted by apeschel on Aug. 19, 2015, 6:08 p.m. 

## Abstract

A question was raised recently about what the impact of hosting extremely large amounts of SSL certificates via Nginx would be. This document provides an explanation of how Nginx manages SSL, provides data about the impact of hosting large numbers of certificates, and provides an analysis of what the data means.

## How Nginx Manages SSL Certificates

Logically, each SSL certificate will be associated with a separate host. In Nginx, separate hosts are managed via VHosts. A VHost is a collection of the IP, Port, Hostname, and other relevant data about a host. Nginx collects the information relevant to a VHosts into a `ngx_http_core_srv_conf_t` struct. These structs are contained within the `server_names_hash`, which uses the hostname string as a key.

Nginx defers SSL management to OpenSSL. It stores certificates in OpenSSL's `SSL_CTX` global context structure, and uses the OpenSSL API for any SSL-related interaction between the client and server.

When a connection is made to an nginx server, nginx looks up the given hostname in its server names hash, to find the relevant structure containing the information for that VHost. Once the struct is found, it can use the OpenSSL API to establish the SSL connection with the correct certificate for the requested domain.

There is an upper limit on the number of VHosts that Nginx can fit into its server hash. This upper limit is configurable via two settings:

```
server_names_hash_bucket_size
server_names_hash_max_size
```

For optimal performance, server_names_hash_bucket_size should be set to the same size as your processor's cache line. In Linux, you can find this value at `/sys/devices/system/cpu/cpu0/cache/index*/coherency_line_size`. This leaves you with `server_names_hash_max_size` as the value that should be modified to increase the VHosts maximum.

For more information about how Nginx manages this stuff, refer to the following Nginx docs:

* [Setting up hashes](http://nginx.org/en/docs/hash.html)
* [How nginx processes a request](http://nginx.org/en/docs/http/request_processing.html)

## Testing and Analyzing Nginx Performance

### Setup

A simple Nginx host was set up in Digital Ocean for testing. The tests were run on a 2CPU / 2GB instance.

For each iteration, all VHosts were running SSL on the same IP:Port pair, utilizing SNI to do so. Each VHost was configured with an SSL certificate, a corresponding private key, and a unique static file to host. All SSL certificates were generated using a 4096-bit RSA key.

siege was used for stress testing. All the hostnames were put into urls.txt, which is what siege uses to determine what hosts to send traffic to. Siege was configured to use 100 concurrently running clients, with each client attempting to retrieve 50 pages from hosts listed in urls.txt. This totals to 5000 total pages requested from the Nginx host, spread across all the VHosts running on it.

It was noted to me after initially writing this article that siege did not support SNI. It seemed unlikely that SNI support on the client side would have any impact on the results. In the interest of accuracy, I worked with the siege maintainer to add support for SNI and reran the tests. As expected, there was no impact on the results.

The goal of this testing was to determine the memory and performance overhead of hosting a large number of SSL certificates. This means the servers do not very closely reflect what an actual production host would look like. This is a deliberate attempt to isolate only the dependent variables for testing.

## Summarized Results

```
|Num Certs|Memory (KiB)|Time (seconds)|
---------------------------------------
|        1|        3000|            41|
|        2|        3200|            42|
|      100|        9600|            41|
|     1000|       40000|            44|
|    10000|      370000|            43|
```

### Analysis

Nginx took approximately the same amount of time, regardless of the number of VHosts. This is what would be expected from Nginx's use of a hash for managing the VHosts.

Performing a linear regression against the memory data provides us with the following relationship:

```
m = 3900 + 37 * n

Where

m = Expected Memory Usage (in KiB)
n = Num of VHosts using SSL
```

This demonstrates that each additional VHost consumes approximately 37 KiB of memory.

## Raw Data

The following values are used unless otherwise noted;

```
server_names_hash_bucket_size: 64
server_names_hash_max_size: 512
```

### No Certificates

```
root      1740  0.0  0.1  85880  1344 ?        Ss   18:36   0:00 nginx: master process /usr/sbin/nginx
www-data  1741  0.0  0.2  86224  2268 ?        S    18:36   0:00  \_ nginx: worker process
www-data  1742  0.0  0.2  86224  2276 ?        S    18:36   0:00  \_ nginx: worker process
www-data  1743  0.0  0.2  86224  2408 ?        S    18:36   0:00  \_ nginx: worker process
www-data  1744  0.0  0.1  86224  2028 ?        S    18:36   0:00  \_ nginx: worker process
```

```
siege -q -b -c 100 -r 50 http://1.apestest/

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  17.87 secs
Data transferred:               1.83 MB
Response time:                  0.30 secs
Transaction rate:             279.80 trans/sec
Throughput:                     0.10 MB/sec
Concurrency:                   83.88
Successful transactions:        5000
Failed transactions:               0
Longest transaction:            4.11
Shortest transaction:           0.05
```

### One Certificate

```
root      1740  0.0  0.2  85992  3012 ?        Ss   18:36   0:00 nginx: master process /usr/sbin/nginx
www-data  2377  0.0  0.1  86276  1904 ?        S    19:10   0:00  \_ nginx: worker process
www-data  2378  0.0  0.1  86276  1904 ?        S    19:10   0:00  \_ nginx: worker process
www-data  2379  0.0  0.1  86276  1904 ?        S    19:10   0:00  \_ nginx: worker process
www-data  2380  0.0  0.3  86276  3116 ?        S    19:10   0:00  \_ nginx: worker process
```
```
siege -q -b -c 100 -r 50 https://1.apestest/

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  40.97 secs
Data transferred:               1.83 MB
Response time:                  0.73 secs
Transaction rate:             122.04 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   89.43
Successful transactions:        5000
Failed transactions:               0
Longest transaction:            5.80
Shortest transaction:           0.12
```

### Two Certificates

```
root      1740  0.0  0.3  86680  3216 ?        Ss   Aug10   0:00 nginx: master process /usr/sbin/nginx
www-data 22502  0.0  0.1  86680  1992 ?        S    21:32   0:00  \_ nginx: worker process
www-data 22503  0.3  0.3  86680  3440 ?        S    21:32   0:00  \_ nginx: worker process
www-data 22504  0.0  0.1  86680  1992 ?        S    21:32   0:00  \_ nginx: worker process
www-data 22505  0.0  0.1  86680  1992 ?        S    21:32   0:00  \_ nginx: worker process
```
```
siege -q -b -c 100 -r 50

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  41.92 secs
Data transferred:               1.83 MB
Response time:                  0.75 secs
Transaction rate:             119.27 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   89.68
Successful transactions:        5000
Failed transactions:               0
Longest transaction:            7.86
Shortest transaction:           0.12
```

### One Hundred Certificates

```
root      1740  0.0  0.9  92476  9560 ?        Ss   Aug10   0:00 nginx: master process /usr/sbin/nginx
www-data 30236  0.0  0.8  92476  8156 ?        S    13:13   0:00  \_ nginx: worker process
www-data 30237  0.0  0.8  92476  8156 ?        S    13:13   0:00  \_ nginx: worker process
www-data 30238  0.0  0.8  92476  8156 ?        S    13:13   0:00  \_ nginx: worker process
www-data 30239  0.0  0.8  92476  8156 ?        S    13:13   0:00  \_ nginx: worker process
```

```
siege -q -b -c 100 -r 50

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  40.50 secs
Data transferred:               0.67 MB
Response time:                  0.73 secs
Transaction rate:             123.46 trans/sec
Throughput:                     0.02 MB/sec
Concurrency:                   90.17
Successful transactions:           0
Failed transactions:               0
Longest transaction:            6.91
Shortest transaction:           0.00
```

### One Thousand Certificates

```
server_names_hash_max_size: 1024
```

```
root       912  0.0  1.9 124704 40272 ?        Ss   Aug17   0:00 nginx: master process /usr/sbin/nginx
www-data  6543 18.2  1.9 124704 40288 ?        S    02:54   1:01  \_ nginx: worker process
www-data  6544 17.3  1.9 124704 40288 ?        S    02:54   0:58  \_ nginx: worker process
www-data  6545 18.2  1.9 124704 40288 ?        S    02:54   1:01  \_ nginx: worker process
www-data  6546 16.3  1.9 124704 40288 ?        S    02:54   0:55  \_ nginx: worker process
```
```
10079> siege -q -b -c 100 -r 50

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  43.83 secs
Data transferred:               1.83 MB
Response time:                  0.77 secs
Transaction rate:             114.08 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   87.71
Successful transactions:        5000
Failed transactions:               0
Longest transaction:           17.73
Shortest transaction:           0.13
```

### Ten Thousand Certificates

```
server_names_hash_max_size: 16384
```
```
root       912  0.0 17.8 449596 365304 ?       Ss   Aug17   0:04 nginx: master process /usr/sbin/nginx
www-data 15625  0.0 17.7 449596 363992 ?       S    03:09   0:00  \_ nginx: worker process
www-data 15626  0.0 17.7 449596 363992 ?       S    03:09   0:00  \_ nginx: worker process
www-data 15627  0.0 17.7 449596 363992 ?       S    03:09   0:00  \_ nginx: worker process
www-data 15628  0.0 17.7 449596 363992 ?       S    03:09   0:00  \_ nginx: worker process
```
```
siege -q -b -c 100 -r 50

Transactions:                   5000 hits
Availability:                 100.00 %
Elapsed time:                  42.71 secs
Data transferred:               1.83 MB
Response time:                  0.74 secs
Transaction rate:             117.07 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   87.15
Successful transactions:        5000
Failed transactions:               0
Longest transaction:           14.30
Shortest transaction:           0.12
```