An In-Depth Guide to Nginx Metrics
Tags: programming, monitoring, software
Published: 2019-10-15 16:39:20

<!-- wp:paragraph -->
<p>In our guides&nbsp;<a href="https://www.scalyr.com/community/guides/zen-and-the-art-of-system-monitoring.md">Zen and the Art of System Monitoring</a>&nbsp;and&nbsp;<a href="https://www.scalyr.com/community/guides/how-to-monitor-nginx-the-essential-guide.md">How to Monitor Nginx: The Essential Guide</a>, we cover our monitoring philosophy. We also recommend a specific set of metrics to monitor and alerts to set for maximum Nginx happiness.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Here, we'd like to dive into the nitty-gritty of those essential Nginx metrics. We'll discuss what exactly they mean and why they're important. This will also serve as a primer for some of the (perhaps esoteric) terminology associated with web servers.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You can think of this as a companion the&nbsp;<a href="http://nginx.org/en/docs/" target="_blank" rel="noopener noreferrer">official Nginx documentation</a>&nbsp;and as an appendix to our&nbsp;<a href="https://www.scalyr.com/community/guides/how-to-monitor-nginx-the-essential-guide.md">Nginx monitoring guide</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>For now, this guide covers only the metrics available via&nbsp;<a href="http://nginx.org/en/docs/http/ngx_http_stub_status_module.html" target="_blank" rel="noopener noreferrer"><code>ngx_http_stub_status_module</code></a>, plus those associated with the F/OSS version of Nginx. More comprehensive metrics are available at<a href="http://nginx.org/en/docs/http/ngx_http_status_module.html" target="_blank" rel="noopener noreferrer"><code>ngx_http_status_module</code></a>. (This is included with the commercial version, Nginx Plus.)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So roll up your sleeves, grab your Slanket or Snuggie, and let's talk Nginx metrics.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":2524,"align":"center"} -->
<div class="wp-block-image"><figure class="aligncenter"><img src="https://library.scalyr.com/2018/04/30192836/Ngnix_shape_with_Scalyr_colors_An_In-Depth_Guide_to_Nginx_Metrics.png" alt="" class="wp-image-2524"/></figure></div>
<!-- /wp:image -->

<!-- wp:more -->
<!--more-->
<!-- /wp:more -->

<!-- wp:heading -->
<h2>Data Sources</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Metrics are available from two sources:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ol>
 	<li><strong>Nginx Status Modules:</strong> This is the most direct way to get the goods. You can get data by polling a configurable web page (such as /status), or through variables written to log files. Polling is the best way to go, since Nginx does not provide embedded variables for everything in the status module.</li>
 	<li><strong>Log Files:</strong> Nginx, like most web servers, maintains an "access log" with a record of each request handled. You can use <a href="http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format" target="_blank" rel="noopener noreferrer">log_format</a> to instruct Nginx to include variable values in each log record. Monitoring tools that can read log records can use these variables to create more metrics.</li>
</ol>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>It's All About the Connections</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The status module metrics are mostly about&nbsp;<strong>connections</strong>, so it's worth talking about those in more detail.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>An Nginx server generally plays two roles. Its primary role is to serve content to clients. Content can be web pages, images, video, or raw API data. Clients used to be only web browsers, but now they include native mobile apps and other servers that consume your APIs.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":2268,"align":"right"} -->
<div class="wp-block-image"><figure class="alignright"><img src="https://blog.scalyr.com/wp-content/uploads/2018/10/As-a-proxy-nginx-can-do-a-few-things-really-really-well-1.png" alt="" class="wp-image-2268"/></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Nginx's secondary role is as a proxy between clients and "upstream servers." Upstream servers do more of the heavy lifting for a given request. They run application servers&nbsp;like Unicorn for Ruby on Rails, or Tomcat for Java. These application servers handle the "dynamic" part of dynamic web pages.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>As a proxy, Nginx can do a few things really, really well. It can serve local static content (images and non-dynamic HTML files) very quickly. It can act as a cache for content from upstream servers, and it can load-balance requests among several upstream servers.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Establishing Connections</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Before servers and clients can talk to each other,&nbsp;they have to establish a formal network-level <strong>connection.&nbsp;</strong>Application-level data flows over these connections via the higher-level protocols that Nginx supports. These protocols are most commonly HTTP. But they can also be WebSockets, SPDY, TCP streams, and even several mail protocols.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>One layer down from the application data (at the TCP layer of networking), a "handshake" process occurs. This handshake negotiates the establishment of a connection. The specific details are beyond the scope of this article, but you can learn more <a href="http://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_establishment" target="_blank" rel="noopener noreferrer">on Wikipedia</a>&nbsp;if you're so inclined. For even more on the specifics of connections and how they relate to HTTP, see&nbsp;<a href="http://odetocode.com/articles/743.aspx" target="_blank" rel="noopener noreferrer">A Software Developer's Guide to HTTP - Connections</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Nginx connections can be in one of several states:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<p style="padding-left: 60px;"><strong>Accepted:</strong> A connection moves to the Accepted state after the TCP handshake. It then takes one of three sub-states:</p>
<p style="padding-left: 60px;"><strong>Idle / Waiting:</strong> This connection is not currently sending or receiving data. This happens between the end of the request and the start of the response, or between a finished response and the next request. As of HTTP 1.1, all connections are persistent (i.e., remain open) unless declared otherwise.</p>
<p style="padding-left: 60px;"><strong>Reading:</strong> Nginx is reading data from the client.</p>
<p style="padding-left: 60px;"><strong>Writing:</strong> Nginx is writing data to the client.</p>
<p style="padding-left: 60px;"><strong>Handled:</strong> This means nginx has finished writing data to the client. It has successfully finished and closed the request.</p>
<p style="padding-left: 60px;"><strong>Dropped:</strong> This means nginx ended the connection before finishing the request. (This usually happens because of a resource or configuration limit.)</p>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>Nginx Status Module Metrics</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now let's take a look at the metrics available via&nbsp;<a href="http://nginx.org/en/docs/http/ngx_http_stub_status_module.html" target="_blank" rel="noopener noreferrer"><code>ngx_http_stub_status_module</code></a>:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>Active connections</code></strong>: The&nbsp;<em>current</em>&nbsp;number of active (accepted) connections from clients. Includes all connections with the statuses&nbsp;<strong>Idle / Waiting</strong>,&nbsp;<strong>Reading</strong>, and&nbsp;<strong>Writing</strong>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>accepts</code></strong>&nbsp;: The&nbsp;<em>total</em>&nbsp;number of accepted connections from clients since the nginx master process started. Note that reloading configurations or restarting worker processes will not reset this metric. If you terminate and restart the master process, you will reset the metric.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>handled</code>&nbsp;:&nbsp;</strong>The&nbsp;<em>total</em>&nbsp;number of handled connections from clients since the nginx master process started. This will be lower than&nbsp;<code>accepts</code>&nbsp;only in cases where a connection is dropped before it is handled.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>requests</code></strong>: The&nbsp;<em>total</em>&nbsp;number of client requests since the nginx master process started. A request is an application-level (HTTP, SPDY, etc.) event. It occurs when a client requests a resource via the application protocol. A single connection can (and often does) make many requests. So most of the time, there are more requests than accepted/handled connections.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>Reading</code></strong>&nbsp;: The&nbsp;<em>current</em>&nbsp;number of (accepted) connections from clients where nginx is reading the request. Measured at the time the status module was queried.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>Writing</code></strong>&nbsp;: The&nbsp;<em>current</em>&nbsp;number of connections from clients where nginx is writing a response back to the client.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>Waiting</code></strong>&nbsp;: The&nbsp;<em>current</em>&nbsp;number of connections from clients that are in the&nbsp;<strong>Idle / Waiting</strong>&nbsp;state.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Default Access Log Variables</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>It's beyond the scope of this guide to dive into&nbsp;<em>every</em>&nbsp;nginx log variable. Instead, we're going to take a close look at a few variables that are particularly important when it comes to monitoring.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The default nginx access log (obtained by declaring&nbsp;<code>log_format combined</code>&nbsp;in your configuration file) uses the following variables:</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$body_bytes_sent</code></strong>&nbsp;: The number of bytes sent to the client as the response body (not including the response header). This allows you to monitor individual response sizes. It also gives you a rough measure of outbound bandwidth.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$http_referer</code></strong>&nbsp;:&nbsp;The&nbsp;<code>HTTP Referer</code>&nbsp;header from the incoming HTTP request. The browser sets this header to the URL of the page that links to the current requested resource.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Two notes on this:</p>
<!-- /wp:paragraph -->

<!-- wp:list {"ordered":true} -->
<ol><li>The word "referrer," at least in English, is spelled with two Rs, but the original misspelling from the HTTP spec (<a href="http://tools.ietf.org/html/rfc1945" target="_blank" rel="noopener noreferrer">RFC 1945</a>) managed to stick around.</li><li>Since the client that makes the request also sets this header, it may not always be accurate.&nbsp;Malicious clients can change or "spoof" this header, which results in <a href="https://en.wikipedia.org/wiki/Referrer_spam" target="_blank" rel="noopener noreferrer">referral spam</a>.</li></ol>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p><strong><code>$http_user_agent</code></strong>&nbsp;:&nbsp;The&nbsp;<code>User-Agent</code>&nbsp;header from the incoming HTTP request. This identifies the specific browser, bot, or other software that issued the request. It can include other info, too, such as the client's operating system. The format is slightly different between human-operated web browsers and bots, but the theme is the same.&nbsp;<a href="http://en.wikipedia.org/wiki/User_agent" target="_blank" rel="noopener noreferrer">Wikipedia has some awesome nitty-gritty details on user agent strings.</a></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$remote_addr</code></strong>&nbsp;: The&nbsp;<a href="http://en.wikipedia.org/wiki/IP_address" target="_blank" rel="noopener noreferrer">IP address</a>&nbsp;of the client making the request. If the request passed through an intermediate device, such as a NAT firewall, web proxy, or your load balancer, this will be the address of the last device to relay the request.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$remote_user</code></strong>&nbsp;: The username supplied if&nbsp;<a href="http://en.wikipedia.org/wiki/Basic_access_authentication" target="_blank" rel="noopener noreferrer">HTTP Basic authentication</a>&nbsp;is used for the request.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$request</code></strong>&nbsp;: The raw HTTP request line. An example of a (familiar) request line is as follows:</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>GET /community/guides/an-in-depth-guide-to-nginx-metrics/ HTTP/1.1  
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>This is actually a compound variable made from three sub-variables (each of which is accessible individually if needed):</p>
<!-- /wp:paragraph -->

<!-- wp:code -->
<pre class="wp-block-code"><code>$request_method $request_uri $server_protocol
</code></pre>
<!-- /wp:code -->

<!-- wp:paragraph -->
<p>Let's break down of these variables:</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><strong><code>$request_method</code></strong>&nbsp;: The&nbsp;<a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html" target="_blank" rel="noopener noreferrer">HTTP method</a>&nbsp;of the request. The most common methods used by browsers are GET and POST, but the spec also includes HEAD, PUT, DELETE, OPTIONS, TRACE, and CONNECT. (You can find details on each in the preceding link.)</li><li><strong><code>$request_uri</code></strong>&nbsp;: The URI of the requested page, including query arguments. Even if nginx returns a different resource to the client, for instance because mod_rewrite was used, this field will still log the URI of the original request.</li><li><strong><code>$server_protocol</code></strong>&nbsp;: The application-level protocol and version used in the request. You'll most likely see HTTP/1.0 or HTTP/1.1, but nginx supports SPDY, WebSockets, and several mail protocols as well.</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p><strong><code>$time_local</code></strong>&nbsp;: The local (server) time the request was received. Shown in the format&nbsp;<code>dd/mm/yyyy:hh:mm:ss UTC-offset</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$status</code></strong>&nbsp;: The numeric&nbsp;<a href="http://en.wikipedia.org/wiki/List_of_HTTP_status_codes" target="_blank" rel="noopener noreferrer">HTTP Status Code</a>&nbsp;of the response. This is a key variable to monitor. It tells you about errors, missing pages, and other unusual events.</p>
<!-- /wp:paragraph -->

<!-- wp:heading {"level":3} -->
<h3>Additional Request Variables</h3>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>On top of the defaults, check out these variables to track even more metrics.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$bytes_sent</code></strong>&nbsp;: The total number of bytes sent to the client in the response, including headers. This is similar to $body_bytes_sent, but it provides a more complete picture.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$content_length</code></strong>&nbsp;: The HTTP content length request header field. This is the total size (in bytes) of the body of the request, as sent by the client.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$request_length</code></strong>&nbsp;: The full request length (in bytes). This includes the request line, header, and body, as calculated by nginx.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you're interested in monitoring overall incoming bandwidth, use&nbsp;<code>$request_length</code>. Because&nbsp;<code>$content_length</code>&nbsp;is drawn from a request header, the client calculates it. This means it has the potential to be spoofed (in the case of a DDoS attack, for example).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$request_time</code></strong>and&nbsp;<strong><code>$upstream_response_time</code></strong>&nbsp;— <code>$request_time</code>&nbsp;is the total time taken for nginx (and any upstream servers) to process a request and send a response. Nginx measures time in seconds with millisecond resolution. This is the primary source you should use for your server's response time metric.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Nginx starts the clock as soon as it reads the first bytes from the client request. It stops the clock after it sends the last bytes to the client. Note that this&nbsp;<em>includes</em>&nbsp;the processing time for upstream servers. If you want to break out those metrics, use&nbsp;<code>$upstream_response_time</code>, which only measures the upstream server's response time.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The language can be a bit confusing here, so let's clarify. Even though the variable is "request" time, it actually measures the elapsed time of the full request-response cycle (from the nginx server's perspective).</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$uri</code></strong>&nbsp;: The current URI of the request. Internal to nginx, this value may change during request processing (i.e., in the case of rewrites). The value that nginx logs represents the URI of the resource the client receives.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>This differs from&nbsp;<code>$request_uri</code>&nbsp;in that&nbsp;<code>$request_uri</code>&nbsp;does not reflect any URL rewrites internal to the nginx server.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$gzip_ratio</code></strong>&nbsp;: The compression ratio of the response (the ratio between the original and compressed response sizes). This applies if you've enabled&nbsp;<a href="http://www.gzip.org/" target="_blank" rel="noopener noreferrer">gzip</a>&nbsp;response compression. Through&nbsp;<a href="http://nginx.org/en/docs/http/ngx_http_gzip_module.html" target="_blank" rel="noopener noreferrer"><code>ngx_http_gzip_module</code></a>, this feature pipes responses through gzip before sending them to the client. It can reduce the size of responses by 50% or more and provide significant outbound bandwidth savings. Even though gzip is extremely fast, there is still material overhead in the compression process, both in terms of CPU usage and response time. Some of this overhead is worth the savings in transfer time from serving a smaller file.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Be sure to monitor your resource usage closely if you decide to use gzip compression. Further details about compression and decompression&nbsp;<a href="http://nginx.com/resources/admin-guide/compression-and-decompression/" target="_blank" rel="noopener noreferrer">are available from NGINX directly</a>.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$connection</code></strong>&nbsp;: The connection serial number. Nginx assigns a unique number to each connection. When multiple requests occur on a single connection, they will all have the same connection serial number. Serial numbers reset when the master nginx process terminates. This means serial numbers get reused, so they won't be unique for long periods of time.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$connection_requests</code></strong>&nbsp;: The number of requests made through this&nbsp;<code>$connection</code>.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Additional Server Level Variables</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p><strong><code>$host</code></strong>&nbsp;: The client used this DNS name to find your server, as seen in the&nbsp;<code>Host</code>&nbsp;HTTP header. If that header is empty or missing, nginx substitutes the name in the first&nbsp;<code>server_name</code>&nbsp;directive in your nginx configuration.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$http_HEADERNAME</code></strong>&nbsp;and&nbsp;<strong><code>$upstream_http_HEADERNAME</code></strong>&nbsp;: This follows the pattern of&nbsp;<code>$http_referer</code>&nbsp;and&nbsp;<code>$http_user_agent</code>&nbsp;above. Nginx allows you to log any&nbsp;<a href="http://en.wikipedia.org/wiki/List_of_HTTP_header_fields" target="_blank" rel="noopener noreferrer">HTTP request headers</a>&nbsp;by referencing&nbsp;<code>$http_</code>&nbsp;and the header name. (You must convert the header name to lowercase and replace dashes with underscores.)</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>You can also access the headers returned by any upstream servers by adding "upstream_" to the front of the desired header name.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$msec</code></strong>&nbsp;: The current time, in milliseconds, from the Unix epoch of 1/1/1970. This allows you to determine the exact time at which a request took place.</p>
<!-- /wp:paragraph -->

<!-- wp:list -->
<ul><li><strong>Open Question</strong>: Is this the time that nginx receives the request, or the time that nginx writes the log statement after the completion of the request?</li></ul>
<!-- /wp:list -->

<!-- wp:paragraph -->
<p><strong><code>$pid</code></strong>&nbsp;: The Process ID (PID) of the nginx worker that handled the request. This is to track metrics and workload of each worker individually.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Note: Worker processes can (and do) crash. The nginx master process can also stop and restart worker processes, so PIDs can come and go. The underlying server can also reuse PIDs.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong><code>$server_addr</code></strong>&nbsp;and&nbsp;<strong><code>$server_name</code></strong>&nbsp;: The IP address (<code>$server_addr</code>) or name (<code>$server_name</code>) of the nginx server that accepted a request. This is useful in a multi-server, load-balancing environment. In that case, you need to monitor which requests (and which metrics) each server handles.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Note: Computation of the IP address requires a system call unless you specifically bind an address using the&nbsp;<code>listen</code>directive in your configuration. Keep in mind that system calls can add significant overhead and impact server performance. Unless you need the full IP, $server_name is a better choice for many installations.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><em>This post was updated by Casey Dunham.&nbsp; <a href="http://www.caseydunham.com/" target="_blank" rel="noopener noreferrer">Casey </a>recently launched his own security business, is known for his unique approaches to all areas of application security, stemming from his 10+ year career as a professional software developer. His strengths include secure SDLC development consulting; threat modeling; developer training; and auditing web, mobile, and desktop applications for security flaws.</em></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><strong>To find out more on Scalyr and monitoring Nginx&nbsp;</strong><a href="https://www.scalyr.com/help/monitors/nginx"><strong>check out our docs</strong></a><strong>. Or <a href="https://www.scalyr.com/signup">Try Scalyr</a> for yourself.</strong></p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><em>This blog is an updated version of one originally written in 2017.</em></p>
<!-- /wp:paragraph -->