Parsing nginx logs using python
Tags: nginx, logs, python, pandas
Published: 2023-02-26 19:13:29

**Introduction**

Nginx is a popular web server software used to serve web pages and other content on the internet. Nginx produces logs that contain information about the requests it receives and the responses it sends. Parsing these logs can provide valuable insights into website traffic and usage patterns. In this article, we will explore how to parse Nginx logs using Python.

Step 1: Understanding Nginx Log Format

Nginx logs are stored in a file, usually located in the /var/log/nginx directory. The log format can be configured using the nginx.conf file. The default log format for Nginx is the Combined Log Format, which includes the following fields:

    The remote IP address
    The time of the request
    The request method (GET, POST, etc.)
    The requested URL
    The HTTP version
    The HTTP status code
    The size of the response sent to the client
    The referrer URL
    The user agent string

The log format can be customized to include or exclude specific fields, or to add custom fields.

Step 2: Installing Required Libraries

To parse Nginx logs using Python, we need to install the following libraries:

    pandas: used for data manipulation and analysis.

You can install these libraries using the following command:
   `pip install pandas`

Step 3: Parsing Nginx Logs Using Python

To parse Nginx logs using Python, we can use the pandas library. The pandas library provides a powerful data structure called a DataFrame that allows us to manipulate and analyze data easily.

Here's an example Python script that reads an Nginx log file and creates a DataFrame:

```
import re
import shlex
import pandas as pd

class Parser:
    IP = 0
    TIME = 3
    TIME_ZONE = 4
    REQUESTED_URL = 5
    STATUS_CODE = 6
    USER_AGENT = 9

    def parse_line(self, line):
        try:
            line = re.sub(r"[\[\]]", "", line)
            data = shlex.split(line)
            result = {
                "ip": data[self.IP],
                "time": data[self.TIME],
                "status_code": data[self.STATUS_CODE],
                "requested_url": data[self.REQUESTED_URL],
                "user_agent": data[self.USER_AGENT],
            }
            return result
        except Exception as e:
            raise e


 if __name__ == '__main__':
    parser = Parser()
    LOG_FILE = "access.log"
    with open(LOG_FILE, "r") as f:
        log_entries = [parser.parse_line(line) for line in f]

    logs_df = pd.DataFrame(log_entries)
    print(logs_df.head())
```

Step 4: Data Analysis 

Once we have the Nginx log data in a DataFrame, we can perform various data analysis tasks.for example :

All requests with status code 404
`logs_df.loc[(logs_df["status_code"] == "404")]`

Requests from unique ip addresses
`logs_df["ip"].unique()`

Get all distinct user agents
`logs_df["user_agent"].unique()`

Get most requested urls
`logs_df["requested_url"].value_counts().to_dict()`

Conclusion

Parsing Nginx logs using Python can provide valuable insights into website traffic and usage patterns. By using the pandas library, we can easily read and manipulate the log data. With the right analysis, we can gain insights into website performance, user behavior, and potential security threats.

Github link :https://gist.github.com/ksn-developer/4072a9e092bccf68559c21f1c5ac2de2