semvec/pypi-packages huggingface dataset:
nginx2es 0.3.0
Put parsed Nginx access.log to Elasticsearch
License: MIT
Keywords: nginx access.log elasticsearch

Put parsed Nginx access.log to Elasticsearch
============================================

Nginx access.log have to be formatted with this format:

.. code-block:: nginx

    log_format main_ext
        '$remote_addr $http_host $remote_user [$time_local] "$request" '
        '$status $body_bytes_sent "$http_referer" '
        '"$http_user_agent" "$http_x_forwarded_for" '
        'rt=$request_time ua="$upstream_addr" '
        'us="$upstream_status" ut="$upstream_response_time" '
        'ul="$upstream_response_length" '
        'cs=$upstream_cache_status';

Install
-------

Install with pip:

.. code-block:: bash

    pip install nginx2es

Features
--------

- Stable log record ID (hostname + file inode number + timestamp + file
  position). It makes possible to import log file more than once (adding some
  additional processing to ``nginx2es``, or dropping a daily index containing
  only a half of records, etc) without creating a duplicate records.

- Parse query params and split request uri path components to separate fields
  for complex log filtering / aggregations.

- Optional use of the GeoIP database (requires the ``geoip`` module and the
  ``GeoIPCity.dat`` database file) - adds ``city`` and ``region_name`` fields.

- Correctly parse log records containing information about multiple upstream
  responses.

- The ``tail -F``-like mode implemented with inotify.