pmacct-to-elasticsearch 0.3.3a1
A Python script designed to read output from pmacct daemons, to process it and to store it into ElasticSearch.
License: MIT
Keywords: pmacct,NetFlow,accounting,BGP

pmacct-to-elasticsearch
=======================

**pmacct-to-elasticsearch** is a python script designed to read output from **pmacct** daemons, to process it and to store it into **ElasticSearch**. It works with both *memory* and *print* plugins and, optionally, it can perform **manipulations on data** (such as to add fields on the basis of other values).

.. image:: img/data_flow.png
        :align: center

1. **pmacct daemons** collect IP accounting data and process them with their plugins;
2. data are stored into **in-memory-tables** (*memory* plugins), **JSON or CSV files** (*print* plugins);
3. **crontab jobs** (*memory* plugins) or **trigger scripts** (*print* plugins) are invoked to execute pmacct-to-elasticsearch;
4. pmacct's output records are finally processed by **pmacct-to-elasticsearch**, which reads them from stdin (*memory* plugins) or directly from file.

Optionally, some **data transformations** can be configured, to allow pmacct-to-elasticsearch to **add or remove fields** to/from the output documents that are sent to ElasticSearch for indexing. These additional fields may be useful to enhance graphs and reports legibility, or to add a further level of aggregation or filtering.

Installation
------------

Install the program using pip:

.. code:: bash

  pip install pmacct-to-elasticsearch
        
Then clone the repository and run the ./install script to setup your system:

.. code:: bash

  cd /usr/local/src/
  git clone https://github.com/pierky/pmacct-to-elasticsearch.git
  cd pmacct-to-elasticsearch/
  ./install
  
Configuration
-------------

Please refer to the `CONFIGURATION.md`_ file. The `TRANSFORMATIONS.md`_ file contains details about data transformations configuration.

.. _CONFIGURATION.md: CONFIGURATION.md
.. _TRANSFORMATIONS.md: TRANSFORMATIONS.md

A simple tutorial on pmacct integration with ElasticSearch/Kibana using pmacct-to-elasticsearch can be found at http://blog.pierky.com/integration-of-pmacct-with-elasticsearch-and-kibana.

Future work
-----------

- Add support of more pmacct output formats (Apache Avro, ...).

Author
------

Pier Carlo Chiodi - https://pierky.com/

Blog: https://blog.pierky.com/ Twitter: `@pierky <https://twitter.com/pierky>`_


Changelog
=========

0.3.3
-----

- Fix: ES 7 compatibility.

  `Issue #14 <https://github.com/pierky/pmacct-to-elasticsearch/issues/14>`.

0.3.2
-----

- Code linting and fixing import and pip3 errors.

  Thanks to `@palisadoes (Peter Harrison) <https://github.com/palisadoes>` for these fixes.

0.3.1
-----

- Fix: ES 6.2 compatibility.

  `Issue #9 <https://github.com/pierky/pmacct-to-elasticsearch/issues/9>`.

0.3.0
-----

- New: **CSV** output support.

  The ``InputFormat`` option in the plugin configuration file can be used to instruct pmacct-to-elasticsearch to parse CSV output from pmacct.

- New: **Multithreading** support.

  The ``ReaderThreads`` option in the plugin configuration file sets the number of threads used to process pmacct's output.

- New: More command line arguments.

  The command line arguments under the *Configuration options* group can be used to override settings done on the plugin configuration file.

- Fix issue with index creation on ElasticSearch 5.x.

  Thanks to Kristoffer Olsson and Daniel Lindberg for reporting this and for their extensive support.

- Improved template for index creation.

- Fix an issue with transformations.

0.2.0
-----

- New feature: HTTP Authentication support for ES API.

0.1.0
-----

First release.