kafka_influxdb 0.9.3
A Kafka consumer for InfluxDB
License: Apache
Keywords: kafka influxdb metrics consumer

Kafka-InfluxDB
==============

| |Build Status|
| |Coverage Status|
| |Code Climate|
| |PyPi Version|
| |Scrutinizer|

| A Kafka consumer for InfluxDB written in Python.
| Supports InfluxDB 0.9.x and up. For InfluxDB 0.8.x support, check out the `0.3.0 tag <https://github.com/mre/kafka-influxdb/tree/v0.3.0>`__.

Use cases
---------

| Kafka will serve as a buffer for your metric data during high load.
| Also it's useful for sending metrics from offshore data centers with unreliable connections to your monitoring backend.

|image5|

Quickstart
----------

For a quick test, run kafka-influxdb inside a container alongside Kafka and InfluxDB. Some sample messages are generated automatically on startup (using kafkacat).

Python 2:
^^^^^^^^^

::

    make
    docker exec -it kafkainfluxdb
    python -m kafka_influxdb -c config_example.yaml -s

Python 3:
^^^^^^^^^

::

    make RUNTIME=py3
    docker exec -it kafkainfluxdb
    python -m kafka_influxdb -c config_example.yaml -s

PyPy 5.x
^^^^^^^^

::

    make RUNTIME=pypy
    docker exec -it kafkainfluxdb
    pypy3 -m kafka_influxdb -c config_example.yaml -s --kafka_reader=kafka_influxdb.reader.kafka_python

(Note that one additional flag is given: ``--kafka_reader=kafka_influxdb.reader.kafka_python``. This is because PyPy is incompabile with the confluent kafka consumer which is a C-extension to librdkafka. Therefore we use the kafka\_python library here, which is compatible with PyPy but a bit slower.)

Installation
------------

::

    pip install kafka_influxdb
    kafka_influxdb -c config-example.yaml

Performance
-----------

| The following graph shows the number of messages/s read from Kafka for various Python versions and Kafka consumer plugins.
| This is testing against a Kafka topic with 10 partitions and five message brokers.
| As you can see the best performance is achieved on Python 3 using the ``-O`` flag for bytecode optimization in combination with the ``confluent-kafka`` reader (default setup). Note that encoding and sending the data to InfluxDB might lower this maximum performance although you should still see a significant performance boost compared to logstash.

|Benchmark results|

Benchmark
---------

For a quick benchmark, you can start a complete ``kafkacat -> Kafka -> kafka_influxdb -> Influxdb`` setup with the following command:

::

    make

This will immediately start reading messages from Kafka and write them into InfluxDB. To see the output, you can use the InfluxDB cli.

::

    docker exec -it docker_influxdb_1 bash # Double check your container name
    influx
    use metrics
    show measurements

Supported formats
-----------------

You can write a custom encoder to support any input and output format (even fancy things like Protobuf). Look at the examples inside the ``encoder`` directory to get started. The following formats are officially supported:

Input formats
^^^^^^^^^^^^^

-  `Collectd Graphite ASCII format <https://collectd.org/wiki/index.php/Graphite>`__: :

::

    mydatacenter.myhost.load.load.shortterm 0.45 1436357630

-  `Collectd JSON format <https://collectd.org/wiki/index.php/JSON>`__:

.. code:: json

    [{
        "values":[
           0.6
        ],
        "dstypes":[
           "gauge"
        ],
        "dsnames":[
           "value"
        ],
        "time":1444745144.824,
        "interval":10.000,
        "host":"xx.example.internal",
        "plugin":"cpu",
        "plugin_instance":"1",
        "type":"percent",
        "type_instance":"system"
     }]

-  `Raw InfluxDB line protocol (e.g. for Telegraf support) <https://github.com/mre/kafka-influxdb/issues/40>`__:

Output formats
^^^^^^^^^^^^^^

-  `InfluxDB 0.9.2+ line protocol format <https://influxdb.com/docs/v0.9/write_protocols/line.html>`__: :

::

    load_load_shortterm,datacenter=mydatacenter,host=myhost value="0.45" 1436357630

-  `InfluxDB 0.8.x JSON format <https://influxdb.com/docs/v0.8/api/reading_and_writing_data.html#writing-data-through-http>`__ (*deprecated*)

Configuration
-------------

| Take a look at the ``config-example.yaml`` to find out how to create a config file.
| You can overwrite the settings from the commandline. The following parameters are allowed:

+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| Option                                                      | Description                                                                                             |
+=============================================================+=========================================================================================================+
| ``-h``, ``--help``                                          | Show help message and exit                                                                              |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--kafka_host KAFKA_HOST``                                 | Hostname or IP of Kafka message broker (default: localhost)                                             |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--kafka_port KAFKA_PORT``                                 | Port of Kafka message broker (default: 9092)                                                            |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--kafka_topic KAFKA_TOPIC``                               | Topic for metrics (default: my\_topic)                                                                  |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--kafka_group KAFKA_GROUP``                               | Kafka consumer group (default: my\_group)                                                               |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--kafka_reader KAFKA_READER``                             | Kafka client library to use (kafka\_python or confluent) (default: kafka\_influxdb.reader.confluent)    |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_host INFLUXDB_HOST``                           | InfluxDB hostname or IP (default: localhost)                                                            |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_port INFLUXDB_PORT``                           | InfluxDB API port (default: 8086)                                                                       |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_user INFLUXDB_USER``                           | InfluxDB username (default: root)                                                                       |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_password INFLUXDB_PASSWORD``                   | InfluxDB password (default: root)                                                                       |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_dbname INFLUXDB_DBNAME``                       | InfluxDB database to write metrics into (default: metrics)                                              |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_use_ssl``                                      | Use SSL connection for InfluxDB (default: False)                                                        |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_verify_ssl``                                   | Verify the SSL certificate before connecting (default: False)                                           |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_timeout INFLUXDB_TIMEOUT``                     | Max number of seconds to establish a connection to InfluxDB (default: 5)                                |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_use_udp``                                      | Use UDP connection for InfluxDB (default: False)                                                        |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_retention_policy INFLUXDB_RETENTION_POLICY``   | Retention policy for incoming metrics (default: autogen)                                                |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--influxdb_time_precision INFLUXDB_TIME_PRECISION``       | Precision of incoming metrics. Can be one of 's', 'm', 'ms', 'u' (default: s)                           |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--encoder ENCODER``                                       | Input encoder which converts an incoming message to dictionary (default: collectd\_graphite\_encoder)   |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--buffer_size BUFFER_SIZE``                               | Maximum number of messages that will be collected before flushing to the backend (default: 1000)        |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``-c CONFIGFILE``, ``--configfile CONFIGFILE``              | Configfile path (default: None)                                                                         |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``-s``, ``--statistics``                                    | Show performance statistics (default: True)                                                             |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``-v``, ``--verbose``                                       | Set verbosity level. Increase verbosity by adding a v: -v -vv -vvv (default: 0)                         |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| ``--version``                                               | Show version                                                                                            |
+-------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+

Comparison with other tools
---------------------------

There is a Kafka input plugin and an InfluxDB output plugin for **logstash**. It supports Influxdb 0.9+. We've achieved a message throughput of around **5000 messages/second** with that setup. Check out the configuration at docker/logstash/config.conf. You can run the benchmark yourself:

::

    make RUNTIME=logstash
    docker exec -it logstash
    logstash -f config.conf

Please send a Pull Request if you know of other tools that can be mentioned here.

.. |Build Status| image:: https://travis-ci.org/mre/kafka-influxdb.svg?branch=master
   :target: https://travis-ci.org/mre/kafka-influxdb
.. |Coverage Status| image:: https://codecov.io/gh/mre/kafka-influxdb/branch/master/graph/badge.svg
   :target: https://codecov.io/gh/mre/kafka-influxdb
.. |Code Climate| image:: https://codeclimate.com/github/mre/kafka-influxdb/badges/gpa.svg
   :target: https://codeclimate.com/github/mre/kafka-influxdb
.. |PyPi Version| image:: https://badge.fury.io/py/kafka_influxdb.svg
   :target: https://badge.fury.io/py/kafka_influxdb
.. |Scrutinizer| image:: https://scrutinizer-ci.com/g/mre/kafka-influxdb/badges/quality-score.png?b=master
   :target: https://scrutinizer-ci.com/g/mre/kafka-influxdb/?branch=master
.. |image5| image:: https://raw.githubusercontent.com/mre/kafka-influxdb/master/assets/schema-small.png
.. |Benchmark results| image:: assets/benchmark.png