tailchaser 0.2.8
the ultimate tailer
License: BSD
Keywords: tail,watch,logs

========
Overview
========



The ultimate rotation and windows friendly log tailer plus a lot more...

- backfills rotated files
- handles gz and bz2 files
- handles multi line records
- doesn't break windows rotating log handlers
- easy to extend
- Free software: BSD license
- other than six pure Python stdlib.

Installation
============ 

::
    
    pip install tailchaser
    
Thsi will install the tailchaser library in your site-packages and it will also add the script tailchase to your Scripts directory. 


Usage
===== 
   
::

    $ tailchase /where/my/logs/*.log

    $ tailchase -h
    
    usage: tailchase [-h] [--only-backfill] [--dont-backfill]
                   [--clear-checkpoint] [--read-period READ_PERIOD]
                   [--read-pause READ_PAUSE] [--reading-from {unix,win}]
                   [--temp-dir TEMP_DIR]
                   [--logging {DEBUG,INFO,WARN,ERROR,CRITICAL}]
                   file-pattern


    positional arguments:
      file-pattern          The file pattern to tail, such as /var/log/access.*

    optional arguments:
      -h, --help            show this help message and exit
      --only-backfill       dont't tail, default: False
      --dont-backfill       basically only tail, default: False
      --clear-checkpoint    start form the begining, default: False
      --read-period READ_PERIOD
                        how long you read before you pause. If zero you don't
                        pause, default: 1
      --read-pause READ_PAUSE
                        how long you pause between reads, default: 0
      --reading-from PLATFORM
                        sets how long you read and then pause can be one of {unix,win}, default: win
      --temp-dir TEMP_DIR   on back fill files are copied to a temp directory.Use
                        this to set this directory, default: None
      --logging LEVEL
                        logging level it can be one of  DEBUG,INFO,WARN,ERROR,CRITICAL, default: ERROR



In its simplest form
--------------------

In its simplest form tailchase works like a combination of cat and tail -f except it will start with the oldest rotated file. So if you were to have ::

    /var/log/opendirectoryd.log
    /var/log/opendirectoryd.log.0
    /var/log/opendirectoryd.log.1
    /var/log/opendirectoryd.log.2
    /var/log/opendirectoryd.log.3
    /var/log/opendirectoryd.log.4

it would first output  the text of /var/log/opendirectoryd.log.4, then /var/log/opendirectoryd.log.3, etc. until it reached the newest file then when it  has reached the end of /var/log/opendirectoryd.log revert to a "tail -f" behaviour.

So if you were to do ::
    
    tailchase 'tests/logs/opendirectoryd.*'
    
"*note: In bash you need to quote the wildcard otherwise it will be expanded into the scripts argv array.*"

you would get something like ::

    2015-12-24 15:39:56.733754 EST - AID: 0x0000000000000000 - Registered node with name '/Contacts'
    2015-12-24 15:39:56.733933 EST - AID: 0x0000000000000000 - Registered node with name '/LDAPv3' as hidden
    2015-12-24 15:39:56.736154 EST - AID: 0x0000000000000000 - Registered node with name '/Local' as hidden
    2015-12-24 15:39:56.736868 EST - AID: 0x0000000000000000 - Registered node with name '/NIS' as hidden
    2015-12-24 15:39:56.737134 EST - AID: 0x0000000000000000 - Discovered configuration for node name '/Search' at path '       2015-12-24 15:39:56.737151 EST - AID: 0x0000000000000000 - Registered node with name '/Search'
    2015-12-24 15:39:56.738794 EST - AID: 0x0000000000000000 - Loaded bundle at path '/System/Library/OpenDirectory/Modules     2015-12-24 15:39:56.740509 EST - AID: 0x0000000000000000 - Loaded bundle at path '/System/Library/OpenDirectory/Modules/


Coding with tailchaser
======================


Using the tailchaser library in a project is probably best done by example.


Example 1 - Tailchase to a REST service.
----------------------------------------

::

    #
    # Example 1 - Tail to Elastic
    #

    import requests

    import tailchaser

    class TailToElastic(tailchaser.Tailer):
        def handoff(self, file_tailed, checkpoint, record):
            """ Expect a record like:

            20160204 10:28:15,525 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/lme-market.properties]
            20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-database.properties]
            20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-hibernate.properties]
            """

            date, time, level, source, _, message = record.split(5)
            result = requests.json("http://someelacticserver.com:9200/myindex/log", json={
                            'timestamp': '{}T{}'.format(date, time)
                            'level': level,
                            'source': source,
                            'message': message
                            })
            return result.status_code == requests.codes.ok


Example 2 - Tailchase to  Kafka
-------------------------------

::
    
    #
    # Example 2 - Tail to Kafka - shows how to add your own arguments and then send messages to kafka.
    #

    from kafka import KafkaProducer
    rom tailchaser.tailer import Tailer
    
    
    class TailToKafka(Tailer):
        def add_arguments(cls, parser=None):
            parser = super(TailToKafka, cls).add_arguments(parser)

        HOSTS = 'localhost:1234'
        TOPIC = b'log'
        def startup(self):
            self.kafka_producer = KafkaProducer(bootstrap_servers=self.HOSTS,value_serializer=msgpack.dumps)


        def handoff(self, file_tailed, checkpoint, record):
            """ Expect a record like:

            20160204 10:28:15,525 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/lme-market.properties]
            20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-database.properties]
            20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-hibernate.properties]
            """
            self.kafka_producer.send(self.TOPIC, record).get(timeout=10)
            return True
             


Example 3 - Saving and Loading last Offset from Zookeeper 
---------------------------------------------------------

::

    #
    # Example 3 - Saving and Loading last Offset from Zookeeper 
    #
    import platform
    import pickle
    import six
    from kazoo.client import KazooClient
    from tailchaser.tailer import Tailer

    class TailToElastic(Tailer):
        def start(self):
            self.zk = KazooClient(hosts=self.config.zookeeper_host)
            self.zk.start()
            
            
        def stop():
            self.zk.stop()
            
        def load_checkpoint(self):
            zk_path = self.config.checkpoint_filename
            try:
                checkpoint = pickle.loads(self.zk.get(zk_path))
                log.debug('loaded: %s %s', zk_path, checkpoint)
                return checkpoint
            except (IOError, EOFError):
                log.debug('failed to load: %s', zk_path)
                return self.INIT_CHECKPOINT

        def save_checkpoint(self, checkpoint):
            log.debug('dumping %s %s', self.config.checkpoint_filename, checkpoint)
            return self.zk.set(self.config.checkpoint_filename, six.b(pickle.dumps(checkpoint)))


        @staticmethod
        def make_checkpoint_filename(source_pattern, path=None):
            zk_path =  '/'.join(['tailchase', platform.node(), slugify(source_pattern)]
            self.zk.ensure_path(zk_path)
            return zk_path


Documentation
=============

https://tailchaser.readthedocs.org/

Development
===========

To run the all tests run::

    tox

Note, to combine the coverage data from all the tox environments run:

.. list-table::
    :widths: 10 90
    :stub-columns: 1

    - - Windows
      - ::

            set PYTEST_ADDOPTS=--cov-append
            tox

    - - Other
      - ::

            PYTEST_ADDOPTS=--cov-append tox



Changelog
=========

0.2.7 (2016-07-24)
--------------------
* added --filter-re to filter out lines #15

0.2.1 (2016-04-30)
--------------------
* Fixed #12 where Handoff is getting the tmp file name...
* removed temp files after handoff #14

0.2.0 (2016-04-30)
--------------------
* Added pipe code from ilabs code base.
* Added example of storing check points remotely #8
* Added REST configuration #9
* Added INIT_CHECKPOINT class attribute to Tailer #11



0.1.3 (2016-04-12)
--------------------
* Added break tail tests and fixes drived from it -  #6
* fixed app description ! - #7

0.1.2 (2016-04-05)
--------------------
* Add multi-line record support -  #3
* add --start-of-record-re option for multi-line records. - #4
* move argparser definition  from cli to Tailer class. - #5

0.1.0 (2016-02-21)
------------------
* First release on PyPI.