bes 0.3
Log actions to Elastic Search via bulk upload
License: BSD License

.. -*- coding: utf-8 -*-

========================
Getting Started With bes
========================

bes is a flexible `bulk uploader`_ for `Elastic Search`_.  At the
moment, it only supports index uploads `over UDP`_.  You can retrieve
the logged data from Elastic Search and analyze it dynamically using
Kibana_ (source__).  This package just makes the initial upload to
Elastic Search as painless as possible.

__ `Kibana source`_

The connection parameters and other configuration options are stored
in ``bes.DEFAULT``.  Override them as you see fit (e.g. to connect to
a remote Elastic Search server)::

  >>> import bes
  >>> bes.DEFAULT['host'] = 'my-es-server.example.com'

Then log away::

  >>> bes.log(type='record', user='jdoe', action='swims')

If you are so inclined, you can override the ``DEFAULT`` index_ on a
per-call basis::

  >>> bes.log(index='my-index', type='my-type', user='jdoe', action='bikes')

The log generated by the above will look like::

  {
    "@timestamp": "2013-09-26T16:34:09.179048",
    "@version": 1,
    "action": "bikes",
    "user": "jdoe"
  }

following Jordan Sissel's `new format`__ for Logstash_:

__ `Logstash format`_

You should specify a unique type_ for each record you create, because
Elastic Search doesn't recalculate its mapping_ if you post a new
event that uses an old key with a new data type.  If you are only
logging a single record type, you can configure a global default::

  >>> bes.DEFAULT['type'] = 'record'
  >>> bes.log(user='jdoe', action='runs')

Django
======

Although the core of the library is framework-agnostic, we'll be using
this to log events in a Django_ website.  There are a few helpers for
extracting information from HttpRequest_ to make this as easy as
possible.  Drop it into your views__ with somthing like::

  import django.http
  imort django.shortcuts
  import bes.django
  from polls.models import Poll

  def detail(request, poll_id):
      try:
          p = Poll.objects.get(pk=poll_id)
      except Poll.DoesNotExist:
          bes.django.log_user_request_path(
              request=request, type='404', poll_id=poll_id)
          raise django.http.Http404
      bes.django.log_user_request_path(
          request=request, type='poll-detail',
          poll_id=poll_id, poll_name=p.name)
      return django.shortcutsrender_to_response(
          'polls/detail.html', {'poll': p})

__ `Django views`_

You can also override bes' defaults in your Django config::

  BULK_ELASTIC_SEARCH_LOGGING_HOST = 'my-es-server.example.com'

The Django config names match the uppercased ``DEFAULT`` keys with a
``BULK_ELASTIC_SEARCH_LOGGING_`` prefix (for example, ``host`` →
``BULK_ELASTIC_SEARCH_LOGGING_HOST``).

Tracing
=======

To log activity for critical functions, bes has tracing decorators::

  import bes.trace

  @bes.trace.trace()
  def my_function():
      return 1 + 2 / 3

By default this logs with ``bes.log`` using the decorated function's
name (e.g. ``my_function``) as the log type.  You can override this if
you want something different::

  def my_logger(*args, **kwargs):
      print((args, kwargs))

  @bes.trace.trace(type='my-type', logger=my_logger)
  def my_function():
      return 1 + 2 / 3

Additional ``trace`` arguments are passed straight through to the
logger::

  @bes.trace.trace(index='my-index', type='my-type', sport='triathalon')
  def my_function():
      return 1 + 2 / 3

By default, you'll get separate logged messages before the decorated
function starts, after it completes, or if it raises an exception.  An
``action`` argument is passed to the logger in each case, with values
of ``start``, ``complete``, and ``error`` respectively.  In the error
case, the logger is also passed the string-ified exception
(``error=str(e)``) to help with debugging the problem.  If you don't
want one or more of these logged messages, you can turn them off::

  @bes.trace.trace(start=False, error=False, complete=False):
  def my_function():
      return 1 + 2 / 3

although if you turn them all off, you're not going to get any
messages at all ;).

The trace decorator uses wrapt_ (if it's installed) to create
introspection-preserving decorators.  If wrapt is not installed, we
fallback to the builtin `functools.wraps`_ which handles the basics
like docstrings, but doesn't do as well for more involved
introspection (e.g. `inspect.getsource`_).

Testing
=======

Testing uses unittest's `automatic test discovery`_.  Run the test
suite with::

  $ python -m unittest discover

For Python <3.3, the test suite requires the external mock_ package,
which is bundled as `unittest.mock`_ in Python 3.3.

.. _Elastic Search: http://www.elasticsearch.org/
.. _bulk uploader: http://www.elasticsearch.org/guide/reference/api/bulk/
.. _over UDP: http://www.elasticsearch.org/guide/reference/api/bulk-udp/
.. _Kibana: http://www.elasticsearch.org/overview/kibana/
.. _Kibana source: https://github.com/elasticsearch/kibana
.. _index: http://www.elasticsearch.org/guide/reference/glossary/#field
.. _type: http://www.elasticsearch.org/guide/reference/glossary/#type
.. _mapping: http://www.elasticsearch.org/guide/reference/mapping/
.. _Logstash: http://logstash.net/
.. _Logstash format: https://logstash.jira.com/browse/LOGSTASH-675
.. _Django: https://www.djangoproject.com/
.. _HttpRequest:
   https://docs.djangoproject.com/en/dev/ref/request-response/#httprequest-objects
.. _Django views: https://docs.djangoproject.com/en/dev/topics/http/views/
.. _wrapt: https://github.com/GrahamDumpleton/wrapt
.. _functools.wraps:
   http://docs.python.org/3/library/functools.html#functools.wraps
.. _inspect.getsource:
   http://docs.python.org/3/library/inspect.html#inspect.getsource
.. _automatic test discovery:
   http://docs.python.org/3/library/unittest.html#unittest-test-discovery
.. _mock: https://pypi.python.org/pypi/mock
.. _unittest.mock: http://docs.python.org/3/library/unittest.mock.html