context-log 0.1.2
A simple library to emit contextual information in structured logs (JSON)
License: MIT
Requires Python: >=3.6

# Python context-log library

## About

context-log is a simple library to emit contextual information in structured logs (JSON).

It works particularly well in a Docker or Serverless (e.g. AWS Lambda) environment where a single thread executes a request and produces a response.

The library uses python threading to store contextual information that is automatically added to all subsequent logs in a `contextMap` field.

Because the library uses the Python thread local context it works across packages and modules in a given project.

The approach is loosely based on the [Log4j 2 API Thread Context](https://logging.apache.org/log4j/2.x/manual/thread-context.html).

## Usage

Structured logging can be achieved with the [python-json-logger library](https://pypi.org/project/python-json-logger/).

Simply add project dependencies to requirements.txt:

```python
python_json_logger
PyYAML
context-log
```

Add the code below to the main code module.

Add the following YAML configuration in the `resources/logging.yaml` file, which outputs JSON structured logs to `stdout`.

```yaml
version: 1
formatters:
  json:
    class:  .jsonlogger.JsonFormatter
    format: '%(asctime)s %(name)s %(levelname)s %(message)s %(filename)s'
handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: json
root:
  level: DEBUG
  handlers:
    - console
```

Use the context_log library to emit logs. Example below.

```python
import logging.config
import yaml

with open('resources/logging.yaml', 'r') as log_config_file:
  logging.config.dictConfig(yaml.safe_load(log_config_file))

from context_log import ContextLog

def handler(event, context):
    # Clear context (e.g. re-use) and get logger
    log = ContextLog.get_logger('handler', True)
    log.info('start')

    ContextLog.put('ip', '1.2.3.4')

    # Helper to add start time in ISO and epoch time
    ContextLog.put_request_start_time()

    # Process request
    sleep(0.1)

    # Helper to add end time in ISO and epoch time
    # as well as duration in milliseconds
    ContextLog.put_request_end_time()

    log.info('end')
```

First log info event:

```json
{
    "asctime": "2019-09-19 11:53:20,479",
    "name": "handler",
    "levelname": "INFO",
    "message": "start",
    "filename": "test_example.py",
    "contextMap": {}
}
```

Second log info event:

```json
{
  "asctime": "2019-09-19 11:53:20,580",
  "name": "handler",
  "levelname": "INFO",
  "message": "end",
  "filename": "test_example.py",
  "contextMap": {
      "ip": "1.2.3.4",
      "start-time": "2019-09-19T11:53:20.480085",
      "epoch-start-time": 1568890400.480085,
      "end-time": "2019-09-19T11:53:20.580513",
      "epoch-end-time": 1568890400.580513,
      "duration": 100.428}
}
```

## The Detail

The standard logger is wrapped by a `LoggerAdapter`. It is therefore imperative that the `ContextLog.get_logger(name='<name>', clear=True|False)` call is made to get the logger to emit contextual logs.

Use `clear=True` when starting a new request in order to clear the previous context if the thread is re-used. This is typically the case in thread pools and in AWS Lambda's.

To manipulate or retrieve the `contextMap` use the following methods:

* `clear()`
* `put(key, value)`
* `get(key)`
* `get_map()`

There are also a number of helpers in an attempt to standardise log output `contextMap` fields:

* `put_request_id(request_id)`
* `put_request_method(request_method)`
* `put_request_path(request_path)`
* `put_response_status(response_status)`
* `put_start_time()`
* `put_end_time()`
* `put_request_user_id(request_user_id)`
* `put_request_client_id(request_client_id)`
* `put_request_primary_ip(primary_ip)`
* `put_request_client_ip(client_ip)`
* `put_request_viewer_country(viewer_country)`
* `put_trigger_source(trigger_source)`

## Contributing

Pull requests are more than welcome.

## Running pytest

Create virtualenv, download dependencies and run tests:

```bash
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r tests/requirements.txt
pip3 install -e .
pytest
```

## Running tox

```bash
pip3 install --user --upgrade tox
tox
```

## Releasing library to PyPI

Short version from the [Packaging Python Projects](https://packaging.python.org/tutorials/packaging-projects/) site.

Install the release tools:

```bash
python3 -m pip install --user --upgrade setuptools wheel twine
```

Remove old distribution(s):

```
rm -rf dist/
```

Build the context-log package:

```bash
python3 setup.py sdist bdist_wheel
```

Upload context-log first to Test PyPI:

```bash
python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*
```