modern-pylogging 0.4.0
Helper library for configuring python logging
Requires Python: >=3.10

# Modern PyLogging

A library to simplify logging configuration.  

Inspired by `litestar`'s logging configuration.

[![Python Version](https://img.shields.io/pypi/pyversions/modern-pylogging.svg)](https://pypi.org/project/modern-pylogging/)
[![PyPI Version](https://img.shields.io/pypi/v/modern-pylogging.svg)](https://pypi.org/project/modern-pylogging/)
[![License](https://img.shields.io/github/license/wwarne/modern_pylogging.svg)](https://github.com/wwarne/modern_pylogging/blob/main/LICENSE)

## Overview

ModernPyLogging provides a wrapper around Python’s standard `logging` library,  
offering structured logging with JSON output, sensible defaults, and simplified configuration.

It is designed primarily for `async` applications and allows you to `bind` context so that every log entry contains it.

It implements the [recommended approach](https://docs.python.org/3.12/howto/logging-cookbook.html#dealing-with-handlers-that-block) to setting up loggers.

There are notable differences in the `logging` module between Python < 3.12 and >= 3.12  
(including a bug in 3.12.4 regarding the `queue` parameter in `QueueHandler`).

Due to these variations, I decided to extract my logging configuration into a reusable library for multiple projects.

## Features

- 🚀 **Simple API** – Get started with just a few lines of code  
- 🧱 **Built on the standard library** – Uses Python’s built-in logging module  
- 🔍 **Structured logging** – JSON output for easy parsing and analysis  
- 🎨 **Console output** – Human-readable logs during development  
- 🔧 **Flexible configuration** – Via code or configuration files  
- 🔄 **Tool integration** – Can use `orjson` and `picologging` if installed  
- 📊 **Context management** – Easily add context to your logs  

## Installation

```bash
pip install modern-pylogging
````

## Quick Start

```python
import modern_pylogging

config = modern_pylogging.LoggingConfig()
get_logger = config.configure()  # sets up logging and returns a logger getter (like logging.getLogger)

# Default logging level is INFO

# Get a logger for your module
logger = get_logger(__name__)

# Start logging!
logger.info("Application started")
logger.debug("Debug information", extra={"user_id": 12345})
logger.warning("Something might be wrong")
logger.error("An error occurred", exc_info=True)
```

## Lazy get_logger

The LoggerProxy mechanism allows you to call get_logger(__name__) safely at import time, 
even if logging hasn't been configured yet. 

This is useful for libraries and applications where the logging configuration is deferred 
until runtime, but logger references are needed earlier.

```python
# my_module.py
from modern_pylogging import get_logger

logger = get_logger(__name__)

def do_something():
    logger.info("This works even before logging is set up!")

```
```python
# main.py
import modern_pylogging
from my_module import do_something

config = modern_pylogging.LoggingConfig()
get_logger = config.configure()
# after calling configure()`logger` in the my_module will be the correct type 
# for example if we use LoggingConfig(logging_module='logging') it will be python logging.Logger
# and if we use LoggingConfig(logging_module='picologging') it will be picologging.Logger
do_something()
```

If the `logger` instance is called before configuring logging at all - it will use default `logging`
module to emit a warning message and then your message.

#### Performance note

LoggerProxy adds ~1% overhead vs native loggers, due to dynamic attribute proxying.



## Default Configuration (Short Version)

* All logs are sent to `sys.stdout` (useful for Kubernetes or containers).
* All `filters`, `handlers`, etc., are attached to the root logger.
* `propagate` is not disabled – all logs from your app **and** external libraries are captured by the root logger.
* Default logging level is `INFO` (can be changed via the `LOGGING_LEVEL` env variable or manually).
* Logs are output in JSON format (Kubernetes-friendly).
* The environment (`env`, `system`, `inst`) is included in every log record, sourced from environment variables.
* `QueueHandler` is used to avoid blocking the main thread. It stores log records in a queue, and a corresponding
  `QueueListener` processes them separately (helpful for async event loops).
* If `orjson` is installed, it's used automatically for JSON dumping.
* If `picologging` is installed, it's used as a drop-in replacement for the standard `logging` module.
* Both `orjson` and `picologging` usage can be configured manually.

## Default Configuration (Extended Version)

```text
         MAIN THREAD             │      ANOTHER THREAD   
                                │                       
    APP─────┐                   │                       
           │                    │                       
           │                    │                       
     ┌─────▼──────────────────┐ │  ┌──────────────┐     
     │   Root logger          │ │  │QueueListener │     
     ├────────────────────────┤ │  └▲──────┬──────┘     
     │ Handler: QueueHandler  │ │   │      │            
     │                        │ │   │      │            
     └────┬───────────────────┘ │   │   ┌──▼─────────────┐       
          │                     │   │   │                │       
          │   ┌──────┐          │   │   │     STDOUT     │       
          └──►│QUEUE ├──────────┼───┘   └────────────────┘                
              └──────┘          │                       
                                │                       
```

### Differences Between Python ≤ 3.11 and ≥ 3.12

* **Python ≤ 3.11**: `QueueHandler` must have its own `Formatter` and sends pre-formatted data (usually `str`) to the queue.
  `QueueListener` simply passes them to its handlers (e.g., `stdout`). So the main thread performs all formatting and JSON serialization.

* **Python ≥ 3.12**: `QueueHandler` places `LogRecord` objects into the queue and must **not** have a `Formatter`.
  `QueueListener` receives raw `LogRecord`s and formats them in its own thread.

⚠️ Because of this, logging configs must differ depending on your Python version.

## Configuration

### Basic Configuration

The simplest way to configure logging:

```python
import modern_pylogging

# Configure with sensible defaults
config = modern_pylogging.LoggingConfig()
get_logger = config.configure()
```

Parameters of LoggingConfig:

`logging_module (Literal['logging', 'picologging']):`

Specifies the backend logging module to use. 'logging' is the standard library,
while 'picologging' is a faster, drop-in replacement.
'picologging' will be used by default if installed.

`json_dumps_module (Literal['json', 'orjson']):`

Module used to serialize log records to JSON format. 'orjson' provides better
performance and type handling. 'orjson' will be used by default if installed.

`version (Literal[1]):`

Schema version for the logging configuration.
The only valid value at present is 1

`disable_existing_loggers (bool):`

Whether to disable loggers that were previously configured before applying this config.

`filters (dict[str, dict[str, Any]] | None):`

A dict in which each key is a filter id and each value is a dict describing how to configure the
corresponding Filter instance.
https://docs.python.org/3/library/logging.html#filter-objects.

`formatters (dict[str, dict[str, Any]] | None):`

A dict in which each key is a formatter and each value is a dict describing how to configure the
corresponding Formatter instance. A 'standard' and 'json_fmt' formatter is provided.
standard - write everything as strings, good for development.
json_fmt - write output as json, good for production.
https://docs.python.org/3/library/logging.html#formatter-objects

`handlers (dict[str, dict[str, Any]] | None):`

A dict in which each key is a handler id and each value is a dict describing how to configure the
corresponding Handler_ instance. Two handlers are provided, 'console' and 'queue_handler'.
https://docs.python.org/3/library/logging.html#handler-objects

`loggers (dict[str, dict[str, Any]] | None):`

A dict in which each key is a logger name and each value is a dict describing how to configure the
corresponding Logger_ instance.
https://docs.python.org/3/library/logging.html#logger-objects

`root (dict[str, dict[str, Any] | list[Any] | str] | None):`

This will be the configuration for the root logger.
Processing of the configuration will be as for any logger,
except that the propagate setting will not be applicable.

`override_formatters (dict[HandlerName, FormatterName] | None):`

Sometimes you want just use default settings but change formatters
(e.g. to write to console as text instead of json)
You could provide a dict like {'console': 'standard', 'queue_handler': 'standard'}

`level (int | str | None):`

Optional default logging level to be applied globally (e.g., 'DEBUG', 10).

`capture_extra_fields (bool):`

Whether to capture additional fields from log records' `extra` parameter
into the structured output (especially relevant for JSON loggers).
This does not work for 'picologging'.
It's where you use like this - logger.info('abc', extra={'key': 'value'})


## Context Management

`modern_pylogging.update_log_extra` allows you to bind context to the current coroutine.

This context propagates to all child coroutines created with `await`, `TaskGroup`, `create_task`, or `gather`.

Example:

If the name of parameter has a dot in it - it would be a nested dictionary in json output.

```python
import modern_pylogging

modern_pylogging.update_log_extra({
    'request.id': 'super ID',
    'request.method': 'GET',
})

logger.info('Done processing')
# Extra data is nested in JSON output:
# {'request': {'id': 'super ID', 'method': 'GET'}, ...}
```

All coroutines in the same `asyncio` Task share context (if you’re using `await`).
So nested coroutines can modify the shared context.

```python
import modern_pylogging

async def func_a():
    modern_pylogging.update_log_extra({'a': 'AAA'})
    await func_to_call()

async def func_to_call():
    modern_pylogging.update_log_extra({'b': 'BBB'})
```

Child tasks, however, run with a copy of the context and **cannot** change the parent’s context.
(Applies when using `asyncio.gather`, `create_task`, or `TaskGroup`.)

```python
from asyncio import TaskGroup
import modern_pylogging

async def func_a():
    modern_pylogging.update_log_extra({'a': 'AAA'})
    async with TaskGroup() as tg:
        tg.create_task(func_to_call())

async def func_to_call():
    modern_pylogging.update_log_extra({'b': 'BBB'})
```

### Logging Context Example

```python
import asyncio
import modern_pylogging
import logging

modern_pylogging.LoggingConfig(logging_module='logging').configure()
logger = logging.getLogger('example')

async def main():
    logger.info('log from main')
    modern_pylogging.update_log_extra({'user_id': 42})
    logger.info('log from main [updated]')
    await another_async_func()
    logger.info('log after another_async_function')

async def another_async_func():
    modern_pylogging.update_log_extra({'service': 'k'})
    logger.info('log from another_async_func')
    await asyncio.gather(task1(), task2())

async def task1():
    logger.info('log from task1')

async def task2():
    logger.info('log from task2')
```

**Expected Output:**

```json
{"message":"log from main"}
{"message":"log from main [updated]", "user_id":42}
{"message":"log from another_async_func", "user_id":42, "service":"k"}
{"message":"log from task1", "user_id":42, "service":"k"}
{"message":"log from task2", "user_id":42, "service":"k"}
{"message":"log after another_async_function", "user_id":42, "service":"k"}
```

## Overriding Formatters

During development, you may prefer string output instead of JSON in the console.
Instead of redefining the entire logger config, use the `override_formatters` parameter.

`queue_handler` sends data to `console` handler and depending on python version you're using
it's even queue_handler's Formatter or console's Formatter.

```python
import logging
import os
import modern_pylogging

if os.environ.get('DEV_MODE'):
    override_formatters = {'queue_handler': 'standard', 'console': 'standard'}
else:
    override_formatters = {}

modern_pylogging.LoggingConfig(logging_module='logging', override_formatters=override_formatters).configure()

logger = logging.getLogger('example')
logger.info("Application started")
# Output (in dev mode):
>> 2025-05-09 20:24:08,888 | INFO | __main__:doc_file:13 | Application started |
```

## Configuring Existing Loggers

```python
import modern_pylogging

config = modern_pylogging.LoggingConfig(
    logging_module='logging',
    level='DEBUG',
    loggers={
        'httpx': {'handlers': [], 'level': 'INFO', 'propagate': True},
        'peewee': {'handlers': [], 'level': 'INFO', 'propagate': True},
        'aiokafka': {'handlers': [], 'level': 'INFO', 'propagate': True},
    }
)
config.configure()
# All logs use DEBUG level, except specified loggers which remain at INFO.
```