wmlog 1.0.4
Centralized Structured Logging for Microservices
License: Apache-2.0
Requires Python: >=3.8
Keywords: logging, websocket, mqtt, distributed, microservices, structured-logging, real-time, monitoring, observability, asyncio, broadcasting, centralized-logging

# wmlog - Centralized Structured Logging

[![PyPI version](https://badge.fury.io/py/wmlog.svg)](https://badge.fury.io/py/wmlog)
[![Python versions](https://img.shields.io/pypi/pyversions/wmlog.svg)](https://pypi.org/project/wmlog/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Downloads](https://pepy.tech/badge/wmlog)](https://pepy.tech/project/wmlog)
[![Code Style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Build Status](https://travis-ci.com/tom-sapletta-com/wmlog.svg?branch=main)](https://travis-ci.com/tom-sapletta-com/wmlog)

**wmlog** is a production-ready centralized logging system that enables structured logging with real-time broadcasting over WebSocket and MQTT. Designed for microservices, embedded systems, and distributed applications that need unified, observable logging across multiple services and environments.

> 🚀 **Part of the modular ecosystem**: [ProServe](https://pypi.org/project/proserve/) (core framework) • [Servos](https://pypi.org/project/servos/) (isolation) • [EDPMT](https://pypi.org/project/edpmt/) (hardware) • **wmlog** (logging) • [SELLM](https://pypi.org/project/sellm/) (AI-powered manifest generator)

## 🚀 Features

- **Structured Logging**: Built on `structlog` with rich context and metadata
- **Real-time Broadcasting**: WebSocket and MQTT streaming for live log monitoring
- **Multiple Formatters**: JSON, Rich Console, Compact, and Structured output
- **Custom Handlers**: WebSocket, MQTT, Redis, File, and more
- **CLI Tools**: Command-line interface for log management and monitoring
- **Context Enrichment**: Automatic service, environment, and request context
- **Async Support**: Full async/await support for high-performance applications
- **Redis Integration**: Optional Redis backend for log storage and analysis
- **Rich Console Output**: Beautiful console logging with syntax highlighting

## 📦 Installation

```bash
# Install from PyPI
pip install wmlog

# Install with optional dependencies
pip install wmlog[redis,mqtt,websocket]

# Development installation
git clone https://github.com/tom-sapletta-com/wmlog.git
cd wmlog
pip install -e .
```

## 🏗️ Architecture

### wmlog Ecosystem Integration
```mermaid
graph TB
    subgraph "Modular Ecosystem"
        ProServe[ProServe Framework<br/>Core Microservices]
        Servos[Servos<br/>Environment Isolation] 
        EDPMT[EDPMT Framework<br/>Hardware Control]
        wmlog[wmlog<br/>Centralized Logging]
        SELLM[SELLM<br/>AI-powered Manifest Generation]
    end
    
    ProServe --> wmlog
    Servos --> wmlog
    EDPMT --> wmlog
    SELLM --> wmlog
    
    subgraph "wmlog Core Architecture"
        WMLLogger[WMLLogger<br/>Singleton Instance]
        LogContext[LogContext<br/>Enrichment]
        LoggingConfig[LoggingConfig<br/>Configuration]
    end
    
    wmlog --> WMLLogger
    wmlog --> LogContext
    wmlog --> LoggingConfig
    
    subgraph "Formatters"
        JSON[JSON Formatter]
        Rich[Rich Console]
        Compact[Compact Format]
        Structured[Structured Text]
    end
    
    WMLLogger --> JSON
    WMLLogger --> Rich
    WMLLogger --> Compact
    WMLLogger --> Structured
    
    subgraph "Handlers & Outputs"
        Console[Console Output]
        FileHandler[File Logging]
        WebSocket[WebSocket Broadcasting]
        MQTT[MQTT Publishing]
        Redis[Redis Storage]
    end
    
    JSON --> Console
    Rich --> Console
    JSON --> FileHandler
    JSON --> WebSocket
    JSON --> MQTT
    JSON --> Redis
    
    subgraph "External Systems"
        WebClients[Web Clients<br/>Real-time Monitoring]
        MQTTBroker[MQTT Broker<br/>Message Distribution]
        RedisDB[Redis Database<br/>Log Storage]
        LogFiles[Log Files<br/>Persistent Storage]
    end
    
    WebSocket --> WebClients
    MQTT --> MQTTBroker
    Redis --> RedisDB
    FileHandler --> LogFiles
```

### Detailed wmlog Architecture
```
┌─────────────────────────────────────────────────────────────────────────────┐
│                             wmlog Architecture                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│ ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐              │
│ │   ProServe      │  │     Servos      │  │     EDPMT       │              │
│ │   Services      │──│   Isolation     │──│   Framework     │              │
│ │                 │  │                 │  │                 │              │
│ └─────────────────┘  └─────────────────┘  └─────────────────┘              │
│          │                    │                    │                       │
│          └─────────────────────┼─────────────────────┘                       │
│                                │                                             │
│                                ▼                                             │
│          ┌─────────────────────────────────────────────────────────────┐     │
│          │                 wmlog Core                                  │     │
│          │                                                             │     │
│          │  ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐   │     │
│          │  │ WMLLogger   │ │ LogContext  │ │  LoggingConfig      │   │     │
│          │  │             │ │             │ │                     │   │     │
│          │  │• Singleton  │ │• Service    │ │• Environment Vars   │   │     │
│          │  │• Async      │ │• Environment│ │• File/Console       │   │     │
│          │  │• Structured │ │• Custom     │ │• WebSocket/MQTT     │   │     │
│          │  │• Context    │ │  Fields     │ │• Redis Integration  │   │     │
│          │  └─────────────┘ └─────────────┘ └─────────────────────┘   │     │
│          └─────────────────────────────────────────────────────────────┘     │
│                                │                                             │
│                                ▼                                             │
│  ┌───────────────────────────────────────────────────────────────────────┐   │
│  │                         Formatters                                   │   │
│  │  ┌──────────┐ ┌─────────────┐ ┌──────────┐ ┌─────────────────────┐   │   │
│  │  │   JSON   │ │    Rich     │ │ Compact  │ │     Structured      │   │   │
│  │  │          │ │   Console   │ │  Format  │ │       Text          │   │   │
│  │  │• Strict  │ │• Colors     │ │• Minimal │ │• Key-Value Pairs    │   │   │
│  │  │• Machine │ │• Syntax     │ │• High    │ │• Human Readable     │   │   │
│  │  │  Readable│ │  Highlight  │ │  Volume  │ │• Debugging          │   │   │
│  │  └──────────┘ └─────────────┘ └──────────┘ └─────────────────────┘   │   │
│  └───────────────────────────────────────────────────────────────────────┘   │
│                                │                                             │
│                                ▼                                             │
│  ┌───────────────────────────────────────────────────────────────────────┐   │
│  │                    Handlers & Broadcasting                            │   │
│  │  ┌─────────────┐ ┌──────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐   │   │
│  │  │  Console    │ │   File   │ │ WebSocket│ │   MQTT   │ │  Redis   │   │   │
│  │  │             │ │          │ │          │ │          │ │          │   │   │
│  │  │• Stdout     │ │• Rotation│ │• Real-   │ │• Pub/Sub │ │• Storage │   │   │
│  │  │• Stderr     │ │• Size    │ │  time    │ │• Topics  │ │• TTL     │   │   │
│  │  │• Colors     │ │  Limits  │ │• Live    │ │• QoS     │ │• Search  │   │   │
│  │  │• Rich       │ │• Backup  │ │  Monitor │ │• Retain  │ │• Analysis│   │   │
│  │  │  Format     │ │  Count   │ │• Multiple│ │• Auth    │ │• Metrics │   │   │
│  │  │             │ │          │ │  Clients │ │          │ │          │   │   │
│  │  └─────────────┘ └──────────┘ └─────────┘ └──────────┘ └──────────┘   │   │
│  └───────────────────────────────────────────────────────────────────────┘   │
│                                │                                             │
│                                ▼                                             │
│  ┌───────────────────────────────────────────────────────────────────────┐   │
│  │                      External Systems                                │   │
│  │  ┌─────────────┐ ┌──────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐   │   │
│  │  │    Local    │ │   Log    │ │   Web   │ │   MQTT   │ │  Redis   │   │   │
│  │  │  Terminal   │ │  Files   │ │ Clients │ │  Broker  │ │ Database │   │   │
│  │  │             │ │          │ │         │ │          │ │          │   │   │
│  │  │• Development│ │• Prod    │ │• Dashbd │ │• Message │ │• Analytics│   │   │
│  │  │• Debugging  │ │• Audit   │ │• Monitor│ │  Queue   │ │• Long-term│   │   │
│  │  │• Rich UI    │ │• Archive │ │• Alerts │ │• Fanout  │ │  Storage  │   │   │
│  │  └─────────────┘ └──────────┘ └─────────┘ └──────────┘ └──────────┘   │   │
│  └───────────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────────┘
```

## 🛠️ Quick Start

### Basic Logging

```python
from wmlog import WMLLogger, LoggingConfig

# Create logging configuration
config = LoggingConfig(
    service_name="my-service",
    log_level="info",
    console_enabled=True,
    console_format="rich"
)

# Get logger instance
logger = WMLLogger.get_logger(config)

# Log messages with structured data
logger.logger.info("Service started", port=8080, environment="production")
logger.logger.error("Database connection failed", 
                   error="Connection timeout", 
                   retry_count=3)
```

### WebSocket Broadcasting

```python
from wml import WMLLogger, LoggingConfig

config = LoggingConfig(
    service_name="websocket-service",
    websocket_enabled=True,
    websocket_port=8765,
    websocket_host="0.0.0.0"
)

logger = WMLLogger.get_logger(config)

# Logs will be automatically broadcast to WebSocket clients
logger.logger.info("WebSocket message", client_count=42)
```

### MQTT Integration

```python
from wml import WMLLogger, LoggingConfig

config = LoggingConfig(
    service_name="mqtt-service",
    mqtt_enabled=True,
    mqtt_broker="mqtt://localhost:1883",
    mqtt_topic="logs/my-service"
)

logger = WMLLogger.get_logger(config)

# Logs will be published to MQTT broker
logger.logger.warning("High memory usage", memory_percent=85)
```

### Context Enrichment

```python
from wml import WMLLogger, LoggingConfig, LogContext

# Create context with additional metadata
context = LogContext(
    service_name="api-service",
    environment="staging",
    version="1.2.3",
    custom_data={"region": "us-east-1", "datacenter": "dc-01"}
)

config = LoggingConfig(service_name="api-service")
logger = WMLLogger.get_logger(config, context)

# All logs will include context information
logger.logger.info("Request processed", 
                   request_id="req-123", 
                   response_time=0.045)
```

## 🖥️ CLI Usage

WML provides a comprehensive CLI for log management and monitoring:

### Send Log Messages

```bash
# Send a simple log message
wmlog log "Hello, World!"

# Send with specific service and level
wmlog log --service my-service --level error "Something went wrong"

# Send with MQTT broadcasting
wmlog log --mqtt-broker localhost:1883 --mqtt-topic logs/test "MQTT log message"
```

### Monitor Logs

```bash
# Monitor MQTT logs
wmlog monitor --broker localhost:1883 --topic "logs/#"

# Monitor with filtering
wmlog monitor --broker localhost:1883 --filter-service my-service --filter-level error

# Monitor WebSocket logs
wmlog websocket-monitor --port 8765 --host localhost
```

### Start Broadcasting Server

```bash
# Start WebSocket and MQTT broadcasting server
wmlog server --websocket-port 8765 --mqtt-broker localhost:1883

# Start WebSocket-only server
wmlog server --websocket-port 8765
```

### Test Configuration

```bash
# Test with default configuration
wmlog test

# Test with custom configuration file
wmlog test --config-file config.json
```

### Package Information

```bash
# Show package information
wmlog info
```

## 📚 Documentation

- [API Documentation](docs/API_DOCUMENTATION.md) - Detailed reference for wmlog APIs

### Python Packages

Explore the ecosystem of Python packages related to wmlog:

- [ProServe](https://pypi.org/project/proserve/) - Core microservices framework
- [Servos](https://pypi.org/project/servos/) - Environment isolation and orchestration
- [wmlog](https://pypi.org/project/wmlog/) - Centralized structured logging
- [SELLM](https://pypi.org/project/sellm/) - AI-powered manifest generator
- [EDPMT](https://pypi.org/project/edpmt/) - Hardware control framework for IoT

## Why wmlog?

- **Unified Logging**: Centralized logging for microservices and distributed systems
- **Real-time Monitoring**: Live log streaming with WebSocket and MQTT
- **Flexible Configuration**: Customizable logging formats, handlers, and outputs
- **Async Support**: High-performance logging for async applications
- **Redis Integration**: Optional Redis backend for log storage and analysis
- **Rich Console Output**: Beautiful console logging with syntax highlighting

## 🔧 Advanced Usage

### Custom Formatters

```python
from wml.formatters import JSONFormatter, RichConsoleFormatter

# Custom JSON formatter with additional fields
json_formatter = JSONFormatter(
    include_extra_fields=True,
    timestamp_format="iso",
    pretty_print=True
)

# Custom Rich formatter with specific styling
rich_formatter = RichConsoleFormatter(
    show_time=True,
    show_level=True,
    show_path=False,
    markup=True
)
```

### Custom Handlers

```python
from wml.handlers import WebSocketHandler, MQTTHandler, RedisHandler

# WebSocket handler for real-time streaming
ws_handler = WebSocketHandler(
    host="0.0.0.0",
    port=8765,
    path="/logs"
)

# MQTT handler for pub/sub messaging
mqtt_handler = MQTTHandler(
    broker_url="mqtt://localhost:1883",
    topic="logs/my-service",
    qos=1,
    retain=False
)

# Redis handler for log storage
redis_handler = RedisHandler(
    redis_url="redis://localhost:6379/0",
    key_prefix="logs:",
    expire_seconds=86400
)
```

### Async Logging

```python
import asyncio
from wml import WMLLogger, LoggingConfig

async def async_application():
    config = LoggingConfig(
        service_name="async-service",
        websocket_enabled=True
    )
    
    logger = WMLLogger.get_logger(config)
    
    # Async logging with context
    async with logger.context(request_id="req-456"):
        logger.logger.info("Processing async request")
        
        # Simulate async work
        await asyncio.sleep(1)
        
        logger.logger.info("Async request completed")

# Run async application
asyncio.run(async_application())
```

## 🔌 Integration Examples

### Flask Application

```python
from flask import Flask, request
from wml import WMLLogger, LoggingConfig, LogContext

app = Flask(__name__)

# Initialize WML logger
config = LoggingConfig(
    service_name="flask-api",
    console_format="rich",
    websocket_enabled=True,
    mqtt_enabled=True,
    mqtt_broker="mqtt://localhost:1883"
)

logger = WMLLogger.get_logger(config)

@app.before_request
def log_request_info():
    logger.logger.info("Request started",
                      method=request.method,
                      url=request.url,
                      remote_addr=request.remote_addr)

@app.after_request
def log_response_info(response):
    logger.logger.info("Request completed",
                      status_code=response.status_code,
                      content_length=response.content_length)
    return response

@app.route('/api/health')
def health():
    logger.logger.info("Health check requested")
    return {"status": "healthy"}

if __name__ == '__main__':
    app.run(debug=True)
```

### FastAPI Application

```python
from fastapi import FastAPI, Request
from wml import WMLLogger, LoggingConfig
import time

app = FastAPI()

# Initialize WML logger
config = LoggingConfig(
    service_name="fastapi-service",
    console_format="rich",
    websocket_enabled=True
)

logger = WMLLogger.get_logger(config)

@app.middleware("http")
async def log_requests(request: Request, call_next):
    start_time = time.time()
    
    logger.logger.info("Request started",
                      method=request.method,
                      url=str(request.url))
    
    response = await call_next(request)
    
    process_time = time.time() - start_time
    logger.logger.info("Request completed",
                      status_code=response.status_code,
                      process_time=process_time)
    
    return response

@app.get("/")
async def root():
    logger.logger.info("Root endpoint accessed")
    return {"message": "Hello World"}
```

### Docker Integration

```dockerfile
FROM python:3.9-slim

WORKDIR /app

# Install WML
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

# Set environment variables for WML
ENV WML_SERVICE_NAME=docker-service
ENV WML_LOG_LEVEL=info
ENV WML_CONSOLE_FORMAT=json
ENV WML_WEBSOCKET_ENABLED=true
ENV WML_WEBSOCKET_PORT=8765
ENV WML_MQTT_ENABLED=true
ENV WML_MQTT_BROKER=mqtt://mqtt-broker:1883

EXPOSE 8000 8765

CMD ["python", "app.py"]
```

```yaml
# docker-compose.yml
version: '3.8'

services:
  app:
    build: .
    ports:
      - "8000:8000"
      - "8765:8765"
    environment:
      - WML_MQTT_BROKER=mqtt://mqtt:1883
    depends_on:
      - mqtt
      - redis

  mqtt:
    image: eclipse-mosquitto:2.0
    ports:
      - "1883:1883"
      - "9001:9001"
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  log-monitor:
    image: wml-cli
    command: wml monitor --broker mqtt:1883 --topic "logs/#"
    depends_on:
      - mqtt
```

## 📊 Monitoring and Observability

### Grafana Dashboard

WML integrates seamlessly with monitoring stacks:

```python
# Send metrics alongside logs
logger.logger.info("Request processed",
                  request_count=1,
                  response_time=0.045,
                  memory_usage=256.5,
                  cpu_percent=12.3)
```

### Prometheus Integration

```python
from prometheus_client import Counter, Histogram, Gauge
from wml import WMLLogger, LoggingConfig

# Prometheus metrics
REQUEST_COUNT = Counter('requests_total', 'Total requests', ['method', 'endpoint'])
REQUEST_DURATION = Histogram('request_duration_seconds', 'Request duration')
MEMORY_USAGE = Gauge('memory_usage_bytes', 'Memory usage')

config = LoggingConfig(service_name="metrics-service")
logger = WMLLogger.get_logger(config)

def process_request(method, endpoint):
    REQUEST_COUNT.labels(method=method, endpoint=endpoint).inc()
    
    with REQUEST_DURATION.time():
        # Process request
        logger.logger.info("Request processed",
                          method=method,
                          endpoint=endpoint,
                          metrics_exported=True)
```

## 🧪 Testing

Run the test suite:

```bash
# Install test dependencies
pip install -e .[dev]

# Run unit tests
pytest tests/

# Run integration tests
pytest tests/integration/

# Run with coverage
pytest tests/ --cov=wml --cov-report=html

# Test CLI commands
wml test
wml info
```

## 📚 API Reference

### Core Classes

- **`WMLLogger`**: Main logging class with structured logging support
- **`LoggingConfig`**: Configuration class for logger settings
- **`LogContext`**: Context enrichment for adding metadata to logs

### Formatters

- **`JSONFormatter`**: JSON output with structured data
- **`RichConsoleFormatter`**: Rich console output with colors and styling
- **`StructuredFormatter`**: Structured text format with key-value pairs
- **`CompactFormatter`**: Minimal compact format for high-volume logging

### Handlers

- **`WebSocketHandler`**: Real-time WebSocket broadcasting
- **`MQTTHandler`**: MQTT pub/sub messaging
- **`RedisHandler`**: Redis storage and retrieval
- **`BufferedHandler`**: Buffered output with configurable flushing

### Broadcasters

- **`WebSocketBroadcaster`**: WebSocket server for real-time log streaming
- **`MQTTBroadcaster`**: MQTT client for pub/sub log distribution

## 🤝 Contributing

We welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.

### Development Setup

```bash
# Clone the repository
git clone https://github.com/your-org/wml.git
cd wml

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install development dependencies
pip install -e .[dev]

# Install pre-commit hooks
pre-commit install

# Run tests
pytest
```

### Code Style

We use:
- **Black** for code formatting
- **isort** for import sorting
- **flake8** for linting
- **mypy** for type checking

## 📄 License

wmlog is released under the Apache Software License 2.0. See the [LICENSE](LICENSE) file for details.

## 👨‍💻 Author

**Tom Sapletta**
- Email: info@softreck.dev
- GitHub: [@tom-sapletta-com](https://github.com/tom-sapletta-com)
- Website: [softreck.dev](https://softreck.dev)

## 🙏 Acknowledgments

- Built with [structlog](https://www.structlog.org/) for structured logging
- Uses [rich](https://github.com/Textualize/rich) for beautiful console output
- MQTT support via [paho-mqtt](https://pypi.org/project/paho-mqtt/)
- WebSocket support via [aiohttp](https://aiohttp.readthedocs.io/) and [websockets](https://websockets.readthedocs.io/)

## 🔗 Related Projects

- **[ProServe](https://pypi.org/project/proserve/)**: Professional microservice framework using wmlog
- **[Servos](https://pypi.org/project/servos/)**: Environment isolation and Docker orchestration
- **[EDPMT Framework](https://github.com/tom-sapletta-com/edpmt)**: Embedded development platform with wmlog integration
- **[SELLM](https://pypi.org/project/sellm/)**: AI-powered manifest generator

---

**wmlog** - Unifying logs across services, platforms, and environments. 🚀

---

**Made with ❤️ by [Tom Sapletta](https://softreck.dev) • Part of the modular microservices ecosystem**