django-query-log 1.0.0
A Django package for logging database query execution time and performance analysis for all Django requests
License: MIT
Requires Python: >=3.8
Keywords: django, database, query, logging, performance, monitoring, sql, middleware, n+1, optimization, inspector, debugging

# Django Query Log

[![PyPI version](https://badge.fury.io/py/django-query-log.svg)](https://badge.fury.io/py/django-query-log)
[![Python Support](https://img.shields.io/pypi/pyversions/django-query-log.svg)](https://pypi.org/project/django-query-log/)
[![Django Support](https://img.shields.io/badge/django-3.2%2B-brightgreen.svg)](https://www.djangoproject.com/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

A comprehensive Django package for logging database query execution time and performance analysis for all Django requests. Monitor your application's database performance with detailed insights, N+1 query detection, and customizable logging formats. Works with regular Django views, class-based views, and Django REST Framework APIs.

## Features

- 🚀 **Real-time Query Monitoring**: Track database queries for all Django requests
- 📊 **Performance Analysis**: Automatic detection of slow queries and performance bottlenecks
- 🔍 **N+1 Query Detection**: Identify and warn about N+1 query problems
- 🔄 **Duplicate Query Detection**: Find and report duplicate queries
- 📝 **Multiple Log Formats**: Support for JSON, compact, detailed, and custom formatters
- ⚙️ **Highly Configurable**: Extensive configuration options for different use cases
- 🎯 **Universal Django Support**: Works with function views, class-based views, and Django REST Framework
- 🔐 **Security**: Automatic sanitization of sensitive data in logs
- 📈 **Stack Trace Support**: Optional stack trace logging for query sources
- 🎨 **Custom Formatters**: Create your own log formatting logic

## Installation

Install using pip:

```bash
pip install django-query-log
```

For Django REST Framework support:

```bash
pip install django-query-log[drf]
```

Or install with development dependencies:

```bash
pip install django-query-log[dev]
```

## Quick Start

1. **Add to Django Settings**:

```python
# settings.py
INSTALLED_APPS = [
    # ... your other apps
    'django_query_log',
]

MIDDLEWARE = [
    # ... your other middleware
    'django_query_log.middleware.QueryLogMiddleware',
    # ... rest of your middleware
]

# Optional: Configure django-query-log
DJANGO_QUERY_LOG = {
    'ENABLED': True,
    'LOG_LEVEL': 'INFO',
    'LOGGER_NAME': 'django_query_log',
}
```

2. **Configure Logging** (optional):

```python
# settings.py
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'query_log': {
            'level': 'INFO',
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
    },
    'loggers': {
        'django_query_log': {
            'handlers': ['query_log'],
            'level': 'INFO',
            'propagate': False,
        },
    },
    'formatters': {
        'verbose': {
            'format': '{levelname} {asctime} {module} {message}',
            'style': '{',
        },
    },
}
```

3. **Start your Django server** and make requests to any Django view. Query logs will appear in your console!

## Usage Examples

### Basic Usage

Once installed and configured, django-query-log automatically logs database queries for all Django requests (views, APIs, admin, etc.):

```
[2024-01-15T10:30:45.123456] GET /api/users/ - Queries: 3, Total Time: 0.0157s
  User: admin (ID: 1)
  Performance Analysis:
    Average Query Time: 0.0052s
    Duplicate Queries: 0 queries in 0 groups
  Queries:
    1. [0.0031s] SELECT "auth_user"."id", "auth_user"."username", "auth_user"."email" FROM "auth_user" WHERE "auth_user"."is_active" = TRUE
    2. [0.0089s] SELECT "user_profile"."id", "user_profile"."user_id", "user_profile"."avatar" FROM "user_profile" WHERE "user_profile"."user_id" IN (1, 2, 3)
    3. [0.0037s] SELECT COUNT(*) AS "__count" FROM "auth_user" WHERE "auth_user"."is_active" = TRUE
  Response: 200 (1024 bytes)
```

### Using Different Formatters

**JSON Formatter** (great for structured logging):

```python
# settings.py
DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'django_query_log.formatters.JSONFormatter',
}
```

**Compact Formatter** (minimal output):

```python
# settings.py
DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'django_query_log.formatters.CompactFormatter',
}
```

Output:

```
[2024-01-15T10:30:45.123456] GET /api/users/ | queries=3 | time=0.016s | user=admin
```

**Detailed Formatter** (comprehensive information):

```python
# settings.py
DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'django_query_log.formatters.DetailedFormatter',
}
```

### Works with All Django Views

Django Query Log works seamlessly with all types of Django views:

**Function-based views:**

```python
def my_view(request):
    users = User.objects.all()  # Logged automatically
    return render(request, 'users.html', {'users': users})
```

**Class-based views:**

```python
class UserListView(ListView):
    model = User  # All queries logged automatically
    template_name = 'users.html'
```

**Django REST Framework views:**

```python
from rest_framework.viewsets import ModelViewSet

class UserViewSet(ModelViewSet):
    queryset = User.objects.all()  # All queries logged automatically
    serializer_class = UserSerializer
```

### Context Manager

Temporarily disable or enable logging for specific code blocks:

```python
from django_query_log.middleware import DjangoQueryLogContext

# Disable logging for this block
with DjangoQueryLogContext(enabled=False):
    users = User.objects.all()  # This won't be logged

# Enable logging (if disabled globally)
with DjangoQueryLogContext(enabled=True):
    users = User.objects.all()  # This will be logged
```

## Configuration

### Complete Configuration Options

```python
# settings.py
DJANGO_QUERY_LOG = {
    # Enable/disable query logging
    'ENABLED': True,

    # Minimum query execution time to log (in seconds)
    'MIN_QUERY_TIME': 0.001,

    # Maximum number of queries to log per request
    'MAX_QUERIES_PER_REQUEST': 100,

    # Log level for query information
    'LOG_LEVEL': 'INFO',  # DEBUG, INFO, WARNING, ERROR, CRITICAL

    # Logger name to use
    'LOGGER_NAME': 'django_query_log',

    # Include request information in logs
    'INCLUDE_REQUEST_INFO': True,

    # Include response information in logs
    'INCLUDE_RESPONSE_INFO': True,

    # Include user information in logs
    'INCLUDE_USER_INFO': True,

    # Include SQL query text in logs
    'INCLUDE_SQL': True,

    # Include query execution stack trace
    'INCLUDE_STACK_TRACE': False,

    # Sensitive parameters to exclude from logs
    'SENSITIVE_PARAMS': ['password', 'token', 'secret', 'key'],

    # Only log queries for specific paths (empty list means all paths)
    'INCLUDE_PATHS': [],

    # Exclude queries for specific paths
    'EXCLUDE_PATHS': ['/admin/', '/static/', '/media/'],

    # Custom formatter class
    'FORMATTER_CLASS': 'django_query_log.formatters.DefaultFormatter',

    # Include duplicate query detection
    'DETECT_DUPLICATE_QUERIES': True,

    # Warn about N+1 query problems
    'DETECT_N_PLUS_ONE': True,

    # Include query performance analysis
    'INCLUDE_PERFORMANCE_ANALYSIS': True,
}
```

### Configuration Examples

**Production Environment** (minimal logging):

```python
DJANGO_QUERY_LOG = {
    'ENABLED': True,
    'MIN_QUERY_TIME': 0.01,  # Only log slow queries
    'MAX_QUERIES_PER_REQUEST': 50,
    'LOG_LEVEL': 'WARNING',
    'INCLUDE_SQL': False,  # Don't log SQL in production
    'FORMATTER_CLASS': 'django_query_log.formatters.CompactFormatter',
    'EXCLUDE_PATHS': ['/admin/', '/static/', '/media/', '/health/'],
}
```

**Development Environment** (detailed logging):

```python
DJANGO_QUERY_LOG = {
    'ENABLED': True,
    'MIN_QUERY_TIME': 0.001,
    'LOG_LEVEL': 'DEBUG',
    'INCLUDE_STACK_TRACE': True,
    'FORMATTER_CLASS': 'django_query_log.formatters.DetailedFormatter',
    'DETECT_N_PLUS_ONE': True,
    'DETECT_DUPLICATE_QUERIES': True,
}
```

**API-Only Logging**:

```python
DJANGO_QUERY_LOG = {
    'ENABLED': True,
    'INCLUDE_PATHS': ['/api/'],  # Only log API requests
    'EXCLUDE_PATHS': ['/api/health/', '/api/metrics/'],
}
```

## Performance Features

### N+1 Query Detection

Django Query Log automatically detects potential N+1 query problems:

```
N+1 Query Issues: 1 detected
  - Pattern executed 25 times (0.1234s total)
```

### Duplicate Query Detection

Identifies repeated identical queries:

```
Duplicate Queries: 15 queries in 3 groups
```

### Slow Query Identification

Highlights the slowest queries in each request:

```
Slowest Queries:
  1. [0.0892s] SELECT * FROM "products" WHERE "products"."category_id" = 1
  2. [0.0445s] SELECT COUNT(*) FROM "orders" WHERE "orders"."user_id" = 123
```

## Custom Formatters

Create your own custom formatter:

```python
# myapp/formatters.py
from django_query_log.formatters import BaseFormatter

class MyCustomFormatter(BaseFormatter):
    def format_log_entry(self, log_data):
        request_info = log_data.get('request', {})
        performance = log_data.get('performance_analysis', {})

        return f"API {request_info.get('method')} {request_info.get('path')} " \
               f"executed {performance.get('total_queries', 0)} queries " \
               f"in {performance.get('total_time', 0):.3f}s"

# settings.py
DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'myapp.formatters.MyCustomFormatter',
}
```

## Integration with Logging Systems

### Structured Logging with JSON

```python
import json
import logging

class JSONLogHandler(logging.StreamHandler):
    def emit(self, record):
        try:
            # Parse the log message as JSON if possible
            log_data = json.loads(record.getMessage())
            # Send to your logging service (e.g., ELK, Splunk, etc.)
            super().emit(record)
        except json.JSONDecodeError:
            super().emit(record)

# settings.py
LOGGING = {
    'handlers': {
        'json_query_log': {
            'level': 'INFO',
            '()': 'myapp.logging.JSONLogHandler',
        },
    },
    'loggers': {
        'django_query_log': {
            'handlers': ['json_query_log'],
            'level': 'INFO',
        },
    },
}

DJANGO_QUERY_LOG = {
    'FORMATTER_CLASS': 'django_query_log.formatters.JSONFormatter',
}
```

## Requirements

- Python 3.8+
- Django 3.2+
- Django REST Framework 3.12.0+ (optional)

## Compatibility

- **Django**: 3.2, 4.0, 4.1, 4.2, 5.0
- **Python**: 3.8, 3.9, 3.10, 3.11, 3.12
- **Django REST Framework**: 3.12.0+ (optional for API logging)

## Contributing

We welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md) for details.

### Development Setup

1. Clone the repository:

```bash
git clone https://github.com/dipee/django-query-log.git
cd django-query-log
```

2. Install development dependencies:

```bash
pip install -e .[dev]
```

3. Run tests:

```bash
pytest
```

4. Run linting:

```bash
black .
flake8 .
isort .
mypy .
```

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Changelog

See [CHANGELOG.md](CHANGELOG.md) for release history.

## Support

- **Documentation**: [https://django-query-log.readthedocs.io/](https://django-query-log.readthedocs.io/)
- **Issues**: [https://github.com/dipee/django-query-log/issues](https://github.com/dipee/django-query-log/issues)
- **Discussions**: [https://github.com/dipee/django-query-log/discussions](https://github.com/dipee/django-query-log/discussions)

## Acknowledgments

- Inspired by Django Debug Toolbar's query logging capabilities
- Built for Django developers who need production-ready query monitoring for all types of views
- Thanks to all contributors who help improve this package

---