setu-trafficmonitor 2.0.1
A comprehensive Django app for logging HTTP requests with analytics dashboard
License: MIT
Requires Python: >=3.8
Keywords: django, request, logging, analytics, monitoring, middleware, dashboard, traffic, http

# Setu TrafficMonitor

A comprehensive Django application for logging HTTP requests with a beautiful analytics dashboard. Monitor your API traffic, analyze performance metrics, and gain insights into your application's usage patterns.

![Version](https://img.shields.io/badge/version-1.0.0-blue.svg)
![Python](https://img.shields.io/badge/python-3.8%2B-blue.svg)
![Django](https://img.shields.io/badge/django-3.2%2B-green.svg)
![License](https://img.shields.io/badge/license-MIT-green.svg)

## Features

### Request Logging
- **Comprehensive Logging**: Captures all HTTP requests with detailed information
- **Performance Metrics**: Track response times and database query counts
- **User Tracking**: Associate requests with authenticated users
- **IP Address Tracking**: Monitor requests by IP address with X-Forwarded-For support
- **Request/Response Bodies**: Store request and response payloads (with size limits)
- **Exception Tracking**: Automatic logging of exceptions with full tracebacks
- **Configurable Exclusions**: Exclude specific paths (static files, health checks, etc.)

### Analytics Dashboard
- **Real-time Visualizations**: Interactive charts powered by Chart.js
- **Time-based Analysis**: View metrics by today, last 7 days, last 30 days, or custom ranges
- **Status Code Distribution**: Track success rates and error patterns
- **HTTP Method Breakdown**: Analyze request types (GET, POST, PUT, DELETE, etc.)
- **Endpoint Performance**: Identify slowest endpoints and optimization opportunities
- **Hourly Heatmap**: Discover traffic patterns throughout the day
- **Top IPs & Users**: Monitor most active clients and users
- **Error Trending**: Track 4xx and 5xx errors over time
- **Advanced Filtering**: Filter by method, status code, path, and user

### REST API Endpoints
- **Analytics Overview API**: Get comprehensive analytics data in JSON format
- **Chart-specific APIs**: Fetch individual chart data for custom integrations
- **Admin-only Access**: Secure endpoints with Django permissions

## Installation

### Using pip

```bash
pip install <path-to-your-private-setu-trafficmonitor-package>
```

### From source

```bash
git clone <internal-repo-url>
cd setu-trafficmonitor
pip install -e .
```

## Quick Start

### 1. Add to Installed Apps

Add `trafficmonitor` to your `INSTALLED_APPS` in `settings.py`:

```python
INSTALLED_APPS = [
    # ... other apps
    'rest_framework',  # Required dependency
    'trafficmonitor',
]
```

### 2. Add Middleware

Add the middleware to your `MIDDLEWARE` setting (preferably near the top):

```python
MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'trafficmonitor.middleware.RequestLoggingMiddleware',  # Add here
    # ... other middleware
]
```

### 3. Configure URLs

Include the TrafficMonitor URLs in your project's `urls.py`:

```python
from django.urls import path, include

urlpatterns = [
    # ... your other URLs
    path('', include('trafficmonitor.analytics.urls')),
]
```

### 4. Run Migrations

```bash
python manage.py migrate trafficmonitor
```

### 5. Access the Dashboard

Start your development server and navigate to:
- **Dashboard**: `http://localhost:8000/analytics/dashboard/`
- **API Overview**: `http://localhost:8000/api/analytics/overview/`

**Note**: Dashboard access is restricted to staff users. Make sure you have a staff user:

```bash
python manage.py createsuperuser
```

## Configuration

### Middleware Settings

You can customize the middleware behavior by subclassing and overriding:

```python
from trafficmonitor.middleware import RequestLoggingMiddleware

class CustomRequestLoggingMiddleware(RequestLoggingMiddleware):
    # Maximum body size to log (in characters)
    MAX_BODY_LENGTH = 20000

    # Paths to exclude from logging
    EXCLUDED_PATHS = [
        '/admin/jsi18n/',
        '/static/',
        '/media/',
        '/__debug__/',
        '/favicon.ico',
        '/health/',  # Add custom paths
    ]
```

Then use your custom middleware in `settings.py`:

```python
MIDDLEWARE = [
    # ...
    'myapp.middleware.CustomRequestLoggingMiddleware',
    # ...
]
```

### Database Optimization

For high-traffic applications, consider:

1. **Database Indexing**: The package includes optimized indexes for common queries
2. **Partitioning**: Consider partitioning the `request_log` table by date
3. **Archiving**: Implement a cleanup strategy for old logs:

```python
from django.utils import timezone
from datetime import timedelta
from trafficmonitor.models import RequestLog

# Delete logs older than 90 days
cutoff_date = timezone.now() - timedelta(days=90)
RequestLog.objects.filter(timestamp__lt=cutoff_date).delete()
```

4. **Read Replicas**: Route analytics queries to read replicas if available

## Usage Examples

### Accessing Analytics Data Programmatically

```python
from trafficmonitor.analytics.views import AnalyticsQueryHelper
from django.utils import timezone
from datetime import timedelta

# Get date range
end_date = timezone.now()
start_date = end_date - timedelta(days=7)

# Get total requests
total = AnalyticsQueryHelper.get_total_requests(start_date, end_date)

# Get top endpoints
top_endpoints = AnalyticsQueryHelper.get_top_endpoints(
    start_date, end_date, limit=10
)

# Get slowest endpoints
slow_endpoints = AnalyticsQueryHelper.get_slowest_endpoints(
    start_date, end_date, limit=10
)

# Get comprehensive analytics
analytics = AnalyticsQueryHelper.get_comprehensive_analytics(
    start_date, end_date
)
```

### Filtering Analytics

```python
# Filter by HTTP method
get_requests = AnalyticsQueryHelper.get_total_requests(
    start_date, end_date, method='GET'
)

# Filter by status code range
errors = AnalyticsQueryHelper.get_total_requests(
    start_date, end_date, status_code_range=(400, 599)
)

# Filter by path
api_requests = AnalyticsQueryHelper.get_total_requests(
    start_date, end_date, path_contains='/api/'
)

# Filter by user
user_requests = AnalyticsQueryHelper.get_total_requests(
    start_date, end_date, user_id=user.id
)
```

### Custom Celery Task for Log Cleanup

```python
from celery import shared_task
from django.utils import timezone
from datetime import timedelta
from trafficmonitor.models import RequestLog

@shared_task
def cleanup_old_request_logs():
    """Delete request logs older than 90 days"""
    cutoff_date = timezone.now() - timedelta(days=90)
    deleted_count, _ = RequestLog.objects.filter(
        timestamp__lt=cutoff_date
    ).delete()
    return f"Deleted {deleted_count} old request logs"
```

## API Reference

### REST API Endpoints

#### Analytics Overview
```
GET /api/analytics/overview/
```

Query Parameters:
- `range`: `today`, `yesterday`, `last_7_days`, `last_30_days`, `custom`
- `start_date`: ISO format date (YYYY-MM-DD) for custom range
- `end_date`: ISO format date (YYYY-MM-DD) for custom range
- `method`: Filter by HTTP method (GET, POST, etc.)
- `status`: Filter by status code
- `path`: Filter by path contains
- `user`: Filter by user ID

#### Chart Data
```
GET /api/analytics/chart/<chart_type>/
```

Chart Types:
- `time-series`: Requests over time
- `status-codes`: Status code distribution
- `methods`: HTTP methods breakdown
- `endpoints`: Top endpoints
- `performance`: Slowest endpoints
- `heatmap`: Hourly request heatmap
- `errors`: Error trend over time

## Database Schema

### RequestLog Model

| Field | Type | Description |
|-------|------|-------------|
| id | UUID | Primary key |
| method | CharField | HTTP method (GET, POST, etc.) |
| path | CharField | Request path/URL |
| full_url | TextField | Complete URL with query params |
| status_code | IntegerField | HTTP response status code |
| user | ForeignKey | Authenticated user (nullable) |
| ip_address | GenericIPAddressField | Client IP address |
| user_agent | TextField | User agent string |
| request_headers | JSONField | Request headers (sensitive ones excluded) |
| request_body | TextField | Request body (truncated) |
| response_body | TextField | Response body (truncated) |
| response_time_ms | FloatField | Response time in milliseconds |
| query_count | IntegerField | Database queries executed |
| exception | TextField | Exception traceback if failed |
| content_length | IntegerField | Response size in bytes |
| timestamp | DateTimeField | Request timestamp |

## Performance Considerations

- **Indexes**: The package includes optimized database indexes for common queries
- **Query Optimization**: Analytics queries use `select_related()` and `annotate()` for efficiency
- **Async Logging**: Consider implementing asynchronous logging for high-traffic applications
- **Body Truncation**: Request/response bodies are automatically truncated to prevent database bloat
- **Excluded Paths**: Static files and health checks are excluded by default

## Security

- **Sensitive Headers**: Authorization headers and cookies are automatically excluded from logging
- **Admin Access**: Dashboard and API endpoints require staff/admin permissions
- **User Authentication**: Built-in Django authentication and authorization
- **CSRF Protection**: Standard Django CSRF protection applies

## Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## Versioning

This project uses [Semantic Versioning](https://semver.org/):
- **MAJOR** version for incompatible API changes
- **MINOR** version for backwards-compatible functionality additions
- **PATCH** version for backwards-compatible bug fixes

See [CHANGELOG.md](CHANGELOG.md) for version history.

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Support

- **Issues**: [GitHub Issues](https://github.com/farmsetu/django-trafficmonitor/issues)
- **Documentation**: [GitHub Wiki](https://github.com/farmsetu/django-trafficmonitor/wiki)

## Credits

Developed by [FarmSetu Team](https://farmsetu.com)

## Changelog

See [CHANGELOG.md](CHANGELOG.md) for a detailed version history.