sify-queue-sdk 0.1.2
Enterprise Kafka Event Framework
License: MIT
Requires Python: >=3.8
Keywords: kafka, events, queue, messaging

# Sify Queue SDK

Enterprise Queue management Framework for building robust, scalable event-driven applications.

## Features

- **Event-Driven Architecture**: Built on Apache Kafka for reliable message streaming
- **Type Safety**: Pydantic models for event validation and serialization
- **Retry & Resilience**: Exponential backoff retry with configurable policies
- **Idempotency**: Built-in duplicate event detection and prevention
- **Dead Letter Queue**: Automatic failed event routing for error handling
- **Stage-Based Processing**: Multi-stage event pipelines with stage-specific handlers
- **Comprehensive Logging**: Structured logging throughout the framework
- **Exception Handling**: Custom exception types for better error management

## Installation

```bash
pip install sify-queue-sdk
```

For development with additional tools:
```bash
pip install sify-queue-sdk[dev]
```

## Quick Start

### Basic Producer

```python
from datetime import datetime, timezone
from queue_sdk import QueueConfig, Producer, KafkaEvent
import uuid

# Configure producer
config = QueueConfig(bootstrap_servers="localhost:9092")
producer = Producer(config)

# Create and send event
event = KafkaEvent(
    eventType="USER_CREATED",
    eventVersion="1.0",
    source="user-service",
    tenantId="tenant-123",
    timestamp=datetime.now(timezone.utc),
    payload={
        "userId": "user-456",
        "email": "user@example.com",
        "name": "John Doe"
    }
)

try:
    producer.send("user-events", event)
    print(f"Event sent: {event.eventId}")
finally:
    producer.close()
```

### Basic Consumer

```python
from queue_sdk import QueueConfig, Consumer, Dispatcher, event_handler, registry

# Define event handler
@event_handler("USER_CREATED", stage="EMAIL_NOTIFICATION")
def send_welcome_email(event: KafkaEvent):
    print(f"Sending welcome email to: {event.payload['email']}")
    # Your email sending logic here

# Configure consumer
config = QueueConfig(
    bootstrap_servers="localhost:9092",
    group_id="email-service-group",
    stage="EMAIL_NOTIFICATION",
    enable_auto_commit=False,
    enable_idempotency=True,
    enable_retry=True,
    dead_letter_topic="failed-events"
)

# Set up consumer
dispatcher = Dispatcher(registry, config)
consumer = Consumer(config, "user-events", dispatcher)

# Start consuming
try:
    consumer.start()
except KeyboardInterrupt:
    consumer.stop()
```

### Error Handling and DLQ

```python
from queue_sdk import QueueConfig, Consumer, Dispatcher, event_handler, registry
from queue_sdk.exceptions import EventProcessingError

@event_handler("PAYMENT_PROCESS", stage="PAYMENT")
def process_payment(event: KafkaEvent):
    try:
        # Payment processing logic
        if event.payload["amount"] <= 0:
            raise ValueError("Invalid payment amount")
        
        # Process payment
        print(f"Processing payment of ${event.payload['amount']}")
        event.payload["processed"] = True
        
    except ValueError as e:
        # This will trigger retry and eventually DLQ if retries exhausted
        raise EventProcessingError(f"Payment validation failed: {e}")

# Configure with DLQ
config = QueueConfig(
    bootstrap_servers="localhost:9092",
    group_id="payment-service-group",
    stage="PAYMENT",
    enable_retry=True,
    dead_letter_topic="failed-payments"
)
```

### Custom Configuration

```python
from queue_sdk import QueueConfig

# Full configuration
config = QueueConfig(
    bootstrap_servers="localhost:9092,localhost:9093",
    group_id="my-service-group",
    enable_auto_commit=True,
    auto_offset_reset="latest",
    enable_idempotency=True,
    enable_retry=True,
    dead_letter_topic="my-dlq-topic",
    stage="MY_STAGE"
)
```

## Configuration Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `bootstrap_servers` | str | Required | Kafka bootstrap servers |
| `group_id` | str | None | Consumer group ID |
| `enable_auto_commit` | bool | False | Enable auto commit of offsets |
| `auto_offset_reset` | str | "earliest" | Offset reset policy |
| `enable_idempotency` | bool | True | Enable duplicate detection |
| `enable_retry` | bool | True | Enable retry with backoff |
| `dead_letter_topic` | str | None | Topic for failed events |
| `stage` | str | None | Processing stage name |

## Event Model

```python
class KafkaEvent(BaseModel):
    eventId: str                    # Auto-generated UUID
    eventType: str                  # Event type identifier
    eventVersion: str               # Event version
    source: str                     # Service name
    tenantId: str                   # Tenant identifier
    timestamp: datetime             # Event timestamp (UTC)
    correlationId: Optional[str]    # Correlation ID for tracing
    traceId: Optional[str]          # Trace ID for distributed tracing
    priority: Optional[str]         # Priority level (LOW/MEDIUM/HIGH)
    retryCount: int                 # Current retry count
    maxRetries: int                 # Maximum retry attempts
    payload: Dict[str, Any]         # Event payload data
    metadata: Optional[Dict[str, Any]]  # Additional metadata
```

## Exception Types

- `QueueSDKError`: Base exception for all SDK errors
- `ConfigurationError`: Configuration-related errors
- `HandlerNotFoundError`: No handler found for event type/stage
- `EventProcessingError`: Event processing failures
- `ProducerError`: Producer operation failures
- `ConsumerError`: Consumer operation failures
- `SerializationError`: Event serialization/deserialization errors

## Testing

Run the test suite:

```bash
# Install development dependencies
pip install sify-queue-sdk[dev]

# Run tests
pytest

```

## Architecture

```
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│    Producer     │───▶│    Kafka Topic  │───▶│    Consumer     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                       │
                                                       ▼
                                            ┌─────────────────┐
                                            │   Dispatcher    │
                                            └─────────────────┘
                                                       │
                                                       ▼
                                            ┌─────────────────┐
                                            │ Event Registry  │
                                            └─────────────────┘
                                                       │
                                                       ▼
                                            ┌─────────────────┐
                                            │ Event Handler   │
                                            └─────────────────┘
```

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Changelog

### v0.1.0
- Initial release
- Basic producer/consumer functionality
- Event validation with Pydantic
- Retry and idempotency features
- Dead letter queue support
- Comprehensive logging
- Unit test coverage