ai-monitor 2.0.2
Ultra-lightweight AI monitoring - pure Python, no compilation required! Docker-friendly.
License: MIT
Requires Python: >=3.8
Keywords: ai, monitoring, llm, agent, observability, prometheus, opentelemetry, tracing, metrics, quality-analysis, hallucination-detection, drift-detection, dashboard, alerting, security, cost-optimization, async

# AI Monitor - Plug & Play AI Agent Monitoring

A comprehensive, zero-configuration monitoring solution for AI agents that requires no source code changes. Simply import and use decorators or context managers.

## 🚀 Quick Start

### Installation (No Cloning Required!)

**One command installs everything:**
```bash
pip install ai-monitor
```

**That's it!** No cloning, no source code changes, no complex setup. Your AI monitoring is ready to use immediately.

### Basic Usage (Plug & Play)

#### 1. Ultra Simple - One Line Setup
```python
from ai_monitor import ultra_simple_setup

# Add this ONE line to enable comprehensive monitoring
ultra_simple_setup()

# Your existing AI code works unchanged!
import openai
response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)
# 🎉 Automatically monitored: latency, tokens, quality, traces
```

#### 2. Enterprise Setup with Traceloop
```python
from ai_monitor import setup_with_traceloop

# Enable enterprise-grade tracing
setup_with_traceloop("my_app_name")

# Your AI code - now with distributed tracing!
```

#### 3. Flask App Integration
```python
from flask import Flask
from ai_monitor import flask_plug_and_play

app = Flask(__name__)

# One line to monitor entire Flask app
flask_plug_and_play(app)

# All your routes are now monitored automatically!
```

### What You Get Immediately

After `pip install ai-monitor`, you get:
- ✅ **HTTP Interception** - Automatic OpenAI API monitoring
- ✅ **Prometheus Metrics** - Real-time metrics at `http://localhost:8000/metrics`
- ✅ **Quality Analysis** - Hallucination detection & drift analysis
- ✅ **OpenTelemetry Traces** - Ready for Jaeger/Traceloop integration
- ✅ **Zero Code Changes** - Works with your existing AI applications

#### 1. Decorator Style (Easiest)
```python
from ai_monitor import monitor_llm_call, monitor_agent, monitor_tool_use

# Monitor LLM calls automatically
@monitor_llm_call(model="gpt-4")
def call_openai(prompt):
    # Your existing LLM code - no changes needed!
    response = openai.ChatCompletion.create(...)
    return response

# Monitor agent sessions  
@monitor_agent(name="my_assistant")
def run_agent():
    # Your agent code
    return agent_result

# Monitor tool usage
@monitor_tool_use(tool_name="web_search")
def search_web(query):
    # Your tool code
    return search_results
```

#### 2. Context Manager Style
```python
from ai_monitor import monitor_llm_call, monitor_agent_session

# Monitor individual LLM calls
with monitor_llm_call("gpt-4", "Hello world") as monitor:
    response = call_llm("Hello world")
    monitor.record_response(
        response="Hello! How can I help?",
        input_tokens=2,
        output_tokens=5,
        cost=0.001
    )

# Monitor complete agent sessions
with monitor_agent_session("customer_service") as session:
    # Record LLM calls within session
    session.record_llm_call("gpt-4", "Help request", "I can help with that", 10, 15, 0.002)
    
    # Record tool usage
    session.record_tool_use("knowledge_base", "search query", "results", success=True, duration=0.5)
```

#### 3. Global Monitor Instance
```python
from ai_monitor import get_monitor

# Get the default monitor instance
monitor = get_monitor()

# Record data manually
call_id = monitor.record_llm_call(
    model="gpt-3.5-turbo",
    prompt="What is AI?",
    response="AI is artificial intelligence...",
    input_tokens=4,
    output_tokens=20,
    latency=1.2,
    cost=0.001
)

# Get monitoring data
stats = monitor.get_summary_stats()
metrics = monitor.get_metrics()
recent_calls = monitor.get_llm_calls(limit=10)
```

## 📊 What Gets Monitored

### All OpenAI API Endpoints (Enhanced Coverage)
- **Chat Completions**: `client.chat.completions.create()` ✅
- **Text Completions**: `client.completions.create()` ✅  
- **Embeddings**: `client.embeddings.create()` ✅
- **Image Generation**: `client.images.generate()` (DALL-E) ✅
- **Audio Transcription**: `client.audio.transcriptions.create()` (Whisper) ✅
- **Audio Translation**: `client.audio.translations.create()` (Whisper) ✅
- **Content Moderation**: `client.moderations.create()` ✅
- **Model Operations**: `client.models.list()`, file operations, fine-tuning ✅
- **Azure OpenAI**: All deployment endpoints with enhanced model detection ✅

### Automatic Metrics (All Endpoints)
- **Latency & Performance**: Response time, tokens per second, throughput  
- **Token Usage**: Input/output tokens, token costs, efficiency ratios
- **Quality Indicators**: Response completeness, structure, consistency
- **Error Tracking**: Failure rates, error types, recovery times
- **Resource Usage**: Memory, CPU (when psutil installed)

### Advanced Detection
- **Hallucination Detection**: Pattern-based detection of uncertain or fabricated content
- **Model Drift**: Automatic detection of performance degradation over time
- **Cost Monitoring**: Real-time cost tracking and budgeting

### Trace & Context
- **Request Tracing**: End-to-end request flows with Jaeger integration
- **Agent Sessions**: Multi-step agent conversations and decision paths
- **Tool Usage**: External API calls and tool performance

## 📈 Data Export Options

### Built-in Exporters
- **Prometheus**: Metrics collection with built-in HTTP server
- **Jaeger**: Distributed tracing for request flows
- **Structured Logs**: JSON logs for analysis
- **Console**: Real-time monitoring output
- **JSON Files**: Local data storage

### Configuration Example
```python
from ai_monitor import init_monitoring, MonitoringConfig

config = MonitoringConfig(
    # Exporters
    enable_prometheus=True,
    enable_jaeger=True, 
    enable_logging=True,
    
    # Prometheus
    prometheus_port=8000,
    
    # Features
    detect_hallucination=True,
    detect_drift=True,
    track_costs=True,
    
    # Sampling
    trace_sampling_rate=1.0
)

monitor = init_monitoring(config)
```

## 🔍 Monitoring Features

### Hallucination Detection
```python
from ai_monitor.detectors import HallucinationDetector

detector = HallucinationDetector()
result = detector.detect(
    prompt="What is the capital of France?",
    response="I think it might be Paris, but I'm not entirely sure...",
    context="France is a country in Europe."
)

print(f"Is hallucination: {result.is_hallucination}")
print(f"Confidence: {result.confidence_score}")
print(f"Reasons: {result.reasons}")
```

### Drift Detection
```python
from ai_monitor.detectors import DriftDetector

drift_detector = DriftDetector()

# Update with each LLM call
drift_detector.update(
    latency=1.2,
    input_tokens=10,
    output_tokens=50,
    cost=0.002,
    response="AI response text"
)

# Check for drift
drift_results = drift_detector.detect_drift()
for drift in drift_results:
    if drift.has_drift:
        print(f"Drift detected in {drift.drift_type}")
        print(f"Baseline: {drift.baseline_value}, Current: {drift.current_value}")
```

## 📋 Dashboard & Visualization

### Prometheus Metrics (localhost:8000)
```
ai_monitor_llm_calls_total{model="gpt-4"} 150
ai_monitor_llm_latency_seconds_bucket{model="gpt-4",le="1.0"} 120
ai_monitor_tokens_total{model="gpt-4",type="input"} 5000
ai_monitor_tokens_total{model="gpt-4",type="output"} 15000
ai_monitor_cost_total{model="gpt-4"} 2.50
```

### Getting Summary Statistics
```python
from ai_monitor import get_monitor

monitor = get_monitor() 
stats = monitor.get_summary_stats()

print(f"Total LLM calls: {stats['total_calls']}")
print(f"Total tokens: {stats['total_tokens']}")
print(f"Total cost: ${stats['total_cost']:.4f}")
print(f"Average latency: {stats['average_latency']:.2f}s")
print(f"Tokens per second: {stats['tokens_per_second']:.1f}")
```

## 🛠️ Integration Examples

### OpenAI Integration
```python
from ai_monitor import monitor_llm_call
import openai

@monitor_llm_call()  # Automatically detects OpenAI format
def chat_with_gpt(messages):
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=messages
    )
    return response

# Usage - monitoring happens automatically!
result = chat_with_gpt([{"role": "user", "content": "Hello"}])
```

### Anthropic Claude Integration
```python
@monitor_llm_call(model="claude-3")
def chat_with_claude(prompt):
    response = anthropic.completions.create(
        model="claude-3-sonnet-20240229",
        prompt=prompt,
        max_tokens_to_sample=100
    )
    return response
```

### Custom Agent Integration
```python
@monitor_agent(name="research_assistant")
def research_agent(query):
    with monitor_agent_session("research_session") as session:
        
        # Search step
        search_results = search_web(query)
        session.record_tool_use("web_search", query, search_results, True, 2.1)
        
        # Analysis step
        analysis = analyze_with_llm(search_results)
        session.record_llm_call("gpt-4", f"Analyze: {search_results}", analysis, 100, 200, 0.01)
        
        # Summary step
        summary = summarize_findings(analysis)
        session.record_llm_call("gpt-3.5-turbo", f"Summarize: {analysis}", summary, 200, 50, 0.002)
        
        return summary
```

## 📊 Grafana Dashboard Setup

1. **Install Grafana and Prometheus**
2. **Configure Prometheus** to scrape `localhost:8000/metrics`
3. **Import dashboard** with these key metrics:
   - LLM call rate and latency
   - Token usage and costs
   - Error rates and drift detection
   - Agent session success rates

## 🚨 Alerting

### Prometheus Alerting Rules
```yaml
groups:
- name: ai_monitor_alerts
  rules:
  - alert: HighLatency
    expr: ai_monitor_llm_latency_seconds > 5
    labels:
      severity: warning
    annotations:
      summary: "High LLM latency detected"
      
  - alert: CostThreshold
    expr: increase(ai_monitor_cost_total[1h]) > 10
    labels:
      severity: critical
    annotations:
      summary: "Hourly cost exceeds $10"
```

## 🔧 Advanced Configuration

### Custom Exporters
```python
from ai_monitor import configure_exporters, get_monitor

monitor = get_monitor()
configure_exporters(monitor, [
    {'type': 'console'},
    {'type': 'json_file', 'file_prefix': 'my_app_monitor'}
])
```

### Performance Profiling
```python
from ai_monitor.utils import PerformanceProfiler

profiler = PerformanceProfiler()

with profiler.profile_operation("llm_inference"):
    result = call_llm("prompt")

summary = profiler.get_performance_summary()
```

## 🎯 Key Benefits

- **Zero Code Changes**: Drop-in decorators and context managers
- **Comprehensive Monitoring**: Tracks everything from tokens to traces
- **Production Ready**: Prometheus, Jaeger, and structured logging
- **Intelligent Detection**: Automated hallucination and drift detection  
- **Cost Awareness**: Real-time cost tracking and optimization
- **Performance Insights**: Detailed latency and throughput analysis

## 📝 License

MIT License - feel free to use in your projects!

---

**Ready to monitor your AI agents?** Just `pip install` the dependencies and add a single decorator! 🚀