llm-observability 1.0.2
Universal LLM observability middleware with multi-provider support (Langfuse v2/v3, MLflow, OpenTelemetry) and healthcare-specific features
License: MIT
Requires Python: <3.14,>=3.11
Keywords: llm, observability, monitoring, langfuse, mlflow, opentelemetry, healthcare, ai, tracing

# LLM Observability

A comprehensive observability solution for LLM applications, supporting both Langfuse V2 and V3 with healthcare compliance features.

## Table of Contents

- [Features](#features)
- [Installation](#installation)
- [Quick Start](#quick-start)
- [Configuration](#configuration)
- [API Reference](#api-reference)
- [Advanced Usage](#advanced-usage)
- [Testing](#testing)
- [Local Development](#local-development)
- [Integration Testing with AIOA-Core](#integration-testing-with-aioa-core)
- [Performance and Timeout Considerations](#performance-and-timeout-considerations)
- [Langfuse 3.0+ Compatibility](#langfuse-30-compatibility)
- [Contributing](#contributing)
- [Pull Request Process](#pull-request-process)
- [Support](#support)

## Features

- **Multi-Version Support**: Seamless support for both Langfuse V2 and V3
- **Healthcare Compliance**: Built-in PII anonymization for HIPAA compliance
- **Healthcare-Specific Middleware**: Patient context tracking, case management, and HIPAA-compliant tracing
- **Local Langfuse Detection**: Automatic detection and connection to local Langfuse instances
- **Trace Collection**: Comprehensive trace lifecycle management and statistics collection
- **Graceful Degradation**: Continues working even when observability services are unavailable
- **Version-Agnostic API**: Same interface works with both V2 and V3
- **Comprehensive Testing**: Unit, integration, and version-specific test suites
- **Real-Time Monitoring**: Live trace and span management
- **Context Management**: User, session, and patient context tracking

## Current Test Status

The project maintains excellent test coverage across all environments:

| Environment | Status | Passed | Failed | Skipped | Coverage |
|-------------|--------|--------|--------|---------|----------|
| **V2 Environment** | ✅ **100% Success** | 298 | 0 | 35 | 100% |
| **V3 Environment** | ✅ **100% Success** | 326 | 0 | 7 | 100% |
| **Mock Environment** | ✅ **100% Success** | 260+ | 0 | 70+ | 100% |
| **CI/CD Pipeline** | ✅ **All Gates Pass** | - | - | - | 100% |

**Recent Improvements (December 2024):**
- ✅ **Test Isolation**: Fixed global middleware mocking issues
- ✅ **V2 Environment**: Achieved 100% test success rate
- ✅ **Refactoring**: Completed modular middleware architecture
- ✅ **Quality**: Enhanced CI/CD pipeline with comprehensive checks

## Installation

### Basic Installation

```bash
pip install llm-observability
```

### Installation with Healthcare Features

For healthcare applications with patient context tracking and HIPAA compliance:

```bash
pip install llm-observability[healthcare]
```

### Installation with Extended Features

For applications requiring local Langfuse detection and trace collection:

```bash
pip install llm-observability[extended]
```

### Installation with All Features

For complete functionality including all providers and features:

```bash
pip install llm-observability[all]
```

### Development Installation

```bash
git clone https://github.com/qventus/llm-observability.git
cd llm-observability
pip install -e .[dev]
```

### Installation Options

| Option | Description | Dependencies |
|--------|-------------|--------------|
| `llm-observability` | Core functionality only | Minimal dependencies |
| `llm-observability[healthcare]` | Healthcare-specific features | `aiohttp`, `requests`, `python-dotenv` |
| `llm-observability[extended]` | Extended features (local detection, trace collection) | `aiohttp`, `requests`, `python-dotenv` |
| `llm-observability[all]` | All features and providers | All dependencies |
| `llm-observability[dev]` | Development tools and testing | All dependencies + development tools |

## Quick Start

```python
from llm_observability import trace_operation, set_user_context, score_current_trace

# Set user context
set_user_context(user_id="user123", patient_id="patient456")

# Trace an operation
with trace_operation("llm-generation", "generation") as trace:
    if trace:
        trace.update(input="User query", output="AI response")
        trace.add_score("quality", 0.95)

# Or use convenience functions
score_current_trace("accuracy", 0.95, "High confidence response")
```

## Migrated Components

The following components have been migrated from aioa-core and are now available in llm-observability:

### New Convenience Functions

The following convenience functions have been migrated from aioa-core and provide simplified access to observability features:

```python
from llm_observability import (
    trace_operation, trace_tool_execution, score_current_trace,
    update_current_trace, set_user_context, clear_user_context, create_trace_id
)

# Create a trace ID
trace_id = create_trace_id("my-workflow")

# Set user context
set_user_context(user_id="user123", patient_id="patient456")

# Trace operations
with trace_operation("llm-generation", "generation") as trace:
    if trace:
        trace.update(input="User query", output="AI response")

# Trace tool executions
with trace_tool_execution("database-query", {"query": "SELECT * FROM users"}) as tool_trace:
    if tool_trace:
        tool_trace.update(output={"result": "data"})

# Score and update traces
score_current_trace("accuracy", 0.95, "High confidence response")
update_current_trace(status="completed", metadata={"final_result": "success"})

# Clear context when done
clear_user_context()
```

### Quick Test

Verify your installation with a quick test:

```bash
# Test V2 environment
source dev_environments/langfuse_v2/bin/activate
python -m pytest tests/ -v | tail -5
# Expected: 298 passed, 0 failed, 35 skipped

# Test V3 environment
source dev_environments/langfuse_v3/bin/activate
python -m pytest tests/ -v | tail -5
# Expected: 326 passed, 0 failed, 7 skipped
```
```

### Healthcare Observability Middleware

Healthcare-specific observability with HIPAA compliance and patient context tracking:

```python
from llm_observability import HealthcareObservabilityMiddleware

# Initialize healthcare middleware
healthcare_middleware = HealthcareObservabilityMiddleware()

# Set patient context
healthcare_middleware.set_patient_context(
    patient_id="patient-123",
    mrn="MRN-456",
    org_id="org-789"
)

# Set case context
healthcare_middleware.set_case_context(
    case_id="case-abc",
    workflow_id="workflow-xyz"
)

# Trace healthcare-specific operations
with healthcare_middleware.trace_patient_interaction("sms", "patient-123") as trace:
    if trace:
        trace.update(channel="sms", priority="high")

with healthcare_middleware.trace_agent_execution("nurse-bot", "specialist") as trace:
    if trace:
        trace.update(workflow_stage="triage", decision_point="escalation")
```

### Local Langfuse Detection

Automatic detection and connection to local Langfuse instances:

```python
from llm_observability import LocalLangfuseDetector

# Initialize detector
detector = LocalLangfuseDetector()

# Scan for local instances
open_ports = detector.scan_local_ports()
print(f"Found {len(open_ports)} open ports")

# Get best local configuration
config = detector.get_best_local_langfuse_config()
if config:
    print(f"Local Langfuse found at {config['host']}")
```

### Trace Collection

Comprehensive trace lifecycle management and statistics collection:

```python
from llm_observability import TraceCollector

# Initialize collector
collector = TraceCollector()

# Start a trace
trace_id = collector.start_trace("workflow-execution", "Patient care workflow")

# Add spans and metrics
collector.add_span_metrics(trace_id, "llm-call", duration_ms=1500, tokens_used=500)

# Complete trace
collector.complete_trace(trace_id)

# Get statistics
stats = collector.get_trace_statistics()
print(f"Active traces: {stats['active_traces']}")
print(f"Completed traces: {stats['completed_traces']}")
```

## Configuration

### Environment Variables

```bash
# Langfuse Configuration
LANGFUSE_VERSION=auto          # "2", "3", or "auto"
LANGFUSE_HOST=https://cloud.langfuse.com
LANGFUSE_PUBLIC_KEY=your_public_key
LANGFUSE_SECRET_KEY=your_secret_key
LANGFUSE_ENVIRONMENT=production
LANGFUSE_TIMEOUT=30            # Default timeout in seconds

# V2/V3 Specific Timeouts (for integration tests)
LANGFUSE_V2_TIMEOUT=120        # 2 minutes for V2 (VPN access)
LANGFUSE_V3_TIMEOUT=60         # 1 minute for V3 (direct cloud)

# Observability Configuration
OBSERVABILITY_ENABLED=true
OBSERVABILITY_FAIL_SILENTLY=true
```

### Configuration File

Create `password.json` for local development:

```json
{
  "langfuse_v2_external": {
    "host": "https://langfuse-dev.qventus-int.com/",
    "publickey": "your_v2_public_key",
    "secretkey": "your_v2_secret_key"
  },
  "langfuse_v3_production": {
    "host": "https://langfuse-prod.qventus.com/",
    "public_key": "your_v3_public_key",
    "secret_key": "your_v3_secret_key"
  }
}
```

## API Reference

### Core Functions

#### **`trace_operation(name, operation_type="span", model=None, metadata=None)`**
Create a new trace operation.

**Parameters:**
- `name` (str): Name of the operation
- `operation_type` (str): Type of operation (span, generation, workflow, etc.)
- `model` (str, optional): Model used for the operation
- `metadata` (dict, optional): Additional metadata

**Returns:** Trace context manager

**Example:**
```python
with trace_operation("llm-generation", "generation", model="gpt-4") as trace:
    if trace:
        trace.update(input="User query", output="AI response")
```

#### **`trace_tool_execution(tool_name, input_data, metadata=None)`**
Trace a tool execution within a trace.

**Parameters:**
- `tool_name` (str): Name of the tool being executed
- `input_data` (any): Input data for the tool
- `metadata` (dict, optional): Additional metadata

**Returns:** Tool trace context manager

**Example:**
```python
with trace_tool_execution("database-query", {"query": "SELECT * FROM users"}) as tool_trace:
    if tool_trace:
        tool_trace.update(output={"results": [{"id": 1}]})
```

#### **`score_current_trace(name, value, comment=None)`**
Add a score to the current trace.

**Parameters:**
- `name` (str): Name of the score
- `value` (float): Score value (0.0-1.0)
- `comment` (str, optional): Comment about the score

**Returns:** True if successful, False otherwise

**Example:**
```python
score_current_trace("accuracy", 0.95, "High confidence response")
```

#### **`update_current_trace(status=None, output=None, metadata=None, **kwargs)`**
Update the current trace with new information.

**Parameters:**
- `status` (str, optional): New status
- `output` (any, optional): Output data
- `metadata` (dict, optional): Additional metadata
- `**kwargs`: Additional fields to update

**Returns:** True if successful, False otherwise

**Example:**
```python
update_current_trace(
    status="completed",
    output="Final result",
    processing_time=1.23,
    tokens_used=150
)
```

#### **`set_user_context(user_id=None, session_id=None, patient_id=None, case_id=None, tags=None)`**
Set user context for the current session.

**Parameters:**
- `user_id` (str, optional): User identifier
- `session_id` (str, optional): Session identifier
- `patient_id` (str, optional): Patient identifier (healthcare)
- `case_id` (str, optional): Case identifier (healthcare)
- `tags` (dict, optional): Additional tags

**Returns:** True if successful, False otherwise

**Example:**
```python
set_user_context(
    user_id="user123",
    patient_id="patient456",
    case_id="case789",
    tags={"environment": "production"}
)
```

#### **`clear_user_context()`**
Clear the current user context.

**Returns:** True if successful, False otherwise

#### **`create_trace_id()`**
Generate a new trace ID.

**Returns:** String trace ID

### Prompt Management Functions (Langfuse 2.0+ and 3.0+)

✅ **Note:** These functions work with both Langfuse 2.0 and 3.0+. The implementation automatically adapts to the installed version.

**Langfuse 2.0 Features:**
- Basic prompt creation and retrieval
- Version management with labels
- Prompt caching with TTL
- Text and chat prompt types
- Limited template listing (no offset support)

**Langfuse 3.0+ Features:**
- All v2 features plus:
- Advanced template listing with pagination
- Enhanced metadata support
- Better error handling
- OpenTelemetry integration

**Important:** In Langfuse V3, prompts are identified by name rather than numeric IDs, and updates create new versions of prompts.

#### **`create_prompt_template(name, prompt, description=None, tags=None, metadata=None, **kwargs)`**
Create a new prompt template in Langfuse V2 or V3.

**Parameters:**
- `name` (str): Name of the prompt template (used as the ID in V3)
- `prompt` (str): The prompt template string with variable placeholders like `{{variable_name}}`
- `description` (str, optional): Description of the template (stored as commit message)
- `tags` (list, optional): List of tags for categorization
- `metadata` (dict, optional): Additional metadata (stored as config)
- `**kwargs`: Additional arguments passed to the Langfuse client

**Returns:** Dictionary containing template information with `name` as the ID, or None if not supported

**Example:**
```python
template = create_prompt_template(
    name="customer-support",
    prompt="You are a helpful customer support agent. User question: {{question}}. Please provide a {{tone}} response.",
    description="Template for customer support responses",
    tags=["support", "customer-service"],
    metadata={"category": "customer-service", "version": "1.0"}
)
print(f"Created template: {template['id']}")  # Returns the name as ID
```

#### **`get_prompt_template(template_id, **kwargs)`**
Retrieve a prompt template by name (ID in V2/V3).

**Parameters:**
- `template_id` (str): Name of the prompt template to retrieve (acts as ID in V3)
- `**kwargs`: Additional arguments passed to the Langfuse client

**Returns:** Dictionary containing template information, or None if not found/supported

**Example:**
```python
template = get_prompt_template("customer-support")
if template:
    print(f"Template: {template['name']}")
    print(f"Prompt: {template['prompt']}")
    print(f"Description: {template['description']}")
    print(f"Tags: {template['tags']}")
```

#### **`list_prompt_templates(limit=None, offset=None, tags=None, **kwargs)`**
List available prompt templates.

**Parameters:**
- `limit` (int, optional): Maximum number of templates to return
- `offset` (int, optional): Number of templates to skip (not supported in V2, limited in V3)
- `tags` (list, optional): Filter templates by tags (only first tag is used in V3, limited in V2)
- `**kwargs`: Additional arguments passed to the Langfuse client

**Returns:** List of template dictionaries, or empty list if not supported

**Note:** Langfuse V2 has limited listing capabilities and may return an empty list.

**Example:**
```python
# List all templates
templates = list_prompt_templates()

# List templates with specific tags
support_templates = list_prompt_templates(tags=["support"])

# Limited listing
templates = list_prompt_templates(limit=10)
```

#### **`update_prompt_template(template_id, name=None, prompt=None, description=None, tags=None, metadata=None, **kwargs)`**
Update an existing prompt template by creating a new version.

**Note:** In both Langfuse V2 and V3, updates create new versions of prompts rather than modifying existing ones.

**Parameters:**
- `template_id` (str): Name of the prompt template to update (acts as ID in V3)
- `name` (str, optional): New name for the template (not recommended in V3)
- `prompt` (str, optional): New prompt string
- `description` (str, optional): New description (stored as commit message)
- `tags` (list, optional): New tags list
- `metadata` (dict, optional): New metadata dictionary (stored as config)
- `**kwargs`: Additional arguments passed to the Langfuse client

**Returns:** Dictionary containing updated template information, or None if not supported

**Example:**
```python
updated = update_prompt_template(
    template_id="customer-support",
    prompt="You are a helpful customer support agent. User question: {{question}}. Please provide a {{tone}} response with examples.",
    description="Updated with examples and tone parameter",
    tags=["support", "updated", "examples"]
)
```

#### **`delete_prompt_template(template_id, **kwargs)`**
Delete a prompt template.

**Note:** This function is not supported in Langfuse 3.0+ API. Prompts are versioned and can be managed through labels instead of deletion. The function returns `False` to indicate that deletion is not supported.

**Parameters:**
- `template_id` (str): Name of the prompt template to delete (acts as ID in V3)
- `**kwargs`: Additional arguments passed to the Langfuse client

**Returns:** `False` (deletion not supported in V3)

**Example:**
```python
success = delete_prompt_template("customer-support")
if not success:
    print("Template deletion not supported in Langfuse 3.0+")
```

**Parameters:**
- `template_id` (str): Name of the prompt template to delete (acts as ID in V3)
- `**kwargs`: Additional arguments passed to the Langfuse client

**Returns:** False (not supported in V3)

**Example:**
```python
success = delete_prompt_template("customer-support")
if not success:
    print("Template deletion not supported in Langfuse V3")
    print("Use labels to manage prompt versions instead")
```

#### **`use_prompt_template(template_id, variables=None, **kwargs)`**
Use a prompt template with variable substitution.

**Parameters:**
- `template_id` (str): Name of the prompt template to use (acts as ID in V2/V3)
- `variables` (dict, optional): Dictionary of variables to substitute in the template
- `**kwargs`: Additional arguments passed to the Langfuse client

**Returns:** Rendered prompt string, or None if not supported

**Features:**
- Supports nested variable substitution (e.g., `{{user.name}}`, `{{user.age}}`, `{{task.description}}`)
- Handles complex data types (lists, dictionaries, nested objects)
- Uses `{{variable}}` syntax (Langfuse style) instead of `${variable}` (string.Template style)
- Works with both V2 and V3 prompt templates
- Graceful handling of missing variables (leaves placeholders unchanged)

**Example:**
```python
# Use template with variables
rendered_prompt = use_prompt_template(
    template_id="customer-support",
    variables={
        "question": "How do I reset my password?",
        "tone": "professional"
    }
)

# Use in LLM call with tracing
with trace_operation("llm-generation", "generation") as trace:
    if trace:
        response = llm_call(rendered_prompt)
        trace.update(
            input=rendered_prompt,
            output=response,
            template_id="customer-support",
            variables={"question": "How do I reset my password?", "tone": "professional"}
        )

# Complex variable substitution with nested objects
complex_prompt = use_prompt_template(
    template_id="healthcare-analysis",
    variables={
        "patient": {
            "name": "John Smith",
            "age": "45",
            "symptoms": ["headache", "fever"],
            "preferences": ["email", "phone"]
        },
        "task": {
            "description": "Diagnostic analysis",
            "priority": "high"
        },
        "analysis_type": "diagnostic"
    }
)
print(complex_prompt)
# Output: Template with {{patient.name}}, {{patient.age}}, {{patient.symptoms}},
#         {{task.description}}, {{task.priority}} substituted
```

### Configuration Functions

#### **`configure_middleware(langfuse_version="auto", public_key=None, secret_key=None, host=None, environment="development", enabled=True, fail_silently=True, debug=False, timeout=30, **kwargs)`**
Configure the global middleware with specific Langfuse settings.

**Parameters:**
- `langfuse_version` (str): Langfuse version to use ("2", "3", or "auto")
- `public_key` (str, optional): Langfuse public key
- `secret_key` (str, optional): Langfuse secret key
- `host` (str, optional): Langfuse host URL
- `environment` (str): Environment name (development, staging, production)
- `enabled` (bool): Whether observability is enabled
- `fail_silently` (bool): Whether to fail silently on connection errors
- `debug` (bool): Whether to enable debug mode
- `timeout` (int): Connection timeout in seconds
- `**kwargs`: Additional configuration options

**Returns:** The configured ObservabilityMiddleware instance

**Example:**
```python
from llm_observability import configure_middleware

# Configure for V2
middleware = configure_middleware(
    langfuse_version="2",
    public_key="pk-lf-1234567890abcdef",
    secret_key="sk-lf-1234567890abcdef",
    host="https://cloud.langfuse.com"
)

# Configure for V3
middleware = configure_middleware(
    langfuse_version="3",
    public_key="pk-lf-1234567890abcdef",
    secret_key="sk-lf-1234567890abcdef",
    host="https://cloud.langfuse.com"
)
```

## Advanced Usage

### Healthcare-Specific Features

```python
from llm_observability import HealthcareObservabilityMiddleware

# Healthcare middleware with PII anonymization
healthcare_middleware = HealthcareObservabilityMiddleware()

# Set patient context
healthcare_middleware.set_patient_context(
    patient_id="patient123",
    case_id="case456"
)

# Trace patient interaction
with healthcare_middleware.trace_patient_interaction(
    "consultation",
    patient_id="patient123"
) as trace:
    if trace:
        trace.update(
            input="Patient symptoms",
            output="Diagnosis and treatment plan"
        )
```

### Prompt Management (Langfuse 2.0+ and 3.0+)

```python
from llm_observability import (
    create_prompt_template,
    get_prompt_template,
    list_prompt_templates,
    update_prompt_template,
    use_prompt_template,
    trace_operation
)

# Create a prompt template for healthcare consultations
template = create_prompt_template(
    name="healthcare-consultation",
    prompt="""
    You are a healthcare AI assistant. Please help with the following:

    Patient Information:
    - Age: {{age}}
    - Symptoms: {{symptoms}}
    - Medical History: {{medical_history}}

    Please provide a {{analysis_type}} response while being mindful of healthcare guidelines.
    Focus on {{focus_area}} and provide recommendations.
    """,
    description="Template for healthcare consultation responses",
    tags=["healthcare", "consultation", "medical"],
    metadata={"category": "healthcare", "version": "1.0"}
)

print(f"Created template: {template['id']}")  # Returns the name as ID

# List all available templates
templates = list_prompt_templates(limit=10)
print(f"Found {len(templates)} templates")

# Use the template in a trace
with trace_operation("healthcare-consultation", "generation") as trace:
    if trace:
        # Render the prompt with variables
        rendered_prompt = use_prompt_template(
            template_id=template["id"],  # Use name as ID
            variables={
                "age": "45",
                "symptoms": "headache and fever",
                "medical_history": "hypertension",
                "analysis_type": "diagnostic",
                "focus_area": "symptom analysis"
            }
        )

        # Use in LLM call
        response = llm_call(rendered_prompt)

        # Update trace with results and template information
        trace.update(
            input=rendered_prompt,
            output=response,
            template_id=template["id"],
            variables={
                "age": "45",
                "symptoms": "headache and fever",
                "medical_history": "hypertension",
                "analysis_type": "diagnostic",
                "focus_area": "symptom analysis"
            }
        )

        # Add evaluation scores
        trace.add_score("relevance", 0.95)
        trace.add_score("safety", 0.98)
        trace.add_score("template_effectiveness", 0.92)

# Update the template (creates a new version in V3)
updated_template = update_prompt_template(
    template_id="healthcare-consultation",
    prompt="""
    You are a healthcare AI assistant. Please help with the following:

    Patient Information:
    - Age: {{age}}
    - Symptoms: {{symptoms}}
    - Medical History: {{medical_history}}

    Please provide a {{analysis_type}} response while being mindful of healthcare guidelines.
    Focus on {{focus_area}} and provide detailed recommendations.
    Include potential differential diagnoses if applicable.
    """,
    description="Updated template with differential diagnoses",
    tags=["healthcare", "consultation", "medical", "updated"]
)

# Complex variable substitution example
complex_prompt = use_prompt_template(
    template_id="healthcare-consultation",
    variables={
        "age": "58",
        "symptoms": ["chest pain", "shortness of breath", "fatigue"],
        "medical_history": {
            "conditions": ["hypertension", "diabetes"],
            "medications": ["lisinopril", "metformin"]
        },
        "analysis_type": "urgent",
        "focus_area": "cardiac symptoms"
    }
)
```

### Prompt Management Version Differences

The prompt management functionality works with both Langfuse V2 and V3, but there are some key differences:

#### **Langfuse V2 Features:**
- ✅ Basic prompt creation and retrieval
- ✅ Version management with labels
- ✅ Prompt caching with TTL
- ✅ Text and chat prompt types
- ✅ Variable substitution (including nested variables)
- ⚠️ Limited template listing (no pagination support)
- ❌ No direct template deletion

#### **Langfuse V3 Features:**
- ✅ All V2 features plus:
- ✅ Advanced template listing with pagination
- ✅ Enhanced metadata support
- ✅ Better error handling
- ✅ OpenTelemetry integration
- ✅ More robust API

#### **Migration from V2 to V3:**
```python
# V2 usage (still works in V3)
template = create_prompt_template(
    name="my-template",
    prompt="Hello {{name}}",
    description="Simple template"
)

# V3 enhanced usage
template = create_prompt_template(
    name="my-template",
    prompt="Hello {{name}}",
    description="Simple template",
    tags=["greeting", "simple"],
    metadata={"category": "greeting", "version": "1.0"}
)

# Both versions support the same API
rendered = use_prompt_template("my-template", {"name": "Alice"})
```

### Version-Specific Configuration

```python
from llm_observability import ObservabilityConfig, LangfuseConfig

# V2 Configuration
config_v2 = ObservabilityConfig()
config_v2.langfuse = LangfuseConfig()
config_v2.langfuse.version = "2"
config_v2.langfuse.host = "https://langfuse-dev.qventus-int.com/"
config_v2.langfuse.timeout = 120  # Longer timeout for VPN access

# V3 Configuration
config_v3 = ObservabilityConfig()
config_v3.langfuse = LangfuseConfig()
config_v3.langfuse.version = "3"
config_v3.langfuse.host = "https://langfuse-prod.qventus.com/"
config_v3.langfuse.timeout = 60   # Shorter timeout for direct access
```

## Testing

### Test Categories

- **Unit Tests**: Test individual components without external dependencies
- **Integration Tests**: Test with real Langfuse credentials
- **Version-Specific Tests**: Test V2 and V3 functionality separately
- **No-Langfuse Tests**: Test graceful degradation when Langfuse is not available

### Running Tests

```bash
# All tests
python -m pytest

# Unit tests only
python -m pytest tests/unit/ -m "not integration"

# Integration tests only
python -m pytest tests/integration/ -m "integration"

# V2 tests only
python -m pytest tests/unit/ -k "v2" -m "not integration"

# V3 tests only
python -m pytest tests/unit/ -k "v3" -m "not integration"

# Prompt management tests (V2 and V3)
python -m pytest tests/unit/test_prompt_management.py -v

# V2 prompt management tests
python -m pytest tests/unit/test_providers/test_langfuse_v2_provider.py::TestLangfuseV2ProviderPromptManagement -v

# V2 prompt management integration tests
python -m pytest tests/integration/test_prompt_management_v2_integration.py -v

# CI/CD prompt management tests
python -m pytest tests/integration/test_prompt_management_cicd.py -v

# Tests without Langfuse
python -m pytest tests/unit/ -m "not integration"
```

### Testing with Different Langfuse Versions

The project supports testing with both Langfuse V2 and V3. Use the appropriate environment variable to specify the version:

```bash
# Test with V2
LANGFUSE_VERSION=2 python -m pytest tests/integration/

# Test with V3 (default)
LANGFUSE_VERSION=3 python -m pytest tests/integration/

# Test in V2 virtual environment
source dev_environments/langfuse_v2/bin/activate
LANGFUSE_VERSION=2 python -m pytest tests/integration/

# Run V2 prompt management example
source dev_environments/langfuse_v2/bin/activate
python examples/langfuse_v2_prompt_management_example.py

# Run automated V2 prompt management tests
./run_v2_prompt_tests.sh

# Test in V3 virtual environment
source dev_environments/langfuse_v3/bin/activate
python -m pytest tests/integration/
```

### Testing Prompt Management

For local testing of prompt management features:

```bash
# Test with local credentials (password.json)
python examples/test_prompt_creation.py

# Test with environment variables (CI/CD)
export LANGFUSE_HOST="https://your-langfuse-server.com"
export LANGFUSE_PUBLIC_KEY="your-public-key"
export LANGFUSE_SECRET_KEY="your-secret-key"
export LANGFUSE_VERSION="3"
python examples/test_prompt_creation.py
```

### Test Environments

The project uses separate virtual environments for V2 and V3 testing:

```bash
# V2 Environment
python -m venv venv_v2
source venv_v2/bin/activate  # On Windows: venv_v2\Scripts\activate
pip install "langfuse>=2.0.0,<3.0.0"
pip install -e .[dev]

# V3 Environment
python -m venv venv_v3
source venv_v3/bin/activate  # On Windows: venv_v3\Scripts\activate
pip install "langfuse>=3.0.0"
pip install -e .[dev]
```

### CI/CD Testing Strategy

The CI/CD pipeline includes:

- **Test Without Langfuse**: Ensures graceful degradation
- **Test Langfuse 2.0**: V2-specific functionality
- **Test Langfuse 3.0**: V3-specific functionality
- **Test Version-Agnostic Code**: Cross-version compatibility

## Langfuse 3.0+ Compatibility

The `llm-observability` package has been fully updated for Langfuse 3.0+ compatibility. For detailed information, see the [Langfuse 3.0+ Compatibility Guide](docs/LANGFUSE_3_COMPATIBILITY.md).

### Key Features

- **✅ Full Langfuse 3.0+ Support**: All prompt management functions work correctly
- **✅ Enhanced Variable Substitution**: Supports nested variables like `{{user.name}}`, `{{task.description}}`
- **✅ Robust Error Handling**: Graceful handling of API differences between V2 and V3
- **✅ Backward Compatibility**: V2 functionality remains fully supported
- **✅ Comprehensive Testing**: 263 passing tests with full CI/CD compatibility

### Quick Migration

If you're upgrading from Langfuse 2.0:

```python
# Update your configuration
from llm_observability import configure_middleware

middleware = configure_middleware(
    langfuse_version="3",  # or "auto" for automatic detection
    # ... your existing configuration
)

# Your existing code continues to work
from llm_observability import create_prompt_template, use_prompt_template

# Create templates with nested variables
template = create_prompt_template(
    name="my-template",
    prompt="Hello {{user.name}}, your task is {{task.description}}"
)

# Use with complex variables
rendered = use_prompt_template(
    "my-template",
    variables={
        "user": {"name": "John Doe"},
        "task": {"description": "Complete analysis"}
    }
)
```

## Recent Improvements and Fixes

### Test Isolation and Quality Improvements (December 2024)

**Latest Fixes:**
- **✅ Test Isolation**: Fixed global middleware mocking issues that caused inconsistent test results
- **✅ V2 Environment**: Achieved 100% test success rate (298 passed, 0 failed)
- **✅ V3 Environment**: Maintained 100% test success rate (326 passed, 0 failed)
- **✅ Middleware Refactoring**: Completed modular architecture with proper separation of concerns
- **✅ Quality Gates**: Enhanced CI/CD pipeline with comprehensive pre-commit hooks

### Langfuse 3.0+ API Compatibility (Latest)

The package has been fully updated for Langfuse 3.0+ compatibility with comprehensive fixes:

**Key Improvements:**
- **✅ Full Langfuse 3.0+ Support**: All prompt management functions now work correctly with the new API
- **✅ Enhanced Variable Substitution**: Supports nested variables like `{{user.name}}`, `{{task.description}}`
- **✅ Robust Error Handling**: Graceful handling of API differences between V2 and V3
- **✅ CI/CD Compatibility**: All tests pass in both local and CI environments
- **✅ Backward Compatibility**: V2 functionality remains fully supported

**API Enhancements:**
- **Prompt Creation**: Uses `client.create_prompt()` with proper name and label parameters
- **Prompt Retrieval**: Implements label-based retrieval with fallback to common labels
- **Prompt Updates**: Creates new versions instead of modifying existing ones (V3 pattern)
- **Variable Substitution**: Custom regex-based substitution supporting `{{variable}}` syntax
- **Nested Variables**: Full support for complex nested object access
- **List Support**: Proper handling of list variables in templates

**Testing Achievements:**
- **✅ 263 passing tests** with comprehensive coverage
- **✅ All CI/CD pipelines passing** (Security Scan, V2/V3 tests, Version-Agnostic tests)
- **✅ Integration tests working** with real Langfuse credentials
- **✅ Error handling tests** for graceful degradation scenarios

**Breaking Changes:**
- `delete_prompt_template()` now returns `False` in V3 (deletion not supported)
- `list_prompt_templates()` returns empty list in V3 (listing not available)
- Variable substitution uses `{{variable}}` syntax exclusively

## Local Development

### Setting Up a Local Langfuse Server

1. **Using Docker Compose**:
```bash
# Clone Langfuse
git clone https://github.com/langfuse/langfuse.git
cd langfuse

# Start local server
docker-compose up -d

# Access at http://localhost:3000
```

2. **Using Docker**:
```bash
docker run -p 3000:3000 langfuse/langfuse:latest
```

3. **Configuration for Local Development**:
```bash
export LANGFUSE_HOST=http://localhost:3000
export LANGFUSE_PUBLIC_KEY=your_local_public_key
export LANGFUSE_SECRET_KEY=your_local_secret_key
```

### Development Workflow

```bash
# Install in development mode
pip install -e .[dev]

# Run tests
python -m pytest

# Run linting
flake8 llm_observability tests
black llm_observability tests

# Run type checking
mypy llm_observability
```

## Integration Testing with AIOA-Core

### Overview

The `llm-observability` package is designed to integrate seamlessly with the `aioa-core` project, providing comprehensive observability capabilities for healthcare AI applications.

### Architecture

**Before Integration:**
```
aioa-core/
├── aioa_core/
│   ├── middleware/
│   │   └── observability.py (duplicate code)
│   ├── config/
│   │   └── observability.py (duplicate code)
│   └── utils/
│       └── observability.py (duplicate code)
└── tests/
    └── observability tests (duplicate tests)
```

**After Integration:**
```
aioa-core/
├── aioa_core/
│   ├── middleware/
│   │   └── healthcare_observability.py (extends llm-observability)
│   └── (observability code removed)
└── tests/
    └── (observability tests updated to use llm-observability)

llm-observability/
├── llm_observability/
│   ├── middleware/
│   │   └── observability.py (shared implementation)
│   ├── config/
│   │   └── observability.py (shared configuration)
│   └── utils/
│       └── observability.py (shared utilities)
└── tests/
    └── comprehensive test suite
```

### Prerequisites

1. **Langfuse Credentials**: Ensure you have valid Langfuse V2 and/or V3 credentials
2. **Network Access**: V2 tests require VPN access to internal servers
3. **Dependencies**: Both projects must have compatible Python versions

### Running Integration Tests

```bash
# From llm-observability directory
./run_v2_integration_tests.sh  # V2 tests with real credentials
./run_v3_integration_tests.sh  # V3 tests with real credentials

# From aioa-core directory
python -m pytest tests/ -k "observability"  # Updated tests using llm-observability
```

### Testing with Different Langfuse Versions

```bash
# Test V2 functionality
export LANGFUSE_VERSION="2"
python -m pytest tests/integration/ -m "integration"

# Test V3 functionality
export LANGFUSE_VERSION="3"
python -m pytest tests/integration/ -m "integration"

# Test version-agnostic code
python -m pytest tests/unit/ -m "not integration"
```

## Performance and Timeout Considerations

### V2 vs V3 Performance Differences

**V2 Tests (Slower)**:
- Require VPN access to internal servers (`langfuse-dev.qventus-int.com`)
- Network latency through corporate VPN
- Default timeout: 120 seconds (2 minutes)
- CI/CD timeout: 180 seconds (3 minutes)

**V3 Tests (Faster)**:
- Direct cloud access (`langfuse-prod.qventus.com`)
- Lower network latency
- Default timeout: 60 seconds (1 minute)
- CI/CD timeout: 60 seconds (1 minute)

### Timeout Configuration

```bash
# Environment variables for timeout control
export LANGFUSE_V2_TIMEOUT="120"  # 2 minutes for V2 (VPN access)
export LANGFUSE_V3_TIMEOUT="60"   # 1 minute for V3 (direct cloud)

# Programmatic configuration
from llm_observability import configure_middleware

# V2 with longer timeout
middleware_v2 = configure_middleware(
    langfuse_version="2",
    timeout=120,  # 2 minutes for VPN access
    host="https://langfuse-dev.qventus-int.com/"
)

# V3 with standard timeout
middleware_v3 = configure_middleware(
    langfuse_version="3",
    timeout=60,   # 1 minute for direct access
    host="https://langfuse-prod.qventus.com/"
)
```

### Integration Test Behavior

Integration tests are designed to handle timeout scenarios gracefully:

1. **Timeout Monitoring**: Tests track execution time and warn if approaching limits
2. **Graceful Failure**: Tests don't fail immediately on timeout - they log warnings
3. **Version-Aware Timeouts**: Different timeout values for V2 vs V3
4. **Detailed Reporting**: Clear indication of timeout vs other failures

### Troubleshooting Slow Tests

If V2 tests are consistently slow:

1. **Check VPN Connection**: Ensure stable VPN access to internal servers
2. **Increase Timeout**: Set `LANGFUSE_V2_TIMEOUT` to higher values (180-300 seconds)
3. **Network Diagnostics**: Test connectivity to `langfuse-dev.qventus-int.com`
4. **Consider V3**: Use V3 for faster testing when possible

### CI/CD Performance

The CI/CD pipeline is configured with appropriate timeouts:

- **V2 Tests**: 180 seconds (3 minutes) to account for VPN latency
- **V3 Tests**: 60 seconds (1 minute) for direct cloud access
- **Unit Tests**: No timeout (run locally without network)

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Ensure all tests pass
6. Submit a pull request

### Development Guidelines

- Follow PEP 8 style guidelines
- Add type hints to all functions
- Write comprehensive docstrings
- Include unit tests for new features
- Test with both V2 and V3 when applicable

## Pull Request Process

The project uses a formalized pull request process to ensure code quality and maintainability. All pull requests must pass through comprehensive quality gates before merging.

### Quality Gates

1. **Required Reviewer Approval**: At least one approved review from designated code owners
2. **Code Formatting**: All code must be formatted with Black (100 character line length)
3. **Import Sorting**: Imports must be properly sorted with isort
4. **Linting**: Code must pass flake8 and pylint checks
5. **Type Checking**: All code must pass mypy type checking
6