smart-semantic-cache 0.1.1
A high-performance semantic caching system with FAISS vector similarity search for LLMs
License: MIT
Requires Python: >=3.8
Keywords: semantic, cache, faiss, vector, similarity, llm, ai, ml

# Smart Semantic Cache 🚀

A high-performance, multi-layered semantic caching system that dramatically reduces LLM costs and latency through intelligent similarity-based response caching.

## ✨ Why Semantic Cache?

Traditional caching requires **exact** query matches. Semantic caching understands that "What's the capital of France?" and "Tell me France's capital city" should return the same cached result. This can reduce your LLM API costs by 60-90% in real applications.

## 🎯 Key Features

- **🧠 Multi-Layer Intelligence**: Memory cache → SQLite → FAISS vector similarity
- **⚡ Lightning Fast**: Sub-millisecond memory lookups, <10ms semantic search
- **🎛️ Configurable Similarity**: Fine-tune cache hit sensitivity (0.0-1.0)
- **💾 Memory Efficient**: Optional FAISS quantization for large-scale deployments
- **🔄 Async Ready**: Full async support for high-throughput applications
- **📊 Rich Metrics**: Comprehensive performance monitoring and analytics
- **🧹 Smart Eviction**: LRU-based cache management with intelligent cleanup
- **🛡️ Production Ready**: Thread-safe, error-resilient, battle-tested

## 📦 Installation

```bash
pip install smart-semantic-cache
```

### Optional Dependencies

```bash
# For GPU acceleration (recommended for production)
pip install smart-semantic-cache[quantization]

# For development and testing
pip install smart-semantic-cache[dev]
```

## 🚀 Quick Start

### Method 1: Global Cache Setup (Recommended)

```python
from smart_semantic_cache import ensure_semantic_cache
from langchain_openai import OpenAI

# Initialize semantic cache globally - one line setup!
ensure_semantic_cache(
    similarity_threshold=0.15,  # 80% similarity required for cache hit
    max_cache_size=10000       # Store up to 10k semantic entries
)

# Now ALL your LLMs automatically use semantic caching
llm = OpenAI(temperature=0)

# These queries will hit the cache after the first one:
response1 = llm.invoke("What is the capital of France?")
response2 = llm.invoke("Tell me the capital city of France")  # Cache hit! 
response3 = llm.invoke("France's capital is?")                # Cache hit!
```

### Method 2: Direct Cache Usage

```python
from smart_semantic_cache import SemanticCache
from langchain.globals import set_llm_cache

# Create cache with custom configuration
cache = SemanticCache(
    database_path="./production_cache.db",
    faiss_index_path="./vector_cache",
    similarity_threshold=0.85,
    max_cache_size=50000,
    memory_cache_size=1000,
    enable_quantization=True  # Enable for large deployments
)

# Set as global cache
set_llm_cache(cache)
```

## ⚙️ Configuration Methods

### Global Configuration (Before First Use)

```python
from smart_semantic_cache import configure_semantic_cache

# Configure before any cache usage
configure_semantic_cache(
    database_path="./my_cache.db",
    similarity_threshold=0.85,
    max_cache_size=20000
)

# Later usage automatically uses these settings
from smart_semantic_cache import ensure_semantic_cache
ensure_semantic_cache()  # Uses config set above
```

### Runtime Configuration

```python
from smart_semantic_cache import ensure_semantic_cache

# Configure at runtime (if not already initialized)
cache = ensure_semantic_cache(
    # Similarity Settings
    similarity_threshold=0.8,      # Higher = more strict matching
    
    # Storage Settings  
    database_path="./cache.db",    # SQLite database location
    faiss_index_path="./vectors",  # FAISS index directory
    
    # Performance Settings
    max_cache_size=10000,          # Max semantic entries
    memory_cache_size=500,         # In-memory cache size
    batch_size=20,                 # Batch processing size
    
    # Advanced Settings
    enable_quantization=False      # Enable for memory efficiency
)
```

### Production Configuration

```python
from smart_semantic_cache import configure_semantic_cache

# High-throughput production setup
configure_semantic_cache(
    database_path="/var/cache/semantic/cache.db",
    faiss_index_path="/var/cache/semantic/vectors",
    similarity_threshold=0.85,     # Stricter for production
    max_cache_size=100000,         # Large semantic store
    memory_cache_size=2000,        # Generous memory cache
    enable_quantization=True,      # Memory optimization
    batch_size=50                  # Larger batches
)
```

## 🎛️ Cache Management

### Getting Cache Instance

```python
from smart_semantic_cache import get_semantic_cache

# Get current cache instance
cache = get_semantic_cache()

if cache:
    print("Cache is active and ready!")
else:
    print("No cache initialized yet")
```

### Resetting Cache

```python
from smart_semantic_cache import reset_semantic_cache

# Reset cache instance (allows reconfiguration)
reset_semantic_cache()

# Now you can configure differently
from smart_semantic_cache import configure_semantic_cache
configure_semantic_cache(similarity_threshold=0.9)
```

### Handling Already Initialized Cache

```python
from smart_semantic_cache import configure_semantic_cache

try:
    configure_semantic_cache(similarity_threshold=0.9)
except ValueError as e:
    print("Cache already initialized!")
    # Either reset or use current instance
    from smart_semantic_cache import reset_semantic_cache
    reset_semantic_cache()
    configure_semantic_cache(similarity_threshold=0.9)
```

## 📊 Performance Monitoring

### Real-time Metrics

```python
from smart_semantic_cache import get_semantic_cache

# Get current cache instance
cache = get_semantic_cache()

if cache:
    # Fetch comprehensive metrics
    metrics = cache.get_metrics()
    
    print(f"🎯 Cache Hit Rate: {metrics['hit_rate']:.1%}")
    print(f"📊 Total Requests: {metrics['total_requests']:,}")
    print(f"⚡ Memory Hits: {metrics['memory_hits']:,}")
    print(f"🧠 Semantic Hits: {metrics['semantic_hits']:,}")
    print(f"⏱️ Avg Embedding Time: {metrics['avg_embedding_time']:.3f}s")
    print(f"🔍 Avg Search Time: {metrics['avg_search_time']:.3f}s")
```

### Cache Cleanup

```python
from smart_semantic_cache import get_semantic_cache

cache = get_semantic_cache()
if cache:
    # Clear all caches (useful for testing)
    cache.clear_cache()
    print("All caches cleared!")
```

## 🏗️ Architecture Overview

```
Query → Memory Cache → SQLite Cache → Semantic Search → LLM API
  ↓         ↓             ↓              ↓            ↓
 <1ms    ~1-2ms        ~2-5ms        ~5-15ms      100-2000ms
```

### How It Works

1. **Memory Cache**: Lightning-fast LRU cache for recently accessed queries
2. **SQLite Cache**: Persistent exact-match cache with indexing
3. **Semantic Search**: FAISS-powered vector similarity search
4. **Embedding Cache**: Cached embeddings to avoid recomputation
5. **Smart Eviction**: Automatic cleanup based on usage patterns

## 🎛️ Advanced Usage

### Async Operations

```python
import asyncio
from smart_semantic_cache import ensure_semantic_cache
from langchain_openai import OpenAI

ensure_semantic_cache()

async def cached_queries():
    llm = OpenAI()
    
    # All async operations benefit from caching
    tasks = [
        llm.ainvoke("Explain quantum computing"),
        llm.ainvoke("What is quantum computing?"),  # Will cache hit
        llm.ainvoke("Define quantum computing")     # Will cache hit  
    ]
    
    results = await asyncio.gather(*tasks)
    return results

# Run async example
results = asyncio.run(cached_queries())
```

### Custom Similarity Thresholds

```python
from smart_semantic_cache import configure_semantic_cache

# Conservative caching (high precision)
configure_semantic_cache(similarity_threshold=0.9)

# Aggressive caching (high recall)  
configure_semantic_cache(similarity_threshold=0.6)

# Balanced (recommended for most use cases)
configure_semantic_cache(similarity_threshold=0.8)
```

### Multiple Cache Instances

```python
from smart_semantic_cache import SemanticCache

# Different caches for different domains
qa_cache = SemanticCache(
    database_path="./qa_cache.db",
    similarity_threshold=0.85
)

summarization_cache = SemanticCache(
    database_path="./summary_cache.db", 
    similarity_threshold=0.75  # More lenient for summaries
)
```

## 🔄 Complete Workflow Example

```python
from smart_semantic_cache import (
    configure_semantic_cache, 
    ensure_semantic_cache,
    get_semantic_cache,
    reset_semantic_cache
)
from langchain_openai import OpenAI

# Step 1: Configure before first use (recommended)
configure_semantic_cache(
    similarity_threshold=0.8,
    max_cache_size=50000,
    enable_quantization=True
)

# Step 2: Initialize cache (uses config from step 1)
cache = ensure_semantic_cache()

# Step 3: Use with any LLM
llm = OpenAI(temperature=0)
response = llm.invoke("What is machine learning?")

# Step 4: Monitor performance
metrics = cache.get_metrics()
print(f"Hit rate: {metrics['hit_rate']:.1%}")

# Step 5: Reset if needed (for testing/reconfiguration)
reset_semantic_cache()

# Step 6: Reconfigure with different settings
configure_semantic_cache(similarity_threshold=0.9)
```

## 📋 Configuration Reference

| Parameter | Default | Description |
|-----------|---------|-------------|
| `database_path` | `.langchain.db` | SQLite database file path |
| `faiss_index_path` | `./semantic_cache_index` | FAISS vector index directory |
| `similarity_threshold` | `0.5` | Semantic similarity threshold (0.0-1.0) |
| `max_cache_size` | `1000` | Maximum entries in vector store |
| `memory_cache_size` | `100` | Maximum entries in memory cache |
| `batch_size` | `10` | Batch size for vector operations |
| `enable_quantization` | `False` | Enable FAISS quantization for memory efficiency |

## 🔧 Troubleshooting

### Common Issues

**Cache not working?**
```python
from smart_semantic_cache import get_semantic_cache
cache = get_semantic_cache()
print(f"Cache active: {cache is not None}")
```

**Configuration errors?**
```python
from smart_semantic_cache import reset_semantic_cache, configure_semantic_cache

# Reset and reconfigure
reset_semantic_cache()
configure_semantic_cache(similarity_threshold=0.7)
```

**Low hit rates?**
```python
from smart_semantic_cache import reset_semantic_cache, configure_semantic_cache

# Try lowering similarity threshold
reset_semantic_cache()
configure_semantic_cache(similarity_threshold=0.7)
```

**Memory issues?**
```python
from smart_semantic_cache import configure_semantic_cache

# Enable quantization for large datasets
configure_semantic_cache(enable_quantization=True)
```

## 🚀 Performance Tips

1. **Start with 0.8 similarity threshold** - good balance of precision/recall
2. **Use `configure_semantic_cache()` for production** - cleaner setup
3. **Enable quantization for >10k cache entries** - significant memory savings
4. **Use larger memory cache in production** - reduces disk I/O
5. **Monitor hit rates** - adjust similarity threshold based on your data
6. **Reset cache in development** - prevents stale entries

## 🛠️ Requirements

- **Python**: 3.8+
- **Core**: FAISS, HuggingFace Transformers, SQLite
- **Optional**: faiss-gpu (for GPU acceleration)

## 📄 License

MIT License - see [LICENSE](LICENSE) file for details.

## 📈 Changelog

### v0.1.2 (Latest)
- 🐛 Fixed metadata configuration
- 📚 Updated documentation with all utility methods
- ⚡ Performance improvements
- 🎛️ Enhanced configuration options

### v0.1.1
- 🎯 Multi-layer caching system
- 🚀 FAISS integration with quantization
- 📊 Comprehensive metrics and monitoring
- 🔄 Full async support

---