log-similarity 1.0.0
Similarity-driven log clustering for error log analysis
License: MIT
Requires Python: >=3.7
Keywords: log, logging, clustering, similarity, error-detection, log-analysis, anomaly-detection, log-parsing

# Log Similarity SDK

A powerful Python SDK for similarity-driven log clustering and error log analysis. Automatically groups similar log messages into clusters and identifies anomalous logs that don't match existing patterns.

## Features

- **Intelligent Log Clustering**: Automatically groups similar logs based on semantic similarity
- **Multi-layer Preprocessing**: Advanced pattern normalization for universal log formats
- **Unmatched Log Detection**: Identifies error logs that don't fit any existing cluster
- **High Performance**: Efficient length-based indexing and similarity calculation
- **Flexible Configuration**: Customizable similarity thresholds and preprocessing rules

## Installation

```bash
pip install log-similarity
```

Or install from source:

```bash
git clone https://github.com/suiwenfeng/log-similarity-sdk.git
cd log-similarity-sdk
pip install -e .
```

## Quick Start

### 1. Build Clusters from Base Error Logs

```python
from log_similarity import SimilarityClusterer

# Sample base error logs
base_logs = [
    "User 12345 logged in from 192.168.1.100",
    "User 67890 logged in from 10.0.0.1",
    "Database connection timeout after 30 seconds",
    "Database connection timeout after 45 seconds",
    "GET /api/users returned 404",
    "GET /api/posts returned 500",
]

# Create and train clusterer
clusterer = SimilarityClusterer(similarity_threshold=0.85)
clusterer.fit(base_logs, verbose=True)

# View cluster summary
clusterer.print_summary()
```

### 2. Find Unmatched Error Logs

```python
# New error logs to check
new_logs = [
    "User 11111 logged in from 172.16.0.1",  # Similar to cluster 1
    "GET /api/products returned 404",         # Similar to cluster 3
    "Critical: Out of memory error",          # NEW - doesn't match any cluster
    "Fatal: Disk space exhausted",            # NEW - doesn't match any cluster
]

# Find logs that don't match any existing cluster
unmatched = clusterer.find_unmatched_logs(new_logs, verbose=True)

# Process unmatched logs
for log in unmatched:
    print(f"Unmatched: {log['original']}")
    print(f"Template:  {log['template']}")
    print(f"Similarity: {log['similarity']:.2f}\n")
```

## API Reference

### SimilarityClusterer

Main class for log clustering and analysis.

#### Constructor

```python
SimilarityClusterer(
    similarity_threshold: float = 0.90,
    merge_threshold: float = 0.95,
    length_tolerance: float = 0.3,
    preprocessor_config: Optional[PreprocessorConfig] = None
)
```

**Parameters:**
- `similarity_threshold`: Minimum similarity score to match a cluster (default: 0.90)
- `merge_threshold`: Threshold for merging logs (default: 0.95)
- `length_tolerance`: Token length tolerance for comparison (default: 0.3)
- `preprocessor_config`: Custom preprocessing configuration

#### Methods

##### `fit(log_messages, verbose=True)`

Build clusters from training logs.

**Parameters:**
- `log_messages` (List[str]): List of log messages to cluster
- `verbose` (bool): Print progress information

**Returns:**
- `self`: The clusterer instance (for method chaining)

**Example:**
```python
clusterer.fit(base_error_logs, verbose=True)
```

##### `predict(log_message)`

Predict which cluster a single log belongs to.

**Parameters:**
- `log_message` (str): Log message to predict

**Returns:**
- `dict`: Prediction result with keys:
  - `cluster_id`: Matched cluster ID or None
  - `template`: Log template
  - `similarity`: Similarity score
  - `match_type`: 'matched' or 'unmatched'
  - `original`: Original log message
  - `preprocessed`: Preprocessed log

**Example:**
```python
result = clusterer.predict("User 999 logged in from 10.0.0.5")
if result['match_type'] == 'matched':
    print(f"Matched cluster: {result['cluster_id']}")
else:
    print("No matching cluster found")
```

##### `predict_batch(log_messages, verbose=False)`

Predict clusters for multiple logs.

**Parameters:**
- `log_messages` (List[str]): List of log messages
- `verbose` (bool): Show progress

**Returns:**
- `List[dict]`: List of prediction results

##### `find_unmatched_logs(log_messages, verbose=False)`

Find logs that don't match any existing cluster.

**Parameters:**
- `log_messages` (List[str]): Logs to check
- `verbose` (bool): Show progress and statistics

**Returns:**
- `List[dict]`: List of unmatched logs with keys:
  - `original`: Original log message
  - `preprocessed`: Preprocessed log
  - `template`: Extracted template
  - `similarity`: Highest similarity score

**Example:**
```python
unmatched = clusterer.find_unmatched_logs(new_error_logs, verbose=True)
print(f"Found {len(unmatched)} new error patterns")
```

##### `get_all_clusters()`

Get all clusters with their details.

**Returns:**
- `List[dict]`: Sorted list of clusters (by frequency)

##### `save(filepath)`

Save clusterer state to JSON file.

**Parameters:**
- `filepath` (str): Path to save file

## Advanced Usage

### Custom Preprocessing

```python
from log_similarity import PreprocessorConfig, LogPreprocessor

# Create custom config
config = PreprocessorConfig()
# Modify patterns as needed
# config.dynamic_patterns.append((r'custom_pattern', '<CUSTOM>'))

clusterer = SimilarityClusterer(preprocessor_config=config)
```

### Adjusting Similarity Threshold

```python
# Stricter matching (higher threshold)
strict_clusterer = SimilarityClusterer(similarity_threshold=0.95)

# More lenient matching (lower threshold)
lenient_clusterer = SimilarityClusterer(similarity_threshold=0.80)
```

### Batch Processing

```python
# Process large log files
with open('error_logs.txt', 'r') as f:
    logs = f.readlines()

# Build clusters from first 10k logs
clusterer.fit(logs[:10000], verbose=True)

# Find unmatched in remaining logs
unmatched = clusterer.find_unmatched_logs(logs[10000:], verbose=True)

# Save results
with open('unmatched_errors.txt', 'w') as f:
    for log in unmatched:
        f.write(log['original'] + '\n')
```

## Use Cases

1. **Error Log Monitoring**: Detect new error patterns in production logs
2. **Log Deduplication**: Group similar logs to reduce noise
3. **Anomaly Detection**: Identify unusual log patterns
4. **Log Template Extraction**: Automatically extract log templates
5. **Log Analysis**: Understand log distribution and patterns

## Algorithm

The SDK uses a similarity-driven clustering algorithm with:

1. **Multi-layer Preprocessing**: Normalizes dynamic parts (IPs, timestamps, UUIDs, etc.)
2. **Template Extraction**: Identifies static and dynamic parts using Logram parser
3. **Similarity Calculation**: Combines Jaccard similarity, prefix matching, and length ratio
4. **Efficient Indexing**: Length-based indexing for fast cluster lookup

## Requirements

- Python >= 3.7
- No external dependencies (pure Python)

## License

MIT License

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Support

For issues and questions, please open an issue on GitHub.