semantic-comparer 0.1.8
Advanced text alignment and semantic containment analysis tool
Requires Python: >=3.12

# Semantic Comparer

Advanced text alignment and semantic containment analysis tool using modern Python practices.

## Features

- **Semantic Alignment**: Uses Smith-Waterman algorithm with sentence transformers for intelligent text comparison
- **Modern CLI**: Built with Typer and Rich for beautiful, user-friendly interface
- **Async Processing**: High-performance asynchronous operations
- **File Support**: Direct text input or file-based processing
- **Rich Output**: Colorized, formatted results with detailed statistics
- **Type Safety**: Full type annotations and modern Python practices

## Usage

### Install

```bash
pip install semantic-comparer
```

### Basic Comparison

```bash
# Compare two texts directly
semantic-comparer compare "This is the first text." "This is the second text."

# Compare with custom parameters
semantic-comparer compare \
  "First text content" \
  "Second text content" \
  --model paraphrase-multilingual-MiniLM-L12-v2 \
  --gap-penalty 0.3 \
  --similarity-threshold 0.5
```

### File-based Comparison

```bash
# Compare text files (prefix with @)
semantic-comparer compare @file1.txt @file2.txt

# Mix direct text and file
semantic-comparer compare "Direct text" @file.txt
```

### Output Options

```bash
# Quiet mode (summary only)
semantic-comparer compare text1 text2 --quiet

# Save detailed results to JSON
semantic-comparer compare text1 text2 --output results.json
```

## Command Line Options

| Option | Short | Description | Default |
|--------|-------|-------------|---------|
| `--model` | `-m` | Sentence transformer model | `paraphrase-multilingual-MiniLM-L12-v2` |
| `--gap-penalty` | `-g` | Penalty for gaps (0.0-1.0) | `0.3` |
| `--similarity-threshold` | `-t` | Minimum similarity for matches (0.0-1.0) | `0.5` |
| `--output` | `-o` | Output file for JSON results | None |
| `--quiet` | `-q` | Suppress detailed output | False |

## Understanding the Results

### Alignment Types

- **✓ Match**: Paragraphs that are semantically similar
- **⚠ Only in A/B**: Paragraphs present in one text but not the other
- **✗ Unaligned**: Paragraphs that couldn't be matched

### Containment Score

The semantic containment score measures how much of text A's semantic content is found in text B:

- **0.0-0.4**: Low similarity (Red)
- **0.4-0.7**: Moderate similarity (Yellow)  
- **0.7-1.0**: High similarity (Green)

## Advanced Usage

### Custom Models

```bash
# Use a different sentence transformer model
semantic-comparer compare text1 text2 --model all-mpnet-base-v2
```

### Fine-tuning Parameters

```bash
# Stricter matching (higher threshold)
semantic-comparer compare text1 text2 --similarity-threshold 0.8

# More lenient gap handling (lower penalty)
semantic-comparer compare text1 text2 --gap-penalty 0.1
```

## Technical Details

### Algorithm

The tool uses the Smith-Waterman algorithm adapted for semantic similarity:

1. **Text Segmentation**: Split texts into paragraphs
2. **Embedding Generation**: Convert paragraphs to semantic vectors
3. **Similarity Calculation**: Compute cosine similarity between vectors
4. **Dynamic Programming**: Apply Smith-Waterman for optimal alignment
5. **Score Calculation**: Weighted containment score based on matches

### Performance

- **Async Processing**: Non-blocking I/O operations
- **Memory Efficient**: Streaming file processing for large texts
- **Progress Tracking**: Real-time progress indicators
- **Error Handling**: Robust error handling with user-friendly messages

### Security

- **Input Validation**: Comprehensive parameter validation
- **File Safety**: Secure file operations with size limits
- **Text Sanitization**: Removal of problematic characters
- **Error Isolation**: Graceful error handling without data exposure

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes with proper type annotations
4. Add tests for new functionality
5. Ensure code passes linting and type checking
6. Submit a pull request

## License

MIT License - see LICENSE file for details.