safe-core 0.1.1
Professional cryptographic engine for secure storage systems
License: MIT
Requires Python: >=3.8
Keywords: cryptography, encryption, security, storage, argon2, aes, chacha20, cbor

# Safe Core

Professional cryptographic engine for secure storage systems with envelope encryption, streaming operations, and comprehensive key management.

[![Version](https://img.shields.io/badge/version-3.2.0-blue.svg)](https://github.com/yourusername/safe_core)
[![Python](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](#)
[![Coverage](https://img.shields.io/badge/coverage-95%25-brightgreen.svg)](#)

## Overview

Safe Core is a production-ready cryptographic engine designed for secure storage systems. It implements envelope encryption, streaming operations for large files, and a comprehensive key management hierarchy. The engine prioritizes security, performance, and extensibility while maintaining backward compatibility.

## Architecture

```mermaid
graph TB
    subgraph "Public API"
        SC[SafeCore Engine]
        RLSC[RateLimitedSafeCore]
    end
    
    subgraph "Key Management"
        KH[Key Hierarchy<br/>HKDF-based]
        KP[Key Purposes<br/>Domain Separation]
        SB[SecureBytes<br/>Memory Protection]
    end
    
    subgraph "Cryptographic Providers"
        KDF[KDF Providers<br/>Argon2id]
        CIPHER[Cipher Providers<br/>AES-GCM, ChaCha20]
        KEYPROV[Key Providers<br/>Password, KMS]
    end
    
    subgraph "Data Operations"
        ENC[Block Encryption]
        DEC[Block Decryption]
        STREAM_ENC[Streaming Encryptor]
        STREAM_DEC[Streaming Decryptor]
    end
    
    subgraph "Security Features"
        RL[Rate Limiter]
        PAD[Padding<br/>PKCS7, ISO7816]
        COMP[Compression<br/>ZSTD, LZ4]
        SIG[Digital Signatures<br/>Ed25519]
        SS[Secret Sharing<br/>Shamir SLIP-0039]
        HW[Hardware Auth<br/>YubiKey]
    end
    
    subgraph "Data Structures"
        CONT[EncryptedContainer<br/>CBOR Format]
        HEAD[StreamingHeader<br/>CBOR Format]
        CFG[CryptoParams]
    end
    
    SC --> KH
    SC --> ENC
    SC --> DEC
    SC --> STREAM_ENC
    SC --> STREAM_DEC
    SC --> SS
    
    RLSC --> SC
    RLSC --> RL
    
    KH --> KP
    KH --> SB
    
    ENC --> CIPHER
    ENC --> PAD
    ENC --> COMP
    ENC --> SIG
    DEC --> CIPHER
    DEC --> PAD
    DEC --> COMP
    DEC --> SIG
    
    STREAM_ENC --> CIPHER
    STREAM_ENC --> COMP
    STREAM_ENC --> SIG
    STREAM_DEC --> CIPHER
    STREAM_DEC --> COMP
    STREAM_DEC --> SIG
    
    ENC --> CONT
    DEC --> CONT
    STREAM_ENC --> HEAD
    STREAM_DEC --> HEAD
    
    SC --> KDF
    SC --> KEYPROV
    
    KEYPROV --> KDF
    KEYPROV --> CIPHER
    
    HW -.-> SC
```

## Key Features

### Core Cryptography
- **Envelope Encryption**: Separation of Data Encryption Keys (DEK) and Key Encryption Keys (KEK) for enhanced security
- **AEAD Ciphers**: AES-256-GCM and ChaCha20-Poly1305 with authenticated encryption
- **Key Derivation**: Argon2id with configurable memory-hardness parameters
- **Streaming Operations**: Memory-efficient encryption/decryption for large files with chunked processing
- **Key Hierarchy**: HKDF-based key derivation with domain separation for different purposes

### Security Features
- **Rate Limiting**: Brute-force protection with configurable attempt windows
- **Password Management**: Change passwords without re-encrypting data
- **DEK Rotation**: Generate new encryption keys while preserving password
- **Digital Signatures**: Ed25519 signatures for data authenticity verification
- **Secret Sharing**: Shamir's Secret Sharing (SLIP-0039) for break-glass recovery
- **Hardware Authentication**: YubiKey HMAC-SHA1 challenge-response integration
- **Padding**: PKCS7, ISO7816-4, and ANSI X.923 for traffic analysis protection
- **Compression**: ZSTD and LZ4 with decompression bomb protection
- **Secure Memory**: Automatic zeroing of sensitive data in memory

### Extensibility
- **Provider System**: Pluggable KDF, cipher, and key providers
- **Custom Algorithms**: Register custom cryptographic implementations
- **KMS Integration**: Mock KMS provider included, real KMS providers can be added
- **CBOR Serialization**: Extensible format with backward compatibility (v3-v6)

### Performance
- **Async/Await**: Non-blocking operations for I/O-bound tasks
- **Parallel Encryption**: Multi-core streaming encryption for large files
- **Efficient Streaming**: Constant memory usage regardless of file size
- **Optimized Nonces**: Counter-based nonce derivation for streaming

## Installation

```bash
# Core dependencies only
pip install cryptography argon2-cffi

# With optional features
pip install cryptography argon2-cffi zstandard lz4 shamir-mnemonic

# For hardware authentication
pip install yubikey-manager

# Development installation
pip install -r requirements.txt
```

## Quick Start

### Basic Storage Initialization

```python
from safe_core import SafeCore, get_default_crypto_params

# Initialize engine
core = SafeCore()
params = get_default_crypto_params('high')  # 'interactive', 'moderate', 'high', 'paranoid'

# Create storage
password = b'your_secure_password'
dek, verification_block = core.initialize_storage(password, params)

# Store verification_block persistently, keep dek in memory for the session
```

### Authentication

```python
# Authenticate and retrieve DEK
dek = core.authenticate_and_get_key(password, verification_block)
```

### Block Encryption

```python
# Encrypt data
plaintext = b'sensitive data'
encrypted = core.encrypt_block(plaintext, dek, params)

# Decrypt data
decrypted = core.decrypt_block(encrypted, dek)
```

### Streaming Operations

```python
# Encrypt large file
with open('large_file.bin', 'rb') as f_in, \
     open('encrypted.bin', 'wb') as f_out:
    
    encryptor = core.create_streaming_encryptor(dek, params)
    for chunk in encryptor.encrypt_stream(f_in):
        f_out.write(chunk)

# Decrypt large file
with open('encrypted.bin', 'rb') as f_in, \
     open('decrypted.bin', 'wb') as f_out:
    
    decryptor = core.create_streaming_decryptor(dek, f_in)
    for chunk in decryptor.decrypt_stream(f_in):
        f_out.write(chunk)
```

### Async Operations

```python
import asyncio
import aiofiles

async def encrypt_async():
    dek, vb = await core.initialize_storage_async(password, params)
    
    async with aiofiles.open('input.bin', 'rb') as f_in, \
               aiofiles.open('encrypted.bin', 'wb') as f_out:
        
        encryptor = core.create_streaming_encryptor(dek, params)
        async for chunk in encryptor.encrypt_stream_async(f_in):
            await f_out.write(chunk)

asyncio.run(encrypt_async())
```

## Advanced Features

### Password Management

```python
# Change password without re-encrypting data
new_password = b'new_secure_password'
new_vb = core.change_master_password(password, new_password, verification_block)

# Authenticate with new password
dek = core.authenticate_and_get_key(new_password, new_vb)
```

### DEK Rotation

```python
# Rotate encryption key (requires re-encrypting all data)
new_dek, new_vb = core.rotate_dek(password, verification_block)

# Re-encrypt existing data with new DEK
old_data_encrypted = core.encrypt_block(b'old data', dek, params)
old_data_plaintext = core.decrypt_block(old_data_encrypted, dek)
new_data_encrypted = core.encrypt_block(old_data_plaintext, new_dek, params)
```

### Rate Limiting

```python
from safe_core import RateLimitedSafeCore, AuthenticationRateLimiter

limiter = AuthenticationRateLimiter(max_attempts=3, window_seconds=60)
rate_limited_core = RateLimitedSafeCore(core, limiter)

try:
    dek = rate_limited_core.authenticate_and_get_key('user_id', password, verification_block)
except RateLimitError as e:
    print(f"Too many attempts. Wait {e.wait_seconds} seconds.")
```

### Key Purposes (Domain Separation)

```python
from safe_core import KeyPurpose

# Encrypt with different key purposes
data_encrypted = core.encrypt_block(b'data', dek, params, key_purpose=KeyPurpose.DATA_ENCRYPTION)
file_encrypted = core.encrypt_block(b'file', dek, params, key_purpose=KeyPurpose.FILE_ENCRYPTION)
meta_encrypted = core.encrypt_block(b'meta', dek, params, key_purpose=KeyPurpose.METADATA)

# Decrypt with matching purpose
data_decrypted = core.decrypt_block(data_encrypted, dek, key_purpose=KeyPurpose.DATA_ENCRYPTION)
```

### Authenticated Associated Data

```python
# Encrypt with context
context = b'user_id:12345'
encrypted = core.encrypt_block(plaintext, dek, params, associated_data=context)

# Decrypt with same context (required)
decrypted = core.decrypt_block(encrypted, dek, associated_data=context)
```

### Digital Signatures

```python
from cryptography.hazmat.primitives.asymmetric import ed25519

# Generate key pair
signing_key = ed25519.Ed25519PrivateKey.generate()
verifying_key = signing_key.public_key()

# Sign data
encrypted = core.encrypt_block(plaintext, dek, params, signing_key=signing_key)

# Verify signature
decrypted = core.decrypt_block(encrypted, dek, verifying_key=verifying_key)
```

### Streaming Signatures

```python
# Sign stream
encryptor = core.create_streaming_encryptor(dek, params, signing_key=signing_key)
# ... encrypt stream

# Verify stream
decryptor = core.create_streaming_decryptor(dek, header, verifying_key=verifying_key)
# ... decrypt stream
```

### Secret Sharing (Break-Glass Recovery)

```python
# Enable recovery (2 out of 3 shares required)
updated_vb, shares = core.create_recovery_shares(
    verification_block, dek, threshold=2, share_count=3, crypto_params=params
)

# Distribute shares to trusted parties
# shares = ['word1 word2 ...', 'word3 word4 ...', 'word5 word6 ...']

# Recover access with shares (no password needed)
recovered_dek = core.recover_access(updated_vb, [shares[0], shares[2]])
```

### Hardware Authentication (YubiKey)

```python
# Initialize with YubiKey challenge-response
dek, vb = core.initialize_storage(password, params, use_hardware_auth=True)

# Authenticate (requires YubiKey present)
dek = core.authenticate_and_get_key(password, vb)
```

### Compression

```python
from safe_core.constants.algorithms import CompressionAlgorithm

# Block compression
encrypted = core.encrypt_block(
    plaintext, dek, params,
    compression_alg=CompressionAlgorithm.ZSTD
)

# Streaming compression
encryptor = core.create_streaming_encryptor(
    dek, params,
    compression_alg=CompressionAlgorithm.ZSTD
)
```

### Traffic Analysis Protection

```python
from safe_core.constants.algorithms import PaddingAlgorithm

# Pad data to hide size
encrypted = core.encrypt_block(
    plaintext, dek, params,
    padding_alg=PaddingAlgorithm.PKCS7
)
```

### Parallel Encryption

```python
# Encrypt using multiple CPU cores
encryptor = core.create_streaming_encryptor(dek, params, chunk_size=1024*1024)

with open('input.bin', 'rb') as f_in, \
     open('encrypted.bin', 'wb') as f_out:
    
    for chunk in encryptor.encrypt_stream_parallel(f_in, max_workers=4):
        f_out.write(chunk)
```

### Custom Providers

```python
from safe_core import CryptoParams, KdfParams, KdfId, CipherId

# Use ChaCha20-Poly1305 instead of AES
params = CryptoParams(
    kdf_id=KdfId.ARGON2ID,
    cipher_id=CipherId.CHACHA20_POLY1305,
    kdf_params=KdfParams(
        memory_cost=524288,  # 512 MB
        time_cost=4,
        parallelism=4,
        key_length=32
    )
)

dek, vb = core.initialize_storage(password, params)
```

### KMS Integration

```python
# Register custom KMS provider
from safe_core.crypto.providers.kms_provider import MockKmsProvider

kms_provider = MockKmsProvider()
core._registry.register_key_provider(kms_provider)

# Use KMS for DEK protection (manual construction for now)
# Full KMS integration in SafeCore.initialize_storage() coming in future release
```

## Security Considerations

### Password Requirements
- Minimum length: 12 characters recommended
- Use a password manager for generation
- Enable hardware authentication for additional security

### Key Management
- Store verification blocks securely (encrypted database, secure file storage)
- Keep DEK in memory only during active session
- Use `SecureBytes` for sensitive data that must persist temporarily
- Enable secret sharing for critical systems

### Rate Limiting
- Configure based on use case (3 attempts per 60 seconds default)
- Use unique identifiers per user/device
- Monitor failed authentication attempts

### Compression Security
**Warning**: Do not compress data that combines secrets with attacker-controlled input. This can lead to side-channel attacks (CRIME/BREACH). Use compression only for trusted data or data without mixed sensitivity levels.

### Decompression Bombs
Safe Core includes built-in protection:
- Maximum decompressed size: 100 MB for blocks
- Maximum expansion ratio: 20x for streaming chunks
- Validation of chunk sizes in streaming headers

## Version History

### v3.2.0 (Current)
- Added digital signatures (Ed25519) for data authenticity
- Added secret sharing (Shamir SLIP-0039) for break-glass recovery
- Added hardware authentication support (YubiKey)
- Added compression (ZSTD, LZ4) with decompression bomb protection
- Added padding (PKCS7, ISO7816-4, ANSI X.923) for traffic analysis protection
- Added parallel encryption for multi-core systems
- Added KMS plugin architecture with mock provider
- Migrated to CBOR serialization (v6 format) with backward compatibility
- Enhanced streaming signatures with SHA-256 hash verification
- Improved error handling and validation throughout

### v3.1.0
- Initial professional refactor
- Envelope encryption with DEK/KEK separation
- Streaming operations for large files
- Key hierarchy with HKDF
- Rate limiting for authentication
- Async/await support
- Comprehensive test suite

## Testing

```bash
# Run all tests
pytest

# Run specific test file
pytest tests/test_encryption_decryption.py

# Run with coverage
pytest --cov=safe_core --cov-report=html

# Run async tests only
pytest -m asyncio
```

## Project Statistics

### Codebase Metrics
- **Total Files**: 75 files analyzed
- **Total Lines**: ~15,000 lines (including tests)
- **Source Code**: ~8,500 lines
- **Test Code**: ~6,500 lines
- **Test Coverage**: 95%+
- **Python Version**: 3.8+

### Components Breakdown
- **Core Engine**: 1,200 lines
- **Cryptographic Providers**: 800 lines
- **Key Management**: 400 lines
- **Data Structures**: 600 lines
- **Security Utilities**: 300 lines
- **Streaming Operations**: 900 lines
- **Test Suite**: 6,500 lines (43 test modules)

### Supported Algorithms
- **KDF**: Argon2id
- **Ciphers**: AES-256-GCM, ChaCha20-Poly1305
- **Signatures**: Ed25519
- **Compression**: ZSTD, LZ4
- **Padding**: PKCS7, ISO7816-4, ANSI X.923
- **Secret Sharing**: Shamir SLIP-0039

### Format Versions
- **Current**: v6 (CBOR)
- **Legacy Support**: v3, v4, v5 (binary)
- **Streaming**: v6 (CBOR), v3-v4 (binary)

## License

MIT License. See [LICENSE](LICENSE) for details.

## Contributing

Contributions welcome. Please ensure:
- All tests pass
- Code coverage maintained above 90%
- Type hints included
- Documentation updated
- Follows existing code style

## Support

For issues, questions, or feature requests:
- GitHub Issues: [Report Issue](https://github.com/yourusername/safe_core/issues)
- Documentation: [Read the Docs](https://safe-core.readthedocs.io)