storage-bucket-s2 0.1.3
High-performance encrypted file uploader for Storage Buckets
License: MIT
Requires Python: >=3.8
Keywords: upload, storage, bucket, encryption, aes, parallel

<p align="center">
  <h1 align="center">storage-bucket-s2</h1>
  <p align="center">
    <strong>High-performance encrypted file uploader for Storage Buckets</strong>
  </p>
  <p align="center">
    <a href="#installation">Installation</a> •
    <a href="#quick-start">Quick Start</a> •
    <a href="#usage-guide">Usage Guide</a> •
    <a href="#api-reference">API Reference</a>
  </p>
</p>

---

**storage-bucket-s2** uploads files to Storage Buckets at maximum speed using a compiled C engine. Files are automatically split into chunks, encrypted with **AES-256-CBC**, and uploaded in parallel — all from a simple 3-line Python API.

### Why?

| | storage-bucket-s2 | Pure Python |
|---|---|---|
| **Speed** | C-native multi-threaded I/O | GIL-limited |
| **Encryption** | OpenSSL AES-256-CBC (hardware-accelerated) | Software AES |
| **Memory** | Streaming chunks, never loads full file | Same |
| **Reliability** | Exponential backoff, per-chunk retry | Manual |
| **Cleanup** | Crash-safe temp file cleanup + orphan detection | None |

---

## Installation

```bash
pip install storage-bucket-s2
```

The C engine compiles automatically during install. If system dependencies are missing, you'll get a clear message:

<details>
<summary><strong>Ubuntu / Debian</strong></summary>

```bash
sudo apt install -y gcc make libcurl4-openssl-dev libssl-dev
pip install storage-bucket-s2
```
</details>

<details>
<summary><strong>Fedora / RHEL / CentOS / Rocky</strong></summary>

```bash
sudo dnf install -y gcc make libcurl-devel openssl-devel
pip install storage-bucket-s2
```
</details>

<details>
<summary><strong>Arch / Manjaro</strong></summary>

```bash
sudo pacman -S gcc make curl openssl
pip install storage-bucket-s2
```
</details>

<details>
<summary><strong>Alpine</strong></summary>

```bash
apk add gcc make musl-dev curl-dev openssl-dev
pip install storage-bucket-s2
```
</details>

<details>
<summary><strong>macOS</strong></summary>

```bash
brew install curl openssl
pip install storage-bucket-s2
```
</details>

---

## Quick Start

```python
from storage_bucket_s2 import StorageClient

client = StorageClient("https://your-server.com")
bucket = client.bucket("my-bucket-id", token="bkt_xxxx")
result = bucket.upload("video.mp4")

print(f"✓ Uploaded {result.file_id} ({result.size_human}) in {result.duration_human}")
# ✓ Uploaded abc123 (1.4 GB) in 2m 15s
```

That's it. The C engine handles chunking, encryption, parallel upload, retry, and cleanup.

---

## Usage Guide

### Basic Upload

```python
from storage_bucket_s2 import StorageClient

client = StorageClient("https://your-server.com")
bucket = client.bucket("bucket-id", token="bkt_your_upload_token")

result = bucket.upload("/path/to/file.mp4")
print(result.file_id)       # "abc123-def456"
print(result.success)       # True
print(result.size_human)    # "1.4 GB"
print(result.duration_human)# "2m 15s"
print(result.speed_human)   # "10.6 MB/s"
```

### Upload with Live Progress

```python
def on_progress(p):
    print(
        f"\r{p.percent:5.1f}% | {p.speed_human:>12} | "
        f"ETA {p.eta_human:>8} | "
        f"{p.completed_chunks}/{p.total_chunks} chunks",
        end="", flush=True
    )

result = bucket.upload("big_file.tar.gz", on_progress=on_progress)
print(f"\nDone! {result.file_id}")
```

Output:
```
 45.2% |   52.4 MB/s |  ETA  1m 12s | 18/40 chunks
```

### Parallel Workers

Control how many chunks upload simultaneously (default: 4, max: 16):

```python
# Fast connection — use more workers
result = bucket.upload("file.iso", workers=8)

# Slow/metered connection — use fewer
result = bucket.upload("file.zip", workers=2)
```

### Error Handling

```python
from storage_bucket_s2 import StorageClient
from storage_bucket_s2.exceptions import (
    AuthenticationError,
    QuotaExceededError,
    UploadError,
    EngineError,
)

client = StorageClient("https://your-server.com")
bucket = client.bucket("my-bucket", token="bkt_xxxx")

try:
    result = bucket.upload("file.mp4")
    print(f"✓ {result.file_id}")

except AuthenticationError:
    print("✗ Invalid or expired token")

except QuotaExceededError:
    print("✗ Bucket is full — free up space or upgrade")

except UploadError as e:
    print(f"✗ Upload failed: {e}")
    print(f"  Failed chunks: {e.failed_chunks}")
    for err in e.errors:
        print(f"  - {err}")

except EngineError as e:
    print(f"✗ C engine issue: {e}")

except FileNotFoundError:
    print("✗ File not found")
```

### List Files in a Bucket

```python
files = bucket.list_files(path="/", limit=50)
for f in files:
    print(f"{f['name']}  ({f.get('size', '?')} bytes)")
```

### Get File Info

```python
file_info = bucket.get_file("file-id-123")
print(file_info)
```

### Server Health Check

```python
client = StorageClient("https://your-server.com")
if client.health():
    print("Server is up")
else:
    print("Server is down")
```

---

## API Reference

### `StorageClient(base_url, timeout=10)`

Create a client connected to your storage server.

| Parameter | Type | Description |
|-----------|------|-------------|
| `base_url` | `str` | Server URL (e.g. `"https://your-server.com"`) |
| `timeout` | `int` | HTTP timeout for health checks (seconds) |

**Methods:**

| Method | Returns | Description |
|--------|---------|-------------|
| `bucket(bucket_id, token)` | `Bucket` | Get a bucket handle |
| `health()` | `bool` | Test server connectivity |

---

### `Bucket`

A handle to a specific bucket. Created via `client.bucket()`.

#### `bucket.upload(file_path, path="/", workers=4, on_progress=None)`

Upload a file to the bucket via the C engine.

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `file_path` | `str` | *required* | Path to file to upload |
| `path` | `str` | `"/"` | Virtual path in bucket |
| `workers` | `int` | `4` | Parallel upload workers (1–16) |
| `on_progress` | `callable` | `None` | Progress callback, receives `UploadProgress` |

**Returns:** `UploadResult`

#### `bucket.list_files(path="/", limit=100)`

List files in the bucket. Returns a list of file dicts.

#### `bucket.get_file(file_id)`

Get info for a specific file. Returns a file dict.

---

### `UploadResult`

Returned by `bucket.upload()`.

| Field | Type | Description |
|-------|------|-------------|
| `file_id` | `str` | Server-assigned file ID |
| `file_name` | `str` | Original filename |
| `total_chunks` | `int` | Number of chunks uploaded |
| `uploaded_bytes` | `int` | Total bytes uploaded |
| `total_bytes` | `int` | Original file size |
| `duration_seconds` | `float` | Upload duration |
| `speed_bps` | `int` | Average speed (bytes/sec) |
| `success` | `bool` | Whether upload succeeded |
| `errors` | `list[str]` | Error messages (if any) |
| `size_human` | `str` | e.g. `"1.4 GB"` |
| `speed_human` | `str` | e.g. `"52.4 MB/s"` |
| `duration_human` | `str` | e.g. `"2m 15s"` |

---

### `UploadProgress`

Passed to the `on_progress` callback during upload.

| Field | Type | Description |
|-------|------|-------------|
| `state` | `str` | `"initializing"`, `"encrypting"`, `"uploading"`, `"completed"`, `"failed"` |
| `percent` | `float` | Upload progress (0–100) |
| `completed_chunks` | `int` | Chunks successfully uploaded |
| `total_chunks` | `int` | Total chunks |
| `failed_chunks` | `int` | Chunks that failed |
| `uploaded_bytes` | `int` | Bytes uploaded so far |
| `total_bytes` | `int` | Total file size |
| `speed_bps` | `int` | Current speed (bytes/sec) |
| `eta_seconds` | `float` | Estimated time remaining |
| `elapsed_seconds` | `float` | Time elapsed |
| `errors` | `list[str]` | Errors so far |
| `speed_human` | `str` | e.g. `"52.4 MB/s"` |
| `eta_human` | `str` | e.g. `"1m 23s"` |

---

### Exceptions

All exceptions inherit from `StorageBucketError`.

| Exception | When |
|-----------|------|
| `StorageBucketError` | Base — catches all library errors |
| `AuthenticationError` | Invalid or expired token |
| `QuotaExceededError` | Bucket storage is full |
| `UploadError` | Upload failed (has `.errors` and `.failed_chunks`) |
| `ServerError` | Server returned 5xx (has `.status_code`) |
| `EngineError` | C engine binary not found or crashed |

---

## How It Works

```
pip install storage-bucket-s2
         │
         ▼
┌─────────────────────────────────┐
│  Build Backend (_build_backend) │
│  1. Detect OS & dependencies    │
│  2. Compile C engine (make)     │
│  3. Bundle binary in package    │
└─────────────────────────────────┘

bucket.upload("file.mp4")
         │
         ▼
┌─────────────────────────────────┐
│  Python (engine.py)             │
│  1. Write config.json           │
│  2. Spawn C engine subprocess   │
│  3. Poll status.json            │
│  4. Return UploadResult         │
└─────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────┐
│  C Engine (bucket_uploader_v2)  │
│  1. Read config.json            │
│  2. POST /upload/init           │
│  3. Split file → encrypt chunks │
│  4. PUT chunks in parallel      │
│  5. POST /upload/complete       │
│  6. Write status.json (live)    │
│  7. Cleanup temp files          │
└─────────────────────────────────┘
```

### Security

- **AES-256-CBC** encryption per chunk with unique keys
- Keys are base64-decoded, used, then securely zeroed (`OPENSSL_cleanse`)
- Encrypted temp files deleted immediately after upload
- All communication over HTTPS

### Reliability

- Exponential backoff on server errors (500, 429)
- Per-chunk retry (up to 5 attempts)
- Connection timeout + stall detection
- Graceful shutdown on SIGINT/SIGTERM
- Orphaned temp file cleanup on startup

---

## License

MIT