magenta-telemetry 0.1.0
Ultra-lightweight binary telemetry serialization format
License: MIT
Requires Python: >=3.8

# Magenta Telemetry Format

**Ultra-lightweight binary telemetry serialization for low-resource African SME environments**

## Overview

Magenta Telemetry Format is a highly optimized binary serialization format designed for IoT and telemetry data transmission in bandwidth-constrained environments. It achieves **50-80% size reduction** compared to JSON while maintaining cross-platform compatibility and schema evolution.

### Key Features

- **Compact Binary Encoding**: Custom wire format optimized for telemetry data
- **LZ4 Compression**: Optional fast compression for additional size reduction
- **Delta Encoding**: Only transmit changed fields to minimize bandwidth
- **Multi-Platform**: Rust core, WASM for browsers, Python bindings
- **Offline Relay**: Support for LAN proxying through neighbor devices
- **Time-Series Ready**: Preserves hardware metrics and temporal patterns
- **Fast**: < 1ms encode/decode for typical telemetry packets

## Architecture

```
┌─────────────────┐
│ Magenta Agent   │
│    (Rust)       │
└────────┬────────┘
         │ Collect telemetry
         ▼
┌─────────────────┐
│  MTF Encoder    │
│   (Rust core)   │
└────────┬────────┘
         │ Compressed bytes
         ▼
┌─────────────────────────────┐
│  Transport: QUIC/HTTP       │
│  ┌─────────────────────┐    │
│  │ Direct → Backend    │    │
│  │   OR                │    │
│  │ LAN Proxy → Backend │    │
│  └─────────────────────┘    │
└─────────────────────────────┘
         │
         ▼
┌─────────────────┐     ┌──────────────────┐
│  Backend        │     │  Dashboard       │
│  (Python/Rust)  │     │  (WASM/Browser)  │
│  Decode → JSON  │     │  Live Decode     │
└─────────────────┘     └──────────────────┘
```

## Quick Start

### Installation

```bash
# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Build all components
cargo build --release

# Run tests
cargo test --all

# Run benchmarks
cargo bench
```

### Using the CLI

```bash
# Encode JSON to MTF binary
magenta-cli encode input.json -o output.mtf

# Decode MTF binary to JSON
magenta-cli decode output.mtf -o output.json

# Inspect MTF file
magenta-cli inspect output.mtf

# Stress test
magenta-cli stress-test --count 10000 --delta
```

### Rust API

```rust
use magenta_telemetry_core::{TelemetryData, encode, decode, CompressionType};

// Create telemetry data
let telemetry = TelemetryData {
    device_id: "device-001".to_string(),
    timestamp: 1735686690000,
    cpu_usage_percent: 45.2,
    memory_used: 8589934592, // 8GB
    // ... other fields
};

// Encode with LZ4 compression
let bytes = encode(&telemetry, CompressionType::Lz4)?;

// Decode
let decoded = decode(&bytes)?;
```

### Python API

```python
from magenta_telemetry import TelemetryData, encode, decode

# Create telemetry
telemetry = TelemetryData(
    device_id="device-001",
    timestamp=1735686690000,
    cpu_usage_percent=45.2,
    memory_used=8589934592,
)

# Encode
data = encode(telemetry, compression="lz4")

# Decode
decoded = decode(data)
```

### WASM (Browser/Node.js)

```javascript
import { TelemetryData, encode, decode } from 'magenta-telemetry-wasm';

// Create telemetry
const telemetry = {
  deviceId: 'device-001',
  timestamp: 1735686690000,
  cpuUsagePercent: 45.2,
  memoryUsed: 8589934592,
};

// Encode
const bytes = encode(telemetry, 'lz4');

// Decode
const decoded = decode(bytes);
```

## Format Specification

See [FORMAT_SPEC.md](./FORMAT_SPEC.md) for detailed binary format documentation.

### Packet Structure Overview

```
┌──────────────────────────────────────────────────────────┐
│ HEADER (15 bytes fixed)                                  │
├──────────────────────────────────────────────────────────┤
│ Version       │ 1 byte  │ Format version (0x01)         │
│ Flags         │ 1 byte  │ Compression, delta, proxy     │
│ Timestamp     │ 8 bytes │ Unix timestamp (milliseconds) │
│ Sequence      │ 4 bytes │ Sequence number               │
│ Device ID Len │ 1 byte  │ Length of device ID (0-255)   │
├──────────────────────────────────────────────────────────┤
│ VARIABLE DATA                                            │
├──────────────────────────────────────────────────────────┤
│ Device ID     │ N bytes │ UTF-8 device identifier       │
│ UUID          │ 16 bytes│ 128-bit UUID                  │
│ Payload       │ M bytes │ Encoded telemetry data        │
│ Checksum      │ 4 bytes │ CRC32 checksum                │
└──────────────────────────────────────────────────────────┘
```

## Benchmarks

Preliminary results show significant improvements over standard formats:

| Format      | Size (bytes) | Encode (μs) | Decode (μs) | Reduction |
|-------------|--------------|-------------|-------------|-----------|
| JSON        | 1,247        | 125         | 95          | baseline  |
| JSON Lines  | 1,198        | 118         | 92          | 3.9%      |
| MessagePack | 687          | 45          | 52          | 44.9%     |
| CBOR        | 712          | 48          | 55          | 42.9%     |
| **MTF**     | **312**      | **18**      | **22**      | **75.0%** |
| **MTF+LZ4** | **198**      | **28**      | **31**      | **84.1%** |

*Run `cargo bench` to see actual results on your hardware.*

## Project Structure

```
magenta-telemetry-format/
├── core/          # Rust core library (encoder/decoder)
├── wasm/          # WebAssembly bindings
├── python/        # Python bindings (PyO3)
├── cli/           # Command-line tools
├── benches/       # Benchmark suite
└── examples/      # Usage examples
```

## Development

### Prerequisites

- Rust 1.70+ (stable)
- wasm-pack (for WASM builds)
- Python 3.8+ (for Python bindings)
- maturin (for Python builds)

### Build WASM

```bash
cd wasm
wasm-pack build --target web
```

### Build Python Package

```bash
cd python
maturin develop  # Development build
maturin build --release  # Production wheel
```

### Run Tests

```bash
cargo test --all --verbose
```

### Generate Documentation

```bash
cargo doc --no-deps --open
```

## Use Cases

- **IoT Telemetry**: Collect metrics from low-power devices
- **Mobile Apps**: Efficient telemetry from mobile agents in areas with expensive data
- **Edge Computing**: Local aggregation before cloud transmission
- **Offline-First**: Store-and-forward architecture for intermittent connectivity
- **Real-Time Monitoring**: Live dashboards with WASM decoding

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.

## License

MIT License - see [LICENSE](./LICENSE) for details.

## Author

Built with ❤️ for the Magenta project by the Magenta Research Team.

---

**Status**: Under active development