datacodec 1.0.0
Binary codec library based upon muesli compile-time codecs
Requires Python: >=3.8
Keywords: binary, codec, serialization, deserialization, muesli

# datacodec

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![Tests](https://github.com/felixjones/datacodec/actions/workflows/tests.yml/badge.svg)](https://github.com/felixjones/datacodec/actions/workflows/tests.yml)
[![License](https://img.shields.io/badge/license-BSD--3--Clause-green.svg)](LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

A Python binary codec library based upon [muesli](https://github.com/felixjones/muesli) compile-time codecs.

## Installation

```bash
pip install datacodec
```

For NumPy support:
```bash
pip install datacodec[numpy]
```

## Quick Start

The easiest way to use datacodec is with automatic codec generation from dataclasses:

```python
from datacodec import dataclass_codec, BinaryFormat
from dataclasses import dataclass
import io

@dataclass
class Vec3:
    x: float  # Automatically uses double_codec
    y: float
    z: float

# Codec is automatically generated from type hints!
vec3_codec = dataclass_codec(Vec3)
format = BinaryFormat(vec3_codec)

# Serialize
output = io.BytesIO()
format.serialize(Vec3(1.0, 2.0, 3.0), output)

# Deserialize
output.seek(0)
result = format.deserialize(output)
print(result)  # Vec3(x=1.0, y=2.0, z=3.0)
```

## Features

- **Automatic codec generation** from dataclasses with type inference
- **Python-native types**: `int` -> varint, `float` -> double, `str` -> string
- **NumPy support**: Automatic detection of NumPy types (optional)
- **Fluent API**: Chain codec operations with `.transform()`, `.constrain()`, `.optional()`
- **Cross-platform**: Generate binary data for ARM32, ARM64, x86-32, x86-64 targets
- **Type-safe**: Full Python typing support
- **Based on muesli**: Binary compatible with muesli C++ library

## Python Type Mappings

When using `dataclass_codec()`, Python types automatically map to appropriate codecs:

| Python Type | Codec | Reason |
|-------------|-------|--------|
| `int` | `signed_varint_codec` | Python ints are arbitrary precision |
| `float` | `double_codec` | Python floats are double precision |
| `bool` | `bool_codec` | Boolean type |
| `str` | `string_codec` | UTF-8 strings |

### NumPy Types (with `pip install datacodec[numpy]`)

| NumPy Type | Codec |
|------------|-------|
| `np.int8/16/32/64` | `int8/16/32/64_codec` |
| `np.uint8/16/32/64` | `uint8/16/32/64_codec` |
| `np.float32` | `float_codec` |
| `np.float64` | `double_codec` |

## Binary Format Details

### String Encoding
Strings use **null-termination** (muesli compatible):
- UTF-8 encoded string data
- Null byte (0x00) terminator

```python
# "hello" serialises as:
# [68 65 6c 6c 6f 00]
#  "hello" UTF-8  null
```

### Integer Encoding
- **Fixed-size integers**: Use native byte order and alignment
- **Varint**: Variable-length encoding (1-10 bytes depending on value)
  - Unsigned: Direct varint encoding
  - Signed: Zigzag encoding (maps negatives to positives efficiently)

### Alignment
Platform-specific alignment is maintained:
- ARM32: 8-byte alignment for int64/double
- ARM64: 8-byte alignment for int64/double
- x86-64: Natural alignment per type

## Examples

### Automatic Codec Generation

```python
from datacodec import dataclass_codec
from dataclasses import dataclass

@dataclass
class Person:
    name: str      # -> string_codec
    age: int       # -> signed_varint_codec
    height: float  # -> double_codec

# Codec automatically generated!
person_codec = dataclass_codec(Person)
```

### Explicit Fixed-Size Types

Use the `@codec` decorator when you need specific fixed-size types:

```python
from datacodec import dataclass_codec, codec, uint8_codec, uint32_codec

@dataclass
class Message:
    msg_type: int = codec(uint8_codec)      # Force uint8
    timestamp: int = codec(uint32_codec)    # Force uint32
    payload_size: int                       # Uses varint (default)

protocol_codec = dataclass_codec(Message)
```

### Manual Codec Construction

For full control, build codecs manually:

```python
from datacodec import tuple_codec, float_codec, BinaryFormat

@dataclass
class Vec3:
    x: float
    y: float
    z: float

# Manually specify each field's codec
vec3_codec = tuple_codec(float_codec, float_codec, float_codec).apply(Vec3)
```

### Dict Codec

Python dicts work like `std::map` in muesli - no special codec needed, just composition:

```python
from datacodec import dict_codec, string_codec, int32_codec

# Dict[str, int]
my_dict_codec = dict_codec(string_codec, int32_codec)
data = {"a": 1, "b": 2, "c": 3}
```

### Cross-Platform

Generate binary data for different architectures:

```python
from datacodec import BinaryFormat, uint32_codec

# Generate data for ARM32 target
format_arm32 = BinaryFormat(uint32_codec, platform='arm32')

# Binary is compatible with muesli C++ on ARM32
```

### Fluent API

Chain transformations and constraints:

```python
from datacodec import int32_codec

validated = int32_codec \
    .constrain(lambda x: x > 0) \
    .transform(lambda x: x * 2, lambda x: x // 2) \
    .optional()
```

## Advanced Usage

### NumPy Arrays

```python
import numpy as np
from datacodec import numpy_array_codec

# Fixed-size 3D vector
vec3 = numpy_array_codec(np.float32, shape=(3,))

# Dynamic-size array
float_array = numpy_array_codec(np.float32)
```

### NumPy in Dataclasses

```python
import numpy as np
from datacodec import dataclass_codec

@dataclass
class Point:
    x: np.float32  # Automatically uses float_codec
    y: np.float32
    z: np.float32

point_codec = dataclass_codec(Point)
```

## License

BSD 3-Clause License