rosetta-data 0.2.0
Universal columnar format bridge - read Parquet, Vortex, Lance with one API
License: Apache-2.0
Requires Python: >=3.9
Keywords: parquet, vortex, lance, arrow, columnar, data, format

# Rosetta

[![CI](https://github.com/rosetta-data/rosetta/actions/workflows/python-ci.yml/badge.svg)](https://github.com/rosetta-data/rosetta/actions/workflows/python-ci.yml)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)

**Universal file format bridge for columnar data formats.**

Read any columnar format (Parquet, Vortex, Lance) through a single unified API. Rosetta auto-detects file formats and provides transparent access to your data, regardless of how it was stored.

## Why Rosetta?

The data lake ecosystem is fragmenting. New formats like [Vortex](https://github.com/spiraldb/vortex) and [Lance](https://github.com/lancedb/lance) offer significant performance improvements over Parquet, but adoption is blocked by compatibility concerns:

- **94% of Parquet files** still use v1 features from 2013 ([CMU research](https://www.cs.cmu.edu/~pavlo/blog/2026/01/2025-databases-retrospective.html))
- Tools like DuckDB [disabled v2 encodings](https://duckdb.org/2025/01/22/parquet-encodings) by default due to compatibility issues
- Migration to new formats requires rewriting data pipelines

Rosetta solves this by providing a **transparent bridge layer** that reads any format and outputs standard Arrow data.

## Performance

Rosetta achieves near-zero overhead for Parquet passthrough. At scale, Rosetta is actually **15-20% faster** than native Parquet:

```
Parquet Read (1M rows):
├── Native Parquet:  28.35 ms
├── Rosetta:         22.63 ms  ← 20% faster!
└── Throughput:      376 MiB/s

Format Detection: ~150 µs per file
```

See **[BENCHMARKS.md](BENCHMARKS.md)** for full methodology and results.

## Installation

```bash
pip install rosetta-data
```

That's it! This installs Rosetta with support for Parquet, Vortex, and Lance formats.

### From Source (Contributors)

For development or to build from source:

```bash
git clone https://github.com/rosetta-data/rosetta
cd rosetta
./scripts/install.sh
```

Or manually:

```bash
# Prerequisites: Python 3.9+, Rust nightly
pip install maturin pyarrow
RUSTUP_TOOLCHAIN=nightly maturin develop --features "python,vortex-reader,lance-reader"
```

### Rust Library

```toml
[dependencies]
rosetta = { version = "0.2", features = ["vortex-reader", "lance-reader"] }
```

> **Note:** Vortex support requires nightly Rust. Use `cargo +nightly build`.

## Quick Start

### Create Sample Data

Before running the examples, create a sample Parquet file:

```python
import pyarrow as pa
import pyarrow.parquet as pq

# Create sample data
table = pa.table({
    'id': [1, 2, 3, 4, 5],
    'name': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'age': [25, 30, 55, 60, 45],
    'status': ['active', 'inactive', 'active', 'active', 'inactive'],
})
pq.write_table(table, 'data.parquet')
```

### Python

```python
import rosetta
from rosetta import col, lit

# Read any supported format - returns PyArrow Table
table = rosetta.read("data.parquet")
# table = rosetta.read("data.vortex")  # Vortex files
# table = rosetta.read("data.lance")   # Lance datasets

# Column projection (pushed to native reader)
table = rosetta.read("data.parquet", columns=["id", "name"])

# Filter pushdown - predicate is pushed to native format reader
# This runs IN the Parquet/Vortex/Lance reader, not after!
table = rosetta.read(
    "data.parquet",
    filter=col("age").gt(lit(30)).and_(col("status").eq(lit("active")))
)

# Detect file format
fmt = rosetta.detect_format("data.parquet")
print(fmt.name)        # "Parquet v1"

# Get file metadata without reading data
info = rosetta.file_info("data.parquet")
print(info["schema"])         # PyArrow Schema
print(info["num_row_groups"]) # Number of row groups

# Get statistics for query optimization
stats = rosetta.statistics("data.parquet")
print(stats.num_rows)
print(stats.column("age").min_value)

# Read as batches for large files
batches = rosetta.read_batches("data.parquet")
```

### Rust

```rust
use rosetta::{UnifiedReader, FormatDetector};

#[tokio::main]
async fn main() -> rosetta::Result<()> {
    // Auto-detect format and read
    let reader = UnifiedReader::open("data.parquet").await?;
    let batches = reader.read_all().await?;

    // Fastest path for known Parquet files (sync, no format detection)
    let reader = UnifiedReader::open_parquet_sync("data.parquet")?;
    let batches = reader.read_all_sync()?;

    // Column projection
    let batches = reader.read_projection(&["id".to_string(), "name".to_string()]).await?;

    // Format detection
    let detector = FormatDetector::new();
    let format = detector.detect_file("data.parquet").await?;
    println!("Format: {}", format.name());

    Ok(())
}
```

### CLI

> **Note:** The CLI is built separately from the Python package. Build it with:
> ```bash
> cargo +nightly build --release --features "vortex-reader,lance-reader"
> # Binary will be at: target/release/rosetta
> ```

```bash
# Detect file format
rosetta detect data.parquet
# Output:
# File: data.parquet
# Format: Parquet v1
# Extension: .parquet

# Show file info (schema, row groups)
rosetta info data.parquet
# Output:
# Format: Parquet v1
# Row Groups: 1
# Schema:
#   id : Int64 (nullable)
#   name : Utf8 (nullable)

# Read and display data
rosetta read data.parquet --limit 10
rosetta read data.parquet --columns id,name --limit 5

# Convert between formats
rosetta convert data.vortex data.parquet

# Benchmark read performance
rosetta bench data.parquet --iterations 100
```

## Supported Formats

| Format | Read | Write | Notes |
|--------|------|-------|-------|
| Parquet v1 | Yes | Yes | Full support, all compression codecs |
| Parquet v2 | Yes | - | Reads v2 encodings (DELTA_*, BYTE_STREAM_SPLIT) |
| Vortex | Yes | - | Requires `--features vortex-reader` |
| Lance | Yes | - | Requires `--features lance-reader` |
| F3 | Planned | - | Self-describing files with embedded WASM decoders |

## Features

Enable optional format support via Cargo features:

```toml
[dependencies]
rosetta = { version = "0.2", features = ["vortex-reader", "lance-reader"] }
```

Available features:
- `parquet-reader` (default) - Parquet v1/v2 support with filter pushdown
- `vortex-reader` - Vortex columnar format (requires nightly Rust)
- `lance-reader` - Lance columnar format
- `datafusion-integration` - DataFusion TableProvider
- `wasm-runtime` - F3-style self-describing files
- `python` - Python bindings via PyO3 (use maturin to build)

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                         │
│      (Spark, DuckDB, Polars, Pandas, Your Code)             │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     Rosetta Bridge                           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │   Format     │  │   Unified    │  │  Transcoder  │      │
│  │  Detector    │  │   Reader     │  │  (optional)  │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
       ┌──────────┐    ┌──────────┐    ┌──────────┐
       │ Parquet  │    │  Vortex  │    │  Lance   │
       │ Reader   │    │  Reader  │    │  Reader  │
       └──────────┘    └──────────┘    └──────────┘
                              │
                              ▼
                     Arrow RecordBatches
```

## Integrations

### DuckDB Extension (Build from Source)

> **Note:** The DuckDB extension requires building from source. See `duckdb_extension/README.md` for instructions.

```sql
-- After building and loading the extension:
LOAD rosetta;

-- Read any format transparently
SELECT * FROM read_parquet('data.vortex') LIMIT 10;
```

### Iceberg FileIO (Java)

> **Note:** Iceberg integration is implemented in Java. See `java/rosetta-iceberg/` for the implementation.

```java
// Java usage with Iceberg
import com.rosetta.iceberg.RosettaFileIO;

Map<String, String> properties = new HashMap<>();
properties.put("io-impl", "com.rosetta.iceberg.RosettaFileIO");
Catalog catalog = CatalogUtil.loadCatalog("my_catalog", properties);
```

### S3 Proxy (Planned)

> **Note:** S3 proxy is planned but not yet implemented.

```bash
# Coming soon:
# rosetta proxy --listen 0.0.0.0:8080 --backend s3://my-bucket
```

## Development

```bash
# Run Rust tests
cargo +nightly test --lib --features "vortex-reader,lance-reader"

# Build and install Python bindings (for development)
pip install maturin pytest pyarrow
RUSTUP_TOOLCHAIN=nightly maturin develop --features "python,vortex-reader,lance-reader"

# Run Python tests
pytest python/tests/ -v

# Run benchmarks
cargo +nightly bench --features "vortex-reader,lance-reader"

# Format and lint
cargo +nightly fmt
cargo +nightly clippy --features "vortex-reader,lance-reader"
```

## Benchmarking

```bash
# Quick CLI benchmark
rosetta bench data.parquet --iterations 100

# Full Criterion benchmarks
cargo bench --bench read_benchmark
```

## Roadmap

See [ROADMAP.md](ROADMAP.md) for the detailed development plan.

- [x] Core readers: Parquet (v1/v2), Vortex, Lance
- [x] Format detection (magic bytes + directory structure)
- [x] Transcoder (any format -> Parquet)
- [x] DuckDB extension (source code, build required)
- [x] Iceberg FileIO integration (Java)
- [x] FUSE filesystem (experimental)
- [x] Python bindings
- [x] Performance optimized (~0% overhead)
- [ ] S3 proxy
- [ ] Progressive upgrade API
- [ ] F3/WASM runtime for self-describing files
- [ ] GPU-accelerated transcoding

## Contributing

Contributions welcome! Please see our [contributing guidelines](CONTRIBUTING.md).

## License

Apache-2.0

## Acknowledgments

- [Apache Arrow](https://arrow.apache.org/) - The foundation for columnar data interop
- [Vortex](https://github.com/spiraldb/vortex) - High-performance columnar format
- [Lance](https://github.com/lancedb/lance) - ML-focused columnar format
- [F3](https://db.cs.cmu.edu/papers/2025/zeng-sigmod2025.pdf) - Self-describing file format research