mimir-io 0.5.0
Declarative dataset processing library with lens-based transformations
License: MIT
Requires Python: >=3.8
Keywords: machine-learning, deep-learning, dataset, data-processing, pytorch, audio-processing, ml-workflow, training, caching, transformations

<div align="center">

<img src="logo.png" alt="mimir_io logo" width="200">

# mimir_io

**End-to-end ML workflow framework with declarative dataset processing, lens-based transformations, automatic caching, and built-in training infrastructure.**

[![Python](https://img.shields.io/badge/Python-3.8+-blue.svg)](https://www.python.org/)
[![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Contributing](https://img.shields.io/badge/contributions-welcome-brightgreen.svg)](CONTRIBUTING.md)

</div>

## Overview

mimir_io is a comprehensive ML workflow framework that simplifies the entire machine learning pipeline from data processing to model training. It provides:

- **Declarative Data Processing**: Define transformations using lenses with automatic caching
- **Training Infrastructure**: Built-in trainers, experiment setup, and model management
- **Minimal Boilerplate**: Start training with just 3 lines of code using `quick_train()`
- **Full Control**: Extend and customize every aspect when needed

Instead of writing hundreds of lines for data loading, preprocessing, and training loops, mimir_io handles it all automatically while giving you full control when needed.

**[Why mimir_io?](why.md)** - Learn about the problems mimir_io solves and why it exists.

**[Roadmap](ROADMAP.md)** - Strategic plan from version 0.1.0 to 1.0.0 to make mimir_io the standard for deep learning experiments.

### Key Features

**Data Processing:**
- Declarative transformations using lenses with automatic caching
- Comprehensive audio processing with specialized lenses
- Dataset splitting utilities for train/validation/test splits
- **GPU-accelerated augmentations:**
  - Audio: SpecAugment, time/frequency masking, gain variation, noise injection, pitch shift, etc.
  - Images: horizontal/vertical flip, rotation, crop, color jitter, random erasing, affine transforms, normalization
- RayFrame data structures for type-safe transformations
- Automatic pipeline optimization

**Training & ML Infrastructure:**
- Quick start: Train models with just 3 lines of code
- Built-in trainer with checkpointing, callbacks, and early stopping
- TensorBoard/WandB integration for experiment tracking
- Mixed precision training (AMP) support
- Model management with YAML configuration
- PyTorch and PyTorch Lightning integration

## Installation

```bash
pip install mimir-io
```

**Audio Format Support:**

For full audio format support (WAV, MP3, FLAC, OGG, M4A, AAC), install with audio extras:

```bash
pip install mimir-io[audio]
```

**System Requirements for MP3/M4A:**

For MP3 and M4A support, you need system-level audio libraries:
- **Linux**: `sudo apt-get install ffmpeg` or `sudo apt-get install sox`
- **macOS**: `brew install ffmpeg` or `brew install sox`
- **Windows**: Download ffmpeg from [ffmpeg.org](https://ffmpeg.org/download.html)

## Quick Start

### Minimal Training (3 Lines)

```python
from mimir_io.experiment import quick_train
import torch.nn as nn
from pathlib import Path

# Create model
model = nn.Sequential(
    nn.Linear(80, 128),
    nn.ReLU(),
    nn.Linear(128, 10),
)

# Prepare data
files = list(Path("./data/raw").glob("*.wav"))
labels = [0, 1, 0, 1, ...]  # Your labels

# Start training (everything else is automatic!)
trainer = quick_train(model=model, file_paths=files, labels=labels)
```

### Data Processing with Automatic Caching

```python
from mimir_io import Dataset
from mimir_io.audio import resample, normalize, log_mel_spectrogram

dataset = Dataset(data_dir="./data")

# Create pipeline - order doesn't matter, automatically optimized!
pipeline = (
    normalize(target_db=-20.0)
    | resample(16000, orig_sample_rate=44100)
    | log_mel_spectrogram(sample_rate=16000, n_mels=80)
)

# Apply with automatic caching (first run: computes, subsequent: loads from cache)
audio_frame = dataset.load_audio("./data/raw/audio.wav")
features = dataset.apply(pipeline, audio_frame.data)
```

### Dataset Splitting

```python
from mimir_io import split_dataset
from pathlib import Path

# Split data into train/val/test with stratification
files = list(Path("./data/raw").glob("*.wav"))
labels = [0, 1, 0, 1, ...]  # Your labels

split = split_dataset(
    file_paths=files,
    labels=labels,
    train_ratio=0.7,
    val_ratio=0.15,
    test_ratio=0.15,
    stratify=True,  # Maintain label distribution
    seed=42,
)

# Save for later use
split.save("./data/split.json")
```

### Custom Pipeline with Augmentations

#### Audio Augmentations

```python
from mimir_io import Dataset
from mimir_io.audio import resample, normalize, log_mel_spectrogram
from mimir_io.audio.augment import spec_augment

dataset = Dataset(data_dir="./data")

# Training pipeline with augmentations
train_pipeline = (
    resample(16000, orig_sample_rate=44100)
    | normalize(target_db=-20.0)
    | log_mel_spectrogram(sample_rate=16000, n_mels=80)
    | spec_augment(time_mask_param=27, num_time_masks=2)  # Data augmentation
)

# Validation pipeline without augmentations
val_pipeline = (
    resample(16000, orig_sample_rate=44100)
    | normalize(target_db=-20.0)
    | log_mel_spectrogram(sample_rate=16000, n_mels=80)
)
```

#### Image Augmentations (GPU-accelerated)

```python
from mimir_io import Dataset
from mimir_io.rayframe.image import (
    random_horizontal_flip,
    random_rotation,
    random_crop,
    color_jitter,
    random_erasing,
    normalize,
    random_affine,
)
from mimir_io.rayframe.image import to_image_frame

dataset = Dataset(data_dir="./data")

# Create image augmentation pipeline
augmentation_pipeline = (
    random_horizontal_flip(probability=0.5, seed=42)
    | random_rotation(degrees=15.0, seed=42)
    | random_crop(size=(224, 224), seed=42)
    | color_jitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1, seed=42)
    | random_erasing(probability=0.5, seed=42)
    | normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))
)

# Apply to image frame
image_frame = to_image_frame()(image_tensor)  # Convert tensor to ImageRayFrame
augmented_frame = dataset.apply(augmentation_pipeline, image_frame)
```

### PyTorch Integration

```python
from mimir_io import Dataset
from mimir_io.torch_dataset import create_dataloader
from mimir_io.audio import resample, log_mel_spectrogram

# Initialize dataset
mimir_dataset = Dataset(data_dir="./data")

# Create processing pipeline
pipeline = (
    resample(16000, orig_sample_rate=44100).no_cache()  # Don't cache cheap ops
    | log_mel_spectrogram(sample_rate=16000, n_mels=80).cache()  # Cache expensive ops
)

# Create DataLoader with automatic caching and augmentation
train_loader = create_dataloader(
    mimir_dataset=mimir_dataset,
    file_paths=train_files,
    transform_pipeline=pipeline,
    labels=train_labels,
    batch_size=32,
    shuffle=True,
    use_augmentation=True,  # Enable augmentations for training
)
```

### Training with Configuration

```python
from mimir_io.experiment import quick_train, ExperimentConfig
import torch.nn as nn

# Create configuration
config = ExperimentConfig(
    batch_size=64,
    epochs=20,
    learning_rate=0.0001,
    n_mels=128,
    use_augmentation=True,
    optimizer="AdamW",
    scheduler="cosine",
)

model = nn.Sequential(
    nn.Linear(config.n_mels, 256),
    nn.ReLU(),
    nn.Linear(256, 10),
)

# Train with configuration
trainer = quick_train(
    model=model,
    file_paths=files,
    labels=labels,
    config=config,
)
```

### Pipeline Optimization

```python
from mimir_io.optimization import optimize_pipeline, analyze_pipeline
from mimir_io.audio import resample, normalize, log_mel_spectrogram

# Create pipeline (order doesn't matter)
pipeline = normalize() | resample(16000) | log_mel_spectrogram(16000)

# Analyze pipeline structure
analysis = analyze_pipeline(pipeline)
print(f"Transformations: {analysis['total_transformations']}")

# Optimize with different algorithms
optimized = optimize_pipeline(pipeline, algorithm="dp")  # Optimal for small pipelines
# or
optimized = optimize_pipeline(pipeline, algorithm="genetic")  # For complex pipelines
```

## Documentation

**[Full Documentation](docs/index.md)** - Complete documentation with guides, tutorials, and API reference

**[Tutorial](docs/tutorial/README.md)** - Step-by-step guide for beginners

## Development

### Setup

```bash
pip install -r requirements-dev.txt
```

### Running Tests

```bash
pytest
```

### Code Formatting

```bash
black mimir_io tests
```

### Publishing to PyPI

For instructions on building and publishing the package to PyPI, see [DEPLOY.md](DEPLOY.md).

### Using Makefile

For easier development, use the Makefile:

```bash
make install-dev    # Install for development
make test           # Run tests
make lint           # Check code
make format         # Format code
make coverage       # Test coverage
make clean          # Clean temporary files
```

### Pre-commit Hooks

Install pre-commit hooks for automatic checks:

```bash
pip install pre-commit
pre-commit install
```

## Contributing

Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on the contribution process.

- [Code of Conduct](CODE_OF_CONDUCT.md) - Community behavior standards
- [Security Policy](SECURITY.md) - How to report security vulnerabilities
- [Roadmap](ROADMAP.md) - Project development plan
- [Feature Ideas](FEATURE_IDEAS.md) - Ideas for new features and enhancements

## License

MIT License - see LICENSE file for details.