spark-dist-fit 2.0.0
Modern Spark distribution fitting library with efficient parallel processing
License: MIT
Requires Python: <3.14,>=3.11
Keywords: distribution, fitting, pyspark, scipy, spark, statistics

# spark-dist-fit
[![CI](https://github.com/dwsmith1983/spark-best-fit/actions/workflows/ci.yml/badge.svg)](https://github.com/dwsmith1983/spark-best-fit/actions/workflows/ci.yml)
[![PyPI version](https://img.shields.io/pypi/v/spark-dist-fit)](https://pypi.org/project/spark-dist-fit/)
[![Documentation Status](https://readthedocs.org/projects/spark-dist-fit/badge/?version=latest)](https://spark-dist-fit.readthedocs.io/en/latest/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

**Modern Spark distribution fitting library with efficient parallel processing**

Automatically fit ~100 scipy.stats distributions to your data using Apache Spark's distributed computing power, with
optimized Pandas UDFs.

[![Python 3.11-3.13](https://img.shields.io/badge/python-3.11--3.13-blue.svg)](https://www.python.org/downloads/)
[![Spark 3.5+](https://img.shields.io/badge/spark-3.5+-orange.svg)](https://spark.apache.org/)

## Features

**Key Optimizations**
- **No data collection**: Computes histograms distributedly, never collects raw data to driver
- **Pandas UDFs**: Uses vectorized Pandas UDFs with Apache Arrow for 10-100x faster processing
- **Broadcast optimization**: Broadcasts tiny histogram (~1KB) instead of duplicating full dataset
- **Zero cross joins**: Completely eliminates expensive cartesian products
- **Adaptive sampling**: Intelligently samples large datasets (100M+ rows) without accuracy loss

**Capabilities**
- Fits ~100 scipy.stats continuous distributions in parallel
- Handles datasets from 100K to 100M+ rows efficiently
- Returns multiple goodness-of-fit metrics (SSE, AIC, BIC)
- Beautiful matplotlib visualizations

## Requirements

### Compatibility Matrix

| Spark Version | Python Versions | NumPy | Pandas | PyArrow |
|---------------|-----------------|-------|--------|---------|
| **3.5.x** | 3.11, 3.12 | 1.24+ (< 2.0) | 1.5+ | 12.0 - 16.x |
| **4.0.x** | 3.12, 3.13 | 2.0+ | 2.2+ | 17.0+ |

**Note:** Spark 3.5.x does not support NumPy 2.0. If using Spark 3.5 with Python 3.12, ensure `setuptools` is installed (provides `distutils`).

## Installation

```bash
# From PyPI
pip install spark-dist-fit

# From source
git clone <repository-url>
cd spark-best-fit
make install

# For development
make install-dev
```

## Quick Start

```python
from pyspark.sql import SparkSession
from spark_dist_fit import DistributionFitter
import numpy as np

# Create Spark session
spark = SparkSession.builder.appName("MyApp").getOrCreate()

# Generate sample data
data = np.random.normal(loc=50, scale=10, size=100_000)
df = spark.createDataFrame([(float(x),) for x in data], ["value"])

# Fit distributions
fitter = DistributionFitter(spark)
results = fitter.fit(df, column="value")

# Get best distribution
best = results.best(n=1)[0]
print(f"Best: {best.distribution} with SSE={best.sse:.6f}")

# Plot
fitter.plot(best, df, "value", title="Best Fit Distribution")
```

## Advanced Usage

### Custom Fitting Parameters

```python
from spark_dist_fit import DistributionFitter

fitter = DistributionFitter(spark, random_seed=123)
results = fitter.fit(
    df,
    column="value",
    bins=100,                    # Number of histogram bins
    support_at_zero=True,        # Only fit non-negative distributions
    enable_sampling=True,        # Enable adaptive sampling
    sample_fraction=0.3,         # Sample 30% of data
    max_distributions=50,        # Limit distributions to fit
)
```

### Working with Results

```python
# Get top 5 distributions by SSE
top_5 = results.best(n=5, metric="sse")

# Get best by AIC
best_aic = results.best(n=1, metric="aic")[0]

# Filter good fits
good_fits = results.filter(sse_threshold=0.01)

# Convert to pandas for analysis
df_pandas = results.to_pandas()

# Use fitted distribution
samples = best.sample(size=10000)  # Generate samples
pdf_values = best.pdf(x_array)     # Evaluate PDF
cdf_values = best.cdf(x_array)     # Evaluate CDF
```

### Plotting

```python
# Basic plot
fitter.plot(best, df, "value", title="Distribution Fit")

# Customized plot
fitter.plot(
    best,
    df,
    "value",
    figsize=(16, 10),
    dpi=300,
    histogram_alpha=0.6,
    pdf_linewidth=3,
    title="Distribution Fit",
    xlabel="Value",
    ylabel="Density",
)
```

### Customizing Excluded Distributions

By default, 17 slow scipy distributions are excluded to prevent fitting from hanging. To include a slow distribution:

```python
from spark_dist_fit import DistributionFitter, DEFAULT_EXCLUDED_DISTRIBUTIONS

# Remove a specific distribution from exclusions
exclusions = tuple(d for d in DEFAULT_EXCLUDED_DISTRIBUTIONS if d != "studentized_range")
fitter = DistributionFitter(spark, excluded_distributions=exclusions)
```

## Architecture

### Key Components

1. **DistributionFitter**: Main orchestrator
2. **HistogramComputer**: Distributed histogram computation
3. **Pandas UDFs**: Vectorized distribution fitting
4. **Broadcast Variables**: Efficient data sharing
5. **FitResults**: Convenient result handling

## API Reference

### DistributionFitter

```python
DistributionFitter(
    spark=None,                    # SparkSession (uses active session if None)
    excluded_distributions=None,   # Distributions to exclude (defaults to slow ones)
    random_seed=42,                # Random seed for reproducibility
)
```

### fit() Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `df` | DataFrame | required | Spark DataFrame with data |
| `column` | str | required | Column name to fit |
| `bins` | int/array | 50 | Number of histogram bins |
| `use_rice_rule` | bool | True | Auto-calculate bins using Rice rule |
| `support_at_zero` | bool | False | Only fit non-negative distributions |
| `max_distributions` | int | None | Limit number of distributions to fit |
| `enable_sampling` | bool | True | Enable sampling for large datasets |
| `sample_fraction` | float | None | Fraction to sample (None = auto) |
| `max_sample_size` | int | 1,000,000 | Max rows to sample |
| `sample_threshold` | int | 10,000,000 | Row count above which to sample |
| `num_partitions` | int | None | Spark partitions (None = auto) |

### plot() Parameters

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `result` | DistributionFitResult | required | Fitted distribution to plot |
| `df` | DataFrame | required | Spark DataFrame with data |
| `column` | str | required | Column name |
| `title` | str | "" | Plot title |
| `xlabel` | str | "Value" | X-axis label |
| `ylabel` | str | "Density" | Y-axis label |
| `figsize` | tuple | (12, 8) | Figure size (width, height) |
| `dpi` | int | 100 | Dots per inch |
| `show_histogram` | bool | True | Show data histogram |
| `histogram_alpha` | float | 0.5 | Histogram transparency |
| `pdf_linewidth` | int | 2 | PDF line width |
| `save_path` | str | None | Path to save figure |
| `save_format` | str | "png" | Save format (png/pdf/svg) |

## Development

### Setup Development Environment

```bash
# Clone repository
git clone <repository-url>
cd spark-best-fit

# Setup development environment
make setup

# This will:
# - Install package in editable mode
# - Install dev dependencies
# - Setup pre-commit hooks
```

### Development Commands

```bash
make help              # Show all available commands
make test              # Run tests
make test-cov          # Run tests with coverage
make pre-commit        # Run pre-commit hooks (lint, format, type check)
make check             # Run pre-commit + tests
make clean             # Clean build artifacts
make build             # Build distribution packages
```

### Pre-commit Hooks

This project uses pre-commit hooks for code quality:
- **ruff**: Fast Python linter
- **black**: Code formatting
- **isort**: Import sorting
- **mypy**: Static type checking

Hooks run automatically on commit. To run manually:
```bash
make pre-commit
```

### Code Style

- Line length: 120 characters
- Python: 3.11+
- Type hints: Encouraged
- Docstrings: Google style

## Testing

```bash
# Run all tests
make test

# Run with coverage
make test-cov

# Run specific test
pytest tests/test_core.py -v
```

## License

MIT License - see LICENSE file for details