semvec/pypi-packages huggingface dataset:
sparktestify 0.1.0
PySpark Data Pipeline Testing Framework
License: MIT
Requires Python: >=3.9

# SparkTestify

**PySpark Data Pipeline Testing Framework**

SparkTestify is a lightweight, modular, and CI/CD-friendly testing framework built for PySpark data pipelines.

### Features
- Pytest Fixtures & Plugins for SparkSession
- DataFrame and Schema Assertions
- Mock Data Sources & Test Data Generation
- Integration Testing Support
- CI/CD ready (GitHub Actions, Pre-commit)

### Installation

```bash
pip install sparktestify
```

### Usage

```python
from sparktest.assertions import assert_dataframe_equal
from sparktest.fixtures import spark


def test_my_transformation(spark):
    input_df = spark.createDataFrame([...])
    output_df = my_transformation(input_df)
    expected_df = spark.createDataFrame([...])
    assert_dataframe_equal(output_df, expected_df)
```

### Development Setup

```bash
pip install -r requirements.txt
pip install -r requirements-dev.txt
pre-commit install
pytest tests/
```

### CI/CD

GitHub Actions are configured to:
- Run pre-commit checks
- Run test cases on every push and PR

Workflow file: `.github/workflows/ci.yml`