semvec/pypi-packages huggingface dataset:
kalyfo-libs-spark-helper 0.1.4
A Spark helper class to create and configure optimized Spark sessions with Delta Lake support
License: MIT
Requires Python: >=3.9
Keywords: spark, pyspark, delta-lake, data-engineering

# Spark Helper

A Python utility library for creating and configuring optimized Apache Spark sessions with Delta Lake support.

## Features

- **Optimized Spark Configuration**: Pre-configured settings for optimal performance
- **Delta Lake Support**: Built-in support for Delta Lake operations
- **Memory Management**: Intelligent memory allocation for driver and executors
- **Customizable**: Extensive configuration options via parameters or environment variables
- **Timezone Support**: Configure session timezone
- **Resource Monitoring**: Built-in configuration verification

## Installation

```bash
pip install kalyfo-libs-spark-helper
```

## Quick Start

```python
from spark_helper import SparkHelper

# Create a Spark helper instance
helper = SparkHelper(
    available_memory_gb=64,
    driver_memory_gb=40,
    cores=16
)

# Create a Spark session
spark = helper.create_spark_session("My Spark App")

try:
    # Your Spark code here
    df = spark.read.parquet("data.parquet")
    df.show()
finally:
    # Always stop the session when done
    helper.stop_spark_session()
```

## Configuration

### Basic Parameters

- `available_memory_gb`: Total memory available for Spark (required)
- `driver_memory_gb`: Memory allocated to the driver (default: 40GB)
- `cores`: Number of CPU cores to use (default: all available cores)
- `cores_per_executor`: Cores per executor (default: 5)
- `enable_delta_lake`: Enable Delta Lake support (default: True)

### Environment Variables

You can also configure Spark Helper using environment variables:

```bash
export AVAILABLE_MEMORY_GB=64
export AVAILABLE_LOGICAL_CORES=16
export SPARK_MEMORY_FRACTION=0.8
export SPARK_MEMORY_STORAGE_FRACTION=0.6
export TIMEZONE="Europe/Athens"
```

or by creating a .env file:

```
AVAILABLE_MEMORY_GB=64
AVAILABLE_LOGICAL_CORES=16
SPARK_MEMORY_FRACTION=0.8
SPARK_MEMORY_STORAGE_FRACTION=0.6
TIMEZONE="Europe/Athens"
```

## Advanced Usage

### Custom Memory Settings

```python
from spark_helper import SparkHelper

helper = SparkHelper(
    available_memory_gb=128,
    driver_memory_gb=40,
    cores=32,
    spark_memory_fraction=0.8,
    spark_memory_storage_fraction=0.6,
    cores_per_executor=5
)
```

## Features in Detail



### Configuration Verification


## Requirements

- Python >= 3.9
- PySpark == 4.0.1
- delta-spark == 4.0.0
- findspark == 2.0.1
- python-dotenv == 1.1.1

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Support

For issues and questions, please open an issue on the [GitHub repository](https://github.com/kalyfo-libs/spark_helper/issues).
blindfile 0.1.1
The virtual blindfold to make truly unbiased file comparisons
License: MIT
Requires Python: >=3.13
Keywords: file, comparison, blind, testing

## `blindfile`

![blindfile logo](docs/images/logo.png)


The virtual blindfold to make truly unbiased file comparisons ⚖️.


## Installation

Install using either [`uv`](https://github.com/astral-sh/uv), [`pipx`](https://github.com/pypa/pipx) or [`pip`](https://github.com/pypa/pip).

```
# install as a tool (recommended)...
$ uv install tool blindfile
$ pipx install blindfile

# or as a a package
$ uv add blindfile
$ pip install blindfile
```

## Usage example

Let's say you want to compare some wav files: `take1.wav`, `take2.wav` and `take3.wav`.
1. **`cover` (hide your filenames)**
   ```
   $ blindfile cover '*.wav'
   __blindfile__/contemporary-tree.wav
   __blindfile__/generous-blade.wav
   __blindfile__/caramel-nut.wav
   ```


1. **compare your files**

   Open files listed by `cover` and compare them. In our example - get your headphones and decide which file sounds best 🎧 !

1. **`uncover` (reveal the originals)**
 
   When you're ready to learn which file is which, run:

   ```
   $ blindfile uncover
   __blindfile__/contemporary-tree.wav -> take2.wav
   __blindfile__/generous-blade.wav -> take3.wav
   __blindfile__/caramel-nut.wav -> take1.wav
   ```