semvec/pypi-packages huggingface dataset:
loadhouse 0.1.4
A data loading and transformation engine for data lakehouses
Requires Python: >=3.10

# Loadhouse

A powerful ETL (Extract, Transform, Load) tool designed for data lakehouse architectures with JSON-based configuration.

## Overview

Loadhouse is a flexible data processing tool that simplifies ETL operations through JSON configuration. It supports various data sources and provides robust data transformation capabilities using Apache Spark.

## Features

- **Configurable Data Sources**
  - File-based (CSV, Delta, etc.)
  - JDBC connections
  - SQL queries
  - DataFrame operations

- **Data Transformations**
  - Expression filtering
  - Custom transformations
  - Data quality validation

- **Multiple Output Formats**
  - Delta Lake
  - File formats (CSV, Parquet, etc.)
  - Console output for debugging
- **Quality Checker**
  - Data quality patterns with Apache Airflow
  - Unit test Spark
  - Data validation with GX
acme_s3 0.0.5
Efficient access to S3

# acme-s3

Efficient operations on S3 with retries, progress tracking and parallel processing

# Problem

Interactive with boto3 client can at times lack some features. This library aims to provide a more user-friendly interface to S3 operations.

# Features

* Multithreaded opeations for opeations on multiple files
* Progress tracking for all operations
* Retries for recoverable errors
* Benchmarking tool to test the performance of the operations
* Graceful error handling with useful messages

# Dev environment

The project comes with a python development environment.
To generate it, after checking out the repo run:

    chmod +x create_env.sh

Then to generate the environment (or update it to latest version based on state of `uv.lock`), run:

    ./create_env.sh

This will generate a new python virtual env under `.venv` directory. You can activate it via:

    source .venv/bin/activate

If you are using VSCode, set to use this env via `Python: Select Interpreter` command.

## Example usage

    as3 bench --bucket $TEST_AWS_BUCKET_NAME --prefix s3-benchmark

# Project template

This project has been setup with `acme-project-create`, a python code template library.

# Required setup post use

* Enable GitHub Pages to be published via [GitHub Actions](https://docs.github.com/en/pages/getting-started-with-github-pages/configuring-a-publishing-source-for-your-github-pages-site#publishing-with-a-custom-github-actions-workflow) by going to `Settings-->Pages-->Source`
* Create `release` environment for [GitHub Actions](https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-deployments/managing-environments-for-deployment#creating-an-environment) to enable uploads of the library to PyPi
* Setup auth to PyPI for the GitHub Action implemented in `.github/workflows/release.yml` via [Trusted Publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) `uv publish` [doc](https://docs.astral.sh/uv/guides/publish/#publishing-your-package)
* Once you create the python environment for the first time add the `uv.lock` file that will be created in project directory to the source control and update it each time environment is rebuilt
* In order not to replicate documentation in `docs/docs/index.md` file and `README.md` in root of the project setup a symlink from `README.md` file to the `index.md` file.
To do this, from `docs/docs` dir run:

    ln -sf ../../README.md index.md