spark-hydro 0.2.2
Advanced Delta-Lake related tooling based on Apache Spark
License: MIT
Requires Python: >=3.7

# hydro 💧

[![build](https://github.com/christophergrant/hydro/actions/workflows/push.yml/badge.svg?branch=main)](https://github.com/christophergrant/hydro/actions/workflows/push.yml)
[![codecov](https://codecov.io/gh/christophergrant/hydro/branch/main/graph/badge.svg?token=Z64814CV1E)](https://codecov.io/gh/christophergrant/hydro)
![pypidownloads](https://img.shields.io/pypi/dm/spark-hydro?color=%23123d0d&label=pypi%20downloads&style=flat-square)

hydro is a collection of Python-based [Apache Spark](https://spark.apache.org/) and [Delta Lake](https://delta.io/) extensions.

See [Key Functionality](#key-functionality-) for concrete use cases.

## Who are hydro's intended users?

hydro is intended to be used by developers and engineers who interact with Delta Lake tables and Spark DataFrames with Python. It can be used by those of all skill levels.

hydro works on the Databricks platform as well as on other platforms where PySpark and Delta Lake can be installed: laptops for example.


## Warning ⚠️

hydro is well tested but not battle hardened, yet. Use it at your own risk.

## Installation

```commandline
pip install spark-hydro
```

## Docs 📖

https://christophergrant.github.io/hydro

## Key Functionality 🔑

- Correctly perform [Slowly Changing Dimensions (SCD)](https://en.wikipedia.org/wiki/Slowly_changing_dimension) on Delta Lake tables - [hydro.delta.scd](https://christophergrant.github.io/hydro/api/delta.html#hydro.delta.scd) and [hydro.delta.bootstrap_scd2](https://christophergrant.github.io/hydro/delta.html#hydro.delta.bootstrap_scd2)
- Issue queries against Delta Log metadata, quickly and efficiently retrieving file-level metadata even on Petabyte-scale tables - [hydro.delta.file_stats](https://christophergrant.github.io/hydro/api/delta.html#hydro.delta.file_stats), [hydro.delta.partition_stats](https://christophergrant.github.io/hydro/api/delta.html#hydro.delta.partition_stats)
- Infer the schema of JSON columns - [hydro.spark.infer_json_schema](https://christophergrant.github.io/hydro/api/delta.html#hydro.delta.infer_json_field)
- Drop nested fields from a Spark DataFrame [hydro.spark.drop_fields](https://christophergrant.github.io/hydro/api/spark.html#hydro.spark.drop_fields)
- Quality of life improvements like [hydro.delta.detail_enhanced](https://christophergrant.github.io/hydro/api/delta.html#hydro.delta.detail_enhanced) and [hydro.spark.fields](https://christophergrant.github.io/hydro/api/spark.html#hydro.spark.fields)
- And more... check the docs!

## Contributions ✨

Contributions are welcome.

Please [create an issue](https://github.com/christophergrant/hydro/issues/new/choose) and discuss before starting work on a feature to make sure that it aligns with the future of the project.

## Naming 🤓

`hydro` is short for hydrologist, where a hydrologist is a person who studies water and its movement. Delta Lake, Data Lake, Lakehouse => water.

## ChatGPT and LLMs 🤖

Some of this project's code and documentation was generated by a Large [Language Model](https://en.wikipedia.org/wiki/Language_model) (LLM), namely [ChatGPT](https://chat.openai.com/chat). Some code was even taken verbatim from the AI (with 0 adaptations). Their outputs have been vetted by humans and tested, so don't worry about quality.

Otherwise, we are proud prompt engineers, so we display the prompt that gave us the code in hydro's source ([example](https://github.com/christophergrant/hydro/commit/8d2d84da4930f14caac62c46ea9a1c07a8bdeac4#diff-4665a0f13cae8eb34e13e308ee3935edf0a63f563ac6301038b0d15f95666446R11)).