semvec/pypi-packages huggingface dataset:
duckdb-spark 1.0.4
A custom PySpark extension for writing data to DuckDB
Requires Python: >=3.6

# DuckDB Extension for PySpark

Since DuckDB supports only a **single writer at a time**, writing directly from PySpark can lead to **locking errors** due to Spark's multi-worker write process.  

This custom **PySpark extension** provides a reliable way to write **DataFrames to DuckDB**, ensuring smooth data transfer without concurrency issues.

## Features

- ✅ **Seamlessly write PySpark DataFrames to DuckDB**  
- ✅ **Supports `overwrite` and `append` modes**  
- ✅ **Automatically detects and adds new columns when appending data**  
- ✅ **Simple integration with PySpark's `DataFrameWriter` API**  

## Installation

You can install the package using `pip`:

```bash
pip install duckdb-spark


## Usage

```bash
from pyspark.sql import SparkSession
from duckdb_extension import register_duckdb_extension

spark = SparkSession.builder.appName("DuckDB Example").getOrCreate()

# Register the DuckDB extension
register_duckdb_extension(spark)

df=spark.read.csv("employe.csv",header=True)

# Use the custom extension to write the DataFrame to DuckDB and specify the table name
df.write.duckdb_extension(database="./company_database.duckdb", table_name="employe_tbl", mode="append")
ddrift 0.1.0
Add your description here
Requires Python: >=3.11

`ddrift` analyzes the differences between data sources, i.e. how much has dataset a drifted from dataset b. The framework is engine agnostic. Each  engine is required to comply with simple abstract protocols in order to enable the standard reporting.

Engines Supported:
- DuckDB

Engines to be Supported:
- Postgres
- Polars
- Pandas

Install with `pip install ddrift` or (prefferebly) `uv pip install ddrift`

## Getting Started

Let's create 2 simple tables and compare them to one another. The fundamental question we're asking is "How much has table2 drifted from table1?"

```
import duckdb

with duckdb.connect() as con:
    con.execute("CREATE TABLE table1 (city VARCHAR, state VARCHAR)")
    con.execute(
        "INSERT INTO table1 VALUES ('New York', 'NY'), ('Los Angeles', 'CA'), ('Chicago', 'IL')"
    )

    con.execute("CREATE TABLE table2 (city VARCHAR, state VARCHAR)")
    con.execute(
        "INSERT INTO table2 VALUES ('New York', 'NY'), ('Phoenix', 'AZ'), ('Philadelphia', 'PA')"
    )

    sql = SQLComparator(df1="table1", df2="table2", con=con)
    sql.comp_freq(vars=("city", "state"))

    print(sql.results) # prints a list of result objects containing an in memory representation
```