chalkdf 3.31.9
DataFrame utilities for Chalk AI, backed by libchalk
Requires Python: <=3.14,>=3.10

# chalkdf

DataFrame utilities for building fast, portable data pipelines on top of
Apache Arrow — powered by Chalk’s libchalk execution engine.

The API centers on two concepts:
- `chalkdf.DataFrame`: a lightweight, eager plan that you can materialize to Arrow.
- `chalkdf.LazyFrame`: a serializable, chainable description of a plan that can
  be round-tripped to/from a protobuf and converted to a `DataFrame` for
  execution.

You build expressions with Chalk’s Python underscore DSL (`from chalk import _`)
and function registry (`import chalk.functions as F`). We also ship pragmatic
testing helpers (`chalkdf.Testing`) to make it easy to compare results in your
unit tests.


## Installation

- Requires Python 3.10–3.13
- Works on Linux and macOS 15+

Install from PyPI:

```
pip install chalkdf chalkpy
```


## Quickstart

Below are minimal, self‑contained examples that mirror the public API used in
the tests.

### Create from Arrow and select columns

```python
import pyarrow as pa
from chalkdf import DataFrame

tbl = pa.table({"a": [1, 2, 3], "b": [10, 20, 30], "c": [100, 200, 300]})
df = DataFrame.from_arrow(tbl)

out = df.select("c", "a")
print(out.run())
# ┌─────┬─────┐
# │  c  │  a  │
# ├─────┼─────┤
# │ 100 │  1  │
# │ 200 │  2  │
# │ 300 │  3  │
# └─────┴─────┘
```


### Add or replace columns with expressions

```python
import pyarrow as pa
import chalk.functions as F
from chalk import _
from chalkdf import DataFrame

tbl = pa.table({"a": pa.array([1, 2, 3], pa.int64()), "b": pa.array([10, 20, 30], pa.int64())})
df = DataFrame.from_arrow(tbl)

out = df.with_columns(
    {
        "sum": _.a + _.b,
        "flag": F.if_then_else(_.a < 2, "small", "big"),
    }
)
print(out.run())
```


### Filter, slice, and order

```python
import pyarrow as pa
from chalk import _
from chalkdf import DataFrame

tbl = pa.table({"a": pa.array([1, 2, 3, 4, 5], pa.int64()), "b": pa.array([10, 20, 30, 40, 50], pa.int64())})
df = DataFrame.from_arrow(tbl)

out = df.filter(_.a > 2).slice(1, 2).order_by(("b", "descending"))
print(out.run())
```


### Rename and explode

```python
import pyarrow as pa
from chalkdf import DataFrame

tbl = pa.table({"id": [1, 2], "vals": pa.array([[1, 2], []], type=pa.list_(pa.int64()))})
df = DataFrame.from_arrow(tbl)

renamed = df.rename({"vals": "values"})
exploded = renamed.explode("values")
print(exploded.run())  # (1,1), (1,2)
```


### Group by and aggregate

```python
import pyarrow as pa
from chalk import _
from chalkdf import DataFrame

tbl = pa.table({"g": [1, 1, 2, 2, 2], "v": pa.array([10, 20, 1, 2, 3], pa.int64())})
df = DataFrame.from_arrow(tbl)

out = df.agg(["g"], _.v.sum().alias("v_sum"))
print(out.run())  # rows may be unordered
```


### Joins

Use a list of join keys when both sides share column names. When names differ,
pass a mapping of left columns to their right-hand counterparts.

```python
import pyarrow as pa
from chalkdf import DataFrame

left = DataFrame.from_arrow(pa.table({"key": [1, 2, 3], "x": [10, 20, 30]}))
right = DataFrame.from_arrow(pa.table({"key": [2, 3, 4], "y": [200, 300, 400]}))

joined = left.join(right, on=["key"], how="inner").select("key", "x", "y")
print(joined.run())

# Or with an explicit mapping of left->right keys
joined2 = left.join(right, on={"key": "key"}, how="inner")

# When column names differ, map each left column to its right counterpart
joined3 = left.join(right, on={"key": "lookup_key"}, how="left")
```


### Scan Parquet files

You can construct a `DataFrame` that scans one or more Parquet files without
loading them eagerly. Use local `file://` URIs (or remote URIs when running in
an environment with appropriate access):

```python
from chalkdf import DataFrame

df = DataFrame.scan(
    ["file:///path/to/data.parquet"],
    name="my_table",
)

print(df.run())  # materializes and prints a preview
```


## Lazy plans (experimental)

`LazyFrame` records a chain of operations and can round‑trip to a protobuf for
transport or persistence. You can reconstruct the same lazy plan later and
convert it to a `DataFrame` to execute.

```python
from chalk import _
from chalkdf import LazyFrame

lf = (
    LazyFrame.from_dict({"c1": [1, 2, 3], "c2": [4, 5, 6]})
    .select("c1", "c2")
    .slice(0, length=10)
    .filter(_.c1 > 0)
)

proto = lf.to_proto()
lf2 = LazyFrame.from_proto(proto)
assert lf == lf2  # structural equality of the recorded plan

df = lf2._convert_to_df()  # convert to a DataFrame for execution
print(df.run())
```

Notes:
- `LazyFrame._convert_to_df()` is currently a private/experimental helper.
- For named tables, use `LazyFrame.named_table("table_name", schema)` as a root.


## Testing helpers

Use `chalkdf.Testing.assert_frame_equal` to compare `DataFrame` results in unit
tests. It materializes both frames to Arrow and supports relaxed comparisons.

```python
import pyarrow as pa
from chalkdf import Testing, DataFrame

left = DataFrame.from_arrow(pa.table({"a": [1.000001], "b": [2.0]}))
right = DataFrame.from_arrow(pa.table({"a": [1.0], "b": [2.0]}))

Testing.assert_frame_equal(left, right, atol=1e-5, rtol=0.0)

# Ignore row or column order when needed
Testing.assert_frame_equal(left, right, check_row_order=False, check_column_order=False)
```


## Why chalkdf?

- Built on Apache Arrow memory formats for zero‑copy interop.
- Runs on Chalk’s native engine for performance and portability.
- Small, predictable API surface suited for services and batch jobs.