semvec/pypi-packages huggingface dataset:
tusk-cluster 0.1.0
Cluster mode plugin for Tusk - distributed queries with DataFusion
Requires Python: >=3.11
Keywords: cluster, datafusion, distributed, plugin, tusk

# tusk-cluster

Cluster mode plugin for Tusk - distributed queries with DataFusion and Arrow Flight.

## Installation

```bash
pip install tusk-cluster
```

Or for development:

```bash
cd plugins/tusk-cluster
pip install -e .
```

## Usage

### CLI Commands

```bash
# Start a local dev cluster (scheduler + workers)
tusk cluster-dev --workers 3

# Or start components separately:
tusk cluster-scheduler --port 8814
tusk cluster-worker --scheduler localhost:8814 --port 8815
```

### Web UI

Once installed, a "Cluster" tab will appear in Tusk Studio where you can:

- Connect to remote schedulers
- Start/stop local clusters
- Monitor workers and jobs
- Submit distributed queries

## Architecture

- **Scheduler**: Coordinates job distribution using Arrow Flight
- **Worker**: Executes queries using DataFusion
- **Communication**: Arrow Flight for efficient data transfer

## Requirements

- `tuskdata>=0.1.0`
- `datafusion>=43.0.0`
- `pyarrow>=18.0.0`