tree-distribution-shift 0.1.0
Tree Distribution Shift benchmark: export HF configs to COCO (train/id_test/ood_test + density buckets).
Requires Python: >=3.9

# tree-distribution-shift

A pip-installable benchmark for **Tree Distribution Shift** — export HuggingFace dataset configs to COCO format on disk.

**No git. No git-lfs. Works on clusters.**

Data lives on the HF Hub: [`aadityabuilds/tree-distribution-shift`](https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift)

## Getting started

```bash
pip install -U pip
pip install tree-distribution-shift

# See all available configs (queries HF Hub live)
tree-shift list

# Export a config to COCO format
tree-shift export --config intl_train_IN__ood_US --out ./data
```

That's it. No cloning, no LFS, no large downloads up front — data streams directly from the Hub.

### Authentication (recommended)

Set an HF token for higher rate limits and faster downloads:

```bash
export HF_TOKEN=hf_...   # recommended
```

Or log in once with the CLI:

```bash
huggingface-cli login
```

You can also pass a token per-command:

```bash
tree-shift list --token hf_...
tree-shift export --config intl_train_IN__ood_US --out ./data --token hf_...
```

### Export only the splits you need

```bash
tree-shift export \
  --config in_state_train_Karnataka__ood_Rajasthan \
  --out ./data \
  --splits train ood_test
```

### Pin a revision for reproducibility

For papers and exact replication, pin a commit hash:

```bash
tree-shift export \
  --config in_state_train_Karnataka__ood_Rajasthan \
  --out ./data \
  --revision abc123def456
```

### Python API

```python
from tree_shift import list_configs, export_coco

print(list_configs())

export_coco(
    config="in_state_train_Karnataka__ood_Rajasthan",
    out_root="./data",
    splits=["train", "id_test", "ood_test"],
    streaming=True,   # default; avoids OOM on low-mem machines
    revision=None,     # or pin a commit hash
)
```

## Output structure

```
./data/<config>/
  train/
    images/
      *.png
    annotations/
      instances_train.json
  id_test/
    images/
    annotations/
  ood_test/
    images/
    annotations/
  ood_same_density/
    images/
    annotations/
  ood_diff_density/
    images/
    annotations/
```

Annotation files follow the [COCO format](https://cocodataset.org/#format-data).

## Dataset contract

Every example row in the HF dataset contains:

| Column | Type | Description |
|--------|------|-------------|
| `image_id` | int | Unique image identifier |
| `filename` | str | Image filename (e.g. `00001.png`) |
| `width` | int | Image width in pixels |
| `height` | int | Image height in pixels |
| `image_bytes` | bytes | Raw image file contents |
| `coco_annotations` | str | JSON string — list of COCO annotation dicts |
| `coco_categories` | str | JSON string — list of COCO category dicts |
| `country` | str | Country code (e.g. `IN`, `US`) |
| `state` | str | State name |
| `zone` | str | Geographic zone |
| `biome` | str | Biome classification |
| `density_category` | str | Tree density bucket |

### Expected splits

- **`train`** — training set
- **`id_test`** — in-distribution test set
- **`ood_test`** — out-of-distribution test set
- **`ood_same_density`** — OOD test filtered to same density bucket as train
- **`ood_diff_density`** — OOD test filtered to different density bucket

> **Note:** Some older configs may not include the `ood_same_density` / `ood_diff_density` splits.

## Resume support

Exports are **resume-safe**. If a job is preempted or interrupted:

```bash
# Just re-run the same command — already-written images are skipped
tree-shift export --config intl_train_IN__ood_US --out ./data
```

Images that already exist on disk are not re-downloaded. The COCO annotation JSON is always regenerated to stay consistent with the full split.

## Rate limits & performance

- **Streaming is on by default** — data is fetched shard-by-shard, avoiding OOM and memory-map errors.
- **`--no-streaming`** downloads the full split into an Arrow table first. This can OOM on large splits or cause `mmap` errors on low-memory machines. Use only if you have plenty of RAM.
- For faster downloads (when available), set the environment variable:
  ```bash
  export HF_HUB_ENABLE_HF_TRANSFER=1
  ```
- Exports call `load_dataset` once per split (not per example), keeping HF API calls minimal.
- If you hit 429 rate-limit errors, wait a few minutes and re-run — resume will skip already-written images.

## Cache location (clusters/shared machines)

The exporter uses the standard HuggingFace cache. On clusters with limited home directory space, set a custom cache location on a large disk:

```bash
export HF_HOME=/path/to/big/disk/hf_cache
```

This prevents re-downloading and reduces load on the HF Hub.

## Diagnostic command

If exports fail or behave unexpectedly, run the doctor command and share the output:

```bash
tree-shift doctor
```

Output includes package versions, environment variables, and Python version — helpful for debugging "works on my machine" issues.

## Manifest file

Every export writes a `manifest.json` in the config output folder:

```json
{
  "repo_id": "aadityabuilds/tree-distribution-shift",
  "config": "intl_train_IN__ood_US",
  "revision": "abc123def456",
  "splits": ["train", "ood_test"],
  "timestamp": "2026-02-17T04:20:00+00:00",
  "counts": {
    "train": {"images": 11783, "annotations": 181208},
    "ood_test": {"images": 2345, "annotations": 36000}
  }
}
```

This is essential for reproducibility: it records the exact revision, splits exported, and counts per split. Include `manifest.json` when sharing or publishing data exports.

## API reference

### `list_configs(repo_id=..., revision=None) -> list[str]`

Returns the list of available dataset configs from the Hub. No data is downloaded.

### `export_coco(config, out_root, repo_id=..., splits=None, revision=None, streaming=True) -> Path`

Exports a config to COCO folders on disk.

| Parameter   | Default                          | Description                                    |
|-------------|----------------------------------|------------------------------------------------|
| `config`    | *(required)*                     | Name of the HF dataset config                  |
| `out_root`  | *(required)*                     | Root output directory                           |
| `splits`    | all 5 splits                     | Which splits to export                          |
| `streaming` | `True`                           | Stream data (recommended for low-mem machines)  |
| `revision`  | `None`                           | HF dataset revision / commit hash               |

## Dependencies

This package is intentionally lightweight:

- `datasets` — HF dataset loading
- `huggingface_hub` — Hub API access
- `orjson` — fast JSON serialization
- `tqdm` — progress display

No `torch`, `pycocotools`, or `opencv` required.

## Why this package?

- **No git / git-lfs** — works on clusters where git-lfs is painful.
- **Streaming by default** — avoids OOM and memory-map errors.
- **Resume-safe** — re-run after interruption without re-downloading.
- **Uses HF cache** — respects revisions, reproducible.
- **Thin client** — new configs on the Hub are visible immediately via `list_configs()` (no pip update needed).

## Versioning policy

The pip package contains tools only. Data versions are pinned with `--revision`.

- **0.1.x** — exporter stability, backward compatible CLI/API.
- **Minor bump** — when on-disk COCO layout or CLI flags change.
- **Major bump** — if split semantics change (unlikely).

New configs on the HF Hub are visible immediately via `tree-shift list`; no pip update required to access new data.

## Maintenance

- **Adding new configs/data** → push to the HF dataset repo only.
- **Package updates** → only needed for new CLI flags or exporter behavior changes.
- **Users see new configs automatically** via `tree-shift list`.

## Citation

If you use this dataset in research, please cite:

```bibtex
@misc{nalawade2025tree_distribution_shift,
  title={Tree Distribution Shift: A Benchmark for Out-of-Distribution Tree Detection},
  author={Nalawade, Aaditya},
  year={2025},
  howpublished={\url{https://huggingface.co/datasets/aadityabuilds/tree-distribution-shift}}
}
```

## License

MIT