annpack 0.1.7
ANNPack builder and tools
Requires Python: >=3.10

# ANNPack

Serverless vector search: static `.annpack` files served over HTTP Range, searched
in the browser via WASM + Transformers.js.

## Badges
[![PyPI version](https://img.shields.io/pypi/v/annpack.svg)](https://pypi.org/project/annpack/)
[![CI](https://github.com/Arjun2729/ANNPACK/actions/workflows/ci.yml/badge.svg)](https://github.com/Arjun2729/ANNPACK/actions/workflows/ci.yml)

## TL;DR / What it is
- A portable, immutable ANN index format plus tooling for build + serve + search.
- Designed for low-ops distribution (CDN/S3/edge) and in-browser query.
- Ships a Python CLI/API and a WASM runtime for HTTP Range search.

Alpha / security posture: ANNPack is in alpha. Security hardening is in progress;
see `SECURITY.md` for current defaults, limits, and runtime caps.

v0.1.5 highlights:
- Deterministic tiny-dataset fallback for centroid training and pinned FAISS
  threads to stabilize builds.
- Hardened registry defaults (dev mode opt-in, JWT required in prod),
  upload/body limits, rate limiting, and path traversal protection.
- C core overflow/alloc guards plus Python metadata size caps
  (`load_meta=False` support), with new security workflows/docs.

## When to use / When not to use
Use it when:
- You want static, cacheable search artifacts you can host anywhere.
- You need reproducible snapshots and lightweight, client-side querying.

Avoid it when:
- You need mutable, transactional, or real-time index updates.
- You want a managed vector database with server-side filtering and writes.

## Quickstart (CLI)
```bash
pip install annpack
ANNPACK_OFFLINE=1 annpack build --input tiny_docs.csv --text-col text --output ./out/tiny --lists 256
annpack serve ./out/tiny --port 8000
annpack smoke ./out/tiny --port 8000
```
Expected output includes `PASS smoke`.

## Installation
```bash
pip install annpack
```
Optional extras:
- `pip install annpack[embed]` for real embeddings (torch + sentence-transformers)
- `pip install annpack[registry]` for the local PackHub service

## Templates (plug-and-play)
```bash
annpack templates
annpack init --template text-search --out ./my-pack
```

## Concepts
- A **pack** is an immutable index directory containing:
  - `*.annpack` (binary index)
  - `*.meta.jsonl` (metadata rows)
  - `*.manifest.json` (shard list; used by the UI)
- **PackSet** supports base + delta packs with tombstones for append-only updates.
- Components:
  - Builder (Python CLI): `annpack build`
  - Runtime (C/WASM): `ann_load_index`, `ann_search` over HTTP Range
  - Frontend (JS): Transformers.js embeddings + WASM ANN search

## Offline mode & determinism
Set `ANNPACK_OFFLINE=1` to use deterministic dummy embeddings (no model
downloads). For real embeddings, install `annpack[embed]` and unset
`ANNPACK_OFFLINE`.

Determinism notes:
- Manifests/meta are deterministic and clustering is seeded.
- Embeddings can vary by device/backend; for strict reproducibility, set
  `ANNPACK_DEVICE=cpu` during builds.

## Demos
10-minute demo (offline, deterministic):
```bash
bash examples/quickstart_10min.sh
```
Expected output includes:
- `PASS smoke`
- `READY: open http://127.0.0.1:<port>/`

Golden demo launcher (build + serve + UI command):
```bash
bash tools/run_demo.sh
```

Optional medium demo assets (downloaded, not in git):
```bash
bash tools/download_demo_assets.sh ./demo_assets
```

## Troubleshooting
- Port in use: pass `--port <free-port>` to `serve`/`smoke`.
- Missing manifest: ensure the output dir contains `*.manifest.json`.
- CORS: `annpack serve` enables permissive CORS headers by default.
- Offline/air-gapped builds: set `ANNPACK_OFFLINE=1` (no embed deps required).
- Real embeddings: install `annpack[embed]` and unset `ANNPACK_OFFLINE`.
- Small datasets: `annpack build` clamps `--lists` to vector count.
- macOS quarantine (console scripts):
  ```bash
  xattr -dr com.apple.quarantine "$(python -c 'import site; print(site.getsitepackages()[0])')"
  ```
- Localhost bind restrictions: set `ANNPACK_SKIP_NET_TESTS=1` to skip network
  smoke in `stage_all.sh`.
- Avoid venvs named `#` (shell treats `#` as a comment).

## Verification / acceptance scripts
Stage 1 acceptance (build + smoke):
```bash
bash tools/stage1_acceptance.sh
```
Expected last line: `PASS stage1 acceptance`.

Stage 2 acceptance (API + determinism):
```bash
bash tools/stage2_acceptance.sh
```
Expected last line: `PASS stage2 acceptance`.

Stage 3: Delta packs (PackSet):
```bash
annpack packset build-base --input tiny_docs.csv --packset ./packset --text-col text --id-col id --lists 4 --seed 123 --offline
annpack packset build-delta --base ./packset/base --add delta_add.csv --delete-ids 1 --out ./packset/deltas/0001.delta --packset ./packset --text-col text --id-col id --lists 4 --seed 123 --offline
annpack packset status --packset ./packset
```

Stage 4 acceptance (distribution + portability):
```bash
bash tools/stage4_acceptance.sh
```
Expected last line: `PASS stage4 acceptance`.

Pre-talk gates (clean repo + clean checkout):
```bash
bash tools/repo_hygiene.sh
bash tools/clean_checkout_gate.sh
```

## Repo layout
- `python/annpack/`: primary Python package and CLI.
- `registry/`: local FastAPI-based pack registry with Range support
  (`registry/README.md`).
- `web/`: client SDK (`web/packages/client`) and UI app (`web/apps/ui`).
- `docs/`: architecture, API, CLI usage, and notes.
- `annpack-v2/`: experimental WASM demo tooling; treat as legacy.
- Legacy notes:
  - `build_fast.py` is legacy; use `annpack build` with MiniLM instead.
  - `ann_query_bytes` in `main.c` is retained for debugging.

## Docs
- `docs/ARCHITECTURE.md`
- `docs/API_USAGE.md`
- `docs/CLI_USAGE.md`
- `docs/WASM.md`
- `docs/POSITIONING.md`
- `docs/AUDIENCE_FRONTEND.md`
- `docs/AUDIENCE_ML_INFRA.md`
- `docs/VERIFY.md`
- `docs/BENCHMARKS.md`
- `docs/benchmarks/README.md`
- `CODE_OF_CONDUCT.md`
- `SUPPORT.md`
- `SECURITY.md`
- `RELEASE.md`
- `docs/PRODUCTION.md`
- `docs/COMPATIBILITY.md`
- `docs/ONE_PAGER.md`
- `docs/DISCOVERY_QUESTIONS.md`
- `docs/DEMO_SCRIPT.md`

## File format
ANNPack is a portable binary format for IVF-based ANN search. The current spec
is in `docs/FORMAT.md`.

Format summary (unchanged):
- 72-byte header: magic, version, endian, header_size, dim, metric, n_lists,
  n_vectors, offset_table_ptr.
- Centroids (float32), then IVF lists:
  `[count:u32][ids:int64*count][vecs:float16*dim*count]`, then an offset table
  `[offset:u64,length:u64]` per list.

Pure Python reader/searcher example:
```python
import numpy as np
from annpack.reader import ANNPackIndex

with ANNPackIndex.open("my_index.annpack") as idx:
    dim = idx.header.dim
    q = np.random.randn(dim).astype(np.float32)
    q /= np.linalg.norm(q)
    hits = idx.search(q, k=10)
    print(hits)
```
Other languages can implement a reader using `docs/FORMAT.md`; the C/WASM runtime
is one consumer.

## License / Contributing / Security
- License: `LICENSE`
- Contributing: `CONTRIBUTING.md`
- Security: `SECURITY.md`