jimmylomro/hercuvit-stitcher

HercuViT Stitcher — learned cross-window instance stitching

HercuViT segments a scroll window by window, so a sheet that spans several windows comes out as several disjoint meshes with unrelated per-window ids. This model is the cross-window stitcher that reconnects them: a small DETR-style decoder reads the per-window embeddings HercuViT stores and predicts a signed affinity between every pair of sheets that meet at a window corner — no hand-crafted geometric matching (no IoU / surface distance), and no manual annotation. The affinities are written into the .asv as an edge table; a signed-graph agglomeration (GASP) then assembles whole, scroll-spanning sheets — in batch, or interactively in VC3D (affinity colouring + auto-stitch).

The method is deliberately primitive-agnostic (any instances + an embedding field in the same space); this checkpoint is trained to stitch HercuViT sheets.

• 📦 Code + docs: https://github.com/jimmylomro/hercUNet/tree/hercuvit · usage: docs/stitcher.md
• 🧠 Companion segmenter: jimmylomro/hercuvit-v0 — produces the sheets and the embeddings this stitcher consumes (one HercuViT forward emits both).
• 🗂️ Corpus manifest: submission/corpus/stitch_corpus_windows.txt — the 22,619 region-sampled corner coordinates the self-supervised samples are extracted at (see submission/corpus/README.md)
• 📄 Methodology: the write-up submission/writeup/hercuvit-stitcher.md

This release: trained on ~22.6k self-supervised corner samples (per corner: the 8 octant windows + a centred pass, overlap-voted into a same-object label matrix — no ground truth), with the v2 label mode (certain negatives — same-window and off-seam pairs — marked as negatives rather than ignored, so the matrix is fully labelled), a deep-supervised focal-BCE + supervised-contrastive loss, over D = 3 per-window normalised-PCA embeddings. The model is updated in place under this jimmylomro/hercuvit-stitcher path as better checkpoints are selected — --stitcher hf://jimmylomro/hercuvit-stitcher always pulls the current revision; pin a specific --revision (a repo commit) for a fixed version.

Files (a single self-contained checkpoint)

A stitcher checkpoint is one .pth that carries both the weights and the config needed to rebuild the decoder — no separate downloads.

File	What
checkpoint_best.pth	the released weights plus the StitchConfig (architecture + the training schedule it was trained under). hercunet.hercuvit.stitch.infer.load_stitcher rebuilds the exact StitchDecoder from the stored config and loads the weights. (A directory resolves checkpoint_best → final → latest.)

Use it

The stitcher consumes HercuViT's stored embeddings, not raw CT. Two ways to run it:

1. Integrated — one command (hercuvit infer --stitcher): segment, store embeddings, and stitch in the leader process, writing one .asv with both the sheets and the affinities. A bare --stitcher pulls this model:

pip install -e ".[hercuvit]"                               # or: pipenv install ".[hercuvit]"
hercuvit infer single-instance PHerc1447 \                 # SOURCE: scroll id, or an s3://|https://|file:// .zarr
  --model-hf jimmylomro/hercuvit-v0 \
  --region 11618:11810,3197:3389,3552:3744 \
  --mesh local --overlap 0.1 \
  --stitcher \                                             # bare = hf://jimmylomro/hercuvit-stitcher
  --out ./demo
# -> ./demo_stitched-meshes.asv   (sheets + affinities, ready for VC3D)

2. Standalone (hercuvit stitcher infer): over the artifacts a previous hercuvit infer --store-embeddings run left behind (the per-window embedding zarr + each sheet's median embedding in the .asv):

hercuvit stitcher infer \
  --embeddings-zarr ./demo_emb.zarr --asv ./demo_meshes.asv \
  --stitcher hf://jimmylomro/hercuvit-stitcher            # adds the `affinities` table into the .asv

Both default --stitcher to hf://jimmylomro/hercuvit-stitcher, resolved like every other model (hf://<repo> | file://<path> | a local .pth/dir). The --emb-dim used when storing embeddings must match this model's pca_dim (3). Stitching is fast — the decoder is tiny and corner-batched, so a whole volume stitches in seconds on one device (CPU or GPU).

⚠️ Inference needs the HercuViT code — and HercuViT-stored embeddings

The checkpoint is loadable by anyone, but running it needs the hercunet.hercuvit code (to assemble each window-corner sample and write the affinities) and the two artifacts a HercuViT run stores: the pooled per-window embedding zarr and each sheet's median embedding in the .asv. It does not run on raw CT, and there is no stock predictor that reproduces it.

What it writes

A signed affinities(src, dst, logit, prob, corner) edge table inside the .asv — both directions stored and indexed on src, so every neighbour of a sheet is a single lookup. VC3D's affinity colouring + auto-stitch read this table directly; a signed-graph agglomeration (GASP) over it assembles the final scroll-spanning sheets. The original sheet meshes are never rewritten.

Recipe & provenance

StitchConfig: pca_dim 3, d_model 128, heads 4, layers 4 (deep-supervised), fourier_bands 8; loss = focal-BCE (γ 2, auto class balance) + supervised-contrastive (τ 0.1); AdamW, base-lr 1e-4 cosine. Full method: docs/stitcher.md (CLI) and the write-up submission/writeup/hercuvit-stitcher.md (primitive-agnostic method).

License: MIT — see LICENSE in the repository.