jimmylomro/hercuvit-v0

HercuViT v0 — sheet instance segmentation on a self-supervised µCT encoder

HercuViT reads the papyrus sheet instances of carbonised Herculaneum scrolls straight out of a self-supervised Dinovol X-ray-µCT encoder. A lightweight CNN pixel-decoder learns a voxel-resolution upsample of the encoder tokens; a decoupled surface head answers is there a sheet; and a Mask2Former-style instance decoder answers which sheet. The sheet primitive is already present in the pretrained features — a small decoder sharpens it to the voxel medial. This is v0: the decoupled (no-null-query) model, stabilised with QK-LayerNorm in the mask-decoder cross-attention and LayerNorm on the pixel-decoder deconvolution blocks, with the encoder fine-tuned on a sigmoid learning-rate ramp.

• 📦 Code + docs: https://github.com/jimmylomro/hercUNet/tree/hercuvit · usage: docs/hercuvit.md
• 🗂️ Training corpus: https://huggingface.co/datasets/jimmylomro/hercunet-corpus
• 🧠 Encoder: scrollprize/dinovol_ps8_9.6um_101626 (the weights are baked into this checkpoint — see below)

This release: train_v5 checkpoint at epoch 462 (decoupled no-null head; QK-LayerNorm + deconv-block LayerNorm stability fixes; sigmoid encoder fine-tune schedule). The model is updated in place under this same jimmylomro/hercuvit-v0 path as better checkpoints are selected — --model-hf jimmylomro/hercuvit-v0 always pulls the current revision, so existing commands upgrade automatically; pin a specific --revision (a repo commit) if you need a fixed version.

Files (a single self-contained checkpoint)

Unlike the nnU-Net models, a HercuViT checkpoint is one .pth that carries everything needed to rebuild and run the network — no separate encoder download, no plans.json.

File

What

checkpoint_latest.pth

the released weights (train_v5 @ epoch 462) plus the Dinovol model_cfg and the training recipe. hercunet.hercuvit.netbuild.build_from_cfg reconstructs the exact architecture from the stored cfg and loads the weights — the fine-tuned encoder and the instance head (needed for the .asv mesh output) are both included, so nothing else is downloaded at inference. (--model-hf resolves this automatically.)

Use it

Install the HercuViT extra from the repo, then run inference straight from this repo (no local data or encoder needed — the model is pulled automatically; needs an NVIDIA GPU):

pip install -e ".[hercuvit]"                              # or: pipenv install ".[hercuvit]"
hercuvit infer single-instance PHerc1447 \                # SOURCE: scroll id, or an s3://|https://|file:// .zarr
  --model-hf jimmylomro/hercuvit-v0 \
  --region 11618:11810,3197:3389,3552:3744 \
  --mesh local --surf off --overlap 0 \
  --out ./hercuvit-demo                                   # -> ./hercuvit-demo_meshes.asv (the sheet store)

Outputs: a surface-probability OME-Zarr (--surf local) and/or per-window fitted sheet meshes in a compact .asv store (--mesh local) — a read-only SQLite of grid-implicit quad meshes in global L0 coordinates. To re-train or fine-tune from this model, pass it to hercuvit fit the same way. See docs/hercuvit.md.

⚠️ Inference needs the HercuViT code — not a stock predictor

HercuViT is a composite network: a Dinovol ViT encoder + a CNN pixel-decoder (voxel-resolution mask features) + a decoupled surface head + a Mask2Former-style ownership decoder, with per-window sheet-mesh fitting on top. The checkpoint is loadable by anyone, but running it requires the hercunet.hercuvit inference code (which builds the composite net, applies the surface gate, and fits the .asv meshes). There is no stock predictor that reproduces it.

⚠️ Faces vs medials

Like HercUNet, HercuViT predicts the medial crest of each sheet — a thin surface at the sheet's centre, a few voxels inboard of the faces that manual / m7-mined labels annotate. This is a deliberate, different target, not an error; we therefore report no Dice against face labels.

Recipe & provenance

Recipe hercunet/hercuvit/recipes/full_decoupled.yml (decoupled no-null head, U1@1/1 32-d mask features, 64 instance queries, QK-LayerNorm attention, sigmoid encoder fine-tune schedule). Methodology: the write-up in the repo (submission/writeup/hercuvit.md).

License: MIT — see LICENSE in the repository.