VECTRA — causal video encoder checkpoints
Frozen DINOv3 ViT-L/16 dense patches → interleaved A-A-M temporal encoder (2 spatial self-attention layers over the 16×16 patch grid + 1 Mamba-3 layer running causally over time, per patch track) → 16×16 region-state grid per frame. Pretrained self-supervised with a region-transport appearance loss and a belief (uncertainty) loss; evaluated with frozen-encoder attentive probes.
Code: https://github.com/fw-ic/ARVR_videoencoder — README.md on main covers
the method, repository layout, environment setup, and how to run training and
evaluation. The full experimental record (per-run metrics, loss curves, ablation
ledgers, research notes, and the EXPERIMENT_LOG.md master index) is preserved
at the results-archive tag in that repository:
git fetch origin tag results-archive && git checkout results-archive
Checkpoints in this upload (2026-08-21)
All are full checkpoints: model_state_dict (student encoder + EMA teacher +
belief heads + frozen anchor pools) and optimizer_state_dict, so each can be
resumed or fine-tuned, not merely probed. Every run's exact hyperparameters are in
checkpoint["train_args"].
| file | size | architecture | pretraining | SSV2 | K400 |
|---|---|---|---|---|---|
grandv2_24L_epoch001.pt |
5.3 GB | 24L, 359M | 1 epoch of 788k mix (K400 200k + SSV2 200k + OpenVid 388k), belief v2 + A1 matchability | 51.42 | 75.99 |
depth42L_epoch001.pt |
9.0 GB | 42L, 609M | same data & recipe, 1 epoch | 39.31 ¹ | 74.48 |
paper_24L_1ep_transportonly.pt |
5.4 GB | 24L, 359M | 1 epoch, transport only (belief off) | 52.12 ² | 77.58 ² |
july_24L_5ep_quantile.pt |
5.0 GB | 24L, 359M | 5 epochs, older quantile-evidence belief | 47.28 ² | 76.82 ² |
belief_champion_6L.pt |
1.6 GB | 6L, 108M | 100k K400 × 3 epochs, belief v2 champion | 42.65 | — |
SSV2 / K400 numbers are top-1 on the full test split unless marked. ¹ Probe-budget confounded — see "Known caveats". ² 5k-subsample monitor protocol.
Evaluation protocol. Frozen encoder; a 3-block bidirectional self-attention + 1 cross-attention probe with factorized 3D RoPE is trained on the dataset's validation split and evaluated on test (the encoder was pretrained on the train splits, so both probe-training and evaluation data are unseen).
Which one do you want?
- Best encoder →
grandv2_24L_epoch001.pt. Everything current references it. - Reproducing the paper →
paper_24L_1ep_transportonly.pt. Note this is the 1-epoch transport-only model; the belief loss is off in it. - Belief / uncertainty work →
belief_champion_6L.pt. Small, and it is the checkpoint behind the calibration and split-conformal study. - Depth studies →
depth42L_epoch001.pt.
Known caveats — please read before quoting numbers
1. The paper checkpoint is transport-only. paper_24L_1ep_transportonly.pt
(52.12 / 77.58) was trained for 1 epoch with the belief loss disabled. The
5-epoch belief run is a different checkpoint that scored 47.28 on SSV2.
Earlier internal notes misattributed 52.12 to the 5-epoch run.
2. Depth did not help. 42L (609M) lost to 24L (359M) on both benchmarks at matched data and recipe. K400 is the clean comparison — the 42L probe there had more optimizer steps than the 24L reference and still scored 1.5 lower. The SSV2 gap of 12 points is not a clean measurement: that probe ran at global batch 128 (3,880 optimizer steps) against the reference's global batch 8 (61,960). Direction is trustworthy; magnitude is not.
3. Most probe numbers are undertrained floors. Probe schedule length alone moved Diving48 from 85.53 (20 epochs) to 91.88 (50 epochs), converged at neither. These encoders' probe curves are strongly back-loaded — treat every number here as a lower bound and never compare across probe budgets.
4. Optimization passes dominate architecture. The same 6L model scores 42.65 at 3 pretraining epochs and 51.10 at 20 — a larger gain than every loss-design improvement in the project combined. Numbers here reflect very few epochs.
5. Offline fixed-window encoder. Despite the causal Mamba backbone, these are trained and evaluated on fixed ~10 s windows at 5 fps with clip-normalized frame positions, and the probes read bidirectionally. They are not streaming models.
Loading
import torch
ckpt = torch.load("grandv2_24L_epoch001.pt", map_location="cpu", weights_only=False)
args = ckpt["train_args"] # every hyperparameter used
state = ckpt["model_state_dict"] # student_encoder.* / teacher_encoder.* / belief heads
Use download/vjepa_single_view_probe/model.py:build_frozen_region_encoder from
the code repo to rebuild the encoder from train_args and load the weights; it
reconstructs the exact architecture (layer count, region grid, residual mode,
register tokens) from the checkpoint rather than requiring you to specify it.
DINOv3 ViT-L/16 weights are not included — the encoder consumes frozen DINOv3 features and expects them supplied at load time.
Older files in this repo
Checkpoints uploaded before 2026-07-28 (under mixed_openvid/ and
k400_ssv2_400k_.../) are from earlier generations, including a 12-layer
d_state=256 era and the 5-epoch quantile run. They predate the belief-v2 recipe
and the region-grid-16 configuration; prefer the files listed above.