Sparse attention for DVLT
Weights for a DVLT variant whose global attention is restricted to a small set of candidates per patch, taken from mutual nearest neighbours between DINOv2 patch tokens. Code: https://github.com/mpilligua/mVGGT/tree/minimal
Two files, one per training stage.
dino_encoder.pt
DINOv2 ViT-B/14 finetuned contrastively so its patch tokens match across views. Pairs are sampled inside a covisibility band measured per dataset rather than taken in order. 86.6M parameters, step 34159.
Mutual-nearest-neighbour matching on the validation split, F1@1 averaged over the 23 datasets that are trained on (Waymo is in the val config but is not one of them):
| encoder | F1@1 |
|---|---|
| DINOv2 off the shelf | 25.1% |
| contrastive finetune, pairs in order | 42.7% |
| this one, pairs sampled per-dataset band | 53.6% |
dvlt_sparse.pt
DVLT trained on top of that index, encoder unfrozen, 117.1M parameters, step 27014 of an unfinished run. Validation over 23 datasets, this time with Waymo in and Spring out: Spring has so little parallax that its AUC moves by 0.15 between identical passes.
| dense DVLT | sparse, frozen encoder | this one | |
|---|---|---|---|
| AUC@5 | 0.6776 | 0.3836 | 0.3955 |
| AUC@30 | 0.8812 | 0.7751 | 0.7862 |
| rotation error (deg) | 0.4673 | 1.5232 | 1.5175 |
| translation error | 0.9697 | 2.9959 | 3.2167 |
| abs_rel | 0.0414 | 0.0586 | 0.0533 |
| rmse | 0.1284 | 0.1261 | 0.1182 |
| delta < 1.25 | 0.9437 | 0.9241 | 0.9291 |
| MVS chamfer | 0.0323 | 0.0545 | 0.0506 |
Depth is close to the dense model and rmse is already better than it. Pose is not: the gap widens as the threshold tightens, from 89% of dense at AUC@30 to 57% at AUC@5, which is what a matching index at 14-pixel granularity would be expected to do.
These weights come from a run that is still training, so treat them as a snapshot rather than a final result.
Reproduce the table with the code repo:
python test.py dvlt dvlt_sparse.pt --scenes 15 --batches 40
Use
import torch
sd = torch.load("dvlt_sparse.pt", map_location="cpu")["model"]
model.load_state_dict(sd)
License
Derived from the nvidia/dvlt checkpoint and therefore covered by the NVIDIA
License, included here as NVIDIA-LICENSE.txt. Non-commercial, research and
evaluation only.