MCVO — Multi-frame, Camera-only Visual Odometry (self-supervised)

Follow-up research to a TU Munich master's thesis (Chair of Computer Vision, Prof. Cremers). A transformer that predicts relative camera pose from images alone, trained on ~80k frames of unlabeled video with no ground truth: pretrained depth (UniDepth) and optical-flow (UniMatch) networks supervise a flow-reprojection loss at training time only and are absent at inference.

Code, benchmark harness and reproduction instructions: https://github.com/kalman17/mcvo (mcvo/).

Model

• Frozen DINOv2-base backbone (86M) + 10 alternating temporal / spatial attention blocks with a per-frame camera token (d=640, 8 heads) → relative pose per adjacent frame pair + per-pixel uncertainty. 67M trained parameters, 154M total.
• Trained 6 epochs on 8-frame clips (RealEstate10K, YouTube-VOS, EpicKitchens, WalkingTours preprocessed with the AnyCam pipeline), batch 4, lr 1.5e-4, AnyCam-style uncertainty-weighted flow-reprojection loss. Single GPU (A40/H100 class), TUM SLURM cluster.
• Input: N frames, 336×336, RGB in [0,1]. Output: N−1 relative poses (cam_i → cam_{i+1}) and per-pixel uncertainty maps.

At a glance: accuracy vs. cost vs. supervision (NVIDIA A40, 4-frame windows, end-to-end per call)

Method	Labels	Params	Peak GPU mem	Latency	Rotation Sintel / TUM / KITTI	Heading KITTI (zero-shot)	Heading Sintel / TUM	Focal error Sintel / TUM / KITTI
This model (mcvo_e3_calib.pt: pose + intrinsics)	none	154 M (67 M trained)	0.69 GiB	75 ms	0.46° / 0.89° / 0.19°	7.0°	81° / 90°	21.8 % / 13.3 % / 42.4 %
π³	GT	959 M	5.5 GiB	171 ms	0.22° / 0.26° / 0.11°	2.2°	27° / 34°	25.2 % / 7.6 % / 28.9 %
VGGT-1B	GT	1257 M	7.0 GiB	203 ms	0.28° / 0.32° / 0.12°	4.6°	38° / 37°	34.0 % / 25.8 % / 37.1 %
Depth Anything 3 (Giant)	GT	1690 M	9.7 GiB	600 ms	0.19° / 0.27° / 0.09°	1.3°	19° / 32°	24.4 % / 4.6 % / 15.8 %
AnyCam (CVPR 2025)	none	115 M + depth/flow nets	3.7 GiB	413 ms	0.50° / 0.74° / 0.20°	28.6°	49° / 50°	70.3 % / 14.6 % / 66.9 %
Thesis pipeline (MCT + AnyCam)	none	460 M	5.0 GiB	820 ms	0.40° / 0.67° / 0.23°	28.2°	47° / 65°	20.7 % / 12.9 % / 20.4 %
Monodepth2 pose net (photometric, trained on KITTI)	none	13 M	0.09 GiB	13 ms	0.80° / 0.77° / 0.30°	1.2° (in-domain)	57° / 86°	—

5–14× lower peak memory and 2–8× lower latency than the billion-parameter supervised models, 5–11× below the self-supervised pipelines, no labels at any stage — at roughly twice their rotation error, competitive heading on driving video, near-chance heading on small-baseline indoor video. Not the cheapest learned pose model: Monodepth2's photometric pose net (13 M, 13 ms, 0.09 GiB; measured on identical windows) is 6× faster and 8× lighter and not far behind — worse rotation on Sintel/KITTI, better on TUM, better heading on Sintel and on KITTI where it was trained. Patch-based SLAM (DPVO) and classical ORB-SLAM are faster per frame with far better trajectories, given known intrinsics and (DPVO) GT-pose training. What this model occupies is the middle: transformer-class window rotation accuracy, image-only, no labels, at a small fraction of the cost of the models it approaches. Measurement: one process per model, same GPU, identical windows, CUDA-synchronised, warm-up excluded; experiments/bench_latency.py, honest_benchmarks/latency_summary.json in the GitHub repo.

Calibration head (new). mcvo_e3_calib.pt adds a 1.9k-parameter linear head on the per-frame camera token that predicts focal length and principal point; it was distilled, with everything else frozen, from the cached AnyCalib per-frame intrinsics that already serve as a training-time teacher — so the pose output is identical to mcvo_e3.pt and cost is unchanged. Focal error (median, square 336 crops): Sintel 21.8 %, TUM-RGBD 13.3 %, KITTI 42.4 % — teacher level (AnyCalib: 20.1 / 11.2 / 18.4 %) on the kind of footage it was trained on, clearly worse than the teacher on KITTI's narrow-FOV driving crops, which lie outside the training corpus' focal range. A joint fine-tune (one epoch) gave 27.3 / 10.2 / 34.4 % and slightly disturbed pose; not released.

Limitations, stated plainly. Translation direction on small-baseline indoor video (Sintel, TUM-RGBD) is near chance and did not improve under longer context, teacher distillation, an epipolar consistency loss, or motion-rich additional training data; it is treated as a structural limit of flow-reprojection self-supervision when parallax is tiny. Trajectory-level ATE trails AnyCam's long-context inference (Sintel 0.18 vs 0.10). Intrinsics come from a distilled head that is teacher-level in-domain and weak on narrow-FOV driving crops; for calibration-critical use pair with thekman17/anycam-mct or AnyCalib.

Files

• mcvo_e3_calib.pt — recommended: E3 weights + calibration head (pose identical to mcvo_e3.pt, plus intrinsics)
• mcvo_e3.pt — pose only; weights (model_state_dict) + training args; load with mcvo.model.MCVO(**args) from the GitHub repo, see experiments/honest_benchmark.py::MCVOModel.

Attribution

Builds on AnyCam (Wimbauer et al., CVPR 2025) for the self-supervised loss and data pipeline, DINOv2 (Meta) as backbone, UniDepth and UniMatch as training-time teachers. Architecture follows the time-space attention design of FVO (Yugay et al., 2025), trained self-supervised instead of with ground-truth poses. Apache-2.0.