SAM-VTP

Weights for SAM-VTP, a continuous traversability segmentation model for RGB-D video. It adapts SAM 2's video memory for prompt-free prediction and fuses geometry measured from depth, producing a per-pixel score in [0,1] at each frame.

• Code: lucashoefs/sam_vtp
• Robot deployment: lucashoefs/sam_vtp_jetacker

Files

File	Description
sam_vtp.pt	Memory, geometry and refinement enabled. Hiera-T encoder, 41.2 M parameters, 32-bit weights
sam_vtp.pt.sha256	SHA-256 checksum

This checkpoint is training seed 42, selected at epoch 286 by validation fuzzy Jaccard. It is the checkpoint used in the robot trials. The embedded configuration has been reduced to architecture and training settings; the weights are unchanged.

Usage

Install the code following its README, then:

hf download lucashoefs/sam-vtp sam_vtp.pt --local-dir .

import torch
from sam_vtp import CameraIntrinsics, build_model

cam = CameraIntrinsics.from_K(K, width=848, height=480)  # native resolution
model = build_model("sam_vtp.pt", device=torch.device("cuda"), camera=cam)

model.reset_memory()
for t, (rgb, depth) in enumerate(stream):
    out = model.forward_frame(
        rgb,                     # (1,3,H,W), RGB in [0,255]
        is_init_frame=(t == 0),
        depth=depth,             # (1,1,H,W), planar z-depth in metres
    )
    traversability = out["logits"].sigmoid()  # (1,1,H,W)

Inputs must be metric depth and your own camera intrinsics. Wrong values produce a plausible but incorrect map, with no error. Frames must be in capture order.

Training

• Initialisation: SAM 2.1 Hiera-T
• Data: TartanGround, with continuous targets generated from simulator depth and semantics
• Input size: 512x512, sequences of 24 frames, 7-frame memory
• Budget: 300 epochs of 1,000 sequences (7.2 M frames)

Citation

TBD

License

Apache 2.0. These weights are fine-tuned from SAM 2.1 Hiera-T, Copyright (c) Meta Platforms, Inc. and affiliates, under Apache 2.0. Training data: TartanGround (Patel et al., 2025), under CC BY 4.0.