Spotter

Spotter is the model of Trainspotting: Learning Object-Centric 3D Reconstruction from Found Photos (Hanyu Chen and Noah Snavely, Cornell University): the released 1B-parameter VGGT-Ω checkpoint fine-tuned on the Trainspotting dataset of found photos of objects from Wikimedia Commons, plus an object mask head. Given a set of photos of one object taken by different people at different times and places, it predicts the camera of every photo, a depth map and the mask of the object.

• Code, demo and evaluation protocol: https://github.com/hanyucc/trainspotting
• Dataset: https://huggingface.co/datasets/hanyucc/trainspotting

Usage

pip install git+https://github.com/hanyucc/trainspotting.git

import torch
from spotter import load_spotter
from spotter.utils.load_fn import load_and_preprocess_images

model = load_spotter(device="cuda")                 # downloads spotter.safetensors from this repository
images = load_and_preprocess_images(["a.jpg", "b.jpg", "c.jpg"]).cuda()
with torch.inference_mode():
    pred = model(images)
pred["extrinsics"], pred["intrinsics"], pred["depth"], pred["object_mask"]

Files

spotter.safetensors (4.6 GB, float32): the full model, 1.15B parameters, loading strictly into spotter.Spotter.

Results

Held-out Trainspotting objects (1,635 samples of 4 or 16 views): camera AUC@30 81.2 (VGGT-Ω: 25.0), depth AbsRel 0.071 (0.201), point-map completeness 1.15% of the object extent (12.68%). NAVI in-the-wild: AUC@30 76.0 (18.8). See the paper and the code repository for the protocol and per-sample results.

License

The weights are a derivative of VGGT-Ω and are distributed under the FAIR Noncommercial Research License (see LICENSE): noncommercial research use only.