nct-tso/VG-Track

Verifier-Guided LiteTracker for STIR 2026

This is the NCT_TSO checkpoint for streaming tissue point tracking in the STIR Challenge 2026. It is a CoTracker3-Online-compatible state dictionary and loads directly into the LiteTracker streaming runtime. The model was fine-tuned with confidence-weighted pseudo-labels fused from six teacher trackers.

The implementation is developed from CoTracker3, which supplies the student architecture and training loss, and uses LiteTracker for causal streaming inference. The release repository adds the STIR-specific multi-teacher verification, pseudo-label fusion, fine-tuning, and evaluation pipeline; it does not vendor CoTracker3 source.

Model details

• Architecture: CoTracker3-Online / LiteTracker-compatible, 25,385,700 parameters
• Checkpoint: student.pth
• Input: RGB video and frame-0 query points in (x, y) pixel coordinates
• Output: per-frame 2D trajectories and visibility estimates
• Window length during training: 16
• Training refinement iterations: 4
• SHA-256: 3b964f18793bab959d1537aa58b0411bb8454e2df5f30bf6d66039a215c8455c
• Code and reproducibility: https://github.com/danushkv/STIR2026_challenge
• Authors: Danush Kumar Venkatesh, Peng Liu, Stefanie Speidel

Intended use

The checkpoint is intended for non-commercial research on streaming point tracking in laparoscopic video, reproduction of the STIR 2026 results, and fine-tuning or benchmarking under the documented protocol. It is not a medical device and must not be used for clinical decisions or autonomous intervention.

Training data and procedure

Training used STIROrig together with STIR Challenge 2024 through a combined, non-colliding patient layout. Six offline teachers (alltracker, cotracker3, locotrack, mft, trackon2, and trackon_r) produced trajectories at a temporal stride of 5. A verifier combined endpoint supervision, cycle consistency, and cross-teacher agreement into per-frame confidence weights.

Evaluation

Evaluation used 32 clips (234 annotated points) from the STIR 2025 test collection under the LiteTracker streaming runtime.

track	refinement iterations	metric	score
2D accuracy	4	delta average at 4/8/16/32/64 px	0.8103
2D latency	1	delta average	0.8085
2D latency	1	p95 latency, RTX A5000	48.7 ms
3D submission-like	4	accuracy at 2/4/8/16/32 mm	0.7031

The offline reproduction protocol uses annotation-derived right-view starting points unavailable to the submission harness and therefore reports a higher 3D score. See docs/CAVEATS.md in the code repository before comparing numbers.

Limitations

• Ground truth exists only at clip endpoints; intermediate trajectory quality and visibility are not directly supervised.
• The training run is seeded but not bit-reproducible because deterministic GPU algorithms and DataLoader worker seeding were not enforced.
• Training includes STIR Challenge 2024 patients that overlap the 2025 test collection at patient level.
• Performance outside laparoscopic tissue video, on unseen devices, or under different image statistics has not been established.
• Inference refinement iterations affect accuracy and latency; benchmark 1, 2, and 4 rather than assuming one operating point.

Licence

The checkpoint is released under CC BY-NC 4.0 because it descends from CoTracker3/LiteTracker weights. The accompanying code is separately MIT licensed. Users must also follow the licences and terms of all upstream models and datasets.

Citation

@software{venkatesh2026stir,
  author = {Venkatesh, Danush Kumar and Liu, Peng and Speidel, Stefanie},
  title = {Verifier-Guided Multi-Teacher Distillation for Streaming Tissue Tracking},
  year = {2026},
  url = {https://github.com/danushkv/STIR2026_challenge}
}

Acknowledgements

We thank the authors of CoTracker3, LiteTracker, Track-On/Track-On2/Track-On-R, AllTracker, LocoTrack, MFT, STIRLoader, and STIRMetrics, as well as the STIR Challenge organisers and dataset contributors. The full linked acknowledgement list is maintained in the code repository README.