skblv/lemonfm-linear-probes-surgical-video

LemonFM linear probes — surgical video understanding benchmarks

Linear-probe heads trained on frozen LEMON (ConvNeXt-Large) features for the surgical video understanding leaderboard. One subfolder per benchmark, each containing probe_weights.pt, metrics.json, and the probe training curve.

Trained as baselines for the SDSC × Chicago Booth surgical video understanding leaderboard.

Protocol

Features are extracted from the frozen LemonFM ConvNeXt-Large encoder; a single linear layer is trained with BCE loss on the training split (seed 42). Decision rules: per-class threshold for multi-label instrument and anatomy tasks, grouped argmax for PitVQA (one phase + one step), plain argmax for the single-label action tasks. Training code: s56_lemonfm_probe.py (instruments) and s62_lemonfm_probe_new.py (newer domains).

The LemonFM encoder itself is not included; obtain it from its authors (Visurg AI) under their terms. This repository contains only the probe heads trained by us.

Results (full validation splits, 95% bootstrap CI)

Benchmark	Task	Exact match	Micro-F1
cholect50/	6-instrument multi-label	67.7% (64.7–70.7)	85.62% (84.16–87.13)
pitvis/	18-instrument multi-label	63.6% (60.3–66.7)	68.75% (65.65–71.63)
surgvu/	17-instrument multi-label	40.0% (37.0–43.1)	68.8% (66.79–70.77)
dsad/	12-structure multi-label anatomy	17.3% (15.7–19.1)	57.6% (56.2–59.1)
pitvqa/	phase + step recognition	63.3% (62.7–63.8)	76.1% (75.7–76.5)
sapbench/	5-way next action	46.2% (41.4–51.6)	= exact match
sarrarp50/	8-way suturing gesture	42.9% (39.3–47.0)	= exact match

For single-label tasks micro-F1 coincides with accuracy. Full comparisons on the leaderboard.

Limitations

Research baselines only. Not a medical device. Probe heads are only meaningful on top of the exact LemonFM encoder and preprocessing used in the training script.