lma-project/selectstream-lite-ego4d

selectstream-lite: Ego4D checkpoints (run full_v3)

Trainable parts only: segment encoder, gated writer, graph reader (DMG/GAR), evidence injector, plus the write-signal scorer. The Qwen2.5-VL-7B backbone is frozen and not included; load it from Qwen/Qwen2.5-VL-7B-Instruct. All runs: 4 × RTX 2080 Ti, fp16 backbone, one seed (42).

Code: branch ego4d-pipeline of the selectstream-lite repo. These checkpoints need the retrieval changes made after commit f7aa8cd (learnable retrieval temperature, zero-shot question prior, learnable score biases, centred segment-encoder input). [[OWNER: replace with the commit that has them once pushed]]

Layout

Path	What
checkpoints/full_v3/R_A/weights.pt	Stage R: reader + writer, arm A (surprise), 2 epochs, 419 steps, 838 train streams
checkpoints/full_v3/R_A/segment_center.npy	train-mean frame feature subtracted before the segment encoder (also stored inside the weights)
checkpoints/full_v3/W_{A,B,C,D}/weights.pt	Stage W: writer only, 50 steps × 4 streams from R_A, one per write arm
checkpoints/full/R_A/weights.pt	reference: the paper's reader as written (run 43319), before the retrieval fixes
runs/full_v3/scorer/scorer.pt	learned importance scorer (arm B's write signal)
runs/full_v3/calibration.json	per-arm write budget, matched to arm A's segment count
runs/full_v3/eval_*, read_*, q2_*	summary.json + predictions.jsonl per evaluation / ablation
runs/full_v3/{R_A,W_*}/	training logs, config, figures
runs/full/, runs/full_v2/	earlier runs: Check 1/2 (scorer, write replay), diagnostics, Stage R of the paper reader
runs/full_ablate_offline/	offline write-side ablations
runs/pilots/, runs/diagnostics/	pilot outputs (smoke tests, not results); probe scripts
config/ego4d_qwen25vl7b.yaml	the run config
ABLATIONS.md	pending ablations and how to run them on Colab

Write arms: A smoothed surprise (paper), B learned scorer, C oracle labels (upper bound), D fixed-length segments (L = 24).

Feature cache (frozen per-frame features, so the vision encoder never runs while streaming): dataset repo amritha27/selectstream-lite-ego4d-features, g/<clip_uid>.npy [frames, 3584] fp32 + meta/. The PNG frames are not uploaded; the code decodes the current frame from your own clips (--video_dir).

Results so far (val, single run)

QA: 243 streams, 3,751 questions. NLQ: 300 streams, 4,552 questions. Paired differences: 95% bootstrap over videos.

	track	Recall@M	best IoU	contains
A	QA	0.518	0.192	0.220
B	QA	0.465	0.168	0.220
A	NLQ	0.522	0.220	—
B	NLQ	0.473	0.190	—
untrained	QA	0.484	0.195	0.043
B − A	QA	−0.053 [−0.074, −0.034]	−0.024 [−0.033, −0.016]	+0.000 [−0.002, +0.001]

Known issue: QA answers barely depend on the retrieved evidence (A and B give the same prediction on 99.4% of questions; contains-match 0.215 when evidence hits the window vs 0.226 when it misses). Treat answer metrics with care until the memory-off / oracle-evidence ablations below are in.

Ablations

Status as of 2026-10-29 12:46 IST. ✅ done · ⏳ running/queued · ⬜ not queued (conditional).

Write side (Q1: learned importance vs surprise)

ID	Varies	Status	Result
AB-B1	write signal A/B/C/D × N ∈ {8, 16, 32, 64}, offline replay	✅	C − A +0.006 [−0.006, +0.018] IoU at N = 16; B − A −0.03
AB-B2/B3	scorer width; scorer vs surprise as a frame predictor	✅	within-video AUC ≈ 0.55 for every signal
AB-G2	minimum segment length L_min ∈ {2, 4, 8}, offline	✅	shorter hurts every arm (A 0.236 → 0.175)
AB-L1/L2	label tolerance; within-video scorer losses	✅	AUC up (0.577–0.592), IoU still below surprise
AB-I1/I2	scorer inputs; surprise + scorer combined	✅	g-only = full; combination no gain
AB-U1	which write decisions take the learned signal	✅	cut by surprise, value by scorer: +0.004 [+0.002, +0.007]
AB-G1	causal event-boundary score as the cut signal	✅	−0.003 [−0.009, +0.002] vs surprise
main	Stage W + eval, arms A/B/C/D, both tracks	⏳ A, B ✅; C, D running	table above
AB-T1	untrained vs trained	✅	Recall@M +0.034; contains +0.178
AB-W1	arm B with Stage-R writer vs retrained writer	⏳
AB-N1	eval-time memory budget N ∈ {16, 64, 128}, 14 longest streams	⏳
AB-W2	Stage W steps 50 vs 200	⬜ only if AB-W1 shows little change
AB-R1	2 more seeds of W_A, W_B	⬜ only if A − B is within ~1 CI
AB-P*	full-model run of offline winners	⬜ none promoted (+0.004 too small)

Read side and query conditioning (trained W_A, 150 val QA streams, paired by question)

ID	Variant	Status
RD-S1..5	score terms: no prior / no cosine / no biases / cosine only / prior only	⏳
RD-G1..3	no neighbourhood hops / no GAR layers / coarse ranking only	⏳
RD-B1/B2	random evidence / oracle evidence	⏳
RD-M	evidence count M ∈ {0 (no memory), 1, 2, 8}; trained at 4	⏳
RD-O	committed nodes only (no open segment)	⏳
QC-1	question over a blank frame	⏳
QC-2	zero-shot prior from question / object phrase / with stopwords / uncentred (offline)	⏳
RD-off	training-free retrieval grid × M × N (offline)	⏳

Q2 (composite questions)

ID	Varies	Status
AB-Q1	single vs decomposed (matched budget) vs unmatched, k ∈ {2, 3}, arms A and B, both tracks	⏳

Caveats

• One seed for every stage; CIs are over videos, not training runs.
• Training-time metrics in logs/ are on the streams being fitted.
• Recall@M counts any overlap; nodes span 120–150 s against ~4 s answer windows, so precision is ~0.03.