kfkas/movinet-a0-streaming-event-detector

MoViNet-A0 Streaming Event Detector

Causal frame-level event detector for deciding when a streaming VLM should process a newly arriving frame. It uses MoViNet-A0 Stream at 2 FPS and emits one of three scheduler labels:

ID	Label	Meaning
0	no_change	The current semantic state continues.
1	visual_only	Camera, shot, fade, or zoom change.
2	semantic_update	Action, actor, object, or semantic state changes.

The PyTorch input is [batch, time, 3, 172, 172]. The repository includes the trained state dictionary, training metadata, source required to load it, and an FP16 ExecuTorch QNN artifact compiled for Snapdragon 8 Elite (SM8750).

The published checkpoint is the best epoch from a 20-epoch unfrozen-backbone run on the locally available Kinetics-GEBD subset. Validation accuracy was 0.601523 and macro-F1 was 0.511131. This is a research checkpoint rather than a full-corpus state-of-the-art claim.

import sys
from pathlib import Path

import torch
from huggingface_hub import snapshot_download

root = Path(snapshot_download("kfkas/movinet-a0-streaming-event-detector"))
sys.path.insert(0, str(root / "src"))

from vlm_eval.events.modeling import MoViNetStreamingEventDetector

model = MoViNetStreamingEventDetector(
    pretrained=False,
    freeze_backbone=False,
).eval()
model.load_state_dict(
    torch.load(root / "event_detector.pt", map_location="cpu", weights_only=True)
)
logits = model(torch.rand(1, 4, 3, 172, 172)).logits

The QNN graph has the fixed streaming ABI:

frame       [1, 3, 172, 172]
frame_state [1, 3, 3, 172, 172]
    -> logits     [1, 3]
    -> next_state [1, 3, 3, 172, 172]

frame_state contains the previous three RGB frames. This explicit fixed-shape contract is portable across ExecuTorch invocations; it is not MoViNet's variable collection of internal activation buffers.

On a Galaxy SM-S938N (SM8750), PyTorch-to-device parity was logits cosine 0.999988, logits max absolute error 0.000795, next-state cosine 0.999999, and matching argmax labels. QAIRT 2.37 delegates 2D compute through QNN HTP while unsupported 5D causal operations use ExecuTorch CPU fallback.

Training annotations are available at kfkas/streaming-gebd-causal, kfkas/kinetics-geb-plus-causal, and kfkas/epic-kitchens-100-causal. Those repositories contain metadata and source-video references, not redistributed videos.

Architecture and pretrained weights follow MoViNets and the MIT-licensed Atze00/MoViNet-pytorch port.