shubhxho/video-benchmark-compact-quality

Compact Operator-Video Quality Model

Rate the quality of first-person / operator video (headband cameras, robotics teleop) — one tiny model, one forward pass, on-device.

Scoring a frame the "proper" way means running a whole stack of models: a learned image-quality ensemble, a CLIP scene classifier, a pile of OpenCV metrics. That's accurate, but it's hundreds of milliseconds and hundreds of megabytes per frame — you can't ship it to a headset or run it live.

So this model learns to imitate that stack. It watches the heavy pipeline label a big corpus of real operator footage, then compresses everything it learned into a single 2.5M-parameter network. You get all the same per-frame quality signals at once, on a plain 0–100 scale (higher is better) — fast enough for real time and small enough to embed.

At a glance

Size	~1.4 MB int4 · ~2.6 MB int8 (fp16 reference ~4.9 MB)
Params	2.5M, one shared backbone + trunk
Speed	a single forward pass replaces the whole teacher stack
Outputs	8 quality signals, each 0–100 (higher is better)
Runtimes	PyTorch, GGUF (candle / Rust), MLX (Apple Silicon)

Try it in 30 seconds

import cv2
from huggingface_hub import hf_hub_download
# grab the model code (video_benchmark.distill) from https://github.com/shubhxho/video-benchmark
from video_benchmark.distill.infer import CompactQualityModel

ckpt = hf_hub_download("shubhxho/video-benchmark-compact-quality", "compact_quality.pt")
model = CompactQualityModel(ckpt)        # picks CUDA / Apple MPS / CPU automatically

frame = cv2.imread("frame.jpg")          # or any decoded video frame (BGR)
print(model.predict(frame))              # {'brightness': ..., 'iqa': ..., 'scene': ...}

Got a batch of frames? Call model.predict_batch([f1, f2, ...]) — it runs them together and, on a GPU, in fp16 for a free speedup.

What you get back

brightness, sharpness, blur, anomaly, iqa, musiq, clipiqa, scene

Two kinds of number live in there, and it's worth being honest about the difference:

• The learned signals (iqa, musiq, clipiqa, scene) are the whole point — perceptual quality and scene judgements that genuinely need a model. This is what the student reproduces.
• The rest are exact OpenCV statistics (brightness, sharpness, and friends). They're bundled so you get one tidy read-out, but if that's all you need, computing them directly is cheaper and exact.

How well does it copy the teacher?

• Throughput: 116.4× the teacher stack (1655 vs 14 fps)
• Composite PLCC: 0.899 (95% BCa CI [0.854, 0.925]) · fitted PLCC: 0.902 · KRCC: 0.714 · Deep PLCC: 0.685

Those are the standard IQA metrics — PLCC / SRCC / KRCC / MAE / RMSE — with a bootstrap 95% CI on the composite, all measured as student vs. teacher agreement. Want the full picture? Per-signal fidelity, training curves and a blur-robustness sweep are in report.md / report.txt, and the "why did we build it this way" story is in WRITEUP.md.

Under the hood

A frozen mobilenetv3_small_100 micro tower turns each frame into an embedding. That embedding gets fused with 14 cheap classical descriptors (luma, contrast, colourfulness, sharpness, edge density, exposure clipping) — the trick that lets the student recover absolute exposure/contrast, which an ImageNet tower otherwise normalises away. A small residual-MLP trunk refines the fused vector, and one tiny MLP head per signal reads off the final scores.

Where the scores come from

Per-signal fidelity is published in HF's eval-results format under .eval_results/results.yaml, keyed to the companion benchmark dataset shubhxho/operator-video-quality — one task per signal. These are self-reported distillation-fidelity numbers (how closely the student tracks the teacher); they roll up into a leaderboard once the benchmark is registered as a Hub Benchmark.

Graphs

Report dashboard

Calibration

Fidelity (PLCC / SRCC vs teacher)

fidelity_by_sigma.png

Residual spread

Robustness vs Gaussian blur

Student vs teacher — every signal

Throughput

Training loss & learning rate

On-device (candle / MLX)

Sub-fp16 weights for Rust / Apple-Silicon inference are bundled. Sizes are the real file bytes (container + weights), not a params×bytes estimate:

file	runtime	precision	bits/weight	size on disk
model.int4.gguf	candle (Rust)	int4	4.50	1.60 MB
model.int4.mlx.safetensors	MLX (Apple Silicon)	int4	5.00	2.05 MB
model.int8.gguf	candle (Rust)	int8	8.50	2.76 MB
model.int8.mlx.safetensors	MLX (Apple Silicon)	int8	8.50	3.01 MB
model.mxfp4.gguf	candle (Rust)	mxfp4 (fp4)	4.25	1.53 MB

To keep int4 genuinely small, weights whose row length isn't a multiple of the quant block (many 1×1 conv / squeeze-excite kernels) are stored transposed so they block-quantise instead of falling back to fp16 — that alone trims the int4 build by ~20%. The transposed tensor names ride along in the metadata (compact.transposed for GGUF, transposed for MLX); a loader transposes them back after dequantising.

• candle (Rust): load the .gguf via candle_core::quantized::gguf_file (ggml Q4_0 / Q8_0). The trunk + heads are plain Linear / LayerNorm / GELU; GGUF metadata (compact.* keys) carries the arch, target order, the fused-descriptor names and the transposed-weight list so the forward pass — including the classical-descriptor fusion — is fully reconstructable.
• model.mxfp4.gguf (smallest): the same GGUF packed in MXFP4, the OCP Microscaling FP4 format frontier open models ship in — 4.25 bits/weight (vs int4's 4.5) for the smallest on-disk build, at a hair more quantisation error. Load it with a recent candle that supports MXFP4.
• MLX (Apple Silicon): load the .mlx.safetensors (grouped affine quant; arch + targets + descriptor names + transposed list in the safetensors metadata).

How it was trained

Self-distillation, end to end. The production pipeline at https://github.com/shubhxho/video-benchmark runs over an operator-video corpus and hands back per-frame teacher labels. Every sampled frame is then multiplied out with kornia Gaussian-blur + photometric augmentation — so signals that would otherwise be near-flat get enough spread to actually learn from — and the student regresses the teacher with SmoothL1 + a correlation loss, AdamW, SGDR cosine warm-restarts and an EMA of the weights.

Want one tuned to your footage? Point it at your own clips: python -m video_benchmark.distill.

License

The trained trunk + heads are MIT (this project). The bundled backbone weights derive from the timm mobilenetv3_small_100 ImageNet tower (Apache-2.0).