edde746/sponsorskip-modernbert

SponsorSkip detector

Word-level token classifier that finds sponsor and self-promotion segments in YouTube transcripts. Built to run in a browser on WebGPU, so a viewer can skip sponsor segments without a server and without crowd-submitted timestamps.

Companion extension: https://github.com/edde746/sponsorskip (GPL-3.0).

Licence, and why it is restrictive

CC BY-NC-SA 4.0 — non-commercial. The training labels come from the SponsorBlock database, which is CC BY-NC-SA 4.0, so these weights inherit that restriction. Attribution and share-alike apply. The extension code is GPL-3.0 and lives in a separate repository precisely so the two licences stay unambiguous.

Files

Two variants. Both are ModernBERT, share one tokenizer, and have their vocabulary pruned from 50,368 to 17,536 rows.

file	size	params	role
detector_fp16.onnx	270 MB	124.4M	base, boundary-weighted
large/detector_fp16.onnx	724 MB	362.2M	large, more accurate
detector_meta.json, large/detector_meta.json	118 KB		vocab map, thresholds, ids
tokenizer/	3.6 MB		shared by both

Fixed-shape graphs. Inputs input_ids and attention_mask, both int64 [1, 2048]; output logits, float32 [1, 2048, 5] over O, B-/I-SPONSOR, B-/I-SELFPROMO.

Results

SponsorBlock crowd labels, tier A, channel-disjoint validation split, held-out half (~200 videos / ~530 segments). Thresholds selected on the disjoint tune half.

metric	base	large
F1 (IoU 0.3)	0.808	0.834
precision / recall	0.799 / 0.817	0.817 / 0.851
segment start, signed	−0.72 s	−0.18 s
mean absolute start error	2.98 s	2.33 s
overshoot at segment end	+1.28 s	+0.46 s
starts more than 0.5 s late	11.8%	11.5%
mean absolute boundary error	2.62 s	2.49 s
token-level F1	0.898	0.909
hysteresis th_hi / th_lo	0.70 / 0.30	0.70 / 0.40

Large is better on every axis. The F1 gap (+0.026) is borderline at this sample size, but the boundary means are estimated far more tightly and are consistently better — most visibly at segment ends, where large overshoots by 0.46 s against base's 1.28 s, so it cuts noticeably less content.

Base additionally uses a boundary-weighted loss (3x weight on tokens within 2 positions of a label transition), since only the first word of a read determines when to skip. That is a modest gain, not a step change: token F1 0.898 vs 0.894, and not separable from noise on F1 alone. Large does not use it.

Evaluate against crowd labels, not LLM labels

An earlier version of this project tuned its threshold on a small hand-labelled set produced by language models. Those labels are systematically tighter than crowd labels: measured on 203 overlapping segments, they start +1.25 s later (p90 +4.0 s) and end 2.04 s earlier. A threshold tuned against them inherits that bias, and the deployed model began sponsor skips +2.4 s late. If you evaluate this model, use crowd labels or your own careful ones.

Usage — the decode matters as much as the model

Do not threshold with a single value. Use hysteresis: a high threshold decides whether a segment exists, a low one decides where its edges are. With one threshold the two jobs fight — raising it improves F1 and pushes starts later, because a run cannot begin until the model is already certain, several words into the sponsor read.

Measured on the same held-out half:

decode	F1	segment start
single 0.60	0.771	+0.86 s (late)
hysteresis 0.70 / 0.30	0.808	−0.72 s

Better on both axes at once. Each variant's detector_meta.json carries its own threshold_hi and threshold_lo (base 0.70/0.30, large 0.70/0.40). threshold_lo is a taste knob — whether you would rather hear a second of ad or lose a second of content:

threshold_lo	mean start
0.40	+0.08 s
0.30	−0.58 s
0.20	−1.05 s

Algorithm: find runs where summed P(classes 1..4) >= threshold_hi, extend each run outward while probability stays >= threshold_lo, resume scanning past the expanded end. Then name each span by its majority ad class, merge same-category spans less than 2 s apart, drop spans under 1 s, and add 0.4 s to the last word's timestamp.

Four things that silently degrade accuracy if wrong

1. No prefix space. The tokenizer has add_prefix_space=False, so HF's is_split_into_words=True is exactly "encode each word alone, concatenate". Prepending spaces produces entirely different ids.
2. Remap ids through keep. Build vmap = zeros(50368); vmap[keep] = arange(len(keep)). The cls/sep/pad ids in the meta are original vocab ids: assemble [CLS] ids [SEP] in original id space, then remap the whole array including the pad filler.
3. Zero attention_mask over padding. Verified live in the graph: changing pad content with mask=0 moves logits by 0.0, while an all-ones mask moves them by 4.37.
4. Window at 2048 with 0.5 stride, pool the first subtoken of each word, and average probabilities across overlapping windows.

A complete, verified reference implementation is src/offscreen/detector.ts.

How it was built

ModernBERT-base fine-tuned as a 5-way token classifier on 21,001 videos from the SponsorBlock database, channel-disjoint train/val split, two-tier PU-aware loss (high-trust labels get full loss; weaker labels have pre-roll/outro zones masked with ignore_index plus a ±2 s collar). 1.5 epochs, 9,166 optimiser steps.

Two things beyond a plain fine-tune:

1. Boundary-weighted loss (base only). See Results above.
2. Vocabulary pruning from 50,368 to 17,536 rows. Only 17,531 ids occur in 98.3M tokens of English ASR. This removed 62% of the model's parameters (149.6M → 124.4M) and is bit-identical: max abs logit delta 0, argmax parity 100%.

fp16 export verified against PyTorch: 99.89% argmax parity over 47,927 live tokens (base). The fp32 reference used for that check is not published; it is regenerable from this checkpoint.

Rejected along the way: truncating to 6 or 12 layers (halves the download but nearly doubles boundary error to 4.4 s), and a 19.3M distilled student for CPU (~11 s mean absolute start error — it only stops being late by being symmetrically wrong, which is worse than not skipping). A backbone bake-off at an equal 4,000-step budget kept ModernBERT: at 149.6M it beat Ettin by +0.065 F1 (95% CI [+0.009, +0.117]).

Limitations — read these

• Per-video variance dwarfs the averages. On one test video the model over-extends ~6 s early and ~9 s late regardless of threshold, because that segment's probability profile is broad rather than sharp. The aggregate numbers above are real; individual videos will still be visibly wrong.
• Short segments are the weak point. Recall degrades sharply under 10 seconds.
• English ASR only. Trained on English auto-captions; other languages and manually-authored caption tracks are out of distribution.
• It detects sponsor reads — spoken advertising in the transcript. It cannot see burned-in banners and has no signal outside the words.
• No CPU path. WebGPU or nothing; see the rejected student above.
• Large costs a 724 MB download and more VRAM. Base is the sensible default unless segment-end precision matters to you.
• Evaluation splits are channel-disjoint from training, but on an earlier hand-labelled test set that was not channel-disjoint, most of the apparent advantage over a reference LLM came from channel memorisation. Treat cross-system comparisons with suspicion unless the split is clean.