Acliptic v5
Finds the moments in a stream or podcast most worth clipping. It listens to the audio, reads the speech, and ranks every 30-second moment in the video.
v5 replaces the DeBERTa text classifier (v2). It isn't a transformers model, so AutoModel.from_pretrained won't load it. Use the code in this repo.
What's in this repo
| Path | What it is | Trained by us? |
|---|---|---|
ranker5.pkl |
The ranker: gradient-boosted regressor, gradient-boosted classifier and logistic regression | Yes |
bge-onnx/ |
BAAI/bge-base-en-v1.5 (ONNX): meaning of the speech |
No |
clap-onnx/ |
CLAP clap-htsat-unfused (Xenova ONNX): sound fingerprints |
No |
faster-whisper-base.en/ |
Speech to text | No |
acliptic/ |
The code that builds the inputs and runs the ranker |
ranker5.pkl is a Python pickle. It needs the acliptic package from this repo and scikit-learn 1.9.1. Only load it from a source you trust.
Use it
import os
import sys
from huggingface_hub import snapshot_download
d = snapshot_download("umarfarzan/acliptic-v5", token=os.environ["HF_TOKEN"])
os.environ["ACLIPTIC_ASR_MODEL"] = f"{d}/faster-whisper-base.en"
os.environ["ACLIPTIC_CLAP_ONNX"] = f"{d}/clap-onnx"
os.environ["ACLIPTIC_TEXT_EMB_ONNX"] = f"{d}/bge-onnx"
sys.path.insert(0, d)
from acliptic.models.base import MediaInfo, ScoreRequest, Segment
from acliptic.models.moment_ranker5 import MomentRanker5Provider
model = MomentRanker5Provider("acliptic-v5", d, "5.0")
# segments: consecutive 30-second transcript chunks. path: the audio or video file.
req = ScoreRequest(segments=[Segment(0, 0, 30, "text of the first 30 seconds"), Segment(1, 30, 60, "text of the next 30 seconds")],
media=MediaInfo(duration=60, has_audio=True, path="video.mp4"))
print(model.score(req))
Scores rank moments within one video. They aren't probabilities, and they can't be compared across videos. There's no threshold: take the top moments per video or hour. MediaInfo(live=True) uses the live model, which only looks at what has already happened.
Data and results
- Training: 19 VODs, 50.6 hours, 8 streamers, labelled with YouTube Most Replayed data.
- Test: 24 unseen VODs, 65 hours, 22 streamers.
| Measure | Random | App with v2 | v5 |
|---|---|---|---|
| AUC | 0.500 | 0.507 | 0.599 |
| Top 6 picks per hour in the top 20% rewatched | 20% | 29.5% | 37.3% |
| Top 6 picks per hour in the top 5% rewatched | 5% | 7.8% | 10.2% |
The gain on the top 5% could be luck. About 6 in 10 picks still miss the top 20%.
Limits
- English only.
- Trained on VODs up to 4.2 hours. Longer videos are untested.
- Doesn't use live chat.
- Weak on some fast gameplay streams. Tfue scored 0.36 AUC, worse than random.
- Without audio it runs from text alone, which hasn't been measured.