IndoVSE — Indonesian Video Salient Entity Detection

IndoVSE is the first model for salient named entity detection in Indonesian news videos. Given a raw video file, the pipeline automatically identifies which named entities are topically central (salient) versus incidentally mentioned.

Installation

pip install indovse

Note: FFmpeg must be installed on your system.

• Linux: apt install ffmpeg
• Mac: brew install ffmpeg
• Windows: ffmpeg.org/download.html

All models (~3GB) are downloaded automatically on first use and cached locally.

Quick Start

from indovse import predict

result = predict("berita.mp4")

print(result["salient_entities"])
# [
#   {"entity_text": "Prabowo Subianto", "entity_label": "PER", "salient_prob": 0.874, ...},
#   {"entity_text": "KPU",             "entity_label": "ORG", "salient_prob": 0.811, ...},
#   {"entity_text": "Pemilu 2024",     "entity_label": "EVT", "salient_prob": 0.743, ...},
# ]

print(result["entity_timeline"])
# {
#   "Prabowo Subianto": [2.3, 45.1, 102.5, 189.0],
#   "KPU": [5.2, 78.9, 134.2],
# }

Output Format

{
  "video_duration": 312.5,        # second
  "total_entities": 18,           # total entity
  "salient_entities": [
    {
      "entity_text":  "Prabowo Subianto",
      "entity_label": "PER",
      "salient_prob": 0.874,      # salient probability [0-1]
      "freq_norm":    0.910,      # appearance frequency (normalized)
      "first_appear": 0.020,      # first appear position (normalized)
      "coverage":     0.840,      # spread (normalized)
      "burstiness":   0.310       # spread pattern (-1 tersebar, +1 menumpuk)
    }
  ],
  "entity_timeline": {
    "Prabowo Subianto": [2.3, 45.1, 102.5, 189.0]  # timestamp in second
  }
}

Parameters

predict(
    video_path,   # path to video file (.mp4, .mkv, .webm, dll)
    top_k=5       # maximum salient entity return
)

How It Works

Video
  → ffmpeg               (ekstrak audio)
  → Whisper large        (speech-to-text + word timestamps)
  → IndoBERT NER         (named entity recognition)
  → Temporal features    (freq_norm, burstiness, first_appear, coverage)
  → IndoBERT CLS         (contextual embedding)
  → IndoVSE MLP          (salience classification)
  → Ranked salient entities

Temporal Features

Feature	Description
freq_norm	Normalized mention frequency
burstiness	Clustering of mentions in time (B > 0 = bursty)
first_appear	Normalized position of first mention
coverage	Fraction of video duration spanned by mentions

Performance

Model	Features	ROC-AUC	F1 Salient	Avg Precision
Baseline LR	freq_norm only	0.8622	0.39	0.4629
TESD-Temporal	4 temporal features	0.9761	0.71	0.7369
IndoVSE (this)	4 temporal + BERT CLS	0.9800	0.75	0.8055

Ablation Study

Features	ROC-AUC	Avg Precision	F1 Salient
All 4 features	0.9791	0.7616	0.6718
w/o burstiness	0.9751	0.7292	0.6420
w/o first_appear	0.9740	0.7264	0.5766
w/o freq_norm	0.9667	0.6942	0.6012
w/o coverage	0.9449	0.5810	0.5262
only coverage	0.9401	0.5968	0.5109
only freq_norm	0.8622	0.4629	0.3468
only burstiness	0.8341	0.4305	0.4615
only first_appear	0.7657	0.1943	0.2791

Dataset

Trained on IndoVSE-dataset — 573 Indonesian news videos (~8.6 hours) from Metro TV, Kompas TV, and tvOne covering 4 domains: politik, ekonomi, kesehatan, pendidikan.

• 27,115 entity instances
• 2,193 salient (8.1%) / 24,922 non-salient (91.9%)
• Human-annotated, inter-annotator agreement κ = 0.931

Models Used

Model	Purpose
openai/whisper-large	Speech-to-text with word timestamps
cahya/bert-base-indonesian-NER	Named entity recognition
indobenchmark/indobert-base-p1	Contextual CLS embeddings
galihkjaya/IndoVSE (this)	Salience classification

Citation

@misc{indovse2025,
  title     = {IndoVSE: Temporal Entity Salience Detection for Indonesian News Videos},
  author    = {Galih Kusuma Wijaya},
  year      = {2026},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/galihkjaya/IndoVSE}
}

License

MIT