Video.js: frame-accurate subtitles
Tags: javascript, video, postproduction, videojs
Published: 2017-12-30 11:03:48

---
title: "Video.js: frame-accurate subtitles"
published: true
description: ""
tags: Javascript, Video, Post-production, Video.js
---

In the post-production and broadcast industry videos are processed frame by frame.
Subtitles follow the same rule : in and out timecodes are frame accurate (ex: `00:00:23:22` -> 23 seconds and 22 images).

Browsers process videos with millisecond timestamps, that's why subtitles have to be converted based on video framerate: for video at 25 images per second the previous timecode would look like `00:00:23.880` in milliseconds format (`1000/25 * 22 = 880`).

Frame accuracy is really important : subtitles need to disappear right before the next cut and appear right after the previous one.
Problem is that default HTML5 video's refresh rate is too low: subtitles appear and disappear too late creating a poor viewer's experience (and a non-broadcast compliant one).

Example below higlights the issue. Subtitles in the fixed video disappear right before the change of plan:

![Demo gif](https://thepracticaldev.s3.amazonaws.com/i/z0g5w833mi9xm72vppxl.gif)

It might seem marginal, but the post production industry needs frame accuracy.

Due to browser limitations, `timeupdate` (event fired when the playing position of a video has changed) is fired every 150-250 milliseconds. It is not enough for frame-accuracy: 25fps means an update every 40ms.
We need to compute which subtitle has to be displayed every frame (instead of doing it every 4-5 frames by default). Video.js subtitles engine does the computation each time the text track's attribute `activeCues` getter is called.

text-track.js extract from Video.js source code:
```js
Object.defineProperty(tt, 'activeCues', {
    get() {
        if (!this.loaded_) {
            return null;
        }
        // nothing to do
        if (this.cues.length === 0) {
            return activeCues;
        }
        const ct = this.tech_.currentTime();
        const active = [];
        for (let i = 0, l = this.cues.length; i < l; i++) {
            const cue = this.cues[i];

            if (cue.startTime <= ct && cue.endTime >= ct) {
                active.push(cue);
            } else if (cue.startTime === cue.endTime &&
                cue.startTime <= ct &&
                cue.startTime + 0.5 >= ct) {
                active.push(cue);
            }
        }
        changed = false;
        if (active.length !== this.activeCues_.length) {
            changed = true;
        } else {
            for (let i = 0; i < active.length; i++) {
                if (this.activeCues_.indexOf(active[i]) === -1) {
                    changed = true;
                }
            }
        }
        this.activeCues_ = active;
        activeCues.setCues_(this.activeCues_);
        return activeCues;
    },
    set() {}
});
```

Then you need to call `trigger('cuechange')` on the text track to make sure the video display is up to date:
```js
player.textTracks()[0].activeCues; // computes the current subtitle based on current time
player.textTracks()[0].trigger('cuechange'); // updates the display
```

`requestAnimationFrame` is optimized for animations and has a lot less delay than `setInterval` or `setTimeout`, so we are going to use it for our time sensitive loop ([Frame rate control source here](http://codetheory.in/controlling-the-frame-rate-with-requestanimationframe/)).

Here is the complete source code:

```js
var fps = 25;
var now;
var then = Date.now();
var interval = 1000/fps;
var delta;

function reloadCues() {
    requestAnimationFrame(reloadCues);

    now = Date.now();
    delta = now - then;
    
    if (delta > interval) {
        then = now - (delta % interval);

        if(videojs.players.player.textTracks().length == 1) {
            videojs.players.player.textTracks()[0].activeCues;
            videojs.players.player.textTracks()[0].trigger('cuechange')
        }
    }
}
reloadCues();
```

Demo project is available [here](/demo/videojs-frame-accurate-subtitles).

**Originally published on [my personal blog](http://cleriotsimon.com/posts/videojs-frame-accurate-subtitles/).**