DeathMark: Programmatically scan videos for points of interest
Tags: video, canvas, electron, javascript
Published: 2021-01-13 03:19:49

---
title: DeathMark: Programmatically scan videos for points of interest
description: I use canvas to fine points of interest in a video
private: false
published: true
tags: video, canvas, electron, js
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/2s87118aahnysqwnrpmz.png
---

## Problem
In 2020 like most people our company transition to more remote work and I started to invest in my home office to prepare for the coming onslaught of zoom meetings. Once I had a comfortable setup, I had everything I needed to start streaming outside of work as well because why not right? I started playing around with twitch and cutting my videos down for youtube.

Most of my streams were 2+ hours and while playing 2+ hours of games with your friends is an investment in relaxation and friendship. Watching back 2+ hours wasn't a time investment I wanted to make.

## Conception
My game of choice is Valorant at the moment but this is true of most games; In the game there is a visual queue to show you have scored a point.

![Frame 7 (1)](https://dev-to-uploads.s3.amazonaws.com/i/6nulab6str8c0mdmtsuz.png)

Because they are UI elements they are usually visually bright, consistently placed, and distinct from the background view of the game.

The score indicator is kind of like a flashing light, if I had something like a Light-Dependent Resistor, I could record when it flashed.

![Frame 8](https://dev-to-uploads.s3.amazonaws.com/i/lm7n4syoidd5w4lte2ix.png)
 
## Application
I have a concept "check for increases in white in the video"
[MDN](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Manipulating_video_using_canvas) has a great example of how to check each frame of a video using HTML canvas. The example is set up like this:


![Frame 9](https://dev-to-uploads.s3.amazonaws.com/i/ikbplf257oqrzuzvtp16.png) 
ctx1 = Canvas 1 context
ctx2 = Canvas 2 context

We are going to borrow their function and focusing on the `computeFrame` section. You can see below the RGB values for each pixel in each frame.

```js
 processor.computeFrame = function computeFrame() {
//drawing the full frame to canvas
    this.ctx1.drawImage(this.video, 0, 0, this.width, this.height);
//get the frame from canvas at 0 x and 0 y 
    let frame = this.ctx1.getImageData(0, 0, this.width, this.height);

    let l = frame.data.length / 4;

    for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
      if (g > 100 && r > 100 && b < 43)
        frame.data[i * 4 + 3] = 0;
    }
    this.ctx2.putImageData(frame, 0, 0);
    return;
  }
```

In the example, above it's checking this range of color and if it falls within the threshold it will make it alpha instead thus creating a green screen or in this case yellow screen.
![Frame 10](https://dev-to-uploads.s3.amazonaws.com/i/19jndpsb2ltcq2qpszpg.png)
 
Simple enough, i will just check for white pixel in the area.
```js
//255,255,255 is white so 240 -> 255 is mostly white
 if (g > 240 && r > 240 && b < 240)
        //is white pixel
    }
```
But the game has complex visuals and many elements would trigger just "white"

![Frame 11](https://dev-to-uploads.s3.amazonaws.com/i/ug7aevusby0rbywcqgjy.png)

Every picture is made up of an almost unique amount of colors and shades so all I needed to do is get as close as possible to that unique number.

![Frame 12](https://dev-to-uploads.s3.amazonaws.com/i/rn27kw2lxiv7i25939k0.png)

```js
let skullFound = []
let white = []
let green = []
let red = []
for (let i = 0; i < l; i++) {
      let r = frame.data[i * 4 + 0];
      let g = frame.data[i * 4 + 1];
      let b = frame.data[i * 4 + 2];
       if (isWhite(r,g,b))
         white.push({r,g,b})
       }
      if (isGreen(r,g,b))
        green.push({r,g,b})
      }
      if (isRed(r,g,b))
        red.push({r,g,b})
      }
}

if(whiteThreshold(white.length) && greenThreshold(green.length) && redThreshold(red.length)) {
 skullFound.push(video.currentTime)
 white = []
 green = []
 red = []
}
```
After 30ish minutes of trial and error, I was able to get about a 99% accuracy rate at 2x speed with the videos tested with 1 main exception being, if the Character Sage is within the cropped section when she is shot with a sniper rifle... which is kinda rare.

![ng7h1c85uhhrl6n4kpwp](https://dev-to-uploads.s3.amazonaws.com/i/a394pgi0dgzxbupal2jr.png)
 

# Conclusion
While the current system is not perfect, It's a simplistic solution to the problem I was facing that can be built upon later.

I believe methods like the one above can be applied to many game videos. I look forward to finding more fun techniques in this area in the future.

## Free Download
https://ko-fi.com/post/DeathMark-W7W8381IO