[TypeScript] Record MediaStream
Tags: typescript
Published: 2021-10-23 17:45:33

## Intro
This time, I will try recording video and audio stream.

* [[ASP.NET Core][TypeScript] Try WebRTC](https://dev.to/masanori_msl/asp-net-core-typescript-try-webrtc-c5m)
* [[Electron][TypeScript][PDF.js] Create images from PDF](https://dev.to/masanori_msl/typescript-play-my-own-voice-58i4)

### Environments
* Node.js ver.16.12.0
* TypeScript ver.4.4.3
* Webpack ver.5.56.0

## Draw video streams on a canvas
Because I want to add images on a video stream, I try drawing it on a canvas.

#### index.html
```html
<!DOCTYPE html>
<html lang="en">
    <head>
        <title>video sample</title>
        <meta charset="utf-8">
        <link rel="stylesheet" href="../css/month_picker.css" />
    </head>
    <body>
        <div>
            <canvas id="local_video_area">
                <video id="local_video"></video>
            </canvas>
        </div>
        <button id="record_button" onclick="Page.record()">Record</button>
        <a id="download_link"></a>
        <script src="./js/main.page.js"></script>
        <script>Page.init();</script>
    </body>
</html>
```

#### main.page.ts
```ts
import { MediaStreamController } from "./mediaStreamController";

let controller: MediaStreamController;
export function init(): void {
    controller = new MediaStreamController();
    controller.init();
}
```

#### mediaStreamController.ts
```ts
export class MediaStreamController {
    private canvas!: HTMLCanvasElement;
    
    public init(): void {        
        this.canvas = document.getElementById("local_video_area") as HTMLCanvasElement;
        const ctx = this.canvas.getContext("2d");
        if(ctx == null) {
            console.error("failed getting context");
            return;
        }
        navigator.mediaDevices.getUserMedia({ video: true, audio: true })
            .then(stream => {
                // add the video stream on a video element
                const localVideo = document.getElementById("local_video") as HTMLVideoElement;
                localVideo.srcObject = stream;
                localVideo.play();

                // must draw the video stream every frames because it is drawn as static images
                setInterval(() => {
                    // clear last images
                    ctx.clearRect(0, 0, localVideo.videoWidth, localVideo.videoHeight);
                    // set canvas size as same as the video element
                    this.canvas.width = localVideo.videoWidth;
                    this.canvas.height = localVideo.videoHeight;
                    ctx.drawImage(localVideo, 0, 0, localVideo.videoWidth, localVideo.videoHeight);
                }, 1000/60);
            })
            .catch(err => console.error(`An error occurred: ${err}`));
    }
}
```

* [Using images - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Using_images)
* [4.12.5.1.13 Drawing focus rings and scrolling paths into view - HTML Standard](https://html.spec.whatwg.org/multipage/canvas.html#dom-context-2d-drawimage)

### Add static images
Because I draw the video capture image every frames, I also have to draw static images every frames.

#### mediaStreamController.ts
```ts
export class MediaStreamController {
    private canvas!: HTMLCanvasElement;
    
    public init(): void {        
...
        navigator.mediaDevices.getUserMedia({ video: true, audio: true })
            .then(stream => {
...
                const glassesImage = document.createElement("img");
                glassesImage.src = "../image/glasses.png";
...
                setInterval(() => {
...
                    ctx.drawImage(localVideo, 0, 0, localVideo.videoWidth / 2, localVideo.videoHeight / 2);
                    ctx.drawImage(glassesImage, 5, 5, glassesImage.width, glassesImage.height);
                }, 1000/60);
            })
            .catch(err => console.error(`An error occurred: ${err}`));
    }
}
```
#### Result
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n1e6h6zis8aqr6bw8j06.png)

### Change color
I can change the image color.

#### mediaStreamController.ts
```ts
export class MediaStreamController {
    private canvas!: HTMLCanvasElement;
    
    public init(): void {        
...
        navigator.mediaDevices.getUserMedia({ video: true, audio: true })
            .then(stream => {
...
                const glassesImage = document.createElement("img");
                glassesImage.src = "../image/glasses.png";
...
                setInterval(() => {
...
                    ctx.drawImage(localVideo, 0, 0, localVideo.videoWidth / 2, localVideo.videoHeight / 2);
                    // change color
                    const canvasImage = ctx.getImageData(0, 0, this.canvas.width / 3, this.canvas.height / 3);
                    for (let i = 0; i < canvasImage.data.length; i += 4) {
                        canvasImage.data[i]= canvasImage.data[i]! * 0;
                        canvasImage.data[i + 1] = canvasImage.data[i + 1]! * 0.1;        
                        canvasImage.data[i + 2] = canvasImage.data[i + 2]! * 1;
                        canvasImage.data[i + 3] = canvasImage.data[i + 3]! * 1;
                    }
                    ctx.putImageData(canvasImage,0,0);

                    ctx.drawImage(glassesImage, 5, 5, glassesImage.width, glassesImage.height);
                }, 1000/60);
            })
            .catch(err => console.error(`An error occurred: ${err}`));
    }
}
```
#### Result
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pj4fxcq6y7btyjylv2vv.png)


* [colors - Changing Image colour through Javascript - Stack Overflow](https://stackoverflow.com/questions/28301340/changing-image-colour-through-javascript)
* [javascript - Change color Image in Canvas - Stack Overflow](https://stackoverflow.com/questions/45706829/change-color-image-in-canvas#:~:text=As%20with%20getImageData%20%28%29%2C%20the%20drawback%20with%20this,draw%20that%20canvas%20back%20onto%20the%20main%20canvas)

## Record the video
I can record the video stream by MediaStream Recording API.

* [MediaStream Recording API - Recording API - Web API | MDN](https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API)

#### main.page.ts
```ts
...
let recording = false;
...
export function record(): void {
    if(recording === true) {
        controller.stopRecording();
        recording = false;
    } else {
        controller.startRecording();
        recording = true;
    }
}
```

#### mediaStreamController.ts
```ts
export class MediaStreamController {
    private canvas!: HTMLCanvasElement;
    private mediaRecorder: MediaRecorder|null = null;
    private dataAvailableEvent = (ev: BlobEvent) => this.handleDataAvailable(ev);
...
    public startRecording(): void {
        if(this.mediaRecorder != null) {
            this.mediaRecorder.removeEventListener("dataavailable", this.dataAvailableEvent);
            this.mediaRecorder = null;
        }
        const stream = this.canvas.captureStream(60);
        const options = { mimeType: "video/webm; codecs=vp9" };
        this.mediaRecorder = new MediaRecorder(stream, options);
        this.mediaRecorder.ondataavailable = this.dataAvailableEvent;
        this.mediaRecorder.start();
    }
    public stopRecording(): void {
        if(this.mediaRecorder == null) {
            return;
        }
        this.mediaRecorder.stop();
    }
    private handleDataAvailable(ev: BlobEvent) {
        if (ev.data.size <= 0) {
            return;
        }
        this.download(ev.data);
    }
    private download(blob: Blob): void {
        const url = URL.createObjectURL(blob);
        const a = document.getElementById("download_link") as HTMLAnchorElement;
        a.style.display = "none";
        a.href = url;
        a.download = "test.webm";
        a.click();
        window.URL.revokeObjectURL(url);
    }
}
```

### MIME TYPE
I want to save data as "mp4".

But when I changed the options of "MediaRecorder" like below, I got an error.

#### options
```ts
const options = { mimeType: 'video/mp4; codecs="avc1.424028, mp4a.40.2"' };
```

#### error
```
Uncaught DOMException: Failed to construct 'MediaRecorder': Failed to initialize native MediaRecorder the type provided (video/mp4;) is not supported.
    at MediaStreamController.startRecording
```

* [MediaRecorder.mimeType - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder/mimeType#:~:text=MediaRecorder.mimeType%20The%20mimeType%20read-only%20property%20returns%20the%20MIME,writing%20all%20of%20the%20recorded%20data%20to%20disk.)