How to create a video editor that works in your browser.
Tags: webdev, tutorial, typescript, nuxt
Published: 2021-11-27 06:47:52

## Intro
Recently, browser features such as SharedArrayBuffer and WebAssembly are increasing. Video and video converter [ffmpeg also works in browsers](https://github.com/Kagami/ffmpeg.js).

Therefore, I tried to create a video editing software that works on a browser experimentally.

You can try the demo from here.
https://vega.toshusai.net/
source
https://github.com/toshusai/vega

## Basic Idea

Using basic below materials.
- images
- video
- audio
- text

In order to display these materials on the screen, a container for information such as playback position, length, and screen position is called a strip.

![Strips](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4vx4f5p239txrfshn5lh.png)

Convert these strips into the final video in a way that suits each material.


FFmpeg inputs are `video`, `image sequences` or `audio`.
If you want to add text, you have to generate image included text.

### Video
So, convert all the images/videos to image sequence once and pass them to ffmpeg. used [three.js](https://github.com/mrdoob/three.js/) for the renderer. Three.js can handle image as Texture, also video as VideoTexture.

The image capture step uses [ccapture](https://github.com/spite/ccapture.js/) to ensure that the playback frame is converted to an image. All frame images to webm video.

### Audio
Next, Merge image sequence and audio. This is ffmpeg part.

For video, get a specific range of audio from the video.
For audio, cut the audio.

Finally, use `filter_compex` to combine all the audio in strip time.

```
  -i _vega_video.webm -i video.mp3 -i audio.mp4
  -filter_complex [2:a]adelay=5715|5715[out2];[1:a]adelay=260|260[out1];[out1][out2]amix=inputs=2[out]
  -map [out]:a
```

This is the overall flow diagram.

![architecture](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nry9otwunuct3lt5ffrt.png)
 

## Thanks Open Source Softwares/Libraries
- JavaScript Framework
  - [Nuxt.js](https://github.com/nuxt/nuxt.js)
    - [Vue.js](https://github.com/vuejs/vue)
- Video/Audio Converter
  - [ffmpeg.wasm](https://github.com/ffmpegwasm/ffmpeg.wasm)
    - [FFmpeg](https://www.ffmpeg.org/)
- Capture canvas
  - [CCapture.js](https://github.com/spite/ccapture.js/)
    - [WebM video encoder](https://github.com/thenickdude/webm-writer-js/)
- 3D library (as Renderer)
  - [three.js](https://github.com/mrdoob/three.js/)
- Audio visualization
  - [wavesufer.js](https://github.com/katspaugh/wavesurfer.js/)
- Monospace font
  - [RictyDimished](https://github.com/edihbrandon/RictyDiminished)
- Icons
  - [Material Design Icons](https://github.com/templarian/MaterialDesign)
  - [Spectrum CSS Workflow Icons](https://github.com/adobe/spectrum-css-workflow-icons)
- Design System
  - [Spectrum Vue](https://github.com/toshusai/spectrum-vue)
    - [Spectrum CSS](https://github.com/adobe/spectrum-css)
- Documentation
  - [Docsify](https://github.com/docsifyjs/docsify/)
- Test Asset
  - Audio
    - [ondoku3](https://ondoku3.com/)
  - Video
    - [Blender Foundation](https://www.bigbuckbunny.org)
  - Image
    - [Kenney](https://www.kenney.nl/)


I also created a UI components to use for this. If you like / are interested, check this out too.
https://github.com/toshusai/spectrum-vue