Part 5 - Video Compiler
Published: 2023-10-08 15:41:17

## Introduction

In the previous parts we have generated and processed all the story content that's text, images and audio. In this section, we will use royalty-free background music from [Pixabay](https://pixabay.com/) and a python library called [moviepy](https://pypi.org/project/moviepy/) to compile the final video.

## High Level Overview

What we have now is a list of images files each represents a page, a list of audio files, each corresponds to a page. So, we want to go from that to a compiled video. To do so, we have to picture the video in our head first then figure out how to implement it using [moviepy](https://pypi.org/project/moviepy/).

![compiled video diagram](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8y6aj6flq513uidz9s4j.png)

As you may already know, the video is a bunch of frames stick together in a sequence, plus audio. So, what we need to do is set each page image as current frame of the video, and the audio part of that page as the audio part of the video. Once the audio part is done (the narrator completes reading the sentence), we want to move to the next frame and the next audio. And so on.

After we finish with all frames and their audio, we blend in the background music with the full audio.

## Image Clips

[moviepy](https://pypi.org/project/moviepy/) offers a simple interface called `ImageClip`. Given an image, we can create a video clip that runs this image X number of seconds. That number of seconds we know we want to be equal to the audio length of that page. We can also add an `AUDIO_GAP` of 0.5 seconds to make sure that there's a tiny little breath between pages (you can play with that, to make the gap longer of shorter).

The below code shows an example of creating an image clip called `page_clip` and adding it to a list of `page_clips`.

```python
page_clip = mpy.ImageClip(page.page_filepath).set_duration(
    page.audio.length_in_seconds + self._AUDIO_GAP
)
page_clips.append(page_clip)
```

We can then concatenate these image clips (`page_clips`) into a single clip `concatenate_videoclips` and `method='compose'`.

```python
clip = mpy.concatenate_videoclips(page_clips, method="compose")
```

## Audio Clips

Similarly, we need to create a list of audio clips from our audio files. But this time, we don't need to set the duration but we want to set when should that audio clip start. So, we can start with a `current_start` pointer at `0.0` and move it as we go.

```python
current_start = 0.0

# for each page
audio_clip = mpy.AudioFileClip(story.pages[i].audio.mp3_file).set_start(
    current_start
)
audio_clips.append(audio_clip)

# keep track of the current length
current_start += page.audio.length_in_seconds + self._AUDIO_GAP
```

We can then combine them together using `CompositeAudioClip` and then add that the audio part of the combined ImageClip from previous section.

```python
full_audio_clip = mpy.CompositeAudioClip(audio_clips)

# clip = mpy.concatenate_videoclips(page_clips, method="compose")
clip.audio = full_audio_clip
```

## Background Music

I opted in for a randomized approach that picks up a random background music file from a predefined folder. The below method lists all music files under `BACKGROUND_MUSIC_PATH` and picks a random background music file.

```python
def _get_background_music_filename(self) -> str:
    """Returns: A path to a mp3 file to be used as a background music."""
    music_files: List[str] = [
        join(self._BACKGROUND_MUSIC_PATH, filename)
        for filename in listdir(self._BACKGROUND_MUSIC_PATH)
        if isfile(join(self._BACKGROUND_MUSIC_PATH, filename))
        and filename.endswith(self._BACKGROUND_MUSIC_EXT)
    ]
    return random.choice(music_files)
```

To add that background music, we first need to read the music file as `AudioFileClip`. Then, we reduce it's volume and combine it with the existing audio.

```python
# read background music file
background_music = mpy.AudioFileClip(background_music_filepath).fx(
    volumex, self._BACKGROUND_MUSIC_VOLUME_FACTOR
)

# set volume
background_music = afx.audio_loop(
    background_music, duration=video_clip.duration
)

# combine it with existing audio
video_clip.audio = mpy.CompositeAudioClip([video_clip.audio, background_music])
```

Finally, we save the video, by calling `clip.write_videofile`.

```python
clip_filepath = os.path.join(workdir, self._FILENAME)
clip.write_videofile(clip_filepath, fps=self._FPS)
```

## Bing it all together

That's the cherry on the cake. Given a `Story` object, this method will return back a filepath that represents the compiled video.

```python
def generate_video(self, workdir: str, story: Story) -> str:
    """Create a video for the given story

    Args:
        workdir: The root workdir for the story to generate video for
        story: The story object that contains all details about the story

    Returns: A local filepath for where the created video is stored
    """
    page_clips = []
    audio_clips = []
    current_start = 0.0

    # Build image and audio clips
    for i in range(len(story.pages)):
        page = story.pages[i]
        page_clip = mpy.ImageClip(page.page_filepath).set_duration(
            page.audio.length_in_seconds + self._AUDIO_GAP
        )
        page_clips.append(page_clip)
        audio_clip = mpy.AudioFileClip(story.pages[i].audio.mp3_file).set_start(
            current_start
        )
        audio_clips.append(audio_clip)
        # keep track of the current length
        current_start += page.audio.length_in_seconds + self._AUDIO_GAP

    # Combine image and audio clips
    clip = mpy.concatenate_videoclips(page_clips, method="compose")
    full_audio_clip = mpy.CompositeAudioClip(audio_clips)
    full_audio_clip.write_audiofile(
        os.path.join(workdir, f"final_audio.wav"), fps=44100
    )

    clip.audio = full_audio_clip

    # Add background music and save
    self._add_background_music(clip)
    clip_filepath = os.path.join(workdir, self._FILENAME)
    clip.write_videofile(clip_filepath, fps=self._FPS)
    return clip_filepath
```

## Conclusion

In this 5 part series, we have gone a step by step in building a fully fledged automated AI visual story generator. We have done text and image generation, text processing, image processing, text to speech and video processing.

Thanks for reading :)