BMF 📹 + Hugging Face🤗, The New Video Processing BFFs
Tags: python, ai, ffmpeg, opensource
Published: 2024-02-01 01:27:15

*TL;DR**if you want to test this tutorial before we start, try it out* *[here](https://colab.research.google.com/drive/1eQxiZc2vZeyOggMoFle_b0xnblupbiXd?usp=sharing)*

[Hugging Face](https://huggingface.co/) has created a major shift in the AI community. It fuels cutting-edge open source machine learning/AI models and datasets. The Hugging Face community is thriving with great ideas and innovations to the point where the possibilities seem endless.

Hugging Face is revolutionizing Natural Language Processing (NLP) with state-of-the-art solutions for tasks like translation, summarization, sentiment analysis, and contextual understanding. Its arsenal of pre-trained models makes it a robust platform for diverse NLP tasks, streamlining the integration of machine learning functionalities. Hugging Face simplifies the training, evaluation, and deployment of models with a user-friendly interface. The more I used Hugging Face in my own personal projects, the more I felt inspired to combine it with [Babit Multimedia Framework (BMF)](https://babitmf.github.io/).

If you're reading this and are not familiar with BMF, it's a cross-platform multimedia processing framework by ByteDance Open Source. Currently, BMF is used to process over 2 billion videos a day across multiple social media apps. Can this get complex? Yes, it sure can. However, in this article, I'll break it all down, so you know how to create unique experiences across any type of media platform.

**Why** **BMF****?**

BMF stands out with its multilingual support, putting it ahead in the video processing game. BMF excels in various scenarios like video transcoding, editing, videography, and analysis. The integration of advanced technologies like Hugging Face with BMF is a game-changer for complex multimedia processing challenges.

Before we get started with the tutorial, let me share with you some ideas I envision coming to life with BMF + Hugging Face:

-   **Multimedia Content Analysis:** Leveraging Hugging Face's NLP models, BMF can delve deep into textual data associated with multimedia content, like subtitles or comments, for richer insights.
-   **Accessibility:** NLP models can automatically generate video captions, enhancing accessibility for the hard-of-hearing or deaf community.
-   **Content Categorization and Recommendation:** These models can sort multimedia content based on textual descriptions, paving the way for sophisticated recommendation systems.
-   **Enhanced User Interaction:** Sentiment analysis on user comments can offer valuable insights into user engagement and feedback for content improvement.

### **What now?**

Open Source AI is creating the building blocks of the future. Generative AI impacts all industries, and this leads me to think about how generative AI can impact the future of broadcasting and video processing. I experimented with BMF and Hugging Face to create the building blocks for a broadcasting service that uses AI to create unique experiences for viewers. So, enough about the background, let's get it going!

### **What we'll build**

Follow along, as we'll build a video processing pipeline with BMF that uses the [runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) model to generate an image to display as an overlayed image ontop of an encoded video. If that didn't make sense, don't worry, here's a picture for reference:

![](https://bytedance.us.larkoffice.com/space/api/box/stream/download/asynccode/?code=ODQ1NDg1YWEyYjg0MjM1NjhkNWIzNTgzZTIyODkxNzFfZE5TNkJsNnRVWUtqakJPd2hnWlNhYmd1UG9LN0l3WmNfVG9rZW46UmE1aGJ4OFJYb0p5dkF4MXVTRHVqamZDc2tlXzE3MDY3NTA2MjQ6MTcwNjc1NDIyNF9WNA)

So why is this significant? The image of the panda is AI generated and combined with BMF , we can put it down a processing pipeline to put it on top of our video. Think about! There could be a scenario where you are creating a video broadcasting service and during live streams, you'd like to display images quickly and display them for your audience with a simple prompt. There can also be a scenario where you are using BMF to edit your videos and you'd like to add some AI-generated art. This tutorial is just one example. BMF combined with models created by the Hugging Face community opens up a whole new world of possibilities.

* * * * *

### **Let's Get Started**

**Prerequisites:**

-   A GPU(I'm using google Colab A100 GPU. You can also use v100 or TP4 GPUs but they will just run a bit slower)
-   Install [BMFGPU](https://babitmf.github.io/docs/bmf/getting_started_yourself/install/#pip)
-   Python 3.9-3.10 (strictly required to work with bmf)
-   FFMPEG

You can find all the BMF installation docs [here](https://babitmf.github.io/docs/bmf/getting_started_yourself/install/#ffmpeg). The docs will highlight more system requirements if you decide to run things locally.

### **Getting Started**

Begin by ensuring that essential toolkits like Hugging Face Transformers and BMF are installed in your Python environment. Use pip for installation:

### **Initial Setup**

1.  First, we'll clone the following repository to get our video that we want to process(If you are coding along and want to use your own video, create your own repo and simply add a video file, preferably a short video and add to easily clone just like I did. You can also just save the video to the directory you're coding in.)

```
git clone https://github.com/Joshalphonse/Bmf-Huggingface.git
```

1.  Install BabitMF-GPU to accelerate your video processing pipeline with BMF's GPU capablities

```
pip install BabitMF-GPU
```

1.  Install the following dependencies

```
pip install requests diffusers transformers torch accelerate scipy safetensors moviepy Pillow tqdm numpy modelscope==1.4.2 open_clip_torch pytorch-lightning
```

1.  Install ffmpeg.BMF framework utilizes the FFmpeg video decoders and encoders as the built-in modules for video decoding and encoding. It's necessary for users to install supported FFmpeg libraries before using BMF.

```
sudo apt install ffmpeg
```

```
dpkg -l | grep -i ffmpeg
```

```
ffmpeg -version
```

This package below is installed to show the BMF C++ logs in the colab console, otherwise only python logs are printed. This step is not necessary if you're not in a Colab or iPython notebook environment.

```
pip install wurlitzer
%load_ext wurlitzer
```

1.  Create a new folder in the directory for the github repository we cloned. We'll need this path later on.

```
import sys
sys.path.insert(0, '/content/Bmf-Huggingface')
print(sys.path)
```

### **Creating the Module**

Now it's time for the fun part. We'll create a module to process the video.Here's the module I created and I'll break it down for you below.

```
import bmf
from bmf import bmf_sync, Packet
from bmf import SubGraph
from diffusers import StableDiffusionPipeline
import torch

model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "a photo of a panda eating waffles"
image = pipe(prompt).images[0]

image.save("panda_photo.png")

class video_overlay(SubGraph):

    def create_graph(self, option=None):
        # create source stream
        self.inputs.append('source')
        source_stream = self.graph.input_stream('source')
        # create overlay stream
        overlay_streams = []
        for (i, _) in enumerate(option['overlays']):
            self.inputs.append('overlay_' + str(i))
            overlay_streams.append(self.graph.input_stream('overlay_' + str(i)))

        # pre-processing for source layer
        info = option['source']
        output_stream = (
            source_stream.scale(info['width'], info['height'])
                .trim(start=info['start'], duration=info['duration'])
                .setpts('PTS-STARTPTS')
        )

        # overlay processing
        for (i, overlay_stream) in enumerate(overlay_streams):
            overlay_info = option['overlays'][i]

            # overlay layer pre-processing
            p_overlay_stream = (
                overlay_stream.scale(overlay_info['width'], overlay_info['height'])
                    .loop(loop=overlay_info['loop'], size=10000)
                    .setpts('PTS+%f/TB' % (overlay_info['start']))
            )

            # calculate overlay parameter
            x = 'if(between(t,%f,%f),%s,NAN)' % (overlay_info['start'],
                                                 overlay_info['start'] + overlay_info['duration'],
                                                 str(overlay_info['pox_x']))
            y = 'if(between(t,%f,%f),%s,NAN)' % (overlay_info['start'],
                                                 overlay_info['start'] + overlay_info['duration'],
                                                 str(overlay_info['pox_y']))
            if overlay_info['loop'] == -1:
                repeat_last = 0
                shortest = 1
            else:
                repeat_last = overlay_info['repeat_last']
                shortest = 1

            # do overlay
            output_stream = (
                output_stream.overlay(p_overlay_stream, x=x, y=y,
                                      repeatlast=repeat_last)
            )

        # finish creating graph
        self.output_streams = self.finish_create_graph([output_stream])
```

### Code Breakdown:

#### Importing Required Modules:

```
import bmf
from bmf import bmf_sync, Packet
from bmf import SubGraph
from diffusers import StableDiffusionPipeline
import torch
```

-   `bmf` and its components are imported to harness the functionalities of the Babit Multimedia Framework for video processing tasks.
-   `SubGraph` is a class in BMF, used to create a customizable processing node.
-   `StableDiffusionPipeline` is imported from the `diffusers` library that allows the generation of images using text prompts.
-   `torch` is the PyTorch library used for machine learning applications, which Stable Diffusion relies on.

### Configuring the Stable Diffusion Model:

```
model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")
```

-   The Stable Diffusion model is loaded with the specified `model_id`.
-   The `torch_dtype` parameter ensures the model uses lower precision to reduce memory usage.
-   `.to("cuda")` moves the model to GPU for faster computation if CUDA is available.

### Generating an Image Using Stable Diffusion:

```
prompt = "a photo of a panda eating waffles"
image = pipe(prompt).images[0]
image.save("panda_photo.png")
```

-   We then set a text prompt to generate an image of "a photo of a panda eating waffles".
-   The image is created and saved to "panda_photo.png".

### Defining a Custom BMF SubGraph for Video Overlay:

```
class video_overlay(SubGraph):
```

-   `video_overlay` class is derived from `SubGraph`. This class will define a custom graph for video overlay operations.

#### Creating the Graph:

```
def create_graph(self, option=None):
```

-   `create_graph` method is where the actual graph (workflow) of the video and overlays are constructed.

#### Processing Source and Overlay Streams:

```
self.inputs.append('source')
source_stream = self.graph.input_stream('source')
overlay_streams = []
```

-   Registers input streams for the source and prepares a list of overlay input streams.

#### Scaling and Trimming Source Video:

```
info = option['source']
output_stream = (
    source_stream.scale(info['width'], info['height']).trim(start=info['start'], duration=info['duration']).setpts('PTS-STARTPTS'))
```

-   The source video is scaled and trimmed according to the specified `option`. Adjustments are made for the timeline placement.

#### Scaling and Looping Overlay Streams:

```
p_overlay_stream = (
    overlay_stream.scale(overlay_info['width'], overlay_info['height']).loop(loop=overlay_info['loop'], size=10000).setpts('PTS+%f/TB' % (overlay_info['start'])))
```

-   Each overlay is scaled and looped as needed, providing a dynamic and flexible overlay process.

#### Overlaying on the Source Stream:

```
output_stream = (
    output_stream.overlay(p_overlay_stream, x=x, y=y,
                          repeatlast=repeat_last))
```

-   Overlays are added to the source stream at the calculated position and with the proper configuration. This allows multiple overlays to exist within the same timeframe without conflicts.

#### Finalizing the Graph:

```
self.output_streams = self.finish_create_graph([output_stream])
```

-   Final output streams are set, which concludes the creation of the graph. Now, after this, it's time for us to encode the video and display it how we want to.

### **Applying Hugging Face Model**

Let's add our image as an overlay to the video file. Let's break down each section of the code to explain how it

```
input_video_path = "/content/Bmf-Huggingface/black_and_white.mp4"
logo_path = "/content/panda_photo.png"
output_path = "./complex_edit.mp4"
dump_graph = 0

duration = 10

overlay_option = {
    "dump_graph": dump_graph,
    "source": {
        "start": 0,
        "duration": duration,
        "width": 1280,
        "height": 720
    },
    "overlays": [
        {
            "start": 0,
            "duration": duration,
            "width": 300,
            "height": 200,
            "pox_x": 0,
            "pox_y": 0,
            "loop": 0,
            "repeat_last": 1
        }
    ]
}

my_graph = bmf.graph({
    "dump_graph": dump_graph
})

logo_1 = my_graph.decode({'input_path': logo_path})['video']

video1 = my_graph.decode({'input_path': input_video_path})

overlay_streams = list()
overlay_streams.append(bmf.module([video1['video'], logo_1], 'video_overlay', overlay_option, entry='__main__.video_overlay')[0])

bmf.encode(
    overlay_streams[0],
    video1['audio'],
    {"output_path": output_path}
    ).run()
```

### Let's break this down too

### Defining Paths and Options:

```
input_video_path = "/content/Bmf-Huggingface/black_and_white.mp4"
logo_path = "/content/panda_photo.png"
output_path = "./complex_edit.mp4"
dump_graph = 0
duration = 10
```

-   `input_video_path`: Specifies the file path to the input video.
-   `logo_path`: File path to the image (logo) you want to overlay on the video.
-   `output_path`: The file path where the edited video will be saved.
-   `dump_graph`: A debugging tool in BMF that can be set to `1` to visualize the graph but is set to `0` here, meaning no graph will be dumped.
-   `duration`: The duration in seconds for the overlay to be visible in the video.

### Overlay Configuration:

```
overlay_option = {
    "dump_graph": dump_graph,
    "source": {
        "start": 0,
        "duration": duration,
        "width": 1280,
        "height": 720
    },
    "overlays": [
        {
            "start": 0,
            "duration": duration,
            "width": 300,
            "height": 200,
            "pox_x": 0,
            "pox_y": 0,
            "loop": 0,
            "repeat_last": 1
        }
    ]
}
```

-   `overlay_option`: A dictionary that defines the settings for the source video and the overlay.
-   For the source, the width and height you want to scale the video to, and when the overlay should start and end are specified.
-   For the overlays, detailed options such as position, size, and behavior (like `loop` and `repeat_last`) are defined.

### Creating a BMF Graph:

```
my_graph = bmf.graph({"dump_graph": dump_graph
})
```

-   `my_graph` is an instance of BMF graph which sets up the processing graph (pipeline), with `dump_graph` passed as an option.

### Decoding the Logo and Video Streams:

```
logo_1 = my_graph.decode({'input_path': logo_path})['video']
video1 = my_graph.decode({'input_path': input_video_path})
```

-   The video and logo are loaded and decoded to be processed. This decoding extracts the video streams to be used in subsequent steps.

### Creating Overlay Streams:

```
overlay_streams = list()
overlay_streams.append(bmf.module([video1['video'], logo_1], 'video_overlay', overlay_option, entry='__main__.video_overlay')[0])
```

-   An empty list `overlay_streams` is created to hold the video layers.
-   The `bmf.module` function is used to create an overlay module, where the source video and logo are processed using the `video_overlay` class defined previously with the corresponding options.

### Encoding the Final Output:

```
bmf.encode(
    overlay_streams[0],
    video1['audio'],{"output_path": output_path}).run()
```

-   The final video stream, with the overlay applied, and the original audio from the input video are encoded together into a new output file specified by `output_path`.
-   The `.run()` method is called to execute the encoding process.

Our final output should look something like this:

![](https://bytedance.us.larkoffice.com/space/api/box/stream/download/asynccode/?code=NTg4MjMwNjMwZWQ3NGJiYTFlNDJiOGM2YWEwMGJhNjZfYlY1Q0FlYUdLaGFZM1dPeUs2aEFxcUcyUWlHNElTSE5fVG9rZW46SXZka2JuQkhyb2lCMFR4aWNkUXVpaWtBc3RkXzE3MDY3NTA2MjQ6MTcwNjc1NDIyNF9WNA)

Thats it! We've explored a practical example of utilizing Babit Multimedia Framework (BMF) a video editing task using AI to create an image we can overlay on a video. Now you know how to set up a BMF graph, decode the input streams, create overlay modules, and finally encode the edited video with the overlay in place. In the future, I will consider adding more AI models, like one to improve the resolution, or even a model that creates a video from text. Through the power of BMF and Hugging Face open source models, users can create complex video editing workflows with overlays that can dynamically change over time, offering vast creative possibilities.

Try it out on CoLab and tell us what you think:

https://colab.research.google.com/drive/1eQxiZc2vZeyOggMoFle_b0xnblupbiXd?usp=sharing

[Join us on our ByteDance Open Source Discord Server!](https://discord.gg/4wXUpdrK2z)