Building a Web-Based Real-Time Computer Vision App with Streamlit
Tags: streamlit, computervision, python, machinelearning
Published: 2021-02-04 17:56:31

This article is based on an older version of the library and out-of-date.
See [this new tutorial](https://dev.to/whitphx/developing-web-based-real-time-videoaudio-processing-apps-quickly-with-streamlit-4k89) ✌️

---

[Streamlit](https://www.streamlit.io/) is a great framework for data scientists, machine learning researchers and developers, and [**`streamlit-webrtc`**](https://github.com/whitphx/streamlit-webrtc) extends it to be able to deal with **real-time video (and audio) streams**.
It means you can implement your computer vision algorithms only in Python on server-side and users can use them as web apps with real-time video inputs from their webcams or smartphones.

[![Streamlit Object Detection App](https://aws1.discourse-cdn.com/business7/uploads/streamlit/original/2X/a/af111a7393c77cb69d7712ac8e71ca862feaeb24.gif)](https://share.streamlit.io/whitphx/streamlit-webrtc-example/main/app.py)

This article is a short tutorial explaining how to use this library to develop such kinds of apps.

# Prerequisites
Install `streamlit`.
```shell
$ pip install streamlit
```

If you are not familiar with Streamlit, I recommend you to see [its tutorial](https://docs.streamlit.io/en/stable/). It takes just a few minutes.

# Using `streamlit-webrtc`

## Install necessary packages
Install `streamlit-webrtc`. This tutorial is based on the version `0.6.3`. The API might be changed in future releases.
```shell
$ pip install streamlit-webrtc
$ # pip install streamlit-webrtc==0.6.3 # Specify the exact version if necessary
```

Install `opencv-python` as well.
```shell
$ pip install opencv-python
```

## First example
First of all, let's simply put the `streamlit_webrtc`'s custom component on the page as an example below.
```python
# simple_loopback.py

import streamlit as st
from streamlit_webrtc import webrtc_streamer

webrtc_streamer(key="example")
```

Note that `key` argument is not optional unlike other Streamlit components. An arbitrary unique string must be set to it.

Then run the app with `streamlit run` command as below.
```shell
$ streamlit run simple_loopback.py
```

After opening the app on the browser, click the "Start" button.
Then a video sourced from your webcam is displayed.
(You may be asked for permission to access the webcam and microphone. If so, allow it.)

This is the simplest example of `streamlit-webrtc` - capturing video from a local webcam, sending it to the Python process, and sending it back to the frontend to display the preview.

![simple_loopback example](https://dev-to-uploads.s3.amazonaws.com/i/xiflxci617a1ivmxw78s.gif)

## Simple frame filter
Next, we will see how to inject computer vision code into the process.

Here is the next example, the Canny filter with OpenCV.
```python
# canny_filter.py

import cv2
from streamlit_webrtc import VideoTransformerBase, webrtc_streamer


class VideoTransformer(VideoTransformerBase):
    def transform(self, frame):
        img = frame.to_ndarray(format="bgr24")

        img = cv2.cvtColor(cv2.Canny(img, 100, 200), cv2.COLOR_GRAY2BGR)

        return img


webrtc_streamer(key="example", video_transformer_factory=VideoTransformer)
```

Run this code and see the result.
```shell
$ streamlit run canny_filter.py
```

You can see the Canny filter is applied to the video stream in real time.

![The Canny filter example](https://dev-to-uploads.s3.amazonaws.com/i/lcme7uab0jgzlu57mvfe.gif)

`webrtc_streamer` can take `video_transformer_factory` argument, which is a `Callable` that returns an instance of a class which has `transform(self, frame)` method.
If you like type annotation, you can use an ABC (abstract base class), `VideoTransformerBase`, which defines this interface, like the example above.

Anyway, as the example above shows, you can implement arbitrary frame filters inside the `transform()` method, which transforms each frame coming from the video stream.
`transform()` takes an argument `frame` and returns a numpy array `img`.
The argument `frame` is an instance of `av.VideoFrame` from [`av` package](https://pypi.org/project/av/) and the returned value `img` is a numpy array representing the output frame picture.

Note that the video stream is processed in a forked thread independent of the Streamlit's script execution flow in the main thread, and `transform()` is called from the forked thread.

## Interactive controls
One of the biggest benefits of using Streamlit is interactive controls such as sliders and radio buttons.
`streamlit-webrtc` works well in combination with them.

See the next example.

```python
# interactive.py

import cv2
import streamlit as st
from streamlit_webrtc import VideoTransformerBase, webrtc_streamer


class VideoTransformer(VideoTransformerBase):
    def __init__(self):
        self.threshold1 = 100
        self.threshold2 = 200

    def transform(self, frame):
        img = frame.to_ndarray(format="bgr24")

        img = cv2.cvtColor(
            cv2.Canny(img, self.threshold1, self.threshold2), cv2.COLOR_GRAY2BGR
        )

        return img


ctx = webrtc_streamer(key="example", video_transformer_factory=VideoTransformer)

if ctx.video_transformer:
    ctx.video_transformer.threshold1 = st.slider("Threshold1", 0, 1000, 100)
    ctx.video_transformer.threshold2 = st.slider("Threshold2", 0, 1000, 200)

```

and run this code.
```shell
$ streamlit run interactive.py
```

![The interactive controls examle](https://dev-to-uploads.s3.amazonaws.com/i/zy2t4ybcaic9vls7m2si.gif)

This example is based on the previous one and extended to use sliders to control the Canny filter parameters.

The `ctx` object returned from `webrtc_streamer()` is used here to refer to the `.video_transformer` attribute. This attribute is an instance of `VideoTransformer` instantiated by `video_transformer_factory`.

Then the values from sliders are passed to be used in `transform()` via the instance attributes, `ctx.video_transformer.threshold1` and `ctx.video_transformer.threshold2`.
This is because `transform()` is called and running in another thread in async way as stated above.
Make sure that this value passing is thread-safe, and be careful especially when you pass complex objects.

# More examples and information
While only a simple example with the Canny filter has been shown in this article,
you can develop more interesting, any kinds of computer vision apps.

For example, I'm hosting a sample Object Detection app with MobileNet SSD at
https://share.streamlit.io/whitphx/streamlit-webrtc-example/main/app.py (Streamlit Sharing)
and
https://streamlit-webrtc-example.herokuapp.com/ (Heroku free plan).
![Object Detection App](https://dev-to-uploads.s3.amazonaws.com/i/x9y9g910qnroenk5kd97.gif)
https://github.com/whitphx/streamlit-webrtc-example is a repository of this app.

You can also try this app on your local environment with the commands below:
```shell
$ pip install streamlit-webrtc opencv-python
$ streamlit run https://raw.githubusercontent.com/whitphx/streamlit-webrtc-example/main/app.py
```
as explained [`streamlit-webrtc`'s README](https://github.com/whitphx/streamlit-webrtc).

[This forum topic](https://discuss.streamlit.io/t/new-component-streamlit-webrtc-a-new-way-to-deal-with-real-time-media-streams/8669) may be useful to know more about this library.

# Hosting on remote servers
We have been working on localhost, and deploying apps on remote hosts might be a next step.
It is a nice usage of `streamlit` and `streamlit-webrtc` as it allows users to easily try the app remotely, with their local webcams or smartphones.

There are some things you need to be aware of to do it, especially with `streamlit-webrtc`.

## Network connectivity
Video streaming does not work in some network environments.
For example, in some office networks, there are firewalls which drop the WebRTC packets.

<!-- In such case,
* you have to ask the administrator.
* you can consider using TURN server.
    * `streamlit-webrtc` uses WebRTC to send and receive media streams between the frontend and the server. WebRTC provides a solution for such situations - **TURN server**. The reason some well-known video chat tools like Zoom and Google Meet work even in such networks may be using TURN server. -->

## HTTPS
`streamlit-webrtc` uses [`getUserMedia()`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia) API to access local webcams, and this method does not work in an insecure context.

[This document](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia#privacy_and_security) says
> A secure context is, in short, a page loaded using HTTPS or the file:/// URL scheme, or a page loaded from localhost.

So, when hosting your app on a remote server, it must be served via HTTPS.

One of the easiest way to do it is to use some hosting services like Heroku, which provides HTTPS serving by default. Especially for apps using `streamlit`, [Streamlit Sharing](https://www.streamlit.io/sharing) can be a choice.
The Object Detection sample app shown above is actually using both Heroku and Streamlit Sharing. You can see the deployment settings of this app for those services in [the repository](https://github.com/whitphx/streamlit-webrtc-example).

If you have to serve the app on hosts inside local networks, you can use some reverse proxy software like **nginx**.
In prototyping phase, https://github.com/suyashkumar/ssl-proxy might be a simple but useful option.