Remote View your Computer Vision Models Running on AWS Panorama
Tags: aws, machinelearning, python, computervision
Published: 2022-05-09 15:41:22

Real-time smart video analytics application development and edge device deployment is a tricks task. In the last few years, the industry players built platforms to support this activity. Notable examples include [NVIDIA Metropolis](https://www.nvidia.com/en-us/autonomous-machines/intelligent-video-analytics-platform/), [Intel’s OpenVINO](https://www.intel.com/content/www/us/en/developer/tools/openvino-toolkit/overview.html), and [AWS Panorama](https://aws.amazon.com/panorama/). While these solutions make some aspects of the video-analytics application development more straightforward, there are still many issues to deal with before deploying a video analytics application in production. This post introduces Telescope, the first in a [series of open-source tools](https://github.com/Neosperience/backpack) to make developing AWS Panorama application simpler.

AWS Panorama is a machine learning appliance and software framework that allows you to deploy video analytics applications on edge. For a thorough introduction and a step-by-step tutorial on deploying a Panorama application, refer to _[Deploy an Object-Detector Model at the Edge on AWS Panorama](https://towardsdatascience.com/deploy-an-object-detector-model-at-the-edge-on-aws-panorama-9b80ea1dd03a)_.

AWS Panorama framework eases many aspects of video analytics application development, including camera connection management, video decoding, frame extraction, model optimization and loading, display output management, over-the-air deployment of your application, and other features. Nevertheless, some tasks are still challenging, including diagnostic tasks when a model does not work as expected.

## Introducing Telescope

The only available way to get visual feedback on the correct functionality of a Panorama application is to physically connect a display to the HDMI port of the appliance. The display will show the output video stream of a single application deployed on the device. However, physically accessing the appliance is not always feasible. Telescope allows to re-stream the output video of any Panorama application to an external service, for example, to [AWS Kinesis Video Streams](https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/what-is-kinesis-video.html). This feature can be very convenient for monitoring an application remotely.

## How does Telescope work?

Telescope instantiates a [GStreamer pipeline](https://gstreamer.freedesktop.org/documentation/application-development/introduction/basics.html) with an [`appsrc`](https://gstreamer.freedesktop.org/documentation/app/appsrc.html) element at the head. An [OpenCV](https://docs.opencv.org/4.5.5/dd/d43/tutorial_py_video_display.html) [`VideoWriter`](https://docs.opencv.org/4.5.5/dd/d43/tutorial_py_video_display.html) is configured to write to the `appsrc`: instead of saving the consecutive frames to a video file, it streams to the output sink. When opening the `VideoWriter` instance, the user should specify the frame width and height and the frame rate of the output stream. You can manually set these parameters or let Telescope infer them from the input dimensions and the frequency you feed new frames to it. If using this auto-configuration feature, some frames (by default 100) will be discarded at the beginning of the streaming because Telescope will use them to calculate statistics of the frame rate and measure the frame dimensions. This is the “warmup” of Telescope. If you send frames with different sizes, Telescope will resize the input, but this comes with a performance penalty. You are also expected to send new frames to Telescope with the frequency specified in the frame-per-second parameter. If frames are sent at a different frequency, the video fragments get out of sync in Kinesis Video Streams, and you won’t be able to replay the video smoothly.

<figure>
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9ptokgyxahjd4syvwvly.jpeg" alt="Cogwheels of a complicated mechanism"><figcaption>Photo by <a href="https://unsplash.com/@joshredd">Josh Redd</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption>
</figure>

`Telescope` is an abstract base class that handles the state machine of the GStreamer pipeline. The concrete implementation KVSTelescope sends frames to [Amazon Kinesis Video Streams](https://aws.amazon.com/kinesis/video-streams/) service. It is easy to extend Telescope to support other services, especially if there is already a GStreamer plugin for that.

## Integrating Telescope into a Panorama application

### Configuring the Docker container

Telescope depends on custom compiled external libraries. All these libraries must be compiled and configured correctly in your application’s docker container to make `Telescope` work. These libraries include:

 - GStreamer 1.0 installed with standard plugins pack, libav, tools, and development libraries;
 - OpenCV 4.2.0, compiled with GStreamer support and Python bindings;
 - numpy (it is typically installed by the base docker image of your Panorama application).

In the case you want to use `KVSTelescope`, the `Telescope` implementation that streams the video to Amazon Kinesis Video Streams, it will need also the following libraries:

 - Amazon Kinesis Video Streams (KVS) Producer SDK compiled with GStreamer plugin support;
 - Environment variable `GST_PLUGIN_PATH` configured to point to the directory where the compiled binaries of KVS Producer SDK GStreamer plugin is placed;
 - Environment variable `LD_LIBRARY_PATH` including the third-party open-source dependencies compiled by KVS Producer SDK;
 - boto3 (it is typically installed by the base docker image of your Panorama application).

In the source code repository of Telescope, a [sample Dockerfile](https://github.com/Neosperience/backpack/blob/master/examples/Dockerfile) is provided in the examples folder that shows how to install these libraries and Telescope in any container correctly. In most cases, it is required just to copy the relevant sections from the sample to your application’s Dockerfile. Please note that the first time you compile the docker container, it might take up to one hour to correctly compile all libraries.

### Setting up AWS IAM privileges for KVSTelescope

`KVSTelescope` uses the Amazon Kinesis Video Streams (KVS) Producer library, wrapped in a GStreamer sink element, to send the processed frames to the KVS service. KVS Producer needs AWS credentials. It can use the credentials associated with the Panorama Application Role, but you must explicitly configure it. `KVSTelescope` needs privileges to execute the following actions:

```text
kinesisvideo:DescribeStream
kinesisvideo:GetStreamingEndpoint
kinesisvideo:PutMedia
```

If KVSTelescope needs to automatically create the Kinesis Video Stream the first time it is used, you should also include the `kinesisvideo:CreateStream` action. An example policy allowing KVSTelescope to write data to Kinesis Video Streams could look like this:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesisvideo:DescribeStream",
                "kinesisvideo:CreateStream",
                "kinesisvideo:GetDataEndpoint",
                "kinesisvideo:PutMedia"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
```

### Setting up credentials

There are two types of AWS security credentials: static and temporary. The former never expire and in case of a leak, you should invalidate them manually and reconfigure the applications. For this reason, their use is strongly discouraged in a production environment. Examples of static AWS credentials include the IAM user’s Access Key Id and Secret Access Key pair.

<figure>
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qrtceqvziyxy34tnreq6.jpeg" alt="A photo of a keychain"><figcaption>Photo by <a href="https://unsplash.com/@mangelinka">Angela Merenkova</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption>
</figure>

Temporary credentials expire after a predefined period. In case of a leak, they can be used only within their expiration time, typically in the order of several hours. Temporary credentials can be renewed before they expire so their lifespan can be extended, or they can be exchanged for new ones with a later expiration time. This process needs additional coordination from the application using this type of credentials. Examples of temporary credentials include the AWS Access Key Id, the Secret Access Key, and the Session Token.

We offer different options to provide credentials with `KVSCredentialsHandler` subclasses provided in the `kvs` module. If you want to use static credentials for testing purposes, [create an IAM user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html) in your AWS account and [attach a policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html) like the one above to the user. You should configure this user to have programmatic access to AWS resources and get the user’s [AWS Access Key and Secret Key pair](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html): create a `KVSInlineCredentialsHandler` or `KVSEnvironmentCredentialsHandler` instance in the application’s code to pass these credentials to KVS Producer Plugin directly in the GStreamer pipeline definition or as environment variables. However, as these credentials do not expire, using this configuration in a production environment is not recommended. Even in a development and testing environment, you should take the appropriate security measures to protect these credentials: never hardcode them in the source code. Instead, use AWS Secret Manager or a similar service to provide these parameters to your application.

`KVSTelescope` can also use the [Panorama Application Role](https://docs.aws.amazon.com/panorama/latest/dev/permissions-application.html) to pass the application's credentials to KVS Producer. These credentials are temporary, meaning that they expire within a couple of hours, and the user should renew them before expiration. The Producer library expects temporary credentials in a text file. `KVSFileCredentialsHandler` manages the renewal of the credentials and periodically updates the text file with the new credentials. If you want to use this method, attach a policy similar to the example above to your Application Role. Remember to always test your Panorama application — KVS integration if it still works after the `KVSFileCredentialsHandler` refreshed the credentials. Let your application run for several hours and periodically check if it continues to stream the video to KVS. You will also find diagnostic information in the CloudWatch logs of your application when the credentials were renewed.

### Environment variables

`KvsTelescope` needs two environment variables to have GStreamer find the KVS Producer plugin. The name of these variables is `GST_PLUGIN_PATH` and `LD_LIBRARY_PATH`. They point to the folder of KVS Producer GStreamer plugin and its 3rd party dependencies. In the sample Dockerfile provided, the correct values of these variables are written to a small configuration file named `/panorama/.env` in the container. You should pass the path of this file to `KvsTelescope` or otherwise ensure that these environment variables contain the correct value.

## Usage example

When initializing a `KVSTelescope` instance, you should pass the AWS region name where your stream is created, the name of the stream, and a credentials handler instance. If you want to configure the frame rate and the dimensions of the frames manually, you should also set them here. When both parameters are specified, Telescope skips the warmup period and sends the first frame directly to KVS. When you are ready to send frames, call the `start_streaming` method which opens the GStreamer pipeline. After this method is called, you are expected to send new frames to the stream calling the `put` method periodically, with the frequency specified in the frame rate or inferred by `KvsTelescope`. You can stop and restart streaming on the same `KvsTelescope` instance.

The following example uses the temporary credentials of the IAM Application Role assumed by your Panorama application:

```python
import panoramasdk
from backpack.kvs import KVSTelescope, KVSFileCredentialsHandler

# You might want to read these values from 
# Panorama application parameters
stream_region = 'us-east-1'
stream_name = 'panorama-video'

# The example Dockerfile writes static 
# configuration variables to this file
# If you change the .env file path in the 
# Dockerfile, you should change it also here
DOTENV_PATH = '/panorama/.env'

class Application(panoramasdk.node):

    def __init__(self):
        super().__init__()
        # ...
        credentials_handler = KVSFileCredentialsHandler()
        self.telescope = KVSTelescope(
            stream_region=stream_region,
            stream_name=stream_name,
            credentials_handler=credentials_handler,
            dotenv_path=DOTENV_PATH
        )
        # This call opens the streaming pipeline:
        self.telescope.start_streaming()

    # called from video processing loop:
    def process_streams(self):
        streams = self.inputs.video_in.get()

        for idx, stream in enumerate(streams):
            
            # Process the stream, for example with:
            # self.process_media(stream)

            # TODO: eventually multiplex streams 
            # to a single frame
            if idx == 0:
                self.telescope.put(stream.image)
```

If everything works well, you can watch the restreamed video on the [Kinesis Video Streams page](https://console.aws.amazon.com/kinesisvideo/home) of the AWS console. Certainly, you can modify the image before sending it to Telescope: draw annotations on it based on the inference result of the deep-learning model.

## Annotations

_Annotations_ and _annotation drivers_ provide a unified way to draw annotations on different rendering backends. Currently, two annotation drivers are implemented:

 - `PanoramaMediaAnnotationDriver` allows you to draw on panoramasdk.media object, and
 - `OpenCVImageAnnotationDriver` allows you to draw on an OpenCV image (numpy array) object.

<figure>
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vkx808nr4cpf2pfceoky.jpeg" alt="A photo of colorful pencils"><figcaption>Photo by <a href="https://unsplash.com/@lucasgwendt">Lucas George Wendt</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption>
</figure>

The library can draw two types of annotations: labels and boxes.

### Using annotations

Depending on the available backends, you can create one or more annotation driver instances at the beginning of the video frame processing loop. During the process of a single frame, you are expected to collect all annotations to be drawn on the frame in a python collection (for example, in a `list`). When the processing is finished, you call the `render` method on any number of drivers, passing the same collection of annotations. All coordinates used in annotation are normalized to the `[0; 1)` range.

Example usage:

```python
import panoramasdk
from backpack.annotation import (
    Point, LabelAnnotation, RectAnnotation, TimestampAnnotation,
    OpenCVImageAnnotationDriver, 
    PanoramaMediaAnnotationDriver
)

class Application(panoramasdk.node):

    def __init__(self):
        super().__init__()
        # self.telescope = ... 
        self.panorama_driver = PanoramaMediaAnnotationDriver()
        self.cv2_driver = OpenCVImageAnnotationDriver()

    # called from video processing loop:
    def process_streams(self):
        streams = self.inputs.video_in.get()
        for idx, stream in enumerate(streams):
            annotations = [
                TimestampAnnotation(),
                RectAnnotation(
                    point1=Point(0.1, 0.1), 
                    point2=Point(0.9, 0.9)
                ),
                LabelAnnotation(
                    point=Point(0.5, 0.5), 
                    text='Hello World!'
                )
            ]
            self.panorama_driver.render(annotations, stream)

            # TODO: eventually multiplex streams to a 
            # single frame
            if idx == 0:
                self.cv2_driver.render(
                    annotations, 
                    stream.image
                )
                # self.telescope.put(stream.image)
```

## Postface

Even if Telescope can be a helpful tool, its usage might raise two concerns that you should consider carefully. We discourage using Telescope in a production environment: it is a development aid or and a debugging tool.

<figure>
<img src="https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vtzdgtuep77n7dxenbw0.jpeg" alt="A photo of a mountain path with a table saying danger"><figcaption>Photo by <a href="https://unsplash.com/@greg_rosenke">Greg Rosenke</a> on <a href="https://unsplash.com/">Unsplash</a></figcaption>
</figure>

The first concern is of technical nature. Currently, the application code in a Panorama app does not have direct access to the onboard GPU; thus, all video encoding done by Telescope run on the device’s CPU. This behavior could slow down the appliance: streaming a single output stream with Telescope could require anything between 10–30% of the CPU capacity of the device.

The second concern is related to data protection. The Panorama appliance is designed to protect the video streams being processed. It has two ethernet interfaces to separate the network of the video cameras (typically a closed-circuit local area network) from the device’s Internet access. Using Telescope, you relay the video stream from the protected, closed-circuit camera network to the public Internet. You should carefully examine the data protection requirements of your application and the camera network before using Telescope. Moreover, you are responsible for keeping private all AWS credentials used by Telescope, as required by [AWS Shared Responsibility Model](https://aws.amazon.com/compliance/shared-responsibility-model/).

## Where to go from here?

`Telescope` is part of `Backpack`, a broader set of tools that aims to help software development on AWS Panorama. Other components will be presented in a future post, so keep tuned and follow us to get updated.

`Backpack` is [open source and available on GitHub](https://github.com/Neosperience/backpack). You can build and run the [example docker container](https://github.com/Neosperience/backpack/blob/master/examples/Dockerfile) on an ARM64 based system. For example, suppose you’ve already set up the [Panorama Test Utility](https://github.com/aws-samples/aws-panorama-samples/blob/main/docs/EnvironmentSetup.md) on a t4g type EC2 instance. In this case, you can build and launch a shell in this container with the following commands on the EC2 instance:

```sh
$ git clone https://github.com/Neosperience/backpack.git
$ cd backpack/examples
$ docker build . -t my_backpack_example:1.0
$ docker run -it my_backpack_example:1.0 bash
```

The container can be built and executed also on a MacBook Pro with the new M1 ARM-based chip.

The backpack library is extensively documented with docstrings. You can read the [detailed API docs online](https://s3.eu-west-1.amazonaws.com/github-ci.experiments.neosperience.com/Neosperience/backpack/docs/index.html) or using the `python3` interpreter in the container shell:

```sh
root@my_backpack_example:/# python3
>>> import backpack.telescope, backpack.kvs, backpack.annotation
>>> help(backpack.telescope)
>>> help(backpack.kvs)
>>> help(backpack.annotation)
```

You can also try other `Telescope` implementations found in Backpack. For example, `RTSPTelescope` allows you to host an RTSP server right in the application container, and connect directly to your Panorama appliance with an RTSP client to monitor your application.

Let me know how you use `Telescope`, which new features you would like to see and stay tuned for future posts about the other components of `Backpack`.

<hr>

## About the author

Janos Tolgyesi is an AWS Community Builder working as Machine Learning Solution Architect at Neosperience. He has worked with ML technologies for five years and with AWS infrastructure for eight years. He loves building things, let it be a [video analytics application on the edge](https://www.neosperience.com/solutions/people-analytics/) or a [user profiler based on clickstream events](https://www.neosperience.com/solutions/user-insight/). You can find me here, on [dev.to](https://dev.to/mrtj), on [Twitter](https://twitter.com/jtolgyesi), [Medium](https://towardsdatascience.com/@janos.tolgyesi), and [LinkedIn](http://linkedin.com/in/janostolgyesi).

The open-source project of Backpack was supported by [Neosperience](https://www.neosperience.com/).

I would like to express my special thanks to 
[Luca Bianchi](https://twitter.com/bianchiluca) for proofreading this article.