Visual Brand Detection with Azure Video Indexer
Tags: machinelearning, azure, ai, datascience
Published: 2020-05-15 23:18:28

---
title: Visual Brand Detection with Azure Video Indexer
published: true
date: 2020-05-15 15:00:03 UTC
tags: machinelearning,azure, ai, datascience
canonical_url: https://medium.com/microsoftazure/visual-brand-detection-with-azure-video-indexer-cd65330b908c
---

![](https://cdn-images-1.medium.com/max/1024/1*TZnVit_dzETymBl-KWW5yw.png)

TLDR; This post will show how to use the Azure Video Indexer, Computer Vision API and Custom Vision Services to extract key frames and detect custom image tags in indexed videos.

All code for the tutorial can be found in the notebook below. This code can be extended to support almost any image classification or object detection task.

[aribornstein/AzureVideoIndexerVisualBrandDetection](https://github.com/aribornstein/AzureVideoIndexerVisualBrandDetection/blob/master/Video%20Indexer%20Keyframe%20Brand%20Detection.ipynb)

The tutorial requires an Azure subscription, however everything can be achieved using the free tier. If you are new to Azure you can get a free subscription here.

[Create your Azure free account today | Microsoft Azure](https://azure.microsoft.com/en-us/free/?WT.mc_id=vikeyframedetection-medium-abornst)

### What is Azure Video Indexer?

![](https://cdn-images-1.medium.com/max/1024/1*jYdZ1vbeTnj6nH2k3XK5-Q.png)

[Azure Video Indexer](https://docs.microsoft.com/en-us/azure/media-services/video-indexer/video-indexer-use-apis?WT.mc_id=vikeyframedetection-medium-abornst) automatically extracts metadata — such as spoken words, written text, faces, speakers, celebrities, emotions, topics, brands, and scenes from video and audio files. Developers can then access the data within their application or infrastructure, make it more discover-able, and use it to create new over-the-top (OTT) experiences and monetization opportunities

[Use the Video Indexer API - Azure Media Services](https://docs.microsoft.com/en-us/azure/media-services/video-indexer/video-indexer-use-apis?WT.mc_id=vikeyframedetection-medium-abornst)

Often, we wish to extract useful tags from videos content.These tags are often the differentiating factor for having successful engagement on social media services such as Instagram, Facebook, and YouTube

This tutorial will show how to use Azure Video Indexer, Computer Vision API, and Custom Vision service to extract key frames and custom tags. We will use these Azure services to detect custom brand logos in indexed videos.

This code can be extended to support almost any image classification or object detection task.

### Step #1 Download A Sample Video with the pyTube API

The first step is to download a sample video to be indexed. We will be downloading an episode of [Azure Mythbusters](https://www.youtube.com/watch?v=I1_kqOIKQTQ) on [Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/?WT.mc_id=vikeyframedetection-medium-abornst) by my incredible Co-Worker [Amy Boyd](https://twitter.com/AmyKateNicho) using the Open Source [pyTube API](https://python-pytube.readthedocs.io/en/latest/)!

{% youtube ijtKxXiS4hE %}

**Installation:**

pyTube can be installed with pip

```
!pip install pytube3 --upgrade
```

**Code:**

```
from pytube import YouTube
from pathlib import Path

video2Index = YouTube('https://www.youtube.com/watch?v=ijtKxXiS4hE').streams[0].download()

video\_name = Path(video2Index).stem
```

### Step #2 Create An Azure Video Indexer Instance

![](https://cdn-images-1.medium.com/max/819/1*nGV770rBR9WMhHeih37MMg.png)

Navigate to [https://www.videoindexer.ai/](https://www.videoindexer.ai/) and follow the instructions to create an Account

For the next steps, you will need your Video Indexer

- Subscription Key
- Location
- Account Id

These can be found in the account settings page in the Video Indexer Website pictured above. For more information see the documentation below. Feel free to comment below if you get stuck.

[Use the Video Indexer API - Azure Media Services](https://docs.microsoft.com/en-us/azure/media-services/video-indexer/video-indexer-use-apis?WT.mc_id=vikeyframedetection-notebook-abornst)

### Step #3 Use the Unofficial Video Indexer Python Client to Process our Video and Extract Key Frames

![](https://cdn-images-1.medium.com/max/1024/0*hDm9h-iNcXoPGHaX)

To interact with the Video Indexer API, we will use the [unofficial Python client](https://github.com/bklim5/python_video_indexer_lib).

**Installation:**

```
pip install video-indexer
```

**Code:**

- Initialize Client:

```
vi = VideoIndexer(vi\_subscription\_key='SUBSCRIPTION\_KEY',
                  vi\_location='LOCATION',
                  vi\_account\_id='ACCOUNT\_ID')
```

- Upload Video:

```
video\_id = vi.upload\_to\_video\_indexer(
              input\_filename = video2Index,
              video\_name=video\_name, #must be unique
              video\_language='English')
```

- Get Video Info

```
info = vi.get\_video\_info(video\_id, video\_language='English')
```

- Extract Key Frame Ids

```
keyframes = []
for shot in info["videos"][0]["insights"]["shots"]:
    for keyframe in shot["keyFrames"]:
        keyframes.append(keyframe["instances"][0]['thumbnailId'])
```

- Get Keyframe Thumbnails

```
for keyframe in keyframes:
    img\_str = vi.get\_thumbnail\_from\_video\_indexer(video\_id,    
                                                  keyframe)
```

### Step #3 Use the Azure Computer Vision API to Extract Popular Brands from Key Frames

![](https://cdn-images-1.medium.com/max/1024/0*uBF1f9xlnIdKD5Fo.jpg)

Out of the box, Azure Video Indexer uses optical character recognition and audio transcript generated from speech-to-text transcription to detect references to popular brands.

Now, that we have extracted the key frames we are going to leverage the [Computer Vision API](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/client-library?pivots=programming-language-python&?WT.mc_id=vikeyframedetection-medium-abornst) to extend this functionality to see if there are any known brands in the key frames.

[Brand detection - Computer Vision - Azure Cognitive Services](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-brand-detection?WT.mc_id=vikeyframedetection-medium-abornst)

- First we will have to create a [Computer Vision API](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/client-library?pivots=programming-language-python&?WT.mc_id=vikeyframedetection-notebook-abornst) key. There is a free tier that can be used for the demo that can be generated with the instructions in the documentation link below. Once done you should get a Computer Vision **subscription key** and  **endpoint**

[Create a Cognitive Services resource in the Azure portal - Azure Cognitive Services](https://docs.microsoft.com/en-us/azure/cognitive-services/cognitive-services-apis-create-account?WT.mc_id=vikeyframedetection-notebook-abornst)

After we have our Azure Computer Vision **subscription key** and **endpoint** , we can then use the Client SDK to evaluate our video’s keyframes:

**Installation:**

```
pip install --upgrade azure-cognitiveservices-vision-computervision
```

**Code:**

- Initialize Computer Vision Client

```
from azure.cognitiveservices.vision.computervision import ComputerVisionClient
from msrest.authentication import CognitiveServicesCredentials

computervision\_client = ComputerVisionClient(endpoint, CognitiveServicesCredentials(subscription\_key))
```

- Send Keyframe To Azure Computer Vision Service to Detect Brands

```
import time

timeout\_interval, timeout\_time = 5, 10.0
image\_features = ["brands"]

for index, keyframe in enumerate(keyframes):

if index % timeout\_interval == 0:
     print("Trying to prevent exceeding request limit waiting {} seconds".format(timeout\_time))
     time.sleep(timeout\_time)

# Get KeyFrame Image Byte String From Video Indexer
img\_str = vi.get\_thumbnail\_from\_video\_indexer(video\_id, keyframe)

# Convert Byte Stream to Image Stream
img\_stream = io.BytesIO(img\_str)

# Analyze with Azure Computer Vision
cv\_results = computervision\_client.analyze\_image\_in\_stream(img\_stream, image\_features)

print("Detecting brands in keyframe {}: ".format(keyframe))

if len(cv\_results.brands) == 0:
    print("No brands detected.")

else:
    for brand in cv\_results.brands:

        print("'{}' brand detected with confidence {:.1f}% at location {}, {}, {}, {}".format( brand.name, brand.confidence \* 100, brand.rectangle.x, brand.rectangle.x + brand.rectangle.w, brand.rectangle.y, brand.rectangle.y + brand.rectangle.h))
```

#### Azure Computer Vision API — General Brand Detection

[Quickstart: Computer Vision client library - Azure Cognitive Services](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/quickstarts-sdk/client-library?pivots=programming-language-python#analyze-an-image)

### Step #4 Use the Azure Custom Vision Service to Extract Custom Logos from Keyframes

![](https://cdn-images-1.medium.com/max/1024/1*tpr1hMApdxIpZV4xLSwQ7A.png)

The [Azure Computer Vision](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-brand-detection?WT.mc_id=vikeyframedetection-medium-abornst) API, provides the ability to capture many of the worlds most popular brands, but sometimes a brand may be more obscure. In the last section, we will use the Custom Vision Service, to train a custom logo detector to detect the Azure Developer Relation Mascot Bit in in the keyframes extracted by Video Indexer.

![](https://cdn-images-1.medium.com/max/1024/1*V9uAlX5vdf7Jcwkjx79BcQ.png)<figcaption>My training set for Custom Bit Detector</figcaption>

This tutorial assumes you know how to train a [Custom Vision Service object detection model for brand detection](http://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/quickstarts/object-detection?pivots=programming-language-python?WT.mc_id=vikeyframedetection-medium-abornst). If not check out the If not, check out the documentation below for a tutorial.

[Tutorial: Use custom logo detector to recognize Azure services - Custom Vision - Azure Cognitive Services](https://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/logo-detector-mobile?WT.mc_id=vikeyframedetection-medium-abornst)

Instead of deploying to mobile, however we will use the python client API for the [Azure Custom Vision Service](http://docs.microsoft.com/en-us/azure/cognitive-services/custom-vision-service/quickstarts/object-detection?pivots=programming-language-python?WT.mc_id=vikeyframedetection-medium-abornst). All the information you’ll need can be found in the settings menu of your Custom Vision project.

![](https://cdn-images-1.medium.com/max/1024/1*ntJkUfIVNhPuWPc32YN0hw.png)<figcaption>Settings menu for Custom Vision Service</figcaption>

**Installation:**

```
pip install azure-cognitiveservices-vision-customvision
```

**Code:**

- Initialize Custom Vision Service Client

```
from azure.cognitiveservices.vision.customvision.prediction import CustomVisionPredictionClient

prediction\_threshold = .8
prediction\_key = "Custom Vision Service Key"
custom\_endpoint = "Custom Vision Service Endpoint"
project\_id = "Custom Vision Service Model ProjectId"
published\_name = "Custom Vision Service Model Iteration Name"

predictor = CustomVisionPredictionClient(prediction\_key, endpoint=published\_name)
```

- Use Custom Vision Service Model to Predict Key Frames

```
import time
timeout\_interval, timeout\_time = 5, 10.0

for index, keyframe in enumerate(keyframes):
    if index % timeout\_interval == 0:
       print("Trying to prevent exceeding request limit waiting {} seconds".format(timeout\_time))
       time.sleep(timeout\_time)

    # Get KeyFrame Image Byte String From Video Indexer
    img\_str = vi.get\_thumbnail\_from\_video\_indexer(video\_id, keyframe)

    # Convert Byte Stream to Image Stream
    img\_stream = io.BytesIO(img\_str)

    # Analyze with Azure Computer Vision
    cv\_results = predictor.detect\_image(project\_id, published\_name, img\_stream)
    predictions = [pred for pred in cv\_results.predictions if pred.probability > prediction\_threshold]
    print("Detecting brands in keyframe {}: ".format(keyframe))

    if len(predictions) == 0:
       print("No custom brands detected.")
    else:
       for brand in predictions:
           print("'{}' brand detected with confidence {:.1f}% at location {}, {}, {}, {}".format( brand.tag\_name, brand.probability \* 100, brand.bounding\_box.left, brand.bounding\_box.top, brand.bounding\_box.width, brand.bounding\_box.height))
```

#### Conclusion

And there we have it! I am able to find all the frames that have either Microsoft for or the Cloud Advocacy Bit Logo in my video.

![](https://cdn-images-1.medium.com/max/716/1*x9w-y9iaZYdmy0TwFEzqig.png)

![](https://cdn-images-1.medium.com/max/709/1*_ZGg2CPHsvCB7_GjfMK95Q.png)<figcaption>Sample Key Frames with Bit</figcaption>

### Next Steps

![](https://cdn-images-1.medium.com/max/1014/1*bXyNWLU6fHK5ZKrF3hicng.png)

You now have all you need to extend the Azure Video Indexer Service with your own custom computer vision models. Below is a list of additional resources to take that will help you take your integration with Video Indexer to the next level.

#### **Offline Computer Vision**

In a production system, you might see request throttling from a huge number of requests. In this case, the Azure Computer Vision service can be run in an offline container

[How to install and run containers - Computer Vision - Azure Cognitive Services](https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/computer-vision-how-to-install-containers?WT.mc_id=vikeyframedetection-medium-abornst)

Additionally, the Custom Vision model can be run locally as well.

[Tutorial - Deploy Custom Vision classifier to a device using Azure IoT Edge](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-deploy-custom-vision?WT.mc_id=vikeyframedetection-medium-abornst)

#### Video Indexer + Zoom Media

[Azure-Samples/media-services-video-indexer](https://github.com/Azure-Samples/media-services-video-indexer/tree/master/Partners/IntegrationWithZoommedia)

#### Creating an Automated Video Processing Flow in Azure

[Creating an automated video processing flow in Azure](https://fabriciosanchez-en.azurewebsites.net/creating-an-automated-video-processing-flow-in-azure/)

### About the Author

[**Aaron (Ari) Bornstein**](https://www.linkedin.com/in/aaron-ari-bornstein-22aa7a77/) is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As an Open Source Engineer at Microsoft’s Cloud Developer Advocacy team, he collaborates with Israeli Hi-Tech Community, to solve real world problems with game changing technologies that are then documented, open sourced, and shared with the rest of the world.

* * *