5 Steps to Training your first Video Classifier in a Flash
Tags: deeplearning, pytorchlightning, pytorch, machinelearning
Published: 2021-05-17 11:22:24

---
title: 5 Steps to Training your first Video Classifier in a Flash
published: True
date: 2021-05-13 11:51:23 UTC
tags: deeplearning,pytorchlightning,pytorch,machinelearning
canonical_url: https://devblog.pytorchlightning.ai/5-steps-to-training-your-first-video-classifier-in-a-flash-dd11d472fded
---

#### Learn step by step how to build, train and make predictions on a video classification task with 3 simple tools: Lightning Flash, PyTorch Video, and Kornia.

![](https://cdn-images-1.medium.com/max/1024/0*U7QYo989MKG86EIZ)

Video Understanding enables computers to recognize actions, objects, and events in videos. From Retail, Health Care to Agriculture, Video Understanding enables automation of [countless industry use cases](https://www.cio.com/article/3431138/ai-gets-the-picture-streamlining-business-processes-with-image-and-video-classification.html).

![](https://cdn-images-1.medium.com/max/1024/0*10HEDLmt5V6GppWI.jpg)<figcaption>“A case in point: Walmart has developed an AI system that inspects fresh food for signs of defects and spoilage. This system is helping Walmart monitor the temperature and freshness of produce and perishable foods, improve visual inspection at distribution centers, and route perishable food to the nearest store.“ [<a href="https://www.cio.com/article/3431138/ai-gets-the-picture-streamlining-business-processes-with-image-and-video-classification.html">Source</a>]</figcaption>

[Facebook AI Research](https://ai.facebook.com)recently released a new library called [**PyTorchVideo**](https://pytorchvideo.org/) powered by [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning) that simplifies video understanding by providing SOTA pre-trained video models, datasets, and video-specific transformers. All the library’s models are highly optimized for inference and support different datasets and transforms.

[**Lightning Flash**](https://github.com/PyTorchLightning/lightning-flash) is a collection of tasks for fast prototyping, baselining, and fine-tuning scalable Deep Learning models, built on [PyTorch Lightning](http://pytorchlightning.ai/). It allows you to train and finetune models without being overwhelmed by all the details, and then seamlessly override and experiment with Lightning for full flexibility. Learn more about Flash [here](https://lightning-flash.readthedocs.io/).

[PyTorchLightning/lightning-flash](https://github.com/PyTorchLightning/lightning-flash)

In its new coming release, [**PyTorch Lightning Flash**](https://github.com/PyTorchLightning/lightning-flash) will provide a deep integration with [**PyTorchVideo**](https://pytorchvideo.org/) and its backbones and transforms can be used alongside [**Kornia**](https://kornia.github.io/) ones to provide a seamless preprocessing, training, and fine-tuning experience of SOTA pre-trained video classification models.

[**Kornia**](https://kornia.github.io/) is a differentiable computer vision library for [PyTorch](https://pytorch.org/) that consists of a set of routines and differentiable modules to solve generic computer vision problems.

In this article you will learn how to train a custom video classification model in 5 simple steps using PyTorch Video, Lightning Flash, and Kornia, using the Kinetics dataset.

![](https://cdn-images-1.medium.com/max/700/0*HDKt6W6q2QGgAMWe.jpg)<figcaption><a href="https://deepmind.com/research/open-source/kinetics">The Kinetics human action video dataset</a> released by DeepMind is comprised of annotated~10s video clips sourced from YouTube.</figcaption>

Keep reading to train your own model in a  **flash**!

### 5 Simple Steps for Video Classification

Before we get started all the steps in this tutorial are reproducible and can be run with the free trial of Grid by clicking the Run button below.

[![](https://cdn-images-1.medium.com/max/262/0*fWRfmrWier8GPzrb.png)](https://platform.grid.ai/#/runs?script=https://github.com/aribornstein/KineticsDemo/blob/4fcf30e1c2fd46247ec0fc1a6cb0886e9838586f/train.py&cloud=grid&instance=g4dn.xlarge&accelerators=1&disk_size=200&framework=lightning&script_args=grid%20train%20--g_name%20kinetics-demo-5%20--g_disk_size%20200%20--g_max_nodes%2010%20--g_instance_type%20g4dn.xlarge%20--g_gpus%201%20train.py%20--gpus%201%20--max_epochs%2010)<figcaption>All the steps in this tutorial are reproducible and can be run with the free trial of Grid by clicking the Run button above.</figcaption>

![](https://cdn-images-1.medium.com/max/1024/1*EV6YVi7658ZChGD2nowG1A.png)

> The [Grid Platform](https://docs.grid.ai/) enables you to seamlessly train hundreds machine learning models on the cloud from your laptop without modifying your code in a fully reproducible manner.

> **Find the full end-to-end code for this tutorial code** [**here**](https://github.com/aribornstein/KineticsDemo) **and the full reproducible run** [**here**](https://platform.grid.ai/#/runs?script=https://github.com/aribornstein/KineticsDemo/blob/188f1948725506914b67d3814073a7bec152ac0a/train.py&cloud=grid&instance=g4dn.xlarge&accelerators=1&disk_size=200&framework=lightning&script_args=grid%20train%20--g_name%20kinetics-demo-7%20--g_disk_size%20200%20--g_max_nodes%2010%20--g_instance_type%20g4dn.xlarge%20--g_gpus%201%20train.py%20--gpus%201%20--max_epochs%203) **on ** [**Grid**](https://www.grid.ai/) **.**

#### Prerequisite Imports

[![](https://cdn-images-1.medium.com/max/1024/1*jDKYE93hbSdqp3qIPNUzaw.png)](https://gist.github.com/tchaton/f3ceafeb246b46f8c8f7d9e4498760b0)<figcaption><a href="https://gist.github.com/tchaton/f3ceafeb246b46f8c8f7d9e4498760b0">https://gist.github.com/tchaton/f3ceafeb246b46f8c8f7d9e4498760b0</a></figcaption>

#### Step 1- Download The Kinetics Dataset

[![](https://cdn-images-1.medium.com/max/1024/1*fWwARV6mhVCN2jSOHm8tJw.png)](https://gist.github.com/tchaton/a4744780de0f534a613fc738afbac8c2)<figcaption><a href="https://gist.github.com/tchaton/a4744780de0f534a613fc738afbac8c2">https://gist.github.com/tchaton/a4744780de0f534a613fc738afbac8c2</a></figcaption>

#### Step 2- Specify Kornia Transforms to Apply to Video

![](https://cdn-images-1.medium.com/max/957/0*qofNAX_qSEYa5-gm.gif)<figcaption>Flash helps you to place your transform exactly where you want and makes Video Pre and Post Processing Simple as easy as a few lines of initialization code</figcaption>

As videos are memory-heavy objects, data sampling and data augmentation play a key role in the video model training procedure and need to be carefully crafted to reach SOTA results.

[**Flash**](https://github.com/PyTorchLightning/lightning-flash) optimizes this process as transforms can be provided with simple dictionary mapping between hooks and the transforms. [Flash hooks](https://lightning-flash.readthedocs.io/en/latest/general/data.html?highlight=hooks#terminology) are simple functions that can be overridden to extend Flash’s functionality.

In the code below, we use the [**post\_tensor\_transform**](https://lightning-flash.readthedocs.io/en/latest/general/data.html?highlight=post_tensor_transform#flash.data.process.Preprocess.post_tensor_transform) hook to uniformly select 8 frames from a sampled video clip and resize them to 224 pixels. This is done in **parallel** within each of the PyTorch DataLoader workers.

We also use the [**per\_batch\_transform\_on\_device**](http://per_batch_transform_on_device) hook to enable Kornia normalization to be applied directly to a batch already moved to GPU or TPU .

[![](https://cdn-images-1.medium.com/max/1024/1*7TOHKpMtv8gLIgYrp80Yiw.png)](https://gist.github.com/tchaton/b627d56e6d3da89eb456aede48c3c171)<figcaption><a href="https://gist.github.com/tchaton/b627d56e6d3da89eb456aede48c3c171">https://gist.github.com/tchaton/b627d56e6d3da89eb456aede48c3c171</a></figcaption>

#### Step 3- Load the Data

Lightning [DataModule](https://pytorch-lightning.readthedocs.io/en/latest/extensions/datamodules.html)s are shareable, reusable objects that encapsulate all data-related code. Flash enables you to quickly load videos and labels from various formats such as config files or folders into [DataModule](https://pytorch-lightning.readthedocs.io/en/latest/extensions/datamodules.html)s.

In this example, we use the Flash from\_folders helper function to generate a [DataModule](https://pytorch-lightning.readthedocs.io/en/latest/extensions/datamodules.html). To use the from\_folders functions videos should be organized into folders for each class as follows:

> dir\_path/<class\_name>/<video\_name>.{mp4, avi}

![](https://cdn-images-1.medium.com/max/1024/1*htT4iyi5MM229K4gjXE1yw.png)<figcaption>Code for Initializing a D<a href="https://pytorch-lightning.readthedocs.io/en/latest/extensions/datamodules.html">ataModule</a> on using Flash on the Kinetics Dataset</figcaption>

#### Step 4- Instantiate the VideoClassifier with a pre-trained Model Backbone .

Training a SOTA video classifier from scratch on Kinetics can take around 2 days on 64 GPUS. Thankfully, the Flash [Backbones Registry API](https://lightning-flash.readthedocs.io/en/latest/general/registry.html) makes it easy to integrate individual models as well as entire model hubs with Flash.

By default, the [PyTorchVideo](https://pytorchvideo.org/) [Model Hub](https://github.com/facebookresearch/pytorchvideo/tree/master/pytorchvideo/models/hub) is pre-registered within the Video Classifier Task [Backbone Registry](https://lightning-flash.readthedocs.io/en/latest/general/registry.html).

You can list any of the available backbones and get more details as follows:

[![](https://cdn-images-1.medium.com/max/1024/1*H031LIOrG3Ool7ecesPP3A.png)](https://gist.github.com/tchaton/d2f2b52f77d7ed5a537d686e124c2b60)<figcaption><a href="https://gist.github.com/tchaton/d2f2b52f77d7ed5a537d686e124c2b60">https://gist.github.com/tchaton/d2f2b52f77d7ed5a537d686e124c2b60</a></figcaption>

Once you pick a backbone, you can initialize it with just one line of code:

[![](https://cdn-images-1.medium.com/max/1024/1*2GtEGLWf7AwWIz1K-eZ3tA.png)](https://gist.github.com/tchaton/0e83bff7e27e125cc675bcdd31d3bf23)<figcaption><a href="https://gist.github.com/tchaton/0e83bff7e27e125cc675bcdd31d3bf23">https://gist.github.com/tchaton/0e83bff7e27e125cc675bcdd31d3bf23</a></figcaption>

> Note: [Serializers](https://lightning-flash.readthedocs.io/en/latest/general/data.html?highlight=serializer#postprocess-and-serializer) are optional components that can be passed to Flash Classifiers that enable you to configure how to format the output of the Task Predictions asLogits, Probabilities , or Labels. Read more [here](https://lightning-flash.readthedocs.io/en/latest/general/data.html?highlight=serializer#postprocess-and-serializer).

#### Step 5- FineTune Model

Once we have a DataModule and VideoClassifier, we can configure the [**Flash** Trainer](https://lightning-flash.readthedocs.io/en/latest/general/training.html?highlight=trainer#trainer-api). Out of the box, the [**Flash** Trainer](https://lightning-flash.readthedocs.io/en/latest/general/training.html?highlight=trainer#trainer-api) supports all the Lightning Trainer flags and [tricks you know and love](https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#trainer-flags) including [distributed training](https://pytorch-lightning.readthedocs.io/en/latest/advanced/multi_gpu.html), [checkpointing](https://pytorch-lightning.readthedocs.io/en/latest/api/pytorch_lightning.callbacks.model_checkpoint.html?highlight=checkpointing), [logging](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html), mixed [precision](https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html?highlight=precision#precision), [SWA](https://pytorch-lightning.readthedocs.io/en/latest/advanced/training_tricks.html?highlight=SWA#stochastic-weight-averaging), [quantization](https://pytorch-lightning.readthedocs.io/en/latest/advanced/pruning_quantization.html), [early stopping](https://pytorch-lightning.readthedocs.io/en/latest/common/early_stopping.html?highlight=early%20stopping), and [more](https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#trainer-flags).

In addition to enabling you to train models from scratch the Flash Trainer provides quick access to a host of [strategies for fine-tuning pre-trained models on new data](https://lightning-flash.readthedocs.io/en/latest/general/finetuning.html).

[![](https://cdn-images-1.medium.com/max/1024/1*o4_yw9gGarEXXbS9CUxZIg.png)](https://gist.github.com/tchaton/685d58357241fe74c656a90798c1e0fe)<figcaption><a href="https://gist.github.com/tchaton/685d58357241fe74c656a90798c1e0fe">https://gist.github.com/tchaton/685d58357241fe74c656a90798c1e0fe</a></figcaption>

In this example, we use the [NoFreeze Strategy](https://lightning-flash.readthedocs.io/en/latest/general/finetuning.html#no-freeze) in which the backbone will be entirely trainable from the start. You can even define your own custom [FineTuning Strategies](https://pytorch-lightning.readthedocs.io/en/latest/extensions/generated/pytorch_lightning.callbacks.BaseFinetuning.html?highlight=finetuning%20strategies) .

#### Conclusion

With these 5 simple steps you can now train your own Video Classification model on any Video Dataset. Once a model is trained, it can easily be saved and reused later for making predictions.

[![](https://cdn-images-1.medium.com/max/1024/1*KJMtomTWlyWCyzhstW0QQw.png)](https://gist.github.com/tchaton/6fc6ba4454a6d7aca0b790f8cbf38fe6)<figcaption><a href="https://gist.github.com/tchaton/6fc6ba4454a6d7aca0b790f8cbf38fe6">https://gist.github.com/tchaton/6fc6ba4454a6d7aca0b790f8cbf38fe6</a></figcaption>

One major advantage of using PyTorch Video backbones is that they are optimized for mobile. With [Lightning](https://github.com/PyTorchLightning/pytorch-lightning), models can be saved and exported as classic PyTorch Checkpoints or optimized for inference on the [edge](https://pytorchvideo.org/docs/tutorial_accelerator_use_model_transmuter) with [ONNX and or TorchScript](https://pytorch-lightning.readthedocs.io/en/latest/common/production_inference.html).

**Find the end-to-end tutorial code ** [**here**](https://github.com/PyTorchLightning/lightning-flash/blob/master/flash_examples/finetuning/video_classification.py) **.**

We thank [Tullie Murrell](https://www.linkedin.com/in/tullie/) from the PyTorchVideo team for the support and feedback leading to this integration in Flash.

### Next Steps

Now that you have train your first model it’s time to experiment. Flash Tasks provide full access to all the Lightning [hooks](https://pytorch-lightning.readthedocs.io/en/latest/api/pytorch_lightning.core.hooks.html) that can be overridden as you iterate towards the state of the art.

Check out the following links to learn more:

- [PyTorch Lightning Repo](https://github.com/PyTorchLightning/pytorch-lightning)
- [Kornia Repo](https://github.com/kornia/kornia)
- [PyTorch Video Repo](https://github.com/facebookresearch/pytorchvideo/)
- [Learn more about Grid](https://www.grid.ai/)

### About the Authors

[**Aaron (Ari) Bornstein**](https://www.linkedin.com/in/aaron-ari-bornstein-22aa7a77/) is an AI researcher with a passion for history, engaging with new technologies and computational medicine. As Head of Developer Advocacy at Grid.ai, he collaborates with the Machine Learning Community, to solve real-world problems with game-changing technologies that are then documented, open-sourced, and shared with the rest of the world.

[**Thomas Chaton**](https://www.linkedin.com/in/thomas-chaton-1529ba128/) is PyTorch Lightning Research Engineering Manager. Previously, Senior Research Engineer at Fujitsu AI and Comcast-Sky Labs, Thomas is also the creator of [TorchPoints3D](https://github.com/nicolas-chaulet/torch-points3d).

* * *