File size: 3,933 Bytes
9514390 01fd5de 9514390 4b66528 95df5fb 4b66528 95df5fb 4b66528 95df5fb 4b66528 95df5fb |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
---
library_name: tf-keras
license: mit
metrics:
- accuracy
pipeline_tag: video-classification
tags:
- videoswin
- vision
---
# [Video Swin Transformer : VideoSwin](https://github.com/innat/VideoSwin)
| Paper | Colab | HF Space | HF Hub |
| :--: | :--: | :---: | :---: |
| [](https://arxiv.org/abs/2106.13230) | [](https://colab.research.google.com/drive/1Q7A700MEI10UomikqjQJANWyFZktJCT-?usp=sharing) | [](https://huggingface.co/spaces/innat/VideoSwin) | [](https://huggingface.co/innat/videoswin) |
VideoSwin is a pure transformer based video modeling algorithm, attained top accuracy on the major video recognition benchmarks. In this model, the author advocates an inductive bias of locality in video transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization. The locality of the proposed video architecture is realized by adapting the [**Swin Transformer**](https://arxiv.org/abs/2103.14030) designed for the image domain, while continuing to leverage the power of pre-trained image models.
- GitHub: https://github.com/innat/VideoSwin
This is a unofficial `Keras` implementation of [Video Swin transformers](https://arxiv.org/abs/2106.13230). The official `PyTorch` implementation is [here](https://github.com/SwinTransformer/Video-Swin-Transformer) based on [mmaction2](https://github.com/open-mmlab/mmaction2).
## Model Zoo
The 3D swin-video checkpoints are listed in [`MODEL_ZOO.md`](https://github.com/innat/VideoSwin/blob/main/MODEL_ZOO.md). Following are some hightlights.
### Kinetics 400
In the training phase, the video swin mdoels are initialized with the pretrained weights of image swin models. In that case, `IN` referes to **ImageNet**.
| Backbone | Pretrain | Top-1 | Top-5 | #params | FLOPs | config |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Swin-T | IN-1K | 78.8 | 93.6 | 28M | ? | [swin-t](https://github.com/SwinTransformer/Video-Swin-Transformer/blob/master/configs/recognition/swin/swin_tiny_patch244_window877_kinetics400_1k.py) |
| Swin-S | IN-1K | 80.6 | 94.5 | 50M | ? | [swin-s](https://github.com/SwinTransformer/Video-Swin-Transformer/blob/master/configs/recognition/swin/swin_small_patch244_window877_kinetics400_1k.py) |
| Swin-B | IN-1K | 80.6 | 94.6 | 88M | ? | [swin-b](https://github.com/SwinTransformer/Video-Swin-Transformer/blob/master/configs/recognition/swin/swin_base_patch244_window877_kinetics400_1k.py) |
| Swin-B | IN-22K | 82.7 | 95.5 | 88M | ? | [swin-b](https://github.com/SwinTransformer/Video-Swin-Transformer/blob/master/configs/recognition/swin/swin_base_patch244_window877_kinetics400_22k.py) |
### Kinetics 600
| Backbone | Pretrain | Top-1 | Top-5 | #params | FLOPs | config |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Swin-B | IN-22K | 84.0 | 96.5 | 88M | ? | [swin-b](https://github.com/SwinTransformer/Video-Swin-Transformer/blob/master/configs/recognition/swin/swin_base_patch244_window877_kinetics600_22k.py) |
### Something-Something V2
| Backbone | Pretrain | Top-1 | Top-5 | #params | FLOPs | config |
| :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| Swin-B | Kinetics 400 | 69.6 | 92.7 | 89M | ? | [swin-b](https://github.com/SwinTransformer/Video-Swin-Transformer/blob/master/configs/recognition/swin/swin_base_patch244_window1677_sthv2.py) |
|