  We introduce Video Transformer (VidTr) with separable-attention for video
classification. Comparing with commonly used 3D networks, VidTr is able to
aggregate spatio-temporal information via stacked attentions and provide better
performance with higher efficiency. We first introduce the vanilla video
transformer and show that transformer module is able to perform spatio-temporal
modeling from raw pixels, but with heavy memory usage. We then present VidTr
which reduces the memory cost by 3.3$\times$ while keeping the same
performance. To further compact the model, we propose the standard deviation
based topK pooling attention, which reduces the computation by dropping
non-informative features. VidTr achieves state-of-the-art performance on five
commonly used dataset with lower computational requirement, showing both the
efficiency and effectiveness of our design. Finally, error analysis and
visualization show that VidTr is especially good at predicting actions that
require long-term temporal reasoning. The code and pre-trained weights will be
released.
