  Performing inference on deep learning models for videos remains a challenge
due to the large amount of computational resources required to achieve robust
recognition. An inherent property of real-world videos is the high correlation
of information across frames which can translate into redundancy in either
temporal or spatial feature maps of the models, or both. The type of redundant
features depends on the dynamics and type of events in the video: static videos
have more temporal redundancy while videos focusing on objects tend to have
more channel redundancy. Here we present a redundancy reduction framework,
termed VA-RED$^2$, which is input-dependent. Specifically, our VA-RED$^2$
framework uses an input-dependent policy to decide how many features need to be
computed for temporal and channel dimensions. To keep the capacity of the
original model, after fully computing the necessary features, we reconstruct
the remaining redundant features from those using cheap linear operations. We
learn the adaptive policy jointly with the network weights in a differentiable
way with a shared-weight mechanism, making it highly efficient. Extensive
experiments on multiple video datasets and different visual tasks show that our
framework achieves $20\% - 40\%$ reduction in computation (FLOPs) when compared
to state-of-the-art methods without any performance loss. Project page:
http://people.csail.mit.edu/bpan/va-red/.
