Paper:
Deep learning (DL) is having a revolutionary impact in image processing, with DL-based approaches now holding the state of the art in many tasks, including image compression. However, video compression has so far resisted the DL revolution, with the very few proposed approaches being based on complex and impractical architectures with multiple networks. This paper proposes what we believe is the first approach to end-to-end learning of a single network for video compression. We tackle the problem in a novel way, avoiding explicit motion estimation/prediction, by formalizing it as the rate-distortion optimization of a single spatio-temporal autoencoder; i.e., we jointly learn a latent-space projection transform and a synthesis transform for low bitrate video compression. The quantizer uses a rounding scheme, which is relaxed during training, and an entropy estimation technique to enforce an information bottleneck, inspired by recent advances in image compression. We compare the obtained video compression networks with standard widely-used codecs, showing better performance than the MPEG-4 standard, being competitive with H.264/AVC for low bitrates.

Review:
This paper presents a spatiotemporal convolutional autoencoder trained for video compression. The basic model follows the logic of traditional autoencoders, with an intermediate quantizer:
input -> convolutional neural network with a skip connection as an encoder -> quantizer -> transposed convolutional neural network with a skip connection as a decoder.
As the quantizer is a non-differentiable operation, the paper proposes to follow (Toderici et al  2016, Balle et al, 2018) and cast quantization as adding uniform noise to the latent variables. The pre-quantized variables are modelled as Gaussians with variance that is predicted by a second "hyperprior" network dedicated to this task. The final model is trained to minimize three losses. The first loss minimizes the difference between the true frame pixel values and the predicted pixel values. The second loss minimizes the entropy of the latent codes. The third loss minimizes the difference between neighboring pixels in subsequent frames, ignoring those pixels that are not linked between frames. The model is trained on 10,000 videos from the Youtub-8M dataset and tested on 10 videos from the MCL-V database, with rather ok results.
Generally, parts of the proposed approach sound logical: an autoencoder like architecture makes sense for this problem. Also, the idea of using uniform noise to emulate quantization is interesting.  However, the paper has also weaknesses.
- The added novelty is limited and unclear.  Conceptually, the paper is overclaiming. Quoting verbatim from the conclusion: "Our work is, as far as we are aware, the first end-to-end learned video compression architecture using DL.", while already citing few works that also rely on deep networks (Wu et al., 2018, Chen et al., 2016). In the related work section it is noted that these works are computationally heavy. However, this doesn't mean they are not end-to-end. The claims appear to be contradicting.
- The technical novelty is also limited. What is new is the combination of existing components for the task of video compression. However, each component in isolation is not novel, or it is not explained as such.
- Parts of the model are unclear. How is the mask M computed in equation (7)? Is M literally the optical flow between frames? If yes, what is the percentage of pixels that is zeroed out? Furthermore, can one claim the model is fully end to end, since a non-differentiable optical flow algorithm is used?
- The purpose of the hyperprior network is unclear. Why not use a VAE that also returns the variance per data point?
- Most importantly, it is not clear whether the model is trained as a generative one, e.g., with using a variational framework to compute the approximate posterior. If the model is not generative, how can the model be used for generation? Isn't it then that the decoder simply works for reconstruction of already seen frames? Is there any guarantee that the model generalizes well to unknown inputs? The fact that the model is evaluated only on 10 video sequences does not help with convincing with the generalization.
- The evaluation is rather weak. The method is tested on a single, extremely small dataset of just 10 videos. In this small dataset the proposed method seems to perform worse in the majority of compression ratios (bits per pixel). The method does seem to perform a bit better on the very low bits per pixel regime. However, given the small size of the dataset, it is not clear whether these results suffice.
- Only two baselines are considered, both hand-crafted codecs: H.264/AVC and MPEG-4. However, in the related work section there are works that could also be applied to the task, e.g., the aforementioned ones. Why aren't these included in the comparison?
- Although it is explained that the focus is on the very low bitrates, it is not clear what part of the model is designed with that focus in mind. Is this just a statement just so to focus on the part of the curve in the experiment where the proposed method is better than the reported baselines? Is there some intrinsic model hypothesis that makes the model suitable for low bit rates?
In general, the paper needs to clarify the model and especially explain if it is (or not a generative one) and why. Also, a more extensitve arrays of experiments need to be executed to give a better outline of the methods capabilities and limitations.