Paper:
Standard compression algorithms work by mapping an image to discrete code using an encoder from which the original image can be reconstructed through a decoder. This process, due to the quantization step, is inherently non-differentiable so these algorithms must rely on approximate methods to train the encoder and decoder end-to-end. In this paper, we present an innovative framework for lossy image compression which is able to circumvent the quantization step by relying on a non-deterministic compression codec. The decoder maps the input image to a distribution in continuous space from which a sample can be encoded with expected code length being the relative entropy to the encoding distribution, i.e. it is bitsback efficient. The result is a principled, end-to-end differentiable compression framework that can be straight-forwardly trained using standard gradient-based optimizers. To showcase the efficiency of our method, we apply it to lossy image compression by training Probabilistic Ladder Networks (PLNs) on the CLIC 2018 dataset and show that their rate-distortion curves on the Kodak dataset are competitive with the state-of-the-art on low bitrates.

Review:
Summary:
This paper aims to circumvent the quantization step and associated gradient approximations for compression algorithms that make use of entropy coding for compression. Entropy coding requires a probability mass function over discrete symbols. As an alternative approach the authors adapt the MIRACLE algorithm by Havasi et al. (2018), which was originally used to compress Bayesian neural networks, to work for lossy source compression with probability density functions over continuous latent variables.
The algorithm is based on taking several importance samples from the prior p(z) with the encoding distribution q(z|x) as the target distribution. The number of samples is equal to the exponent of the KL between the encoding and prior distribution, which can quickly grow to an uncontrollably large number. The index of the sample with the maximum importance weight is used as the code for the original image. If both sender and receiver have access to the same random sampler, the receiver can reproduce the sample by drawing a number of samples equal to this index, and decoding the last sample. Similar in spirit to Havasi et al, the authors take several steps to ensure that the KL divergence (and therefore the number of samples) does not become prohibitively large. The compression performance is evaluated by training the proposed model and the competing neural network-based method [1] on the Clic dataset, and evaluating it on a subset of the images of the Kodak dataset. JPEG is also used as a baseline. The authors show results achievable if coding using the relative entropy was perfect (denoted with ‘theoretical’), and the practically achieved compression performance (‘actual’). 
Decision:
Weak reject: although the idea of circumventing the quantization step required by the use of entropy coders is certainly valid and interesting, the results in the paper show that the resulting compression performance is worse than the competing method that does quantize. Moreover, the encoding time is significantly longer than this same baseline. 
Supporting arguments for decision:
Although the motivation for circumventing the quantization step seems plausible, the authors show no evidence that the competing method [1], which does perform a post-training quantization step, actually suffers from it. The authors even state on page 8 that “ Most notably though, they only used the continuous relaxation during the training of the model, thereafter switching back to quantization and entropy coding, which, they show does not impact the predicted performance, and hence confirming that their relaxation during training is reasonable.“ If post-training quantization is reasonable, then this overthrows the entire motivation. More importantly, the “theoretically” achievable results of the proposed method in seem only competitive in the low-bit rate regime and worse in the higher bit rate regimes. Even more, the “actual” practically achieved compression results are worse than [1] and also considerably worse than the “theoretically” achievable compression results. The authors do provide a reason for why the “theoretical” and “actual” results are so far apart, but are unfortunately not able to overcome this issue.
In the conclusion the authors honestly admit that the runtime of their method is much slower than the competitors (1-5 min for proposed method vs ~0.5 s  for [1] for encoding times). I appreciate that the authors mention this. The authors then state “improving the rate factor and the run-time does not seem too difficult a task, but since the focus of our work was to demonstrate the efficiency of relative entropy coding, it is left for future work.” I do not think this paper demonstrates the efficiency of relative entropy coding, the results simply don’t support this claim, and I therefore think that stating that the issues seem not too difficult to overcome is insufficiently convincing. 
The quality of the empirical study can be improved. Another neural network-based compression baseline would make the empirical evaluation of the proposed method more insightful. Now we only see that the result is worse than [1], but it would be good to know how it compares to other baselines such as [2]. Furthermore, the paper does not show compression results aggregated over the entire Kodak dataset, but rather picks 2 images for the main part of the paper, and shows 3 in the appendix. Showing aggregate results gives a more robust estimate of the performance. Individual image results can just be put in the appendix. 
 
Additional feedback to improve paper (not part of decision assessment):
- In section 5 on the dependency structure of latents in the ladder VAE: is it really necessary to indicate the dependency structure with “topological structure”? Seems unnecessary to me as dependency structure is a clear enough description already without making it sound overly complicated.
- Page 9: “Further, our architecture also only uses convolutions and deconvolutions as non-linearities.” Convolutions and deconvolutions are not non-linearities.
- Fig 2: I’m not sure if this figure is relevant enough for such a prominent placement in the paper. It doesn’t discuss anything relevant to the contributions claimed in this paper. 
- I can’t find a definition of O in line 3 of “procedure” in algorithm 2 and in the return statement.
[1] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. ICLR 2018.
[2] Lucas Theis et al.  Lossy Image Compression with Compressive Autoencoders. ICLR 2017