From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: Summary:
This paper proposes two techniques to improve image/video compression with a neural net encoder-decoder: 1) vector quantization of the latent codes using a uniform grid, 2) "latent shifting", where the latents for an image/video are perturbed using the gradient of the hyperprior entropy as a proxy for minimizing image/video reconstruction error. These techniques improve bit rate but increase computation time on several benchmarks.

Strengths:
* Reasonable approach with theoretical backing
* Plug-and-play: can be used for a range of codecs
* Decent bit rate gains in experiments

Weaknesses:
* High time complexity for encoding
* Using VQ is not entirely novel
* The two technical contributions -- uniform VQ and latent shifting -- are somewhat orthogonal, and neither provides a huge improvement on its own

justification_for_why_not_higher_score: This paper was thoroughly discussed on openreview and in the AC-reviewer meeting. Although reviewers appreciated the theory and acknolwedged the performance gains, they felt the improvements might not be worth the extra time complexity, and did not feel either technical contribution was ground breaking. Therefore the decision is that this paper does not quite meet the ICLR bar.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: This paper was discussed in a virtual AC-reviewer meeting. All reviewers, and the AC, agreed on the main strengths and weaknesses of the paper, listed above. One reviewer had overlooked the time complexity and decided to decrease their score after discussion. Although two reviewers remained borderline positive after discussion, both also agreed with the shortcomings and did not want to champion the paper.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: High complexity of encoding 
comment: We understood why the reviewers thought about encoding complexity of LatentShift. 
We believe the main issue is that we could not explain the finetuning complexity and our complexity well.
Definitely this experiences help us to be more clear on this issue for the future.

Just a little reminder that HexQuant and OctQuant complexity is almost negligible as we showed in the complexity tests thanks to the fast quantization grid search algorithms on uniform grids.

LatentShift encoding complexity (x10 on one thread and one cpu) is almost negligible compare to Finetuning solutions encoding complexity (x4000 on one thread and one cpu). In addition LatentShift can give extra gain after applying Finetuning solutions that Finetuning never reach that performance. Plus both Finetuning and LatentShift can be parallelizable and complexity can decrease 5-10 times. But we did not say this point because we believe that encoding time analysis should be measured on single cpu as it can give some proxy about energy consumption as well unlikely all finetuning papers that report encoding time with massive parallelization.
Even though Finetuning encoding is too costly, still they are comparable with traditional SOTA method with high performance settings (such as VVC). 

Thanks for the fruitful discussion again.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: difference between the proposed uniform VQ and lattice VQ
comment: We also wanted to let the reviewer know that there are some uniform grid that are non-lattice grid which first explored in [1] by gradient descent based optimization. Basically they explored a new non-lattice tessellation map that consist of uniform grids for the dimension 7 and 9 which are better than lattice grids in the same dimension in terms of quantization error for uniform source. They defined the differences in last paragraph of the first column of the 13th page by followings:

-
As mentioned, D_d^+  is not a lattice when d is odd. However, it is still a tessellation, because all its Voronoi regions are congruent. Half of them are upside-down (more precisely, reflected in a point), which disqualifies the tessellation from being a lattice. In a lattice, all Voronoi regions are translations of each other, without scaling, rotation, or reflection 


In plein words, lattice's grids should be in the same orientation/reflection. Even though uniform grids have also the same scale (the area should be the same) but it maybe on different rotation/reflection that breaks being lattice.

However mostly explored optimal  uniform grids create lattice.  

Even though lattice VQ or uniform VQ has quite long history, it is remained unused  in neural codecs before our submission.

[1] Erik Agrell and Thomas Eriksson, Optimization of Lattices for Quantization, IEEE TRANSACTIONS ON INFORMATION THEORY, 1998

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: How to improve the performance of LatentShift
comment: We are glad to hear these comments from the reviewer. Regardless of the rate/scores we are happy to hear that our theoretical contributions are found insightful.



Introducing more steps may search the best step size well and improve the performance slightly more. But note that if there is no correlation, the gradient of the entropy shows a random direction for increasing the PSNR. Thus, in that case it does not help.



The first idea to improve the performance more is related to keeping the true gradient  (unavailable at decoder) with help of a proxy gradient (available at decoder). As we show in the Appendix D1, the true gradient has quite big effects on the PSNR. Since the correlation is not exactly -1 due to the expectation in the training set for the KKT conditions. Thus proxy does not represent the true gradient entirely. We believe upcoming researches can dig this point and improve the performance with the help of our theoretical contributions on connection between these gradients.



The second possible improvement is related to the training procedure. Since the KKT conditions and the connection to the gradients are related to the training procedure, we strongly believe that after our connections on gradients, upcoming researches also will propose something to use this connection during the training phase or post training instead of image specific improvements as we did.



Please see our LatentShift as an idea on how to use the KKT conditions on neural codecs. We believe it is maybe not the best way to use the connection. We would like to show an experimental case how to use it as a very first idea.

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_k9t4
title: Thanks for the feedback
comment: Thanks for the comprehensive explanation and additional experimental results. The proposed latent shifting is interesting and insightful. Even though latent shifting is more computationally efficient than latent finetuning, but the BD-rate savings on complex models (cheng2020-attn and DCVC-intra mode) are still quite small (<0.75% and <0.3%, respectively). Is it possible to improve the R-D performance by introducing more candidate steps?
Considering the feedback, I am willing to increase my rating.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Gain from VQ or high dimensional gaussian entropy model?
comment: In addition to previous discussion, the main gain source of the mentioned paper (Zhu, Xiaosu, et al.)  is not clear. Since the ablation study was done to against existed non-uniform VQ (codebook are learned), but not SQ, it is difficult to say how their quantization effective compare to uniform SQ. 

According to the paper, they also used high dimensional gaussian entropy model in order to take redundancy between latent in different channel into account. Since another baselines that we used in our paper, are all assume the latents have no remaining redundancy, just one dimensional gaussian is used for entropy model. However Zhu, Xiaosu, et al. learn inter channel correlation by their gaussian covariance (full covariance, not diagonal), we cannot say anything if the gain coming from better entropy model or better quantization. But in our case, our gaussian covariance is diagonal. Or we can say our entropy model is 1d, just quantization is on v-dim in order to decrease the quantization error. That is also another differences between ours uniform VQ and Zhu, Xiaosu, et al.

As a result, our gain is coming from better quantization, their gain is coming from mixture of better entropy and better quantization. For now we cannot measure their quantization gain over uniform SQ.

Our solution is more generic and fit on top of any SQ trained neural codec as vast majority of them are.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Comparison to the classic VQ is not possible.
comment: According to ICLR policy, we are not supposed to compare and conduct test on the published peer-reviewed paper  4 months ago before submission deadline (28th May) This paper is published on June 2022 at CVPR, which means we should count it as contemporaneous work. (see last question of ICLR review guide: https://iclr.cc/Conferences/2023/ReviewerGuide#FAQ)    

However, we are not hiding behind that excuse. We already added this work into our citation and discuss it, because it fixed many problems on classic VQ in neural codecs. As it gives quite encouraging ablation study that it increases BD-rate from classic VQ significantly. However, we believe that one main ablation study is missing which is to train the model with uniform SQ. In this way we will be able to see real advantage of their two level of  “multi-codebook” strategy over uniform SQ. Besides, the analysis we conducted for another codec is not appropriate for this one. As we clearly stated that our method does not need training, we can apply it on top of already trained model if the model is trained for uniform SQ. Since this model is trained for VQ, we cannot plug our method up to this one. One thing we can do is to change their quantization map into uniform SQ and train the model. Later we can apply our uniform VQ. Or directly train their model with our uniform VQ, which is in our future work agenda.   

One similarity of this work is that both of us see the drawbacks of the classic VQ in neural codec. Their way is to re-train model with multi codebook strategy and add some tricks to not to make dictionary big and get as diverse as possible codebook. Our way is to use uniform VQ as it is uniform SQ by supplying some theoretical insight why uniform grid will be enough.   

Besides, another method's improvement does not hurt our novelty and our improvement. As they did not propose something to improve SQ and even did not compare, our method is not for improving performance of trained neural codec with classic (non-uniform) VQ. 

We thank to the Reviewer again to point out this new research. Previously we were not aware of this work. It indeed strength our claims in the paper that classic VQ has many drawbacks and should be solved in order to improve RD performance.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Connection to the finetuning approaches
comment: Thanks to the reviewer to point of the connection between finetuning and proposed LatentShift.

We answered this point in an answer of another review. Please have a look at the connection between finetuning solution and our latentshif here:   

  

https://openreview.net/forum?id=QVSoh6VM4nG&noteId=r9a0crcVtsC  

    

In summary, the mentioned two references are typical example of finetuning solution where the latents are finetuned for RD loss for some maximum number of iterations. (typically 1000-10000) This process complexity is the same with training the model for one single image.    

  

  

Basically they finetune the latent by   

  

  

$for ~~iter=1, \dots 10000$   

  

   

  

$~~~~\mathbf{y} \leftarrow \mathbf{y}-\rho.\nabla_{\mathbf{y}}(R(\mathbf{\hat{y}})+\lambda D(\mathbf{\hat{y}}))$   

  

   

  

encode $\mathbf{\hat{y}}=Q(\mathbf{y})$ into bitstream  

  

  

   

As your reference [2] mentioned, these proposals encoding time is $t_e+2*t_d n_s$.  (page 5 last paragraph)  n_s is the max iteration that is 10000 in the paper [2]. Thus,  their complexity is equal one encoding ($t_e$) plus 20000 decoding ($t_d$). If encoding and decoding network is almost the same, their encoding time is 20000 times more complex than baseline encoder. They also reported that by using GPU they still need 10mins to encode one 512x768 resolution image instead of some miliseconds.   

  

  

However, we do not finetune the latent, we just find the optimal step size if we follow gradient of the entropy as we showed theoretically and experimentally that it is correlated with gradient of distortion according to K.K.T conditions.  

   

  

Simply we find the best step size by evaluating 8 predefined step sizes, finally encode the same latent but also single number of step size into the bitstream.  

  

  

  

$for ~~\rho ~ in ~ [ \rho_1,...\rho_8 ]$     

  

   

  

$~~~~\mathbf{y^*} = \mathbf{\hat{y}}-\rho.\nabla_{\mathbf{y}}(R(\mathbf{\hat{y}}))$   

  

   

  

$~~~~if~~g_s(\mathbf{y^*})$ is best, keep $\rho^*=\rho$  

  

   

  

encode $\rho^*$ and $\mathbf{\hat{y}}$ into bitstream  

  

  

  

  

Thus our complexity is $t_e+8 t_d$ (we do not need backward pass. because latent follows gaussian distribution and we can calculate the closed form solution for the gradient of  negative log likelihood of gaussian pdf) )where all 8 evaluation can be parallelize and can have just one single forward pass encoding time. But since we do not find the complexity analysis involving parallel processes realistic, we reported single cpu runtime compared to our latentshift and finetuning solutions. We reported these results as an answer of another review. Please have a look.  

  

https://openreview.net/forum?id=QVSoh6VM4nG&noteId=2kHadYCnmP  

  

  

As it can be seen, finetuning solution needs x4000 (for 2000 iteration) encoding time. In addition, our KKT conditions is even strength after finetuning. Thus, it increases the correlation and also our gain. If the one is happy to have %5-%10saving for x4000 encoding time, the one will be more than happy if he spend extra x10 for extra %1 gain. As a result, relative differences between  x4010 and x4000 is almost negligible. also note that those complexity is in encoder side. Most of the time encoding is done once but decoding is a million times. That is why the compression community pays more attention to decoder but not encoder complexity where our decoder has the same complexity.  

  

Besides that, our proof on correlated gradient is more fundamental than proposed latentShift. Please see LatentShift as an proposal on how to use these correlated gradients. It might not be the perfect usage, but we believe after our proof and showing this forgotten information, coming papers investigate this point and propose maybe better techniques even in training time or post training time as how we implement. KKT condition  is somehow forgotten in neural codec literature and it is fundamentally showed which information we forgot to use in decoding side.

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_tojV
title: Re: Answers to the Questions-2
comment: Thank you for explaining the novelty of the proposed VQ method. However, the method proposed in [1] also achieves good coding performance while maintaining fast encoding/decoding speed. Could you please provide some experiment results of the comparison between the proposed method and the method proposed in [1]?


[1] Zhu, Xiaosu, et al. "Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_tojV
title: Re: Computational Complexity-2 
comment: Thank you for providing the encoding/decoding complexity details. In my opinion, increasing encoding time by 5x~10x but only having about 1% coding gain is not worthwhile. Also, in my understanding, the proposed latent shifting method is similar to some online training methods like [1][2]. Could you please explain the difference between them?


[1] Tsubota K, Akutsu H, Aizawa K. Universal Deep Image Compression via Content-Adaptive Optimization with Adapters[J]. arXiv preprint arXiv:2211.00918, 2022.

[2] Mari D, Milani S. Features Denoising for Learned Image Coding[C]//2022 10th European Workshop on Visual Information Processing (EUVIP). IEEE, 2022: 1-6.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Thanks for insightful questions
comment: Regardless of the decision, we are happy to get acknowledge by domain experts that our theoretical contribution and insightful work. We believe that uniform VQ will be definitely a part of future neural codecs, plus maybe not how we use, but latent-shift (more general KKT conditions on neural codec) will use in training time and/or post training time either in image specific or entire dataset in the future. Hope our work will be starting points of these coming researches.

Particularly for potentially misunderstanding of our previous answer, we would say, only  Latentshift cannot give better result that Finetuning only. Our comments  "However it is noted the encoding the side shifted latent will increase the file size ...." is for our latentshift algorithm to make it more clear. For finetuning, it will either increase side information bitlength but decrease main latent bitlength more (or decrease both), at the end it will  always improve RD performance more than only LatentShift. But as we tested, if the one is happy to spend x4000 encoding time to gain 5-10% bit-saving, the one would be more than happy to spend extra x10 encoding time to gain extra %1. The analysis shows that after finetuning even strength the KKT conditions and improves the correlations. It means whatever you finetune the latent, it will never compensate our extra savings. 

This analysis is shown in  

https://openreview.net/forum?id=QVSoh6VM4nG&noteId=r9a0crcVtsC

https://openreview.net/forum?id=QVSoh6VM4nG&noteId=2kHadYCnmP.

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_7dNo
title: Response to authors
comment: Thanks to the authors for providing a more detailed explanation of why we still need to perform VQ but can make uniform grid assumptions.

I still don't entirely get why latent-shifting the hyperprior is better than just finetuning the hyperprior on the encoder side (e.g. I'm confused by this comment "However it is noted the encoding the side shifted latent will increase the file size , because the side information entropy and main information entropy are contrary to each other"), but maybe in the end it's a matter of runtime. I agree finetuning is slow, and it's good that latent shift takes less time. 

In the end I keep my score: there are some neat insights in the paper that won't drastically improve SOTA R-D curves but provide room for some additional optimization.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Revised Paper
comment: We would like to express our thanks to all the reviewers for their valuable time and feedback. Regardless of the decision/rates, we are happy to get acknowledge by all 4 domain experts about our theoretical contributions and they found the paper is clear and sufficient. Our KKT conditions on neural codec was recognized as novel by 3 out of 4 reviewers. On uniform VQ, 2 out of 4 reviewers think it is novel, but rest think it is an extension on classic VQ. We hope, we explained the differences and why uniform VQ has never been used in neural codec before and why it will be definitely a part of future neural codecs. Specifically latent-shift, or in more general KKT conditions on neural codec is fundamentally novel in neural codec that we first introduced it to the community. We explained the differences from finetuning solutions and how LatentShift is complementary to finetuning and cannot be replaced by finetuning. This result will be used  in training time and/or post training time either in image specific or entire dataset in the future. Hope our work will be the reference of these coming researches.

We pointed out some key questions during rebuttal. Here is the summary of these question and where we answered them.

1. New results on more recent neural codec.
https://openreview.net/forum?id=QVSoh6VM4nG&noteId=Cg7dY888rb
2. Why the performance is relatively lower in autoregressive entropy models.
https://openreview.net/forum?id=QVSoh6VM4nG&noteId=q_ivuRuRmzB
3. Computational complexity of proposed methods.
https://openreview.net/forum?id=QVSoh6VM4nG&noteId=twjVs93NA9
4. The connection between finetuning solution and our Latentshift. The advantage of our model over finetuning methods.
https://openreview.net/forum?id=QVSoh6VM4nG&noteId=r9a0crcVtsC
5. Why SQ is not enough, why uniform VQ is the best choice.
https://openreview.net/forum?id=QVSoh6VM4nG&noteId=iRF-k23k1N
6. The differences between general VQ in neural codecs and our uniform VQ.
https://openreview.net/forum?id=QVSoh6VM4nG&noteId=OwXfG1VGdal


We have uploaded a revised version of the paper. All added parts are highlighted with red color in the paper.  The main changes are summarized as follows:

1. We conducted a new experiments with one recent image and a video codec. We added these results in the table in Result section.
2. We did computational complexity test and report the results in Appendix G.
3. We updated citations and fixed some typos.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Answers-2
comment: ## Shifting the side latents in the encoding

During the encoding, the side latent $\mathbf{\hat{z}}$ is encoded in the bitstream by using shared factorized prior $p_f(.)$ as it is the same in default model. However encoder shift the side latent wrt its gradient, but not encode that "shifted latent". Encoder uses shifted side latent in order to decrease main latent entropy. So encoder's steps can be seen as follows
1. $\mathbf{y}=g_a(\mathbf{x})$

2. $\mathbf{z}=h_a(\mathbf{y})$

3. $\mathbf{\hat{z}}=Q(\mathbf{z})$

4. Encode $\mathbf{\hat{z}}$ with shared $p_f(.)$ into bitstream: encoded latent is original latent not shifted one!

5. Find the best step size $\rho_f^*$ and encode it explicitly

6. Shift the side latent by :  $\mathbf{\hat{z}}^* \leftarrow \mathbf{\hat{z}}-\rho_f^*.\nabla_{\mathbf{\hat{z}}}(R(\mathbf{\hat{z}}))$

7. Find main information distribution parameters $[\mu,\sigma]=h_s(\mathbf{\hat{z}}^*)$

8. $\mathbf{\hat{y}}=Q(\mathbf{y})$

9. Encode $\mathbf{\hat{y}}$ with $N(.;\mu,\sigma)$ into bitstream: encoded latent is original latent not shifted one! but its parameters are obtained by shifted side latent

10. Find the best step size $\rho_h^*$ and encode it explicitly

In decoder side the step by step process can be as follows;

1. Decode $\mathbf{\hat{z}}$ with shared $p_f(.)$ from bitstream: decoded latent is original latent not shifted one!

2. Decode the best step size $\rho_f^*$ from bitstream

3. Shift the side latent by :  $\mathbf{\hat{z}}^* \leftarrow \mathbf{\hat{z}}-\rho_f^*.\nabla_{\mathbf{\hat{z}}}(R(\mathbf{\hat{z}}))$ : now decoder also reach the shifted side information.

4. Find main information distribution parameters $[\mu,\sigma]=h_s(\mathbf{\hat{z}}^*)$: Now decoder can reach the same distribution params

5. Decode $\mathbf{\hat{y}}$ with $N(.;\mu,\sigma)$ from bitstream: decoded latent is original latent not shifted one!

6. Decode the best step size $\rho_h^*$ from bitstream

7. Shift the main latent by :  $\mathbf{\hat{y}}^* \leftarrow \mathbf{\hat{y}}-\rho_h^*.\nabla_{\mathbf{\hat{y}}}(R(\mathbf{\hat{y}}))$

8. Reconstruct image by $\mathbf{\hat{x}}=g_s(\mathbf{\hat{y}}^*)$


We agree the side latent can be shifted during encoding to decrease the file size of the main information. However it is noted the encoding the side shifted latent will increase the file size , because the side information entropy and main information entropy are contrary to each other. The same argument also valid for encoding main shifted latent. Therefore, it is not optimal to perform shifting the latent w.r.t gradients during the encoding. Thus, for encoding we don’t change the side latents and main latents or it’s entropy model, but only update (shift) them during decoding.

## Leveraging the inductive bias of the network itself to “fill in” information that is not provided by the original code

We really appreciate about this comments of the Reviewer.  Indeed we did not explain the Latenshift idea better than this sentences. It is absolutely true.

The main latent shift with respect to entropy of the main latents increase the entropy with zero cost because we do not write that shifted information into bitstream.  As the entropy increases the information content in the latent also increases, more information results better reconstructed image. But every single shifting on the latent that increase the entropy does not give better reconstructed image for sure. If yes, we would just shift the latent randomly and hope to have good reconstruction which we ridiculously tested this idea in Appendix D.2!

## Side effects/downsides of LatentShift

Since the best step size includes 0 step and all predefined  step sizes are tested in encoding time, it is not possible to have worser performance than baseline model. In this worst case scnario, Latentshift set both $\rho_f^*=0$ and $\rho_p^*=0$. In this case, just the cost is to write two scalar  $\rho_f^*$, $\rho_p^*$  into the bitstream where their cost is negligible. Another loss is spent time in encoder in order to find $\rho_f^*$, $\rho_p^*$. Unless there is no another downsides of the LatentShift proposal. It cannot give the wrong information is any cases. However, if the train/test distributions are very similar, the correlation between gradients are strong and LatentShift works well. But if there is no correlation,it does not improve the performance and stay on par with baseline model

## Is it possible that reconstruction error is better “on average” but is worse for specific subsets of images

When we follow the average gradients of the latents in train set, this direction can give global direction to improve the performance. However we always get given single image specific gradient. Thus this gradient cannot say meaningful information for general train set. 

## Runtime Analysis

As an answer of another Reviewer, we did some complexity analysis. Please have a look at this post
https://openreview.net/forum?id=QVSoh6VM4nG&noteId=twjVs93NA9

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Answers-1
comment: We thank the reviewer 7dNo for the constructive comments and for appreciating our theoretical contributions. We are glad to hear that our two proposals are found novel and our paper is easy to read. Many of us shared the same counter intuitive feeling with the reviewer when this idea came into mind. Also glad to hear quite difficult and compulsive questions raised by our reviewer where some of them are into our mind from the beginning but did not find an answers yet!

## Does Theorem 1 extend to VQ

We did not find an affirmative answer to this question yet. To the best of our knowledge, the answer is No. But we might approximate it if there are infinitive layer/neuron of $f$ and $f^{-1}$ function exists which is not realistic. The intuitive idea that we cannot extend it to VQ is based on whether there are continuous smooth invertible function that transforms one high dimensional grids center and perimeters to another's center and perimeters.  In 1-dim case, it is easy because the borders and centers are just a point. So we need to find a transform between these monotonically increasing points. But in high dimensional space, the centers are still points but borders are perimeter of any possible grid. Let's think if any NN can learn transformation between hexagonal grid to square grid in 2d case. First and foremost, the borders are not smooth and NN can approximate that non smooth function very badly (in theory with infinite number of parameters it can approximate). Thus, in our opinion, seems it is not extendable to VQ. But still we use the intuition of proof of 1-dim case and go to high dimensional case. 

Here the one maybe asks then why we use hexagonal grid for 2d but not square grid on 2d (which is equivalent to 1d SQ). We cannot find an invertible transformation from any arbitrary 2d grid to hexagonal or square grid. Sounds both hexagonal and square grid fail here. But since hexagonal grid is the optimal in a sense of inertia on space tessellation, it is the best grid that we can approximate any 2d space partitioning grid with less error.  

## Why can’t a neural network also transform a VQ-map to an SQ one?

Actually this question is the same with the first question. Since SQ is equivalent square grid in 2d case, now the question is whether we can find smooth invertible transformation from any arbitrary 2-dim grid's center and borders to square grid's center and borders. Actually this question was asked ourselves many times as well. Because of non smooth transformation is necessary and NNs are bad on it, we cannot transform in practice. 

However, we think this questions impact and possible future work potentials are a lot. Because if any NN can transform any arbitrary VQ to SQ map, then we plug in that NN after anywhere that we need to apply VQ. Thus, we get rid of computational complexity of VQ by applying additional transformation. 

## Why can't we optimize main and side latent directly for the rate-distortion tradeoff without proxies in encoding time.

We called this line of works as "finetuning solutions" where the main idea is to minimize the amortization gap. The definition of amortization gap, their shortcomings computational costs were explained as an answer of another reviewers question. Pleased have a look.

https://openreview.net/forum?id=QVSoh6VM4nG&noteId=z9jIL8bcUZB

https://openreview.net/forum?id=QVSoh6VM4nG&noteId=r9a0crcVtsC

https://openreview.net/forum?id=QVSoh6VM4nG&noteId=2kHadYCnmP

But specifically to this question, we can optimize $\mathbf{\hat{y}}$  and $\mathbf{\hat{z}}$ directly for the classic RD loss. Those approaches gives around 5% to 15% bd-rate savings which are quite significant. However their encoding time complexity is massive. As we report in the link above, our LatentShift needs around 10 times more encoding time from baseline, while Latent Finetuning [1-2] solution needs around 4000 times more encoding time. All details of the tests are given in above post.

Our reviewers intuition on the possibility that these finetuning approaches and ours are complementary is exactly true. Please have a look at *a. LatentShift is not alternative of Finetuning but complementary of it! LatentShift has more improvement after Finetuning!* headline in post on given link. It will explains how our LatentShift even improve the correlations and gives better gain if we apply LatentShift after mentioned FineTuning solutions.


[1] Yang, Y., Bamler, R., & Mandt, S. (2020). Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33, 573-584.


[2] J. Campos, S. Meierhans, A. Djelouah, and C. Schroers, “Content adaptive optimization for neural image compression,” in CVPR Workshops,June 2019

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Computation Complexity and Performance of Finetuning versus LatentShift
comment: ### b. Computation Complexity and Performance of Finetuning versus LatentShift 

  

In order to validate the run time analysis of finetuning solutions and its gain, we tested our LatentShift and LatentFineTuning described in [1] over mbt2018-mean and bmshj2018 -factorized model in CompressAI library on Kodak and Clic dataset. Again, we use the same complexity measurement settings where we run the model on single cpu. The results can be found below. 

  

| Model | Encoding Time | Decoding Time | Bd-Rate (Kodak) | Correlation (Kodak) | Bd-Rate (Clic)| Correlation (Clic) | 
| ----------- | :----:  | :----:  | :----: | :----: |:----:| ----------- | 
| mbt2018-mean (baseline)  | 0.5274sec| 0.7138sec | 0% | N/A | 0% | N/A | 
| mbt2018-mean + LatentShift | 5.5674sec (x10.1)| 0.7188sec (+0.7%) | -1.27% | -0.1805 | -1.21% | -0.1465| 
| mbt2018-mean + FineTuning|    2312sec (x4380)  | 0.7138sec (+0.0%) | -5.77% | N/A  | -5.53% | N/A | 
| mbt2018-mean + FineTuning + LatentShift  |  2317 (4390x)  | 0.7188sec (+0.7%) | -7.27% | -0.2012 | -6.96% | -0.1596| 
| bmshj2018 (baseline)  |  0.4374sec | 0.6367sec  | 0% |N/A | 0% | N/A| 
| bmshj2018  + LatentShift |  5.351sec (x12) | 0.6876 (+0.7%)  | -0.59% | -0.108 | -0.69% | -0.0952 | 
| bmshj2018  + FineTuning|   2098sec (x4800)   |  0.6367sec (+0%) | -6.52% |  N/A | -6.73% | N/A | 
| bmshj2018  + FineTuning + LatentShift  | 2103sec (x4812)   | 0.6876 (+0.7%)  | -7.38% | -0.135 | -7.66% | -0.1278| 

  

Finetuning solutions needs 4000-5000 times more cpu time in encoding which validates our theoretical analysis. Since our LatentShift needs just 8 forward pass, even though its complexity is high, compared to finetuning solutions it is almost negligible. We also reported average correlation coefficient between gradient of entropy and gradient of reconstruction error. In mbt2018-mean model on Kodak dataset, the average correlation is   -0.1805. But after finetuning the correlation goes up to -0.2012. This correlation increase results in better contribution for LatentShift as we discussed in the paper’s result section. For instance, LatentShift's alone bd-rate is -1.27% (mbt2018-mean + LatentShift) in kodak dataset. However, the differences between bd-rate of with and without LatentShift on mbt2018-mean + FineTuning  is 7.27-5.77=1.50%. This can be explainable by increase in correlation after finetuning from -0.1805 to   -0.2012. 

  

[1] J. Campos, S. Meierhans, A. Djelouah, and C. Schroers, “Content adaptive optimization for neural image compression,” in CVPR Workshops,June 2019

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Q2: Difference between our LatentShift and Finetuning solutions
comment: ## 2. The differences 

  

The reviewer’s interpretation on how finetuning methods and our LatentShift works are accurately true. Thus, we explain the advantages of the LatentShift method. For simplicity, when we neglect the side information and concentrate on the main information, without loss of generality, finetuning solutions update the latent iteratively wrt to classic RD loss such that: 

  

$for ~~iter=1, \dots 2000$ 

  

$~~~~\mathbf{y} \leftarrow \mathbf{y}-\rho.\nabla_{\mathbf{y}}(R(\mathbf{\hat{y}})+\lambda D(\mathbf{\hat{y}}))$ 

  

where $\mathbf{\hat{y}}$ is uniform noise added version of latent as an continuous approximation of the SQ such that $\mathbf{\hat{y}}= \mathbf{y}+U(-0.5,0.5)$.  When maximum iteration is reached, the latent is quantized $\mathbf{\hat{y}}=round(\mathbf{y})$ and encoded it in the bitstream. At the end of the iteration, obtained $\mathbf{\hat{y}}$ will have minimum bitlength $R(\mathbf{\hat{y}})$ and minimum distortion on reconstruction such that $D(\mathbf{\hat{y}})$. This training is the same as training full neural codec not for the entire dataset but just on a single image training set.  

  

Since the tradeoff is defined by hyperparameter $\lambda$, the rate and reconstruction quality can be higher or lower than default model, but always gives better RD performance. Note that this process cannot change anything in decoder side, so no signaling cost is necessary, thus, decoder works as it is in default mode.  

  

### a. LatentShift is not alternative of Finetuning but complementary of it! LatentShift has more improvement after Finetuning! 

  

Now, let's think the gradient's value after finishing the 2000 iteration. It probably converges to any saddle point; thus, the sum of gradient will be almost zero such that $\nabla_{\mathbf{y}}(R(\mathbf{\hat{y}})+\lambda D(\mathbf{\hat{y}})) \approx 0$. Thus, we can conclude that $\nabla_{\mathbf{y}}R( \mathbf{\hat{y}} ) \approx - \nabla_{\mathbf{y}}D( \mathbf{\hat{y}} ) $. This point is the main motivation of our LatentShift mechanism. Basically, LatentShift says, in decoding time when we decode the $\mathbf{\hat{y}}$, we can have access to $\nabla_{\mathbf{y}}R( \mathbf{\hat{y}} )$  but not $\nabla_{\mathbf{y}}D( \mathbf{\hat{y}} )$ which we need to minimize the error in decoder side. However, because of KKT condition, these gradients are likely to be highly correlated and we can use the available one as it is necessary one. Thus, we can shift the latent with $\nabla_{\mathbf{y}}R( \mathbf{\hat{y}} )$. But the problem is that we do not know how much we should shift (the step size). To solve this problem, we propose to test 8 different $\rho$ values and select the one which gives the minimum error and write the best $\rho^* \in  R$ in the bitstream explicitly.  It is trivial to mention that if the gradients are not correlated (it sometimes happens) the best step size will be $\rho^*=0$ means, LatentShift does not improve the reconstruction. 

  

Without finetuning, the gradients are zero in expectation of train set's latent such that $E_{y \sim p} [ \nabla_{\mathbf{y}}(R(\mathbf{\hat{y}})+\lambda D(\mathbf{\hat{y}}))] \approx 0$. However after finetuning, since the loss is minimized for a given single image's latent, we can remove the expectation (it is equivalent to the expectation of single element). Thus, we found even better correlation and thus improved the performance even more as they are given in the following section. Since the finetuning solutions stop after reaching the stopping criteria, they are not aware of the correlated gradients and never attempt to improve the performance more. Thus, finetuning solutions cannot replace the LatentShift's gain but they can be used together.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Why Finetuning solution's are computationally heavy solution.
comment: ### 1. Amortization Gap in Neural Compression 

In traditional compressing methods, all the processes are instance adaptive. However, any ML model should be optimized over the entire train set and hopefully that optimized parameters can fit on given test image well. Since the ML model is optimized over the given train set, it may be optimal for the entire train set, but not optimal for a given test instance (always ML model has some error on given test instance). This gap is usually called the amortization gap [1].  In ML based techniques, we cannot do anything for given test image but just inference. However, in neural compressing, our encoder knowns what is the test instance (image to be compressed), thus it can do some extra process in order to adapt the compressing task better. In literature, some methods update the encoder network parameter for a given single image, some update the encoded latent directly, while some update all the network for a given single image. If the method updates some part of decoder network, those updates should be signaled (transmitted), which introduces another rate term. So far, the very first proposal of updating latent directly is proposed in [2]. Later [3] generalized those techniques and proposed again updating latent for given single image with different quantization relaxation. We call those methods "finetuning solutions", because these techniques should optimize RD loss for just on a given single image in encoding time. So, during encoding, we need to optimize the latent code with RD loss. Indeed, these techniques reported 5%-10% bitrate saving in exchange of massively encoding time. For instance [1] and [2] reported their encoding time in the appendices as it is 400 times, time consuming from baseline models.  Even these numbers were obtained with highly parallelizable GPUs according to these papers which is not realistic. As the traditional methods measure the complexity on single CPU without allowing multi-thread process in order to understand their necessary number of clock cycle as a proxy of energy consumption, we followed the same process and reported the CPU complexity of the simplest model in [2]. Since these techniques need at least 2000 iteration, theoretically necessary computations are equal to 2000 forward pass on g_s block and 2000 backward pass for gradient calculations. Even if we think one forward pass and one backward has the same complexity (usually backward is much more complex) and neglect the latent updating step, we can conclude that we need at least 4000 times forward pass. Even if we simplify the calculations and assume one forward pass of g_s block (decoder network) is almost same with one forward pass of encoder network (g_a block), we can roughly say that their complexity is 4000 times higher which is not realistic in exchange of any bit savings for most of the case (still it may be possible if we once compress something but decompress million times). This problematic side of mentioned works in [1-2] is also mentioned recently in [4]. That is why we call these methods computationally heavy finetuning solution where we reported the CPU time results in the next section. 

  

  

[1] C. Cremer, X. Li, and D. Duvenaud, “Inference suboptimality in variational autoencoders,” ICML, 2018 

  

[2] J. Campos, S. Meierhans, A. Djelouah, and C. Schroers, “Content adaptive optimization for neural image compression,” in CVPR Workshops,June 2019 

  

[3]  Yang, Y., Bamler, R., & Mandt, S. (2020). Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33, 573-584. 

  

[4] Guanbo Pan et al. Content Adaptive Latents and Decoder for Neural Image Compression, ECCV 2022

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Complexity Analysis of methods and Higher bit resolution AE
comment: ### Q3 Complexity Analysis of the Proposals 

  

We totally agree with the Reviewer on his comments about missing complexity analysis. Thanks to the other reviewers pointing out the same fact, we did bunch of complexity analysis and described our proposals and pointed out where the additional complexity comes. For the details, please check this post as a response of another reviewer: 

https://openreview.net/forum?id=QVSoh6VM4nG&noteId=twjVs93NA9 

  

As a summary, our Hex-Quant needs between 0.7% to 3% additional cpu time for encoding where 0.1% to 1.9% additional cpu time for decoding. Oct-Quant needs 1.1%-5% additional cpu time for encoding where 0.1% to 1.9% additional cpu time for decoding. LatentShift method's additional cpu time is negligible as it is 0.07% to 0.7%. However, the most complex part is LatentShift's encoding process where it needs 5 to 10 times more cpu time than baseline encoder. At first sight, that complexity seems a lot. However, compared to the finetuning alternatives such as in [1], this complexity is low and LatentShift can be complementary of those finetuning solutions. We responded to that case in a separate post due to the length. 

  

Please note that all computation complexity results include the entropy encoding and decoding time as well. 

  

  

[1]  Yang, Y., Bamler, R., & Mandt, S. (2020). Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33, 573-584. 

  

  

### Q3 Complexity of Higher bit resolution AE 

  

We agree with the Reviewer that if the AE has higher resolution bit to represent the probabilities, the encoding and decoding becomes much slower, and it introduces another complexity. For practical purposes, somehow most of the researchers find reasonable to use 16-bit resolution in AE to represent the probabilities as we did in our tests. But Appendix D.3 is not for real test, but mostly ablation study in order to show the comparison between Oct-Quant and Hex-Quant if we have infinitive bit resolution AE which is hypothetical.  Thus, just in Appendix D.3, we measure the rate by Shanon's lower bound of bitlength as sum of negative log likelihood of the latent. All rest of the results are based on real bitlength taken from AE.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Novelty of uniform VQ and Small improvements on SOTA neural codec
comment: We thank the Reviewer k9t4 for the comments and we appreciate that the theoretical analysis of our paper is found interesting and insightful. And the reviewer finds our paper is well written and easy to follow. The constructive criticism regarding the complexity issue is very useful. In addition, raising the question on novelty of uniform VQ and differences between our LatentShift with overfitting solutions such as in hybrid amortized-iterative inference paper is very important. The latter issue increases our proposals' importance more.  

  

Below we first respond to general comments.  

  

### Q1 Uniform VQ or Lattice VQ and Novelty  

  

We agree that Uniform VQ and Lattice VQ are similar. However, it is noted that in [1], all lattices may not have necessarily an optimal space embedding. The term uniform VQ mentioned in our paper is a lattice but also with optimal space embedding (see minimum inertia in Remark1). But we agree that using the term “Lattice” may increase the clarity of the proposed method.  

  

Even though lattice vector quantization has been known for decades, researchers have not found it useful in neural codec so far. The reason might be "we researchers" tend to optimize everything end-to-end including VQ's grid center (thus partitioning map). In our Theorem1, we proved that non-uniform quantization map is not necessary for Scalar Quantization. It is because necessary transformation and its inverse can be applied by encoder/decoder network, and it is enough to use uniform quantization map over latent if our network is enough expressive.  In neural codec literature, most of the method applies uniform SQ map while a few methods use VQ with learned partitioning map. We discussed why VQ with learned partitioning map is not necessary and proposed to apply uniform VQ map that is an instance of lattice VQ for to 2-dim and 3-dim.  

  

To the best of our knowledge, lattice (or uniform) VQ has never been used in neural codec so far. The unique paper proposed to use it in model compression (network weights compression) in [2] as a post training quantization process which is out of concern of neural image compressing. Thus, we respectfully disagree with the Reviewer and think our proposal is novel. We are very much interested to hear if we miss any paper that proposes lattice (or uniform) VQ in neural codecs even just in practical usage. 

  

[1]	J.H. Conway, N.J.A. Sloane, and E. Bannai, Spherepackings, lattices, and groups, New York, NY, USA: Springer-Verlag New York, Inc., 1993  

  

[2] Choi, Y., El-Khamy, M., & Lee, J. (2020). Universal deep neural network compression. IEEE Journal of Selected Topics in Signal Processing, 14(4), 715-726. 


### Q3 Small Improvement on SOTA neural codec (Significance of the results)

According to our test, we have seen similar lower results on autoregressive model named InvCompress in [1] like cheng2020 that we reported their performance as an answer of another reviewer here https://openreview.net/forum?id=QVSoh6VM4nG&noteId=Cg7dY888rb In autoregressive model, we cannot reshape all latent together, but use the same spatial location's different channel feature and create pseudo high dimensional tensor for uniform VQ. Since there is redundancy along the spatial dimension but not necessarily in feature dimension, the latent to be packed into the same vector will be diverse (i.e coming from different distribution). However, if we pack the latents which are spatially neighbor is much better than pack the latents which are neighbor along the channel. In autoregressive model we should use latter because another spatial neighbor latent is not available before coding latents on the current spatial position. This may explain the performance degradation in autoregressive entropy models.  We used a more complex encoder /decoder network model that does not use autoregressive probability model named DCVC [2] as an answer of a reviewer 36qZ and found that our VQ method saves more than 1% for image compression and more than 1.5% in video compression from baseline DCVC at the same quality.

Since autoregressive models has a non-parallelizable (or at least hardly parallelizable) complexity in the codec, more recent codecs try to avoid that problem by using some alternatives to the autoregressive model as the one is introduced in [2] which uses two step coding by dividing the latent into two groups. We believe that if neural codecs are standardized, autoregressive prediction will not be part of the codec in the future. Thus, we can see the codec that uses autoregressive entropy model is one of the limitations of our uniform VQ proposal. 



[1] Xie et al., Enhanced invertible encoding for learned image compression. ACM MM 2021.

[2] Li et al., Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression ACM MM 2022.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: follow-up
comment: We thank the reviewer 36qZ for reading our answers and raising the score.  We appreciate that the reviewer acknowledged our additional experiments on the latest codec.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Answers to the Questions-2
comment: ### Q4 Novelty of Uniform Vector Quantization over "Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression." 

  

We agree with the Reviewer that Vector Quantization was used in neural codecs before as we cited the paper by Agustsson et al., 2017, in our paper. Since the studied codecs are a specific version of VAE model, Applied Vector Quantization on general VAE is known by VQ-VAE, and even if they do not use it for compression, it can be seen as our previous works, such as papers in [1,2,3]. Also, the paper in [4] that was pointed out by our Reviewer is a wonderful example of how VQ can be applied on neural codecs. However, all those mentioned models including [4], target to learn grid partitioning in high dimensional space. Thus, their quantization map may not be uniform. The non-uniform quantization map has several disadvantages. First the center should be optimized jointly with the rest of the parameters. Second, continuous relaxation of non-uniform VQ cannot be done with additional uniform noise in training time, which was found so far, the best practice in the literature. Applied VQ's continuous relaxation is based on *softmax* assignment instead of finding the closest grid center (hard assignment). Thus, their algorithm assigns all grid center to the coming latent with different probability in training time. Thus, relaxation of quanta center may be very far from its center in test time. That is why all VQ based VAE model applies the soft assignment with some adaptive temperature parameter. Initially it uses very smooth softmax, but it is going to approximate hard assignment in the later training epochs in order to decrease the mismatch between latent values in train and test time. Third and most important is the complexity of searching for the closest grid center. Since the VQ map is not uniform, there is no fast approximation to find the closest grid. Naive search should be done which increases the encoding complexity exponentially.  That is why in [4], they used multi-codebook structure. Up to these problematic sides of standard VQ, we also prove that learning grid of quantization is not necessary if our transformations are enough rich in our Theorem1. So, we proposed Uniform VQ known as lattice VQ where the space is partitioned by optimal uniform grids. The main idea to apply uniform VQ is that, if the codec needs a non-uniform map, the transformation between necessary non-uniform map to our uniform map can be learnt by neural network. Thus, we do not need to learn partitioning of the space as in mentioned references [1-4]. If we keep the uniform VQ grid size as same with already trained neural codec with SQ, we can improve the performance even without training. But if we re-train the network, we can even reach less quantization error thus, better reconstruction. 

  

So far in neural codec literature, there are two mainstream perspectives on quantization. The one is to use uniform SQ where the grid center is fixed (integer centers), thus no learnable parameter (the vast majority of the models), other use VQ that the centers are learned end-to-end. However, our proposal is to use uniform VQ that takes advantages of both SQ's experimental efficiency and VQ's theoretical power. We are quite confident that our paper is the first paper to use it and explain why we need to use uniform VQ in Theorem1. 

  

[1] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. “Neural Discrete Representation Learning.” In: NeurIPS. 2017 

  

[2] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. “Generating Diverse High-Fidelity Images with VQ-VAE-2.” In: NeurIPS. 2019 

  

[3] Walker, J., Razavi, A., & Oord, A. V. D. (2021). Predicting video with vqvae. arXiv preprint arXiv:2103.01950. 

  

[4] Zhu, Xiaosu, et al. "Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Answers to the Questions-1
comment: 
### Q1: Vector Notation 

We thank the reviewer for carefully catching this point. Indeed, the last paragraph of page3 $u, C_j, x \in \displaystyle R^{v}$ are v-dimensional vector. Since we define its dimension before usage, we suppose we can use the same notation. But apparently, we were wrong on it. We update the paper and use appropriate notation for vectors soon. 

  

### Q2: Separate performance and complexity analysis of each proposal 

We proposed 2 versions of vector quantization named *Hex-Quant* (2d uniform VQ) and *Oct-Quant* (3d uniform VQ) and one version of Latens shifting method named *Latent Shift* in the tables and figures. So, each table's column and each curve on the figures are the result of this method's individual performance. We also use another label named *Join* that refers to using both quantization and latent shift together. If the table or figure includes *Oct-Quant*, then *Join* means applying both *Oct-Quant* and *Latent Shift* together. But if there is no *Oct-Quant* in the table or figure but just *Hex-Quant*, then *Join* means applying both *Hex-Quant* and *Latent Shift* together. Thus, the tables and figures in the paper show the coding performance (the percentage of saved bits for the same quality, known as bd-rate) of each separate proposal as well as together. That is what exactly we understood that our Reviewer wants to see. The missing part was the computational complexity of each method where we responded to that point in the following post separately (https://openreview.net/forum?id=QVSoh6VM4nG&noteId=kLj48PJSpF). To give concrete example, in Table2 in the paper, our Hex-quant on SSF saves 1.31% bitstream from original model on Beauty video, where in running time result says Hex-Quant on SSF needs 2.4% more cpu time in encoding and 1.9% more cpu time in decoding from baseline SSF. 

  

### Q3 Small Improvement on *cheng2020* model. (Significance of the results)

According to our test, we have seen similar lower results on autoregressive model named *InvCompress* in [1] like *cheng2020* that we reported their performance as an answer of another reviewer here https://openreview.net/forum?id=QVSoh6VM4nG&noteId=Cg7dY888rb In autoregressive model, we cannot reshape all latent together, but use the same spatial location's different channel feature and create pseudo high dimensional tensor for uniform VQ. Since there is redundancy along the spatial dimension but not necessarily in feature dimension, the latent to be packed into the same vector will be diverse (i.e coming from different distribution). However, the more similar scalar pack into the same vector (coming from the same distribution) the more accurate quantization we have. Shortly, if we pack the latents which are spatially neighbor is much better than pack the latents which are neighbor along the channel. In autoregressive model we should use latter because another spatial neighbor latent is not available before coding latents on the current spatial position. This may explain the performance degradation in autoregressive entropy models. For the latent shift, indeed, we have seen that the smaller the decoder network, the more correlation we have, thus more improvement on latent shift method. In addition, latent shift performance depends on the training procedure and used dataset, and these can be different on different neural codecs. We used a more complex encoder /decoder network model that does not use autoregressive probability model named DCVC [2] as an answer of a reviewer 36qZ and found that their uniform VQ performance is quite significant.  

The proposed Latentshift method's performance is directly related to the existing correlation between gradients as can be seen in Figure 4c. As our theory shows, the connection between gradients are in expectation of the train set. Thus, the correlations can be different and varies model by model and can be affected by training procedure as well as the used training set. It may explain the reason for divergence of Latentshift’s improvements on different codecs.

Beside autoregressive entropy models, our proposal gives very significant gain over baselines. For instance, it gives more than 2% bd-gain in average over SSF video codec. We also tested very recent video codec (DCVC in [2]) and present results for the comments of Reviewer 36qZ on https://openreview.net/forum?id=QVSoh6VM4nG&noteId=Cg7dY888rb . Again, this test shows more than 2% bd-gain in average. Our proposals gain is much higher in lower rate regime which is more important in general use. In general compression work, these improvements are significant.


[1] Xie et al., Enhanced invertible encoding for learned image compression. ACM MM 2021.

[2] Li et al., Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression ACM MM 2022.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Computational Complexity-2
comment: # Run Time Results 

  

For experimental computational complexity, we run the test on single core of Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz with preventing multi-thread operations.  We measure *cheng2020_attn*, *mbt2018-mean* performance for Kodak dataset and SSF's on bunny dataset. In order to have the same resolution image, we just crop the top-left part of the frames in bunny frame set which has the same resolution as kodak dataset (512x768). In this way, all methods complexity can be comparable between each other.  

  

The results show average encoding/decoding time in seconds for the provided lowest bitrate baseline model and our Hex-Quant, Oct-Quant and Latent Shift improvements. Reported encoder running times is the duration from the input image is read to the memory to the producing the bitstream including arithmetic encoder time. It is the same for decoding. The duration is from the bitstream in the memory to obtain reconstructed image including arithmetic decoder process.  The results are reported in the table below.  

  

| Model ||| Encoding ||| Decoding|||
| ----------- | :----:  | :----: | :----: |:----: |:----:  |:----: |:----:| ----------- |  
|   | Baseline  | Hex Quant | Oct Quant |Latent Shift | Baseline    | Hex Quant  | Oct Quant| Latent Shift | 
| mbt2018-mean  | 0.5274sec| 0.5436sec (+3%)| 0.5548sec (+5%)| 5.5674sec (x10.1)| 0.7138sec | 0.7264sec (+1.8%)  | 0.7265sec (+1.8%) | 0.7188sec (+0.7%)| 
| cheng2020_attn | 3.4982sec  |3.5238sec (+0.7%) |3.5368sec (+1.1%)  | 18.5345sec (x5.3)|8.3065sec | 8.3190sec (+0.1%) | 8.3195sec (+0.1%) | 8.3115sec (+0.06%)| 
| SSF | 1.8618sec  |  1.9068sec (+2.4%)  | 1.9230sec (+3.2%)  |  11.1418sec (x6.0)|1.3025sec | 1.3285sec (+1.9%) | 1.3288sec (+1.9%)  |1.3125sec (+0.07%)| 
  

The main source of additional encoding complexity of Hex-Quant and Oct-Quant is to find the closest codebook in Step2.  The naive search gives around %20 overhead complexity for 3-dimensional VQ. But there are algorithms that find nearest quanta center very fast defined in [1-2]. We adapted this solution to our model and decreased additional cost to 3%-5% in encoding time for *mbt2018-mean* model. Since *cheng2020_attn* model is most complex model in our studied neural codec, even though our extra computation is almost the same with *mbt2018-mean* model (our Hex-Quant and Oct-Quant runtime depends on the resolution of the latent not the architecture), Hex-Quant and Oct-Quant 's relative overhead is between 1.1%-0.7% in encoding time. For SSF, we neglect the I frame compression runtime (because it is the same with *mbt2018-mean*) but measure the average runtime for encoding and decoding of P frames which is done by 2 VAEs, one for motion information another for residual information. Thus, there are two main information (main motion and main residual) and our quantization should be done on these latent. It explains our extra absolute runtime is more than absolute extra runtime in *mbt2018-mean*. Since the P frame compression model is more complex than *mbt2018-mean*, Hex-Quant and Oct-Quant's extra relative runtime is between 2.4%-3.2%. Since the differences between dimension of VQ is just reshaping tensor and number of indices to be found from dictionary in decoding time, the decoding time extra complexity of Hex-Quant and Oct-Quant is negligible and almost the same with hex and oct quantization as it can be seen in the table. 

  

As it is explained in details of *Latent Shift*, its decoding complexity is less than 1% as it can be shown on Table. However, *Latent Shift*'s encoding complexity is between 5-10 magnitude of baseline encoding running time. Almost all complexity comes from finding the best shift step size by brutal force out of 8 candidates. Even though this high complexity in encoding time, we explain the connection to this method to the latent finetuning approaches in response to the Reviewer k9t4 and 7dNo. Thus, this complexity is very low compare to the those similar finetuning solutions. Please follow these answers.

  

  

  

  

[1] E. Agrell, T. Eriksson, A. Vardy, and K. Zeger. Closest point search in lattices. IEEE Transactions on Information Theory, 48(8):2201–2214, 2002. doi: 10.1109/TIT.2002.800499   

  

   

  

[2] Convay, J. H., & Sloane, N. J. A. (1982). Fast quantizing and decoding algorithms for lattice quantizers. IEEE Trans Inform Theory, 28(2), 227-232.

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Computational Complexity-1
comment: We thank reviewer tojV for the time that was spent on our submission and also comments about the clarity, novelty and theoretical sounds of our submission. We also very much appreciated the very useful criticism on our paper which are significance of the results, novelty on uniform VQ, additional computational complexity of proposals and more details on application of VQ. 

Here we start with some implementation details of proposal and explains where the additional complexity comes.


# More Details of Proposals and computation costs 

  

We agree with our reviewer that computational complexity should have been done in order to show the efficiency of the proposed methods. Here we explain how our proposals can be applied at the application level, where the additional computational costs come from and what the runtime results of proposals are as well as the baseline models.   

  

  

## Encoding Time of uniform VQ 

  

In preprocessing, we should define our grid centers as codebook such that $\mathbf{c}^{(i)} \in R^{v}, i=1 \dots M$ be the $M$ grid centers of the $v$-dimensional shape (e.g hexagonal grids center for v=2) and PMF of our v-dimensional grid using learned 1-d PMF by Eq2 in the paper.  

  

  

The extra processes in encoding time are the followings:  

  

1. Reshape the latents into pseudo v-dimensional vector $ \mathbf{y} \in R^{m \times m \times o} \rightarrow \mathbf{\check{y}} \in R^{b \times v}$, where $b =\frac{m.n.o}{v}$ (Before reshaping, the latent can be sorted by their distribution's $\sigma$ params. In this way, latent that belongs to a similar distribution can be dropped into the same latent vector.  

  

2. Find closest codebook from our initial quantization grids for each $b$ vector such $\mathbf{\tilde{y}}_j$=$argmin_i ||\mathbf{\check{y}}_j-\mathbf{c}^{(i)}||$   where   

  

$\mathbf{\tilde{y}}_j \in \{1 \dots M\}, j=1 \dots b$ are the codes to be encoded into bitstream.   

  

   

  

In the shared step, $\mathbf{\tilde{y}}_j,  j=1 \dots b$ should be encoded into bitstream by given PMF.  

  

Thus, in encoding time above 2 step's complexity is the reason of extra complexity.  

  

## Decoding Time of uniform VQ 

  

In decoding time, $\mathbf{\tilde{y}}_j,  j=1 \dots b$ should be decoded from bitstream by provided PMF table. The source of extra complexity is in the following 2 steps in decoding time.   

  

1. De-quantizate the codes: To find the center of selected quanta center such that $\mathbf{\bar{y}}_j=\mathbf{c}^{\mathbf{\tilde{y}}_j},  j=1 \dots b$  

  

2. Reshape dequantized latent into original dimensions such that $\mathbf{\bar{y}} \in R^{b \times v}   \rightarrow \mathbf{\hat{y}} \in R^{m \times m \times o}  $, (if the ordering wrt $\sigma$ is applied, we need to revert the order back in order to have original latent order)  

  

In shared step, these dequantized and reshaped latent is to be fed to the decoder network

## Encoding Time of Latent Shift 

  

In encoding time, *Latent Shift* must calculate the gradient of the entropy $\nabla_{\mathbf{\hat{y}}}(-log(p_h(\mathbf{\hat{y}};\mathbf{\hat{z}},\Theta)));\mathbf{\theta}))$ as the first step. Later, it needs to find the best step size $\rho_h$ that maximize the reconstruction quality when the latent is shifted as $\mathbf{\hat{y}} \leftarrow \mathbf{\hat{y}}+\rho_h^*\nabla_{\mathbf{\hat{y}}}(-log(p_h(\mathbf{\hat{y}};\mathbf{\hat{z}},\Theta)))$. These two steps create additional complexity in encoding time. Calculating gradient of the entropy has negligible complexity. Because we do not need forward pass and there is closed form solution. When the entropy model uses gaussian distribution such as $p_h(\mathbf{\hat{y}};\mathbf{\hat{z}},\Theta):=N(\mathbf{\hat{y}};\mu,\sigma)$, the gradient of the entropy becomes the derivative of  $-log(N(\mathbf{\hat{y}};\mu,\sigma))$   wrt $\mathbf{\hat{y}}$ which has closed form solution where $N(\mathbf{.;\mu,\sigma})$ is PDF of gaussian distribution by given fixed $\mu,\sigma$ parameters. However, finding the best step size $\rho_h$ is a high demanding process. In the application we tested 8 different choices for $p_h$ and selected the best one. Thus, it needs 8 forward passes of decoding and calculation of error between input and reconstruction.  

  

## Decoding Time of Latent Shift 

In decoding time, *Latent Shift* needs to decode latents from bitstream as common step. As an extra process, it needs to read single scalar $\rho_h$ from bitstream and calculates the gradient entropy wrt latents which are both negligible. The last step is to shift the latent by using gradient and decoded step size which is actually adding two tensor and it is negligible compared to the decoding complexity of the baseline model.

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_36qZ
title: Updated review
comment: The authors provided more informative results in the response. I change my recommendation to "accept".

From: ICLR.cc/2023/Conference/Paper4294/Authors
title: Comparison with latest codec
comment: We thank reviewer 36qZ for the comments and we are glad that the reviewer appreciated our method as novel and theoretically sound. Below we address the concerns of the reviewer.  

  

## Comparison with latest codecs is missing:   

  

We have performed experiments with the latest codec suggested by the reviewer 36qz, and the results of image codecs are reported in the below table. The results show that our proposed method has more than 1% average BD gain across different rates. This shows that our proposed method is pretty generic and can be applied on any existing neural codecs.    

  

| Baseline Model || Kodak ||| Clic-2021 || 
| ----------- | :----:  | :----: | :----: |:----: |:----: | ----------- |  
|   | Hex-Quant  | Latent-shift | Join | Hex-Quant  | Latent-shift | Join | 
| InvCompress[1]  | -0.21% |-0.55% | -0.82% | -0.52% | -0.63%  | -1.12% | 
| DCVC [2] All intra mode | -0.97%  |-0.30% |-1.28% |-1.22% |-0.11% |-1.44% | 

  

  

Since InvCompres in [1] uses autoregressive entropy model for main information similar to cheng2020 in the paper, its result on uniform vector quantization is weak as in cheng2020. This can be explainable by the fact that pseudo high dimensional latent cannot be created once using by all latents together. It should be done one by one according to latent's spatial position. Thus, if the latent $y \in R^{m \times m \times o}$,  a latent vector $y_{i,j} \in R^{o}$ should be sorted according to their distribution parameter $\sigma$ and reshaped into $R^{2,o/2}$ dimensional matrix for 2-dim uniform VQ.  Since there are less latent to be packed into 2-dim together, the latent to be quantized together can be more diverse than using all latent together. This point is one of the limitations of applying our quantization as a post training process. However, since autoregressive models has a non-parallelizable (or at least hardly parallelizable) complexity in the codec, more recent codecs try to avoid that problem by using some alternatives to the autoregressive model as the one is introduced in [2] which uses two step coding by dividing the latent into two groups. Thus, we believe that if neural codecs are standardized, autoregressive prediction will not be part of the codec in the future. The reason we did not include one mentioned neural codec in [3] by reviewer is that this codec uses cheng2020 model as I frame compression that we already included in our image result, but also again autoregressive prediction in motion and residual coding.  

  

For neural video codec results, we repeated the same testing procedure with SSF and reported the DCVC [2] performance in the table below. The results show that our proposed method has more than 2% average BD gain across provided 4 different qualities in pre-trained model. 

  

|Video |  Hex-Quant  | Oct-Quant  | Latent-shift | Join |  
| ----------- | :----:  | :----: | :----: |:----: | 
|Beauty  |-1.92% | -1.97% | -0.10%  | -2.02%  |  
|Bosphorus | -1.23% |-1.48% |-0.62% |-2.05% | 
| Honeybee |-1.80% |-2.13% |-0.56% |-2.65% | 
|Jockey | -1.42% |-1.32% |-0.11% |-1.51% | 
|ReadySteadGo |-1.29% |-1.30%|-1.08% |-2.35% | 
|ShakeNDry |-1.55% |-1.65% |-0.23%|-1.92% | 
|YatchRide |-1.42% |-1.61% |-0.18% |-1.83% | 
|UVG Average |-1.52% |-1.64% |-0.41% |-2.05% | 
|Bunny |-0.90% |-1.02% |-1.21% |-2.25% |   

  

We are glad to adapt these results and citations in the revised paper soon. 

  

[1] Xie et al., Enhanced invertible encoding for learned image compression. ACM MM 2021. 

  

[2] Li et al., Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression ACM MM 2022.  

  

[3] Li et al., Deep Contextual Video Compression, NeurIPS 2021

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_36qZ
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes some theoretical insights about quantization and entropy gradient, and show that this can improve the performances of many off-the-shelf codecs. The proposed method is able to save 2-4% of the bit-rate for various pre-trained methods.
strength_and_weaknesses: The proposed method is novel and technically sound. The experiments show that the proposed method is able to improve many learned image/video compression networks. However, the reviewer concerns about the codec used in experiments. The used codecs are all proposed on or before 2020, and the latest codecs are missing from the experiments. The authors should test it on some newer methods with open-sourced codes, e.g.,

Image codec:

Xie et al., ACM MM 2021. Codes: https://github.com/xyq7/InvCompress

Video Codec:

Li et al., NeuIPS 2021. Codes: https://github.com/microsoft/DCVC/tree/main/NeurIPS2021

Li et al., ACM MM 2022. Codes: https://github.com/microsoft/DCVC/tree/main/ACMMM2022

------ After response -------

The authors addressed my questions.

After discussing with other reviews, I think the high complexity of encoding is a shortcoming of the method. I still prefer a positive comment, but slightly adjusted the rating from 8 to 6. 
clarity,_quality,_novelty_and_reproducibility: Good novelty.
summary_of_the_review: The proposed method is novel and technically sound. The experiments show that the proposed method is able to improve many learned image/video compression networks. However, the reviewer concerns about the codec used in experiments. The used codecs are all proposed on or before 2020, and the latest codecs are missing from the experiments. The authors should test it on some newer methods with open-sourced codes.

------ After response -------

The authors addressed my questions.

After discussing with other reviews, I think the high complexity of encoding is a shortcoming of the method. I still prefer a positive comment, but slightly adjusted the rating from 8 to 6. 
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_tojV
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper exploit vector quantization and latent shifting to improve the performance of neural codec. For vector quantization, the author first explans why vector quantization is not often used in neural codec while being theoretically better than scalar quantization. The author then proposes proposed uniform vector quantization can be applied in the off-the-shelf neural codec without re-training and optimal VQ grids with theoretical proof. For latent shifting, the author explans how the gradients of the entropy in the receiver side is used to improve the reconstruction performance and gives theoretical proof.

strength_and_weaknesses: Strengths: 
1. This paper uses uniform vector quantization and latent shifting to impove coding performance. Also, the author test this method on several image and video codecs and proves the proposed method works in many image/videos codecs.
2. This paper gives detailed theoretical proof for the proposed method.

Weaknesses: 
1. The proposed vector quantization and latent shifting method may increase encoding and decoding complexity while the coding improvement may not that great.
2. This paper lacks analysis on the encoding/decoding complexity compared with baseline codecs.
3. The author takes a lot of space to describe how scalar quantization based neural codec works, may be it is better to describe the training and testing process of vector quanization based neural codec more detailedly.

Questions
1. In the last paragraph of page 3, the author describes the process of vector quantiation but uses symbols with no vector notation. Should the symbols metioned in this paragraph vectors or scalars?
2. The author decribes several methods to obtain  and  will using different method affect the coding performance and coding complexity? And what's the complexity and coding performance by using each method? 
3. The proposed method provides relatively small performance imprsovement using image codec cheng2020-atten. Is it because the analysis/synthesis transform is can better catch the spatial correlation of the picture? What will happen when using more complex analysis/synthesis transfoms as baselines?
4. In Zhu, Xiaosu, et al. "Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. vector quantization is also used for image coding. How does the proposed method compare with the method in this article?

clarity,_quality,_novelty_and_reproducibility: Quality: This paper provides new methods to improve the performance of neural image compression. However, the improvement is not that high and lacks some experiment and comparison.
Clarity: The article describes clearly and gives detailed theoretical proof.
Originality: Latent shifting is first used in image compression but vector quantization has been used before.
summary_of_the_review: This paper uses vector quantization and latent shifting to improve the coding performance of the neural codec and provides theoretical proof. However, the performance gain is relatively small while the auther didn't explan how much complexity will be added using the proposed method. Also, vector quantization has been used in image compression before (like Zhu, Xiaosu, et al. "Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.) and it lacks comparison with those works in this paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_k9t4
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes to explore two latent properties (vector quantization and entropy gradient based latent shifting) in off-the-shelf neural codecs. The proposed method replaces uniform scalar quantization with uniform vector quantization to take advantage of space tessellation grid. Moreover, the authors show that the gradients of the reconstruction error are correlated to the gradients of entropy, and propose to update latent using the entropy gradients at decoder side. The proposed two contributions are plug-in-and-play and bring 2-4% rate savings at same quality.
strength_and_weaknesses: The paper is well written and the theoretical analysis of vector quantization and latent shifting is clear and sufficient. 

However, I have several concerns about the novelty and complexity: 
1) what’s the difference between the proposed uniform VQ and lattice VQ? It seems that they are the same thing; 

2) what’s major difference between the proposed latent shifting and hybrid amortized-iterative inference [1]? In my understanding, [1] update latent using both entropy gradients and distortion gradients, and the proposed latent shifting update latent using entropy gradients only. The major difference is that [1] directly encodes the updated latent, but latent shifting encodes the best updated steps and then updates latent at decoder side. Is that right? If so, what’s the advantages of the proposed method? The paper claims that [1] is a computationally heavy finetuning solution. Is there any complexity comparison? 

3) the gains of uniform VQ on the state-of-the-art backbone (chen2020-attn) are not significant ( <0.5% BD-rate saving in Table 1) and the complexity analysis is lacked. In appendix D.3, the frequency table in arithmetic coder is large (over 65536), which may result in slow entropy coding. 

[1] Yang, Y., Bamler, R., & Mandt, S. (2020). Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33, 573-584.

clarity,_quality,_novelty_and_reproducibility: The Clarity is good with fair quality, novelty and reproducibility.  
summary_of_the_review: Overall the paper is well written and easy to follow. The theoretical analysis of the two latent properties (vector quantization and latent shifting) is insightful. However, the novelty of the proposed method is limited and performance gains on the state-of-the-art backbone are not significant.  
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4294/Reviewer_7dNo
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes two “orthogonal” methods to improve existing deep-learning based image and video compression models. First, the authors swap out existing scalar quantization modules with a uniform vector quantization map - they show that this VQ map doesn’t need to be learned but can be uniform (e.g. hexagonal or octahedral), and will still propose improvements over SQ. Second, the authors propose using the gradient of entropy on the decoder side; they use KKT conditions to show that at least for the main code, the gradient of entropy and gradient of reconstruction error are correlated, so even if reconstruction error can’t be calculated we can still follow gradient along entropy for a limited number of steps. 
strength_and_weaknesses: Strengths/questions: 
- Interesting and thorough theoretical treatment of VQ and how a uniform VQ map is sufficient over learned centroids. Though I have two questions there: 1) Does Theorem 1 extend to VQ? It seems primarily focused on scalar quantization. And 2) Following the intuition in Theorem 1 where a NN can transform a non-uniform quantization map into a uniform one, why can’t a neural network also transform a VQ-map to an SQ one? Are there limitations there? 
- I like the analysis of how KKT conditions implies -> gradient of entropy is correlated with gradient of reconstruction error, it’s a novel idea for squeezing out slightly better reconstruction quality (though I do have some concerns, described below). 


Some high-level concerns:
- Why follow gradients on the decoder-side when we can also follow gradients for both the hyperprior and main code on the encoder side We have access to the ground truth $x$ and $y$ on the encoder side, and can optimize $\hat{y}$ and $\hat{z}$ directly for the rate-distortion tradeoff without proxies. Is one approach better than the other in terms of runtime (also see below for runtime concerns)? Are they complementary? 
- For the hyperprior especially, following the gradients on the decoder side only doesn’t make sense. In fact the authors state at the bottom of section 4 that “It is noted that during the encoding, the main latents yˆ are encoded wrt entropy model predicted by the shifted side latent” - which means that this procedure for shifting the hyperprior code $\hat{z}$ is already conducted on the encoder side. But at that point you have access to the ground truth $\hat{y}$, so you can directly follow the gradient $\nabla_{\hat{z}}(-log(p_h(\hat{y}; \hat{z})))$ and you don’t need to follow the proxy $\nabla_{\hat{z}}(-log(p_f(\hat{z})))$ right? Am I misunderstanding something? 
- Intuitively, by having the main code follow the gradient of entropy (increase entropy) to “improve” reconstruction error, this basically means that we’re leveraging the inductive bias of the network itself to “fill in” information that is not provided by the original code right? Are there side effects/downsides to such an approach? One practical concern I’m wondering is 1) if this approach only works when the train/test distributions are very similar (for a new image what if the network “fills in” the wrong info?)? 2) Also by following the gradient to “increase entropy” is it possible that reconstruction error is better “on average” but is worse for specific subsets of images? 

clarity,_quality,_novelty_and_reproducibility: The paper is generally easy to read and from what I can tell the ideas are novel. It does have some vague statements (e.g. “Most of the time, this variation is in favor of the combination” in Section 5) and spelling/grammar mistakes, but those are minor. 
summary_of_the_review: The paper suggested some novel ideas that were not immediately intuitive to me, such as doing uniform vector quantization as well as following the gradients of entropy on the decoder side for the hyperprior/main code. There is a fairly thorough theoretical treatment of some of these ideas. However, I have concerns about the justification for decoder-side “latent shift” (why not just optimize codes on the encoder side), as well as some practical concerns around runtime. 

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold