Original: Flexible Neural Image Compression via Code Editing
Revised: Flexible Neural Image Compression via Code Editing

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Neural image compression (NIC) has outperformed traditional image codecs in ratedistortion (R-D) performance.
Revised: Neural image compression (NIC) has outperformed traditional image codecs in ratedistortion (R-D) performance.

Original: However, it usually requires a dedicated encoderdecoder pair for each point on R-D curve, which greatly hinders its practicaldeployment.
Revised: However, it usually requires a dedicated encoderdecoder pair for each point on R-D curve, which greatly hinders its practicaldeployment.

Original: While some recent works have enabled bitrate control via conditionalcoding, they impose strong prior during training and provide limited flexibility.
Revised: While some recent works have enabled bitrate control via conditionalcoding, they impose strong prior during training and provide limited flexibility.

Original: In this paper we propose Code Editing, a highly flexible coding method for NICbased on semi-amortized inference.
Revised: In this paper we propose Code Editing, a highly flexible coding method for NICbased on semi-amortized inference.

Original: And we further improve our Code Editingvia adaptive quantization.
Revised: And we further improve our Code Editingvia adaptive quantization.

Original: We demonstrate our method under various conditionsand we show that our code editing surpass existing variable-rate methods throughexperiment.
Revised: We demonstrate our method under various conditionsand we show that our code editing surpass existing variable-rate methods throughexperiment.

Original: Our work is a new paradigm for variable bitrate NIC and it is thefirst to achieve continuous bitrate control, ROI coding and perception-distortiontrade-off with a single decoder.
Revised: Our work is a new paradigm for variable bitrate NIC and it is the firstto achieve continuous bitrate control, ROI coding and multi-distortion trade-offwith a single decoder.

Original:  Introduction
Revised:  Introduction

Original: Lossy image compression is a fundamental problem of computer vision.
Revised: Lossy image compression is a fundamental problem of computer vision.

Original: In a simplified setting,the aim of lossy compression is to minimize rate-distortion cost R + λD , with λ as the Lagrangemultiplier controlling R-D trade-off.
Revised: In a simplified setting,the aim of lossy compression is to minimize rate-distortion cost R + λD , with λ as the Lagrangemultiplier controlling R-D trade-off.

Original: Traditional compression algorithms (e.g. JPEG, JPEG2000 and
Revised: Traditional compression algorithms (e.g. JPEG, JPEG2000 and

Original: BPG [Bellard, 2018])utilize regularity of natural statistics, which assumes the sparsity of imagesin transform domain. They first perform a linear transformation (e.g. DCT, wavelet transform) andthen quantize the transformed coefficients for compression. 
Revised: BPG [Bellard, 2018]) first perform a linear transformation (e.g. DCT, wavelet transform) and thenquantize the transformed coefficients for compression. Their bitrate control is achieved by controllingquantization stepsize.

Original: On the other hand, lossy neural imagecompression (NIC) methods have outperformed traditional codecs in recent years [Xie et al., 2021,
Revised: On the other hand, lossy neural image compression (NIC) methods haveoutperformed traditional codecs in recent years [Xie et al., 2021,

Original: He et al., 2022].
Revised: He et al., 2022].

Original: These methods implement powerful non-linear transformation with a neural network,and therefore achieve better R-D performance.
Revised: These methodsimplement powerful non-linear transformation with a neural network, and therefore achieve better R-D performance.

Original: However, in order to realize R-D trade-offs, we needto optimize multiple networks towards different losses with different λ s. In other words, the existingneural image compression methods cannot achieve flexible bitrate control.
Revised: However, in order to realize R-D trade-offs, we need to optimize multiple networkstowards different losses with different λ s, which means that we cannot achieve flexible bitrate control.

Original: To enable NIC models with bitrate control ability, several works adopt conditional networks [Choiet al., 2019, Cui et al., 2021, Sun et al., 2021] .
Revised: To enable NIC models with bitrate control ability, several works adopt conditional networks [Choiet al., 2019, Cui et al., 2021, Sun et al., 2021] .

Original: They use λ as condition and feed it into the encoderand decoder.
Revised: They use λ as condition and feed it into the encoderand decoder.

Original: During training, they sample λ from a pre-defined prior and optimize the E p ( λ ) [ R + λD ]via Monte Carlo.
Revised: During training, they sample λ from a pre-defined prior and optimize the E p ( λ ) [ R + λD ]via Monte Carlo.

Original: Due to the capacity of conditional networks, most of them use simple prior andvery limited sets of λ , such as a uniform distribution over only 5 − 10 discrete λs
Revised: Due to the capacity of conditional networks, most of them use simple prior andlimited sets of λ , such as a uniform distribution over only 5 − 10 discrete λ

Original:  [Choi et al., 2019, Cui et al., 2021].
Revised: s [Choi et al., 2019, Cuiet al., 2021].

Original: And additional effort is required to interpolate the discrete λ s to achieve continuous
Revised: And additional effort is required to interpolate the discrete λ s to achieve continuous

Original: R-D curve [Choi et al., 2019, Cui et al., 2021, Sun et al., 2021], which limits the flexibility of variablebitrate model.
Revised: R-D curve [Choi et al., 2019, Cui et al., 2021, Sun et al., 2021], which limits the flexibility of variablebitrate model.

Original: On the other hand, traditional image codecs achieve flexible R-D trade-off naturally bycontrolling the quantization step.
Revised: On the other hand, traditional image codecs achieve flexible R-D trade-off naturally bycontrolling the quantization step.

Original: This control is fine grain (e.g. 100 levels for JPEG).
Revised: This control is fine grain (e.g. 100 levels for JPEG).

Original: Moreover, theysupport flexible spatial bits-allocation for ROI-based (region of interest) coding.
Revised: Moreover, theysupport flexible spatial bits-allocation for ROI-based (region of interest) coding.

Original: For example, the

Original: H.264 allows 64 level quantization step control for each 16 × 16 block of images.

Original: And the user needonly one decoder for all those flexibility.
Revised: And the user needonly one decoder for all those flexibility.

Original: In this work, we intend to give neural image compression methods the flexibility of traditional codecs:the continuous rate-distortion trade-off with arbitrary spatial bits-allocation by one decoder.
Revised: In this work, we intend to give neural image compression methods the flexibility of traditional codecs:the continuous rate-distortion trade-off with arbitrary spatial bits-allocation by one decoder.

Original: It is a meaningful task for practical NIC considering the growing size of neural decoders and the needfor fine-grain rate control.
Revised: It isa meaningful task for practical NIC considering the growing size of neural decoders and the needfor fine-grain rate control.

Original: In order to achieve this, we propose Code Editing, a semi-amortizedinference [Kim et al., 2018]  approach for flexible R-D trade-off.
Revised: In order to achieve this, we propose Code Editing, a semi-amortized inference [Kim et al., 2018] based approach for flexible R-D trade-off.

Original: Although previous works haveadopted semi-amortized inference to improve R-D performance [Johnston et al., 2018, Yang et al.
Revised: Although previous works haveadopted semi-amortized inference to improve R-D performance [Johnston et al., 2018, Yang et al.

Original: ,2020], we are the first to discover its potential for flexible bitrate control.
Revised: ,2020], we are the first to discover its potential for flexible bitrate control.

Original: Specifically, we edit thelatent code directly to change the optimization target at inference time based on desired λ s and ROIs.
Revised: Specifically, we edit thelatent code directly to change the optimization target at inference time based on desired λ s and ROIs.

Original: However, we find that naïve Code Editing has limited bitrate control ability.
Revised: However, we find that naïve Code Editing has limited bitrate control ability.

Original: We further analyzeand address the problem via quantization step size adaptation, and enhance its speed-performancetrade-off.
Revised: We further analyzeand address the problem via quantization step size adaptation, and enhance its speed-performancetrade-off.

Original: Surprisingly, we find that our Code Editing can achieve not only flexible continuous bitratecontrol and ROI-based coding, but also perception-distortion trade-off, all with one decoder.
Revised: Surprisingly, we find that our Code Editing can achieve not only flexible continuous bitratecontrol and ROI-based coding, but also multi-distortion trade-off, all with one decoder.

Original: To wrap up, our contributions are as follows:
Revised: To wrap up, our contributions are as follows:

Original: • We propose Code Editing, a new paradigm for variable bitrate NIC based on semi-amortized inference.
Revised: • We propose Code Editing, a new paradigm for variable bitrate NIC based on semi-amortizedinference.

Original: It supports continuous bitrate control with a single decoder and outperforms
Revised: It supports continuous bitrate control with a single decoder and outperforms

Original: previous variable bitrate NIC methods.
Revised: previous variable bitrate NIC methods.

Original: To the best of knowledge, we are the first to consider semi-amortized inference for variable bitrtae NIC.
Revised: To the best of knowledge, we are the first to considersemi-amortized inference for variable bitrate NIC.•

Original: • We resolve the performance decay of Code Editing by combining it with adaptive quantiza- tion step size.
Revised: We resolve the performance decay of Code Editing by combining it with adaptive quantization step size.

Original: And we further improve the speed-performance trade-off of Code Editing.
Revised: And we further improve the speed-performance trade-off of Code Editing.•

Original: • We demonstrate the potential of Code Editing in flexible spatial bits-allocation and distortion- perception trade-off, which is beyond the flexibility of traditional codecs.
Revised: We demonstrate the potential of Code Editing in flexible spatial bits-allocation and multidistortion trade-off, which is beyond the flexibility of traditional codecs.

Original:  Code Editing: Flexible Neural Image Compression2.1 Background
Revised:  Code Editing: Flexible Neural Image Compression2.1 Background

Original: The general optimization procedure of NIC methods can be concluded as follows: 1) given animage x , an encoder network produces latent parameter y =
Revised: The general optimization procedure of NIC methods can be concluded as follows: 1) given animage x , an encoder (inference model) produces latent parameter y =

Original: f ϕ ( x ) .
Revised: f ϕ ( x ) .

Original: 2) y goes through unit scalarquantization (rounding) and produces discrete code
Revised: Then, we obtain⌊ y ⌉ by unit scalar quantization. Then, an entropy model (prior) p θ ( y ) parameterized by θis used to compute the probability mass function (pmf) P θ ( ⌊ y ⌉ ) = (cid:81) (cid:82)

Original: ⌊ y ⌉ .
Revised: ⌊ y i

Revised: +0 .

Original: An entropy model p θ (  ⌊ y  ⌉ ) is used toevaluate the likelihood of discrete latent code, and encode it into bitstreams.
Revised: ⌊ y i ⌉− 0 . 5 p θ ( y i ) d y i =(cid:81) ( F θ ( ⌊ y i ⌉ +0 . 5) − F

Original: During this process, weobtain the bitrate − log p θ( ⌊ y  ⌉ ) . 
Revised: θ ⌊ y i ⌉ +0 . 5)) , where F θ is the cdf of p θ . 4) With that pmf, we encode ⌊ y

Original: 4) Then ⌊ y ⌉ is feed into decoder network to obtain reconstructionimage g θ( ⌊ y ⌉ ) , and d ( ., . ) is used to compute the distortion between reconstruction and originalimage.
Revised: i ⌉ withbitrate R = − log P θ ( ⌊ y ⌉ ) by entropy coder. 5) For the decoding process, ⌊ y ⌉ is feed into decodernetwork to obtain reconstruction image ¯ x = g θ ⌊ y ⌉ ) , and d ( · , · ) is used to compute the distortionbetween reconstruction and original image.

Revised: The distortion D can be interpreted as the likelihoodlog p ( x |⌊ y ⌉ ) so long as we treat distortion as energy of Gibbs distribution [Minnen et al., 2018].

Revised: Forexample, when d ( · , · ) is pixel-wise MSE, we can interpret d ( x , ¯ x ) = − log p ( x |⌊ y ⌉ )

Revised: + constant ,where p ( x |⌊ y ⌉ )

Original: 5) the encoder parameter ϕ , decoder and entropy model parameter θ are optimized to minimizethe R-D cost R + λD , where R = − log p θ ( ⌊ y ⌉ ) and D =
Revised: = N ( ¯ x , 1 / 2 λI ) . 6) Finally, the encoder parameter ϕ , decoder and entropy modelparameter θ are optimized to minimize the R-D cost R + λD , where R = − log P θ ( ⌊ y ⌉ ) and D =

Original: d ( x , g θ ( ⌊ y ⌉ )) .
Revised: d ( x , g θ ( ⌊ y ⌉ )) .

Original: This procedure can bedescribed by Eq.
Revised: This procedure can be described by Eq.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: As the rounding operation ⌊ . ⌉ is non-differentiable, some works of NIC adopt additive uniform noise (AUN) to relax it [Ballé et al., 2017, 2018, Cheng et al. , 2020],some use straight through estimator (STE)
Revised: As the rounding operation⌊·⌉ is non-differentiable, the majority works of NIC adopt additive uniform noise (AUN) to relax it[Ballé et al., 2017, 2018, Cheng et al., 2020].

Original: [Toderici et al., 2017, Johnston et al., 2018], and someattempts annealing base approaches [Guo et al., 2021].

Original: [Equation]
Revised: [Equation]

Original: ] with y = f ϕ ( x ) (
Revised: With the AUN relaxed latent code ˜ y = y

Original: [Equation]
Revised: + U ( − 0 .

Original: Among all those methods, the most adopted one is the framework proposed by Ballé et al. It relaxes NIC  as a probabilistic graphic model ˜ y
Revised: 5 , 0 . 5) , the encoding and decoding of NIC canbe formulated as a Variational Autoencoder (VAE) [Kingma and Welling, 2013] on a probabilisticgraphic model ˜ y

Original: → x , where ˜ y are continuous relaxed latentcodes.
Revised: → x , where ˜ y are continuous relaxed latent codes.

Revised: The prior likelihood log p θ ( ˜ y )of such VAE is a continuous relaxation of log P θ ( ⌊ y ⌉ ) .

Revised: The data likelihood log p θ ( x | ˜ y )

Revised: is thedistortion offseted by a constant.

Revised: Moreover, the AUN relaxed latent code ˜

Original: Then the encoding process is connected to sampling from the variational posterior q ϕ ( ˜ y | x )=
Revised: y can be interpreted asthe reparameterized sample through a factorized uniform posterior q ϕ ( ˜ y | x )

Original: Π U
Revised: =

Original:  ( yi − 0 .
Revised: U ( y − 0 .

Original: 5 , yi + 0 . 5) .
Revised: 5 , y + 0 . 5) .

Original: This factorized uniform distribution is used to simulate the quantizationnoise.

Original: And the decoding process is connected to compute the likelihood p θ ( x | ˜ y ) , which is equivalentto distortion evaluation.

Original: Finally, the prior p θ ( ˜ y ) is connected to bitrate.

Original: Under such formulation, theevidence lower bound (ELBO) is directly connected to the negative R-D cost of compression (Eq. 3).
Revised: Under such formulation, the evidence lower bound (ELBO) is directly connected to the negative R-Dcost of compression (Eq. 3).

Original: Then, minimizing R + λD is relaxed into maximizing L trainθ,ϕ .
Revised: Then, minimizing R + λD is relaxed into maximizing L trainθ,ϕ .

Original: [Equation]
Revised: [Equation]

Original: The above process of encoding is also known as fully amortized variational inference, as for each newimage x ′ , its variational posterior is fully determined by amortized paramters ϕ .
Revised: The above process of encoding is also known as fully amortized variational inference, as for each newimage x ′ , its variational posterior is fully determined by amortized paramters ϕ .

Original: Those parametersare called amortized parameters as they are global parameters shared by all data points.
Revised: Those parametersare called amortized parameters as they are global parameters shared by all data points.

Original: The adoptionof fully amortized inference is time efficient but can also lead to inference sub-optimality.

Original: Thisphenomena is known as amortization gap [Cremer et al., 2018].

Original: 2.2 Code Editing Naïve
Revised: 2.2 Code Editing Naïve

Original: Different from previous methods in variable bitrate NIC, we propose a new paradigm of controlling
Revised: Different from previous methods in variable bitrate NIC, we propose a new paradigm of controlling

Original: R-D trade-off by semi-amortized inference, named Code Editing.
Revised: R-D trade-off by semi-amortized inference, named Code Editing.

Original: Consider now we have a pair ofparameters θ λ 0 , ϕ λ 0 optimized for a specific R-D trade-off parameter λ 0 .
Revised: Consider now we have a pair ofparameters θ λ 0 , ϕ λ 0 optimized for a specific R-D trade-off parameter λ 0 .

Original: Then, for a new Lagrangianmultiplier λ 1 , we encode by editing the code.
Revised: Then, for a new Lagrangianmultiplier λ 1 , we encode by editing the code.

Original: To be specific, given an image x , we first initialize theunquantized latent parameters y ← f θ λ 0 ( x ) .
Revised: To be specific, given an image x , we first initialize theunquantized latent parameters y ← f ϕ λ 0 ( x ) .

Original: Note that now y is exactly the same as the latent with
Revised: Note that now y is exactly the same as the latent with

Original: R-D trade-off λ 0 .
Revised: R-D trade-off λ 0 .

Original: Next, we iteratively optimize y to maximize the target L y as Eq.
Revised: Next, we iteratively optimize y to maximize the target L y as Eq.

Original: In other words,we directly edit the code y to adapt to different R-D trade-offs.
Revised: In other words,we directly edit the code y to adapt to different R-D trade-offs.

Original: During the optimization process,the decoder and entropy parameter θ λ 0 remains the same.
Revised: During the optimization process,the decoder and entropy parameter θ λ 0 remains the same.

Original: This process of encoding is known assemi-amortized inference [Kim et al., 2018], as it utilizes the initial value of amortized inference, andoptimizes latent for each data point afterwards.
Revised: This process of encoding is known assemi-amortized inference [Kim et al., 2018], as it utilizes the initial value of amortized inference, andoptimizes latent for each data point afterwards.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Similar to NIC, one challenge remains is that the rounding operation ⌊ .
Revised: Similar to NIC, one challenge remains is that the rounding operation ⌊ .

Original: ⌉ is not-differentiable.
Revised: ⌉ is not-differentiable.

Original: It is notpossible to adopt gradient based optimization approach directly.
Revised: It is notpossible to adopt gradient based optimization approach directly.

Original: Learning from Yang et al.
Revised: Learning from Yang et al.

Original: [2020],we adopt stochastic gumbel annealing (SGA) in lieu of rounding to obtain a surrogate ELBO.
Revised: [2020],we adopt stochastic gumbel annealing (SGA) in lieu of rounding to obtain a surrogate ELBO.

Original: Asshown in Sec.
Revised: Asshown in Sec.

Original: 4.2, adopting SGA here achieves better R-D performance than AUN.
Revised: 4.2, adopting SGA here achieves better R-D performance than AUN.

Original: After that, weuse standard gradient based optimization methods with temperature annealing to optimize latent y .
Revised: After that, weuse standard gradient based optimization methods with temperature annealing to optimize latent y .

Original: And we call this method Code Editing Naïve.
Revised: And we call this method Code Editing Naïve.

Original: 2.3 Code Editing Enhanced
Revised: 2.3 Code Editing Enhanced

Original: Empirically, we find that Code Editing Naive can only achieve continuous R-D trade-off within anarrow range (e.g. ± 0 .
Revised: Empirically, we find that Code Editing Naive can only achieve continuous R-D trade-off within anarrow range (e.g. ± 0 .

Original: 1 bpp).
Revised: 1 bpp).

Original: Out of this range, the R-D performance falls rapidly (See Sec. 4.2).
Revised: Out of this range, the R-D performance falls rapidly (See Sec. 4.2).

Original: One possible reason is the train-test mismatch of the entropy model p θ λ 0 ( y ) .
Revised: One possible reason is the train-test mismatch of the entropy model p θ λ 0 ( y ) .

Original: Naturally, ⌊ y ∗ ⌉grows sparser as bitrate decreases.
Revised: Naturally, ⌊ y ∗ ⌉grows sparser as bitrate decreases.

Original: And this causes an obvious gap between latent distributionoptimized for λ 

Revised: And this causes an obvious gap between latent distributionoptimized for λ

Original: and λ 1 (See Fig. 2).
Revised: and λ 1 (See Fig. 2).

Original: Thus, the entropy model fitted to latent distribution at λ 0might perform poorly in estimating latent density of other λ s. More specifically, the mismatchedbitrate is E p ( x ) [ E q ( y ∗ | x ) [log p θ λ 1 ( ⌊ y ∗ ⌉ ) − log p θ λ 0 ({+
+} ⌊ y ∗ ⌉ )]] .
Revised: Thus, the entropy model fitted to latent distribution at λ 0 mightperform poorly in estimating latent density of other λ s. More specifically, the mismatched bitrateis E p ( x ) [ E q ( y ∗ | x ) [log P θ λ 1 ( ⌊ y ∗ ⌉ ) − log P θ λ {+
+} ⌊ y ∗ ⌉ )]] .

Original: Under the assumption that the variationalposterior is the true posterior, this mismatch equal to D KL ( p θ λ 1 ( ⌊ y ∗ ⌉ )
Revised: Under the assumption that the variationalposterior is the true posterior, this mismatch equal to D KL ( P θ λ 1 ( ⌊ y ∗ ⌉ )

Original: || p θ λ 0{+
+} ( ⌊ y ∗ ⌉ )) , which is justthe distance between two distributions.
Revised: || P θ λ {+
+} ( ⌊ y ∗ ⌉ )) , which is justthe distance between two distributions.

Original: In NIC, the probability mass function (pmf) p θ ( ⌊ y ⌉ ) over quantized y is computed via differentiatingcumulative distribution function.
Revised: In NIC, the probability mass function (pmf) P θ ( ⌊ y ⌉ ) over quantized y is computed by taking thedifference of cumulative distribution function.

Original: In other words, p θ  ( ⌊ y ⌉ ) is not directly computed.
Revised: In other words, P θ λ 0 ( ⌊ y ⌉ ) =

Original: Instead, itis approximated by (cid:81) (cid:82) y i +∆ / 2{+( F θ λ 
 ⌊+} y i  − ∆ / 2 p θ  (  y i )
Revised: (cid:81) {+( F θ λ 
 ⌊+} y i ⌉ + 0 . 5) − F θ λ 0 ( ⌊ y i ⌉ − 0 . 5)) . Following Choi et al.

Original: d y i , with∆ = 1 . The ∆ is the quantization step size.{+step∆ , and this results in P θ λ 
 ( ⌊ y / ∆ ⌉ ; ∆)+}
Revised: [2019], we augmenting it with the quantization {+step∆ , and this results in P θ λ 
 ( ⌊ y / ∆ ⌉ ; ∆)+}

Original: Inoriginal NIC, the quantization operation is simply rounding, and its step size is 1 .
Revised: = (cid:81)

Original: We note thatwe could control the pmf by controlling the quantization step ∆ , and this results in an augmentedprior p θ ( ⌊ y  ⌉ , ∆) .
Revised: ( F θ λ 0 (∆ ⌊ y i / ∆ ⌉ + ∆ / 2) − F θ λ 0 (∆ ⌊ y i / ∆ ⌉ − ∆ / 2)) ,where ∆ ⌊· / ∆ ⌉ means the dequantized result of quantization with step ∆ .

Original: Controlling ∆ results in significant different pmf without changing the underlyingcontinuous density p θ  ( y ) .
Revised: Controlling ∆ results insignificantly different pmf without changing the underlying continuous density p θ λ 0 ( y ) .

Original: Then, we can optimize the ∆ augmented ELBO to find the optimal codey ∗ and quantization step size ∆ ∗ as Eq.
Revised: Then, wecan optimize the ∆ augmented ELBO to find the optimal code y ∗ and quantization step size ∆ ∗ as Eq.

Original: 6 and Eq.
Revised: 6 and Eq.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Empirically, we find that jointly optimizing y and ∆ via gradient based approaches brings significantperformance decay (See Appendix).
Revised: Empirically, we find that jointly optimizing y and ∆ via gradient based approaches brings significantperformance decay (See Appendix).

Original: Instead, we determine quantization step for each R-D trade-off by grid search (See Appendix) and optimize y via gradient descent.
Revised: Instead, we determine quantization step for each R-D trade-offby grid search (See Appendix) and optimize y via gradient descent.

Original: In fact, similar idea of controlling∆ has been adopted in Choi et al.
Revised: In fact, similar idea of controlling ∆ has been adopted in Choi et al.

Original: It implements continuous R-D trade-off within small bitraterange (e.g. ± 0.2 bpp) by controlling ∆ .
Revised: It implements continuous R-D trade-off within small bitraterange (e.g. ± 0.2 bpp) by controlling ∆ .

Original: However, we show that a large range of continuous R-Dtrade-off (e.g. 0 .
Revised: However, we show that a large range of continuous R-Dtrade-off (e.g. 0 .

Original: 1 − 1 .
Revised: 1 − 1 .

Original: 1 bpp) can be achieved by combining the quantization step size control andsemi-amortized inference.
Revised: 1 bpp) can be achieved by combining the quantization step size control andsemi-amortized inference.

Original: 2.4 Speed-Performance Trade-off
Revised: 2.4 Speed-Performance Trade-off

Original: Optimizing y over Eq.
Revised: Optimizing y over Eq.

Original: from f ϕ 0 ( x ) for each x is time-consuming.
Revised: from f ϕ 0 ( x ) for each x is time-consuming.

Original: Naturally, we can accelerate
Revised: Naturally, we can accelerate

Original: Code Editing by reducing the iteration of optimization, and the R-D performance would dropaccordingly.
Revised: Code Editing by reducing the iteration of optimization, and the R-D performance would dropaccordingly.

Original: In practice, we find that reducing the number of iterations to 10% of full convergenceachieves reasonable results for low bitrate regions (with 0.4 db PSNR drop).
Revised: In practice, we find that reducing the number of iterations to 10% of full convergenceachieves reasonable results for low bitrate regions (with 0.4 db PSNR drop).

Original: However, the R-Dperformance in the high bitrate range degrades severely.
Revised: However, the R-Dperformance in the high bitrate range degrades severely.

Original: We propose to enhance the amortizedencoder by finetuning at a high bitrate R-D objective.
Revised: We propose to enhance the amortizedencoder by finetuning at a high bitrate R-D objective.

Original: Then the finetuned encoder can predict a betterinitial y for a high bitrate range, and thus makes computationally scalable Code Editing possible.
Revised: Then the finetuned encoder can predict a betterinitial y for a high bitrate range, and thus makes computationally scalable Code Editing possible.

Original: 2.5 Extension of Code Editing
Revised: 2.5 Extension of Code Editing

Original: With code editing, we can empower NIC with more flexible coding capabilities out of continuousbitrate control.
Revised: With code editing, we can empower NIC with more flexible coding capabilities out of continuousbitrate control.

Original: By extending the optimization target for code editing, we can achieve differentcoding requirements.
Revised: By extending the optimization target for code editing, we can achieve differentcoding requirements.

Original: In this work, we explore the potential of code editing in ROI-based coding anddistortion-perception trade-off.
Revised: In this work, we explore the potential of code editing in ROI-based coding andmulti-distortion trade-off.

Original: ROI-based Coding.
Revised: ROI-based Coding.

Original: The need of ROI-based coding stems from the fact that different pixels inan image have different levels of importance.
Revised: The need of ROI-based coding stems from the fact that different pixels inan image have different levels of importance.

Original: So that spatially different bitrate should be assignedaccording to regions of interest (ROI) during compression.
Revised: So that spatially different bitrate should be assignedaccording to regions of interest (ROI) during compression.

Original: When performing ROI-based coding, aquality map m ∈ R H,W is provided to the encoder in addition to the input image x , which indicatesthe importance of each pixel.
Revised: When performing ROI-based coding, aquality map m ∈ R H,W is provided to the encoder in addition to the input image x , which indicatesthe importance of each pixel.

Original: Here we use a bounded continuous-valued quality map within [0 , 1] ,where 0 and 1 represent the least and the most important pixel respectively.
Revised: Here we use a bounded continuous-valued quality map within [0 , 1] ,where 0 and 1 represent the least and the most important pixel respectively.

Original: To make Code Editingsupport ROI-based coding, we extent the objective in Eq.
Revised: To make Code Editingsupport ROI-based coding, we extent the objective in Eq.

Original: Specifically, we weight the per-pixeldistortion with ROI map m to obtain the optimization target L ROI y , ∆ , where ◦ represents element-wiseproduct.
Revised: Specifically, we weight the per-pixeldistortion with ROI map m to obtain the optimization target L ROI y , ∆ , where ◦ represents element-wiseproduct.

Original: [Equation]
Revised: [Equation]

Original: Note that different from existing ROI-based coding works for NIC [Song et al., 2021], our approachimposes no prior on the shape of ROI during training.
Revised: Note that different from existing ROI-based coding works for NIC [Song et al., 2021], our approachimposes no prior on the shape of ROI during training.

Original: And thus our ROI control is more flexible (Seeresults in Sec. 4.5)
Revised: And thus our ROI control is more flexible (Seeresults in Sec. 4.5)

Original: Distortion-Perception Trade-off.
Revised: Multi-Distortion Trade-off.

Original: When compressing an image, sometimes we want to optimizethe fidelity (measured in MSE), sometimes we want to optimize the subjective quality (measured inperceptual loss), and sometimes we want a balanced trade-off between them.
Revised: When compressing an image, sometimes we want to optimize the MSE,sometimes we want to optimize the perceptual loss, and sometimes we want a balanced trade-offbetween them.

Original: However, the fidelityand perceptual quality are in odds to each other, and they share similar relationship as R-D trade-off[Blau and Michaeli, 2018, 2019].
Revised: However, the different distortion metrics are in odds to each other [Blau and Michaeli,2018, 2019].

Original: For conventional NIC, we can add another perception loss term in Eq.
Revised: For conventional NIC, we can add multiple distortion term in Eq.

Original: 7 to optimize θ, ϕ for a specific weights between rate,distortion and perception.
Revised: 7 to optimize θ, ϕfor a specific weights between rate, and different distortions.

Original: However, justlike R-D trade-off, distortion-perception trade-off also requires multiple, or even infinite number ofdecoders to achieve.
Revised: However, just like R-D trade-off,multi-distortion trade-off also requires multiple, or even infinite number of decoders to achieve.

Original: And it is very annoying for practical deployment.

Original: Again, our Code Editing can achieve the distortion-perception trade-off by extending the objectivein Eq.
Revised: Again, our Code Editing can achieve the multi-distortion trade-off by extending the objective in Eq.

Original: Specifically, we optimize towards R + λ d D  + λ p P , where D is the distortion measured d ( x , g θ λ 0 ( ⌊ y  ⌉ )), P is the perception divergence measure d p ( x , g θ λ 0 ( ⌊ y  ⌉ ))  and λ d , λ p are parameters controlling the trade-off.
Revised: For example, we we want to balance the distortion and perceptual quality, we optimize towards R + λ d D d + λ p D p , where D d = d d ( x , g θ λ 0 (∆ ⌊ y / ∆ ⌉ )) is MSE, D p = d p ( x , g θ λ 0 (∆ ⌊ y / ∆ ⌉ )) isthe LPIPS [Zhang et al., 2018] and λ d , λ p are parameters controlling the trade-off.

Original: The extended target is as Eq. 9.
Revised: The extendedtarget is as Eq. 9.

Original: Similarly, we can achieve this target byoptimizing y and ∆ to maximize L DP y , ∆ .[Equation] perception
Revised: Similarly, we can achieve this target by optimizing y and ∆ to maximize L MD y , ∆ .

Original: [Equation]
Revised: [Equation]

Original: Moreover, it is theoretically possible to achieve a trade-off between multiple loss metrics by extendingthe target as Eq.
Revised: Moreover, it is theoretically possible to achieve a trade-off between multiple loss metrics by extendingthe target as Eq.

Original: For example, we can let d 1 be the MSE , d
Revised: For example, we can let d 1 be the MSE , d

Original: 2 be the VGG loss, d 3 be the styleloss and d 4 be the adversial loss.
Revised: 2 be the VGG loss, d 3 be the style loss.

Original: Then, we can optimize our latent code to achieve the sophisticatedphoto-realistic effect described in Sajjadi et al.
Revised: Then, we can optimize our latent code to achieve the sophisticated photo-realistic effect described in Sajjadi et al.

Original: And it is so flexible that we do not even needto know what those loss functions are during training.
Revised: And it is so flexible that we do not even need to know what those loss functions are during training.

Original: However, we have not conducted empiricalstudy on L MUL y , ∆ as it is a much less common scenario in practice.
Revised: However, we have not conducted empirical study on L MUL y , ∆ as it is a much lesscommon scenario in practice.

Original: [Equation]
Revised: [Equation]

Original: 2.6 Hierarchical Latent Case
Revised: 2.6 Hierarchical Latent Case

Original: Most of the sota NIC methods [Minnen et al., 2018, Minnen and Singh, 2020, Cheng et al., 2020, Heet al., 2022] are based on the hierarchical latent framework proposed by Ballé et al.
Revised: Most of the sota NIC methods [Minnen et al., 2018, Minnen and Singh, 2020, Cheng et al., 2020, Heet al., 2022] are based on the hierarchical latent framework proposed by Ballé et al.

Original: [2018], whichhas graphic model ˜ z
Revised: [2018], whichhas graphic model ˜ z

Original: → ˜ y
Revised: → ˜ y

Original: → x .
Revised: → x .

Original: In this framework, ˜ y , ˜ z are relaxed latent code and x is image.
Revised: In this framework, ˜ y , ˜ z are relaxed latent code and x is image.

Original: To simplify notation, in this section we base our analysis upon 1 level latent framework by Balléet al.
Revised: To simplify notation, in this section we base our analysis upon 1 level latent framework by Balléet al.

Original: [2017] with graphic model ˜ y
Revised: [2017] with graphic model ˜ y

Original: → x .
Revised: → x .

Original: We note that all the formulas in this section can be easilyextended to the hierarchical latent framework.
Revised: We note that all the formulas in this section can be easilyextended to the hierarchical latent framework.

Revised: We also note that for hierarchical latent case, the y ’squantization stepsize ∆ y is directly optimized by gradient descent, and the z ’s quantization stepsize∆ z is optimized by grid search.

Original:  Related Works
Revised:  Related Works

Original: 3.1 Variable Bitrate and ROI-based control for NIC
Revised: 3.1 Variable Bitrate and ROI-based control for NIC

Original: The pioneers of NIC [Toderici et al., 2016, 2017, Johnston et al., 2018]use recurrent neural networksto implement progressive coding. They achieve coarse rate control by growing the number ofresiduals.
Revised: The pioneers of NIC [Toderici et al., 2016, 2017, Johnston et al., 2018] achieve coarse rate control bygrowing the number of residuals.

Original: On the other hand, Theis et al.
Revised: On the other hand, Theis et al.

Original: [2017] learns the scaling vector of latent code for eachbitrate on a pre-trained compression model to control the bitrate.
Revised: [2017] learns the scaling vector oflatent code for each bitrate to control the bitrate.

Original: However, their R-D performance issurpassed by Ballé et al.
Revised: However, their R-D performance is surpassed by Ballé et al.

Original: More recently, based on the framework by Ballé et al.
Revised: More recently, based on the framework by Ballé et al.

Original:  Choi et al.[2019], Cui et al.
Revised: [2017], Choi et al. Cui et al.

Original: [2021], Sun et al.
Revised: [2021], Sun et al.

Original: [2021] uses the one-hot encoded λ as conditionalinput to the auto-encoder where λ is sampled from uniform discrete prior with limited support duringtraining.
Revised: [2021] uses the one-hot encoded λ as conditional input to the auto-encoderwhere λ is sampled from uniform discrete prior during training.

Original: And additional efforts are required to interpolate the R-D curve for continuous rate control.
Revised: And additional efforts are requiredto interpolate the R-D curve for continuous rate control.

Original: Cui et al.
Revised: Cui et al.

Original: [2021] improves Theis et al. [2017] with asymmetrically scaled vectors trained jointly withthe auto-encoder.
Revised: [2021] improves Theis et al.[2017] with asymmetrically scaled vectors.

Original: More recently, Song et al.
Revised: More recently, Song et al.

Original: [2021] inserts Spatial Feature Transform (SFT) moduleinto the auto-encoder tomake the input map modulate the intermediate feature.
Revised: [2021] inserts Spatial Feature Transform (SFT) module into the auto-encoder to modulate the intermediate feature.

Original: However, to the bestof knowledge, we are the first to explore semi-amortized inference based variable bitrate model.
Revised: However, to thebest of knowledge, we are the first to explore semi-amortized inference based variable bitrate model.

Original: For ROI-based control, Agustsson et al.
Revised: For ROI-based control, Agustsson et al.

Original: [2019] distinguishes images into important and unimportantregions, and use a GAN to generate the unimportant regions.
Revised: [2019] distinguishes images into important and unimportantregions, and use a GAN to generate the unimportant regions.

Original: Cai et al.
Revised: Cai et al.

Original: [2020] supports bit allocationby applying different losses to ROI and non-ROI regions.
Revised: [2020] supports bit allocationby applying different losses to ROI and non-ROI regions.

Original: Song et al.
Revised: Song et al.

Original: [2021] uses ROI to modulatethe intermediate features and recover the ROI implicitly from the hyperprior, and it claims that itsupport flexible ROI control.
Revised: [2021] uses ROI to modulatethe intermediate features and recover the ROI implicitly from the hyperprior, and it claims that itsupport flexible ROI control.

Original: However, it requires complicated handcrafted ROI prior during training.
Revised: However, it requires complicated handcrafted ROI prior during training.

Original: Compared with above-mentioned methods, our Code Editing does not require any prior of ROI, nordata contains segmentation map during training.
Revised: Compared with above-mentioned methods, our Code Editing does not require any prior of ROI, nordata contains segmentation map during training.

Original: 3.2 Semi-amortized Inference for NIC
Revised: 3.2 Semi-amortized Inference for NIC

Original: The semi-amortized variational inference is concurrently invented by Kim et al.
Revised: The semi-amortized variational inference is concurrently invented by Kim et al.

Original: [2018] and Marinoet al.
Revised: [2018] and Marinoet al.

Original: The basic idea is that works following Kingma and Welling [2013] lose the goodtradition of learning variational posterior parameters per datapoint.
Revised: The basic idea is that works following Kingma and Welling [2013] lose the goodtradition of learning variational posterior parameters per datapoint.

Original: Instead they learn fully amortizedparameters for the whole dataset.
Revised: Instead they learn fully amortizedparameters for the whole dataset.

Original: And this fully amortized inference might lead to sub-optimalposterior distribution.
Revised: And this fully amortized inference might lead to sub-optimalposterior distribution.

Original: Cremer et al.
Revised: Cremer et al.

Original: [2018] refers to this phenomena as amortization gap and arguesit is a major contributor to inference sub-optimality of VAE.
Revised: [2018] refers to this phenomena as amortization gap and arguesit is a major contributor to inference sub-optimality of VAE.

Original: The semi-amortized inference is toinitialize the variational posterior’s parameter with fully amortized parameters, and conduct stochasticoptimization per data-point.
Revised: The semi-amortized inference is toinitialize the variational posterior’s parameter with fully amortized parameters, and conduct stochasticoptimization per data-point.

Original: Although Kim et al.
Revised: Although Kim et al.

Original: [2018], Marino et al.
Revised: [2018], Marino et al.

Original: [2018] are alluring to apply, they require nested iterativeoptimization within the stochastic gradient descent loop.
Revised: [2018] are alluring to apply, they require nested iterativeoptimization within the stochastic gradient descent loop.

Original: When adopting semi-amortized inferenceto NIC, Djelouah and Schroers [2019], Yang et al.
Revised: When adopting semi-amortized inferenceto NIC, Djelouah and Schroers [2019], Yang et al.

Original: [2020] simplify it into two stages: 1) training afully amortized encoder-decoder pair 2) iteratively optimize latent which is initialized by amortizedencoder.
Revised: [2020] simplify it into two stages: 1) training afully amortized encoder-decoder pair 2) iteratively optimize latent which is initialized by amortizedencoder.

Original: Although this simplification can lead to sub-optimal generative model (recall that now the generative model is paired with sub-optimal inference model), it indeed makes semi-amortizedinference practical for NIC.
Revised: Although this simplification can lead to sub-optimal generative model (recall that nowthe generative model is paired with sub-optimal inference model), it indeed makes semi-amortizedinference practical for NIC.

Original: Our method follows Yang et al.
Revised: Our method follows Yang et al.

Original: [2020] as we explore semi-amortizedinference for R-D trade-off instead of R-D performance.
Revised: [2020] as we explore semi-amortizedinference for R-D trade-off instead of R-D performance.

Original:  Experimental Results
Revised:  Experimental Results

Original: 4.1 Experimental Settings
Revised: 4.1 Experimental Settings

Original: Following He et al.
Revised: Following He et al.

Original: [2021], we train the baseline models on a subset of 8,000 images from ImageNet.
Revised: [2021], we train the baseline models on a subset of 8,000 images from ImageNet.

Original: For training, we use MSE as distortion.
Revised: For training, we use MSE as distortion.

Original: For each baseline method, we train models with 7 fixed R-Dtrade-off points, with λ ∈ { 0 . 0016 , 0 . 0032 , 0 . 0075 , 0 . 015 , 0 .
Revised: For each baseline method, we train models with 7 fixed R-Dtrade-off points, with λ ∈ { 0 . 0016 , 0 . 0032 , 0 . 0075 , 0 . 015 , 0 .

Original: This setup is alignedwith Cheng et al.
Revised: This setup is alignedwith Cheng et al.

Original: The baseline models’ bitrate ranges from 0.1 to 1.0 bpp on Kodak dataset[Kodak, 1993].
Revised: The baseline models’ bitrate ranges from 0.1 to 1.0 bpp on Kodak dataset[Kodak, 1993].

Original: All baseline models are trained using the Adam optimizer for 2000 epochs.
Revised: All baseline models are trained using the Adam optimizer for 2000 epochs.

Original: Batchsizeis set to 16 and the initial learning rate is set to 10 − 4 .
Revised: Batchsizeis set to 16 and the initial learning rate is set to 10 − 4 .

Revised: For all the results reported in the R-D curve, thebitrate is measured by actual bits of range encoder, and the reconstruction is computed by the latentcoded from the actual the range decoder.

Revised: For all visualization involves spatial bitrate distribution, thetheoretical bitrate is used.

Original: More detailed setup can be found in Appendix.
Revised: More detailed setup can be found in Appendix.

Original: 4.2 Ablation Study
Revised: 4.2 Ablation Study

Original: We set Ballé et al.
Revised: We set Ballé et al.

Original: [2018] as the baseline of the ablation study.
Revised: [2018] as the baseline of the ablation study.

Original: All experiments involving Code
Revised: All experiments involving Code

Original: Editing are based on the model trained with λ 0 = 0 . 015 .
Revised: Editing are based on the model trained with λ 0 = 0 . 015 .

Original: In ablation study, all the results are testedon Kodak dataset.
Revised: In ablation study, all the results are testedon Kodak dataset.

Original: Adaptive ∆ .
Revised: Adaptive ∆ .

Original: To reduce the prior mismatch, we propose to control the quantization step ∆ in additionto optimizing y .
Revised: To reduce the prior mismatch, we propose to control the quantization step ∆ in additionto optimizing y .

Original: To verify the effectiveness of this approach, we compare Code Editing Enhnanced(w/ adaptive ∆ ) with Code Editing Naïve (w/o adaptive ∆ ).
Revised: To verify its effectiveness, we compare Code Editing Enhnanced (w/ adaptive ∆ )with Code Editing Naïve (w/o adaptive ∆ ).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, the R-D performanceof Code Editing Naïve drops evidently when the bitrate changes out of a small range ( ± 0.1 bpp).
Revised: 1, the R-D performance of Code Editing Naïve drops evidently when the bitrate changes out of a small range ( ± 0.1 bpp).

Original: Further, we provide statistics of ⌊ y ⌉ generated by Code Editing of different R-D trade-off on the Kodak dataset in Fig 2.
Revised: Further, we providestatistics of ⌊ y ⌉ generated by Code Editing of different R-D trade-off on the Kodak dataset in Fig 2.

Original: It is shown that the distribution mismatch of Code Editing Naïve is moreevident than Code Editing Enhanced.
Revised: It is shown that the distribution mismatch of Code Editing Naïve is more evident than Code Editing Enhanced.

Original: On the other hand, we also show the result of controlling ∆ alone.
Revised: On the other hand, we also show the result of controlling ∆ alone.

Original: As Choi et al.
Revised: As Choi et al.

Original: [2019], we findthat adjusting ∆ can shift the bitrate in a limited range ( ± 0.1 bpp).
Revised: [2019],we find that adjusting ∆ can shift the bitrate in a limited range ( ± 0.1 bpp).

Original: But out of this range, the R-Dperformance drops rapidly.
Revised: But out of this range, the R-D performance drops rapidly.

Original: SGA vs. AUN.
Revised: SGA vs. AUN.

Original: In Code Editing, we adopt SGA instead of additive uniform noise (AUN) as relaxationof discrete latent.
Revised: In Code Editing, we adopt SGA instead of additive uniform noise (AUN) as relaxationof discrete latent.

Original: To verify the effect of SGA, we optimize y for 2,000 iterations using SGA and
Revised: To verify the effect of SGA, we optimize y for 2,000 iterations using SGA and

Original: AUN, respectively.
Revised: AUN, respectively.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, although Code Editing with AUN also outperforms thebaseline, the SGA’s R-D performance is significantly higher than AUN especially in high bitrateregion.
Revised: 1, although Code Editing with AUN also outperforms thebaseline, the SGA’s R-D performance is significantly higher than AUN especially in high bitrateregion.

Original: This is due to the fact that SGA closes the discretization gap
Revised: This is due to the fact that SGA closes the discretization gap

Original: [Yang et al., 2020] via soft tohard annealing.
Revised: [Yang et al., 2020] via soft tohard annealing.

Original: Encoder Fine-tuning for Fast Inference.
Revised: Encoder Fine-tuning for Fast Inference.

Original: We show that we can control the computational complexityof Code Editing by early termination.
Revised: We show that we can control the computational complexityof Code Editing by early termination.

Original: We terminate Code Editing with { 50 , 100 , 200 } iterations,which correspond to { 2 . 5% , 5% , 10% } of full iterations.
Revised: We terminate Code Editing with { 50 , 100 , 200 } iterations,which correspond to { 2 . 5% , 5% , 10% } of full iterations.

Original: The results on the Kodak dataset are shownin Fig.
Revised: The results on the Kodak dataset are shownin Fig.

Original: In low bitrate range, the R-D performance achieves a reasonable point within 10% ofiterations.
Revised: In low bitrate range, the R-D performance achieves a reasonable point within 10% ofiterations.

Original: However, the R-D performance in the high bitrate region degrades severely.
Revised: However, the R-D performance in the high bitrate region degrades severely.

Original: To solve thisproblem, we finetune encoder with λ = 0 .
Revised: To solve thisproblem, we finetune encoder with λ = 0 .

Original: The latent code predicted by the finetuned encoder isused as the initial value for Code Editing in the high bitrate range.
Revised: The latent code predicted by the finetuned encoder isused as the initial value for Code Editing in the high bitrate range.

Original: With a stronger encoder in thehigh bitrate range, we can predict better initial codes and achieve a better R-D performance with veryfew iterations (marked with Encoder FT.
Revised: And a stronger encoder achievesbetter R-D performance with very few iterations (marked with Encoder FT.

Original: in the Fig. 1).
Revised: in the Fig. 1).

Original: 4.3 Variable Rate Coding
Revised: 4.3 Variable Rate Coding

Original: First, we evaluate Code Editing Enhanced for continuous variable bitrate coding.
Revised: First, we evaluate Code Editing Enhanced for continuous variable bitrate coding.

Original: We choose threeestablished works [Ballé et al., 2018, Minnen et al., 2018, Cheng et al., 2020] as baselines.
Revised: We choose threeestablished works [Ballé et al., 2018, Minnen et al., 2018, Cheng et al., 2020] as baselines.

Original: Foreach of the baselines, we set base model’s R-D trade-off parameter λ 0 = 0 .
Revised: Foreach of the baselines, we set base model’s R-D trade-off parameter λ 0 = 0 .

Original: 015 , which results in amid-range bitrate (around 0.5 bpp).
Revised: 015 , which results in amid-range bitrate (around 0.5 bpp).

Original: For each image x to be compressed, we initialize y ← f ϕ λ 0 ( x )and optimize y for 2,000 iterations using SGA, with the learning rate of 5e-5.
Revised: For each image x to be compressed, we initialize y ← f ϕ λ 0 ( x )and optimize y for 2,000 iterations using SGA, with the learning rate of 5 × 10 − 3 .

Original: This setting is alignedwith Yang et al.
Revised: This setting isaligned with Yang et al.

Original: The ∆  is determined by grid search.
Revised: ∆ y is directly optimized by gradient descent and ∆ z is optimized bygrid search.

Original: Fig.
Revised: Fig.

Original: 3 shows the R-D performance ofvariable bitrate via Code Editing.
Revised: 3 shows the R-D performance of variable bitrate via Code Editing.

Original: It can be seen that Code Editing Enhanced can effectively achievewide range continuous R-D trade-off with fixed decoder and entropy model trained at λ 0 .
Revised: It can be seenthat Code Editing Enhanced can effectively achieve wide range continuous R-D trade-off with fixeddecoder and entropy model trained at λ 0 .

Original: [Figure]

Original: Then, we compared Code Editing Enhanced with the sota variable code-rate methods [Theis et al.
Revised: Then, we compared Code Editing Enhanced with the sota variable code-rate methods [Theis et al.

Original: ,2017, Cui et al., 2021, Song et al., 2021].
Revised: ,2017, Cui et al., 2021, Song et al., 2021].

Original: For fairness, we re-implement these methods on thebaselines with the same setting as original papers.
Revised: For fairness, we re-implement these methods on thebaselines with the same setting as original papers.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: Note that we onlyshow the results of Song et al.
Revised: Note that we only [Figure] show the results of Song et al.

Original: [2021] on Ballé et al.
Revised: [2021] on Ballé et al.

Original: [2018] as it produces NaNs on other models.
Revised: [2018] as it produces NaNs on other models.

Original: As shown in Fig 3, Our proposed Code Editing outperforms existing variable bitrate methods withall three baseline.
Revised: As shown in Fig 3, Our proposed Code Editing outperforms existing variable bitrate methods withall three baseline.

Original: Theis et al.
Revised: Theis et al.

Original: [2017] brings larger degradation in R-D performance, especially inthe high bitrate region.
Revised: [2017] brings larger degradation in R-D performance, especially inthe high bitrate region.

Original: Cui et al.
Revised: Cui et al.

Original: [2021] achieves similar R-D performance as the original fix-bitratemodel.
Revised: [2021] achieves similar R-D performance as the original fix-bitratemodel.

Original: The R-D performance of Song et al.
Revised: The R-D performance of Song et al.

Original: [2021] on Ballé et al.
Revised: [2021] on Ballé et al.

Original: [2018] exceeds the fix-bitratemodel in the low bitrate range but is inferior to Code Editing Enhanced.
Revised: [2018] exceeds the fix-bitratemodel in the low bitrate range but is inferior to Code Editing Enhanced.

Original: 4.4 Distortion-Perception Trade-off
Revised: 4.4 Multi-Distortion Trade-off

Original: Next, we explore the potential of Code Editing Enhanced in distortion-perception trade-off.
Revised: Next, we explore the potential of Code Editing Enhanced in multi-distortion trade-off.

Original: Fordistortion metric d d ( ., . ) , we choose standard MSE. For perception metric d p ( ., . ) , we select LPIPS[Zhang et al., 2018], which is widely adoptedas a metric for perceptual quality [Mentzer etal. ,2020, Bhat et al., 2021].
Revised: For experiments, we choose to balance between MSE and LPIPS [Zhang et al., 2018]. MSE is adopted tomeasure fidelity, while LPIPS is widely adopted for perceptual quality [Mentzer et al., 2020, Bhatet al.

Original: The lower the LPIPS is, the better perceptual quality an image has.
Revised: 2021].The lower the LPIPS is, the better perceptual quality an image has.

Original: Inour experiment, we select 3distortion-perception trade-off points by controlling the perception lossweight λ p ∈ { 0 . 1 , 0 . 5 , 1 . 0 } , respectively.
Revised: In our experiment,we select 3 trade-off points by controlling the LPIPS weight λ p ∈ { 0 . 1 , 0 . 5 , 1 . 0 } .

Original: We use the bitrate control scheme in Mentzer et al.
Revised: We use the bitratecontrol scheme in Mentzer et al.

Original: [2020]to ensure the bitrateroughly remains the same (See Appendix for detail).
Revised: [2020] to stabilize bitrate (See Appendix for detail).

Original: Fig 4 shows the DistortionPerception trade-off results based on Ballé et al.
Revised: Fig 4 shows themulti-distortion trade-off results based on Ballé et al.

Original: [2018] on the Kodak dataset.
Revised: [2018] on the Kodak dataset.

Original: As λ p increases, both LPIPS and PSNR decrease, implying better perception and worse distortion is achieved.
Revised: As λ p increases,both LPIPS and PSNR decrease, implying better perceptual quality and worse distortion is achieved.

Original: Further, weshow qualitative results in Fig.
Revised: Further, we show qualitative results in Fig.

Original: We can see that as λ p increases, the detail level of image increases,which indicates a better perceptual quality.
Revised: We can see that as λ p increases, the detail level ofimage increases, which indicates a better perceptual quality.

Original: Both quantitative and qualitative results show that our Code Editing can achieve flexible distortion-perception trade-off.
Revised: Both quantitative and qualitative resultsshow that our Code Editing can achieve flexible multi-distortion trade-off.

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: 4.5 ROI-based Coding

Original: This section shows the performance of Code Editing in terms of ROI-based coding.
Revised: This section shows the performance of Code Editing in terms of ROI-based coding.

Original: In order to verifythe effectiveness of spatially adaptive R-D trade-off, we use various quality maps.
Revised: In order to verifythe effectiveness of spatially adaptive R-D trade-off, we use various quality maps.

Original: We use Ballé et al.[2018] as the baseline model to verify the effectiveness of Code Editing in ROI-based coding on
Revised: We use Ballé et al.[2018] as the baseline model to verify the effectiveness of Code Editing in ROI-based coding on

Original: Kodak and CLIC2022 datasets [CLIC, 2022].
Revised: Kodak and CLIC2022 datasets [CLIC, 2022].

Original: Fig.
Revised: Fig.

Original: 6 illustrates these results.
Revised: 6 illustrates these results.

Original: And our Code Editingcan allocate bitrate spatially according to arbitrary-shape ROI maps, including checkerboard andalphabet shapes.
Revised: And our Code Editingcan allocate bitrate spatially according to arbitrary-shape ROI maps, including checkerboard andalphabet shapes.

Original: Moreover, we compared our Code Editing ROI results with Song et al.
Revised: Moreover, we compared our Code Editing ROI results with Song et al.

Original: [2021] whichis trained with sophisticated prior in Fig.
Revised: [2021] whichis trained with sophisticated prior in Fig.

Original: It can be seen that Song et al.
Revised: It can be seen that Song et al.

Original: [2021]’s result suffers from
Revised: [2021]’s result suffers from

Original: ROI map diminishing issue.
Revised: ROI map diminishing issue.

Original: We can barely sense the effect of ROI map inside the region indicated bythe white rectangle.
Revised: We can barely sense the effect of ROI map inside the region indicated bythe white rectangle.

Revised: In Appendix, we provide more ROI results on different types of masks, includingsegmentation mask.

Original:  Conclusions
Revised:  Conclusions

Original: We propose Code Editing, a new paradigm for continuous variable bitrate NIC based on semiamortized inference.
Revised: We propose Code Editing, a new paradigm for continuous variable bitrate NIC based on semiamortized inference.

Original: It outperforms current variable bitrate NIC methods.
Revised: It outperforms current variable bitrate NIC methods.

Original: Moreover, it achievesflexible spatial bits allocation and distortion-perception trade-off, all with one decoder.
Revised: Moreover, it achievesflexible spatial bits allocation and multi-distortion trade-off, all with one decoder.

Original: Limitationsinclude that the efficiency of Code Editing could be improved, and that the empirical study on morecompounded target mentioned as Eq.
Revised: Limitationsinclude that the efficiency of Code Editing could be improved, and that the empirical study on morecompounded target mentioned as Eq.

Original: 10 should be addressed.
Revised: 10 should be addressed.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .•

Original: • Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answersbelow.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes](c)
Revised: [Yes]

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] SeeAppendix(d)
Revised: [Yes] See Appendix

Original:  Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [No](b) Did you specify all the training details (e
Revised: [No] (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [No](d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes](b) Did you mention the license of the assets?
Revised: [Yes] (b) Did you mention the license of the assets?

Original: [N/A](c)
Revised: [N/A]

Original:  Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: N /A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects.

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A]
Revised: [N/A

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks

Revised: , with

Revised: links

Revised: to Institutional

Revised: Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals , if applicable?

Original: [N/A]
Revised: [N/A

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A