Original: Flexible Neural Image Compression via Code Editing
Revised: Flexible Neural Image Compression via Code Editing

Original: Abstract
Revised: Abstract

Original: Neural image compression (NIC) has outperformed traditional image codecs in ratedistortion (R-D) performance.
Revised: Neural image compression (NIC) has outperformed traditional image codecs in ratedistortion (R-D) performance.

Original: However, it usually requires a dedicated encoderdecoder pair for each point on R-D curve, which greatly hinders its practical deployment.
Revised: However, it usually requires a dedicated encoderdecoder pair for each point on R-D curve, which greatly hinders its practical deployment.

Original: While some recent works have enabled bitrate control via conditional coding, they impose strong prior during training and provide limited flexibility.
Revised: While some recent works have enabled bitrate control via conditional coding, they impose strong prior during training and provide limited flexibility.

Original: In this paper we propose Code Editing, a highly flexible coding method for NIC based on semi-amortized inference and adaptive quantization.
Revised: In this paper we propose Code Editing, a highly flexible coding method for NIC based on semi-amortized inference and adaptive quantization.

Original: Our work is a new paradigm for variable bitrate NIC, and experimental results show that our method surpasses existing variable-rate methods.
Revised: Our work is a new paradigm for variable bitrate NIC, and experimental results show that our method surpasses existing variable-rate methods.

Original: Furthermore, our approach is so flexible that it can also achieves ROI coding and multi-distortion trade-off with a single decoder.
Revised: Furthermore, our approach is so flexible that it can also achieves ROI coding and multi-distortion trade-off with a single decoder.

Original: Our approach is compatible to all NIC methods with differentiable decoder
Revised: Our approach is compatible to all NIC methods with differentiable decoder

Original: NIC, and it can be even directly adopted on existing pre-trained models.
Revised: NIC, and it can be even directly adopted on existing pre-trained models.

Original:  Introduction
Revised:  Introduction

Original: Lossy image compression is a fundamental problem of computer vision.
Revised: Lossy image compression is a fundamental problem of computer vision.

Original: In a simplified setting, the aim of lossy compression is to minimize rate-distortion cost R + λD , with λ as the Lagrange multiplier controlling R-D trade-off.
Revised: In a simplified setting, the aim of lossy compression is to minimize rate-distortion cost R + λD , with λ as the Lagrange multiplier controlling R-D trade-off.

Original: Traditional compression algorithms (e g. JPEG, JPEG2000 and
Revised: Traditional compression algorithms (e g. JPEG, JPEG2000 and

Original: BPG [Bellard, 2018]) first perform a linear transformation (e g. DCT, wavelet transform) and then quantize the transformed coefficients for compression.
Revised: BPG [Bellard, 2018]) first perform a linear transformation (e g. DCT, wavelet transform) and then quantize the transformed coefficients for compression.

Original: Their bitrate control is achieved by controlling quantization stepsize.
Revised: Their bitrate control is achieved by controlling quantization stepsize.

Original: On the other hand, lossy neural image compression (NIC) methods have outperformed traditional codecs in recent years [Xie et al., 2021, He et al., 2022].
Revised: On the other hand, lossy neural image compression (NIC) methods have outperformed traditional codecs in recent years [Xie et al., 2021, He et al., 2022].

Original: These methods implement powerful non-linear transformation with a neural network and therefore achieve better
Revised: These methods implement powerful non-linear transformation with a neural network and therefore achieve better

Original: R-D performance.
Revised: R-D performance.

Original: However, in order to realize R-D trade-offs, we need to optimize multiple networks towards different losses with different λ s, which means that we cannot achieve flexible bitrate control.
Revised: However, in order to realize R-D trade-offs, we need to optimize multiple networks towards different losses with different λ s, which means that we cannot achieve flexible bitrate control.

Original: To enable NIC models with bitrate control ability, several works adopt conditional networks [Choi et al., 2019, Cui et al., 2021, Sun et al., 2021] .
Revised: To enable NIC models with bitrate control ability, several works adopt conditional networks [Choi et al., 2019, Cui et al., 2021, Sun et al., 2021] .

Original: They use λ as condition and feed it into the encoder and decoder.
Revised: They use λ as condition and feed it into the encoder and decoder.

Original: During training, they sample λ from a pre-defined prior and optimize the E p ( λ ) [ R + λD ] via Monte Carlo.
Revised: During training, they sample λ from a pre-defined prior and optimize the E p ( λ ) [ R + λD ] via Monte Carlo.

Original: Due to the capacity of conditional networks, most of them use simple prior and limited sets of λ , such as a uniform distribution over only 5 − 10 discrete λ s
Revised: Due to the capacity of conditional networks, most of them use simple prior and limited sets of λ , such as a uniform distribution over only 5 − 10 discrete λ s

Original: [Choi et al., 2019, Cui et al., 2021].
Revised: [Choi et al., 2019, Cui et al., 2021].

Original: And additional effort is required to interpolate the discrete λ s to achieve continuous
Revised: And additional effort is required to interpolate the discrete λ s to achieve continuous

Original: R-D curve [Choi et al., 2019, Cui et al., 2021, Sun et al., 2021], which limits the flexibility of variable bitrate model.
Revised: R-D curve [Choi et al., 2019, Cui et al., 2021, Sun et al., 2021], which limits the flexibility of variable bitrate model.

Original: On the other hand, traditional image codecs achieve flexible R-D trade-off naturally by controlling the quantization step.
Revised: On the other hand, traditional image codecs achieve flexible R-D trade-off naturally by controlling the quantization step.

Original: This control is fine grain (e.g. 100 levels for JPEG).
Revised: This control is fine grain (e.g. 100 levels for JPEG).

Original: Moreover, they support flexible spatial bits-allocation for ROI-based (region of interest) coding.
Revised: Moreover, they support flexible spatial bits-allocation for ROI-based (region of interest) coding.

Original: And the user need only one decoder for all those flexibility.
Revised: And the user need only one decoder for all those flexibility.

Original: In this work, we intend to give neural image compression methods the flexibility of traditional codecs: the continuous rate-distortion trade-off with arbitrary spatial bits-allocation by one decoder.
Revised: In this work, we intend to give neural image compression methods the flexibility of traditional codecs: the continuous rate-distortion trade-off with arbitrary spatial bits-allocation by one decoder.

Original: It is a meaningful task for practical NIC considering the growing size of neural decoders and the need for fine-grain rate control.
Revised: It is a meaningful task for practical NIC considering the growing size of neural decoders and the need for fine-grain rate control.

Original: In order to achieve this, we propose Code Editing, a semi-amortized inference [Kim et al., 2018] based approach for flexible R-D trade-off.
Revised: In order to achieve this, we propose Code Editing, a semi-amortized inference [Kim et al., 2018] based approach for flexible R-D trade-off.

Original: Although previous works have adopted semi-amortized inference to improve R-D performance [Johnston et al., 2018, Yang et al., 2020], we are the first to discover its potential for flexible bitrate control.
Revised: Although previous works have adopted semi-amortized inference to improve R-D performance [Johnston et al., 2018, Yang et al., 2020], we are the first to discover its potential for flexible bitrate control.

Original: Specifically, we edit the latent code directly to change the optimization target at inference time based on desired λ s and ROIs.
Revised: Specifically, we edit the latent code directly to change the optimization target at inference time based on desired λ s and ROIs.

Original: However, we find that naïve Code Editing has limited bitrate control ability.
Revised: However, we find that naïve Code Editing has limited bitrate control ability.

Original: We further analyze and address the problem via quantization step size adaptation, and enhance its speed-performance trade-off.
Revised: We further analyze and address the problem via quantization step size adaptation, and enhance its speed-performance trade-off.

Original: Surprisingly, we find that our Code Editing can achieve not only flexible continuous bitrate control and ROI-based coding, but also multi-distortion trade-off, all with one decoder.
Revised: Surprisingly, we find that our Code Editing can achieve not only flexible continuous bitrate control and ROI-based coding, but also multi-distortion trade-off, all with one decoder.

Original: To wrap up, our contributions are as follows:
Revised: To wrap up, our contributions are as follows:

Original: • We propose Code Editing, a new paradigm for variable bitrate NIC based on semi-amortized inference.
Revised: • We propose Code Editing, a new paradigm for variable bitrate NIC based on semi-amortized inference.

Original: It supports continuous bitrate control with a single decoder and outperforms previous variable bitrate NIC methods.
Revised: It supports continuous bitrate control with a single decoder and outperforms previous variable bitrate NIC methods.

Original: To the best of knowledge, we are the first to consider semi-amortized inference for variable bitrate NIC.
Revised: To the best of knowledge, we are the first to consider semi-amortized inference for variable bitrate NIC.

Original: • We resolve the performance decay of Code Editing by combining it with adaptive quantization step size.
Revised: • We resolve the performance decay of Code Editing by combining it with adaptive quantization step size.

Original: And we further improve the speed-performance trade-off of Code Editing.
Revised: And we further improve the speed-performance trade-off of Code Editing.

Original: • We demonstrate the potential of Code Editing in flexible spatial bits-allocation and multidistortion trade-off, which is beyond the flexibility of traditional codecs.
Revised: • We demonstrate the potential of Code Editing in flexible spatial bits-allocation and multidistortion trade-off, which is beyond the flexibility of traditional codecs.

Original:  Code Editing: Flexible Neural Image Compression 2.1 Background
Revised:  Code Editing: Flexible Neural Image Compression 2.1 Background

Original: The general optimization procedure of NIC methods can be concluded as follows: 1) given an image x , an encoder (inference model) produces latent parameter y =
Revised: The general optimization procedure of NIC methods can be concluded as follows: 1) given an image x , an encoder (inference model) produces latent parameter y =

Original: f ϕ ( x ) .
Revised: f ϕ ( x ) .

Original: Then, we obtain ⌊ y ⌉ by unit scalar quantization.
Revised: Then, we obtain ⌊ y ⌉ by unit scalar quantization.

Original: Then, an entropy model (prior) p θ ( y ) parameterized by θ is used to compute the probability mass function (pmf) P θ ( ⌊ y ⌉ )
Revised: Then, an entropy model (prior) p θ ( y ) parameterized by θ is used to compute the probability mass function (pmf) P θ ( ⌊ y ⌉ )

Original: = (cid:81) (cid:82)
Revised: = (cid:81) (cid:82)

Original: ⌊ y ( i ) ⌉
Revised: ⌊ y ( i ) ⌉

Original: +0 .
Revised: +0 .

Original: ⌊ y ( i ) ⌉− 0 .
Revised: ⌊ y ( i ) ⌉− 0 .

Original: 5 p θ ( y ( i ) )
Revised: 5 p θ ( y ( i ) )

Original: d y ( i ) =
Revised: d y ( i ) =

Original: (cid:81)
Revised: (cid:81)

Original: ( F θ ( ⌊ y ( i ) ⌉
Revised: ( F θ ( ⌊ y ( i ) ⌉

Original: + 0 . 5)
Revised: + 0 . 5)

Original: − F θ ( ⌊ y ( i ) ⌉
Revised: − F θ ( ⌊ y ( i ) ⌉

Original: + 0 . 5)) , where F θ is the cdf of p θ . 4)
Revised: + 0 . 5)) , where F θ is the cdf of p θ . 4)

Original: With that pmf, we encode
Revised: With that pmf, we encode

Original: ⌊ y ⌉ with bitrate
Revised: ⌊ y ⌉ with bitrate

Original: R = − log P θ ( ⌊ y ⌉ ) by entropy coder. 5)
Revised: R = − log P θ ( ⌊ y ⌉ ) by entropy coder. 5)

Original: For the decoding process, ⌊ y ⌉ is feed into decoder network to obtain reconstruction image ¯ x = g θ
Revised: For the decoding process, ⌊ y ⌉ is feed into decoder network to obtain reconstruction image ¯ x = g θ

Original: ⌊ y ⌉ ) , and d ( · , · ) is used to compute the distortion between reconstruction and original image.
Revised: ⌊ y ⌉ ) , and d ( · , · ) is used to compute the distortion between reconstruction and original image.

Original: The distortion D can be interpreted as the likelihood log p ( x |⌊ y ⌉ ) so long as we treat distortion as energy of Gibbs distribution [Minnen et al., 2018].
Revised: The distortion D can be interpreted as the likelihood log p ( x |⌊ y ⌉ ) so long as we treat distortion as energy of Gibbs distribution [Minnen et al., 2018].

Original: For example, when d ( · , · ) is pixel-wise MSE, we can interpret d ( x , ¯ x ) = − log p ( x |⌊
Revised: For example, when d ( · , · ) is pixel-wise MSE, we can interpret d ( x , ¯ x ) = − log p ( x |⌊

Original: y
Revised: y

Original: + constant , where p ( x |⌊ y ⌉ )
Revised: + constant , where p ( x |⌊ y ⌉ )

Original: =
Revised: =

Original: N ( ¯ x , 1 / 2 λI ) .
Revised: N ( ¯ x , 1 / 2 λI ) .

Original: 6) Finally, the encoder parameter ϕ , decoder and entropy model parameter θ are optimized to minimize the R-D cost R + λD , where R = − log P θ ( ⌊ y ⌉ ) and
Revised: 6) Finally, the encoder parameter ϕ , decoder and entropy model parameter θ are optimized to minimize the R-D cost R + λD , where R = − log P θ ( ⌊ y ⌉ ) and

Original: D =
Revised: D =

Original: d ( x , g θ ( ⌊ y ⌉ )) .
Revised: d ( x , g θ ( ⌊ y ⌉ )) .

Original: This procedure can be described by Eq.
Revised: This procedure can be described by Eq.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: As the rounding operation ⌊·⌉ is non-differentiable, the majority works of NIC adopt additive uniform noise (AUN) to relax it [Ballé et al., 2017, 2018, Cheng et al., 2020].
Revised: As the rounding operation ⌊·⌉ is non-differentiable, the majority works of NIC adopt additive uniform noise (AUN) to relax it [Ballé et al., 2017, 2018, Cheng et al., 2020].

Original: [Equation]
Revised: [Equation]

Original: With the AUN relaxed latent code ˜ y
Revised: With the AUN relaxed latent code ˜ y

Original: =
Revised: =

Original: y
Revised: y

Original: + U ( − 0 .
Revised: + U ( − 0 .

Original: 5 , 0 . 5) , the encoding and decoding of NIC can be formulated as a Variational Autoencoder (VAE)
Revised: 5 , 0 . 5) , the encoding and decoding of NIC can be formulated as a Variational Autoencoder (VAE)

Original: [Kingma and Welling, 2013] on a probabilistic graphic model ˜ y
Revised: [Kingma and Welling, 2013] on a probabilistic graphic model ˜ y

Original: → x , where ˜ y are continuous relaxed latent codes.
Revised: → x , where ˜ y are continuous relaxed latent codes.

Original: The prior likelihood log p θ ( ˜ y ) of such VAE is a continuous relaxation of log P θ ( ⌊ y ⌉ ) .
Revised: The prior likelihood log p θ ( ˜ y ) of such VAE is a continuous relaxation of log P θ ( ⌊ y ⌉ ) .

Original: The data likelihood log p θ ( x | ˜ y )
Revised: The data likelihood log p θ ( x | ˜ y )

Original: is the distortion offseted by a constant.
Revised: is the distortion offseted by a constant.

Original: Moreover, the AUN relaxed latent code ˜
Revised: Moreover, the AUN relaxed latent code ˜

Original: y can be interpreted as the reparameterized sample through a factorized uniform posterior q ϕ
Revised: y can be interpreted as the reparameterized sample through a factorized uniform posterior q ϕ

Original: ( ˜ y
Revised: ( ˜ y

Original: | x )
Revised: | x )

Original: =
Revised: =

Original: U ( y − 0 .
Revised: U ( y − 0 .

Original: 5 , y + 0 . 5) .
Revised: 5 , y + 0 . 5) .

Original: Under such formulation, the evidence lower bound (ELBO) is directly connected to the negative R-D cost of compression (Eq. 3).
Revised: Under such formulation, the evidence lower bound (ELBO) is directly connected to the negative R-D cost of compression (Eq. 3).

Original: Then, minimizing R + λD is relaxed into maximizing L trainθ,ϕ .
Revised: Then, minimizing R + λD is relaxed into maximizing L trainθ,ϕ .

Original: [Equation]
Revised: [Equation]

Original: trainθ
Revised: trainθ

Original: [Equation]
Revised: [Equation]

Original: The above process of encoding is also known as fully amortized variational inference, as for each new image x ′ , its variational posterior is fully determined by amortized paramters ϕ .
Revised: The above process of encoding is also known as fully amortized variational inference, as for each new image x ′ , its variational posterior is fully determined by amortized paramters ϕ .

Original: Those parameters are called amortized parameters as they are global parameters shared by all data points.
Revised: Those parameters are called amortized parameters as they are global parameters shared by all data points.

Original: 2.2 Code Editing Naïve
Revised: 2.2 Code Editing Naïve

Original: Different from previous methods in variable bitrate NIC, we propose a new paradigm of controlling
Revised: Different from previous methods in variable bitrate NIC, we propose a new paradigm of controlling

Original: R-D trade-off by semi-amortized inference, named Code Editing.
Revised: R-D trade-off by semi-amortized inference, named Code Editing.

Original: Consider now we have a pair of parameters θ λ 0 , ϕ λ 0 optimized for a specific R-D trade-off parameter λ 0 .
Revised: Consider now we have a pair of parameters θ λ 0 , ϕ λ 0 optimized for a specific R-D trade-off parameter λ 0 .

Original: Then, for a new Lagrangian multiplier λ 1 , we encode by editing the code.
Revised: Then, for a new Lagrangian multiplier λ 1 , we encode by editing the code.

Original: To be specific, given an image x , we first initialize the unquantized latent parameters y ← f ϕ λ 0 ( x ) .
Revised: To be specific, given an image x , we first initialize the unquantized latent parameters y ← f ϕ λ 0 ( x ) .

Original: Note that now y is exactly the same as the latent with
Revised: Note that now y is exactly the same as the latent with

Original: R-D trade-off λ 0 .
Revised: R-D trade-off λ 0 .

Original: Next, we iteratively optimize y to maximize the target L y as Eq.
Revised: Next, we iteratively optimize y to maximize the target L y as Eq.

Original: In other words, we directly edit the code y to adapt to different R-D trade-offs.
Revised: In other words, we directly edit the code y to adapt to different R-D trade-offs.

Original: During the optimization process, the decoder and entropy parameter θ λ 0 remains the same.
Revised: During the optimization process, the decoder and entropy parameter θ λ 0 remains the same.

Original: This process of encoding is known as semi-amortized inference [Kim et al., 2018], as it utilizes the initial value of amortized inference, and optimizes latent for each data point afterwards.
Revised: This process of encoding is known as semi-amortized inference [Kim et al., 2018], as it utilizes the initial value of amortized inference, and optimizes latent for each data point afterwards.

Original: [Equation]
Revised: [Equation]

Original: Similar to NIC, one challenge remains is that the rounding operation ⌊ .
Revised: Similar to NIC, one challenge remains is that the rounding operation ⌊ .

Original: ⌉ is not-differentiable.
Revised: ⌉ is not-differentiable.

Original: It is not possible to adopt gradient based optimization approach directly.
Revised: It is not possible to adopt gradient based optimization approach directly.

Original: Learning from Yang et al.
Revised: Learning from Yang et al.

Original: [2020], we adopt stochastic gumbel annealing (SGA) in lieu of rounding to obtain a surrogate ELBO.
Revised: [2020], we adopt stochastic gumbel annealing (SGA) in lieu of rounding to obtain a surrogate ELBO.

Original: As shown in Sec.
Revised: As shown in Sec.

Original: 4.2, adopting SGA here achieves better R-D performance than AUN.
Revised: 4.2, adopting SGA here achieves better R-D performance than AUN.

Original: After that, we use standard gradient based optimization methods with temperature annealing to optimize latent y .
Revised: After that, we use standard gradient based optimization methods with temperature annealing to optimize latent y .

Original: And we call this method Code Editing Naïve.
Revised: And we call this method Code Editing Naïve.

Original: 2.3 Code Editing Enhanced
Revised: 2.3 Code Editing Enhanced

Original: Empirically, we find that Code Editing Naive can only achieve continuous R-D trade-off within a narrow range (e.g. ± 0 .
Revised: Empirically, we find that Code Editing Naive can only achieve continuous R-D trade-off within a narrow range (e.g. ± 0 .

Original: 1 bpp).
Revised: 1 bpp).

Original: Out of this range, the R-D performance falls rapidly (See Sec. 4.2).
Revised: Out of this range, the R-D performance falls rapidly (See Sec. 4.2).

Original: One possible reason is the train-test mismatch of the entropy model p θ λ 0 ( y ) .
Revised: One possible reason is the train-test mismatch of the entropy model p θ λ 0 ( y ) .

Original: Naturally, ⌊ y ∗ ⌉ grows sparser as bitrate decreases.
Revised: Naturally, ⌊ y ∗ ⌉ grows sparser as bitrate decreases.

Original: And this causes an obvious gap between latent distribution optimized for λ 0 and λ 1 (See Fig. 2).
Revised: And this causes an obvious gap between latent distribution optimized for λ 0 and λ 1 (See Fig. 2).

Original: Thus, the entropy model fitted to latent distribution at λ 0 might perform poorly in estimating latent density of other λ s. More specifically, the mismatched bitrate is E p ( x ) [ E q ( y ∗ | x )
Revised: Thus, the entropy model fitted to latent distribution at λ 0 might perform poorly in estimating latent density of other λ s. More specifically, the mismatched bitrate is E p ( x ) [ E q ( y ∗ | x )

Original: [log P θ λ 1 ( ⌊ y ∗ ⌉ ) − log P θ λ 0 ( ⌊ y ∗ ⌉ )]] .
Revised: [log P θ λ 1 ( ⌊ y ∗ ⌉ ) − log P θ λ 0 ( ⌊ y ∗ ⌉ )]] .

Original: Under the assumption that the variational posterior is the true posterior, this mismatch equal to D KL ( P θ λ 1 ( ⌊ y ∗ ⌉ )
Revised: Under the assumption that the variational posterior is the true posterior, this mismatch equal to D KL ( P θ λ 1 ( ⌊ y ∗ ⌉ )

Original: || P θ λ 

Revised: || P θ λ

Original: ( ⌊ y ∗ ⌉ )) , which is just the distance between two distributions.
Revised: ( ⌊ y ∗ ⌉ )) , which is just the distance between two distributions.

Original: In NIC, the probability mass function (pmf) P θ ( ⌊ y ⌉ ) over quantized y is computed by taking the difference of cumulative distribution function.
Revised: In NIC, the probability mass function (pmf) P θ ( ⌊ y ⌉ ) over quantized y is computed by taking the difference of cumulative distribution function.

Original: In other words, P θ λ 0 ( ⌊ y ⌉ ) =
Revised: In other words, P θ λ 0 ( ⌊ y ⌉ ) =

Original: (cid:81) ( F θ λ 

Revised: (cid:81) ( F θ λ

Original: ⌊ y ( i ) ⌉
Revised: ⌊ y ( i ) ⌉

Original: +0 . 5) −
Revised: +0 . 5) −

Original: F θ λ 0 ( ⌊ y ( i ) ⌉ − 0 . 5)) .
Revised: F θ λ 0 ( ⌊ y ( i ) ⌉ − 0 . 5)) .

Original: Following Choi et al.
Revised: Following Choi et al.

Original: [2019], we augmenting it with the quantization step ∆ , and this results in P θ λ
Revised: [2019], we augmenting it with the quantization step ∆ , and this results in P θ λ

Original: [Equation]
Revised: [Equation]

Original: ( ⌊ y /
Revised: ( ⌊ y /

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) , where ∆ ⌊· / ∆ ⌉ means the dequantized result of quantization with step ∆ .
Revised: ) , where ∆ ⌊· / ∆ ⌉ means the dequantized result of quantization with step ∆ .

Original: Controlling ∆ results in significantly different pmf without changing the underlying continuous density p θ λ 0 ( y ) .
Revised: Controlling ∆ results in significantly different pmf without changing the underlying continuous density p θ λ 0 ( y ) .

Original: Then, we can optimize the ∆ augmented ELBO to find the optimal code y ∗ and quantization step size ∆ ∗ as
Revised: Then, we can optimize the ∆ augmented ELBO to find the optimal code y ∗ and quantization step size ∆ ∗ as

Original: Eq.
Revised: Eq.

Original: 6 and Eq.
Revised: 6 and Eq.

Original: [Equation]
Revised: [Equation]

Original: In fact, similar idea of controlling ∆ has been adopted in Choi et al.
Revised: In fact, similar idea of controlling ∆ has been adopted in Choi et al.

Original: It implements continuous
Revised: It implements continuous

Original: R-D trade-off within small bitrate range (e.g. ± 0.2 bpp) by controlling ∆ .
Revised: R-D trade-off within small bitrate range (e.g. ± 0.2 bpp) by controlling ∆ .

Original: However, we show that a large range of continuous R-D trade-off (e.g. 0 .
Revised: However, we show that a large range of continuous R-D trade-off (e.g. 0 .

Original: 1 − 1 .
Revised: 1 − 1 .

Original: 1 bpp) can be achieved by combining the quantization step size control and semi-amortized inference.
Revised: 1 bpp) can be achieved by combining the quantization step size control and semi-amortized inference.

Original: 2.4 Speed-Performance Trade-off
Revised: 2.4 Speed-Performance Trade-off

Original: Optimizing y over Eq.
Revised: Optimizing y over Eq.

Original: from f ϕ 0 ( x ) for each x is time-consuming.
Revised: from f ϕ 0 ( x ) for each x is time-consuming.

Original: Naturally, we can accelerate
Revised: Naturally, we can accelerate

Original: Code Editing by reducing the iteration of optimization, and the R-D performance would drop accordingly.
Revised: Code Editing by reducing the iteration of optimization, and the R-D performance would drop accordingly.

Original: In practice, we find that reducing the number of iterations to 10% of full convergence achieves reasonable results for low bitrate regions (with 0.4 db PSNR drop).
Revised: In practice, we find that reducing the number of iterations to 10% of full convergence achieves reasonable results for low bitrate regions (with 0.4 db PSNR drop).

Original: However, the R-D performance in the high bitrate range degrades severely.
Revised: However, the R-D performance in the high bitrate range degrades severely.

Original: We propose to enhance the amortized encoder by finetuning at a high bitrate R-D objective.
Revised: We propose to enhance the amortized encoder by finetuning at a high bitrate R-D objective.

Original: Then the finetuned encoder can predict a better initial y for a high bitrate range, and thus makes computationally scalable Code Editing possible.
Revised: Then the finetuned encoder can predict a better initial y for a high bitrate range, and thus makes computationally scalable Code Editing possible.

Original: 2.5 Extension of Code Editing
Revised: 2.5 Extension of Code Editing

Original: With code editing, we can empower NIC with more flexible coding capabilities out of continuous bitrate control.
Revised: With code editing, we can empower NIC with more flexible coding capabilities out of continuous bitrate control.

Original: By extending the optimization target for code editing, we can achieve different coding requirements.
Revised: By extending the optimization target for code editing, we can achieve different coding requirements.

Original: In this work, we explore the potential of code editing in ROI-based coding and multi-distortion trade-off.
Revised: In this work, we explore the potential of code editing in ROI-based coding and multi-distortion trade-off.

Original: ROI-based Coding.
Revised: ROI-based Coding.

Original: The need of ROI-based coding stems from the fact that different pixels in an image have different levels of importance.
Revised: The need of ROI-based coding stems from the fact that different pixels in an image have different levels of importance.

Original: So that spatially different bitrate should be assigned according to regions of interest (ROI) during compression.
Revised: So that spatially different bitrate should be assigned according to regions of interest (ROI) during compression.

Original: When performing ROI-based coding, a quality map m ∈ R H,W is provided to the encoder in addition to the input image x , which indicates the importance of each pixel.
Revised: When performing ROI-based coding, a quality map m ∈ R H,W is provided to the encoder in addition to the input image x , which indicates the importance of each pixel.

Original: Here we use a bounded continuous-valued quality map within [0 , 1] , where 0 and 1 represent the least and the most important pixel respectively.
Revised: Here we use a bounded continuous-valued quality map within [0 , 1] , where 0 and 1 represent the least and the most important pixel respectively.

Original: To make Code Editing support ROI-based coding, we extent the objective in Eq.
Revised: To make Code Editing support ROI-based coding, we extent the objective in Eq.

Original: Specifically, we weight the per-pixel distortion with ROI map m to obtain the optimization target L ROI y , ∆ , where ◦ represents element-wise product.
Revised: Specifically, we weight the per-pixel distortion with ROI map m to obtain the optimization target L ROI y , ∆ , where ◦ represents element-wise product.

Original: [Equation]
Revised: [Equation]

Original: Note that different from existing ROI-based coding works for NIC [Song et al., 2021], our approach imposes no prior on the shape of ROI during training.
Revised: Note that different from existing ROI-based coding works for NIC [Song et al., 2021], our approach imposes no prior on the shape of ROI during training.

Original: And thus our ROI control is more flexible (See results in Sec. 4.5)
Revised: And thus our ROI control is more flexible (See results in Sec. 4.5)

Original: Multi-Distortion Trade-off.
Revised: Multi-Distortion Trade-off.

Original: When compressing an image, sometimes we want to optimize the MSE, sometimes we want to optimize the perceptual loss, and sometimes we want a balanced trade-off between them.
Revised: When compressing an image, sometimes we want to optimize the MSE, sometimes we want to optimize the perceptual loss, and sometimes we want a balanced trade-off between them.

Original: However, the different distortion metrics are in odds to each other [Blau and Michaeli, 2018, 2019].
Revised: However, the different distortion metrics are in odds to each other [Blau and Michaeli, 2018, 2019].

Original: For conventional NIC, we can add multiple distortion term in Eq.
Revised: For conventional NIC, we can add multiple distortion term in Eq.

Original: 7 to optimize θ, ϕ for a specific weights between rate, and different distortions.
Revised: 7 to optimize θ, ϕ for a specific weights between rate, and different distortions.

Original: However, just like R-D trade-off, multi-distortion trade-off also requires multiple, or even infinite number of decoders to achieve.
Revised: However, just like R-D trade-off, multi-distortion trade-off also requires multiple, or even infinite number of decoders to achieve.

Original: Again, our Code Editing can achieve the multi-distortion trade-off by extending the objective in
Revised: Again, our Code Editing can achieve the multi-distortion trade-off by extending the objective in

Original: Eq.
Revised: Eq.

Original: For example, we we want to balance the distortion and perceptual quality, we optimize towards
Revised: For example, we we want to balance the distortion and perceptual quality, we optimize towards

Original: R + λ d D d + λ p D p , where D d = d d ( x , g θ λ 0 (∆ ⌊ y / ∆ ⌉ )) is MSE, D p
Revised: R + λ d D d + λ p D p , where D d = d d ( x , g θ λ 0 (∆ ⌊ y / ∆ ⌉ )) is MSE, D p

Original: = d p ( x , g θ λ 0 (∆ ⌊ y / ∆ ⌉ )) is the LPIPS [Zhang et al., 2018] and λ d , λ p are parameters controlling the trade-off.
Revised: = d p ( x , g θ λ 0 (∆ ⌊ y / ∆ ⌉ )) is the LPIPS [Zhang et al., 2018] and λ d , λ p are parameters controlling the trade-off.

Original: The extended target is as Eq. 9.
Revised: The extended target is as Eq. 9.

Original: Similarly, we can achieve this target by optimizing y and ∆ to maximize L MD y , ∆ .
Revised: Similarly, we can achieve this target by optimizing y and ∆ to maximize L MD y , ∆ .

Original: [Equation]
Revised: [Equation]

Original: Moreover, it is theoretically possible to achieve a trade-off between multiple loss metrics by extending the target as Eq.
Revised: Moreover, it is theoretically possible to achieve a trade-off between multiple loss metrics by extending the target as Eq.

Original: For example, we can let d 1 be the MSE , d
Revised: For example, we can let d 1 be the MSE , d

Original: 2 be the VGG loss, d 3 be the style loss.
Revised: 2 be the VGG loss, d 3 be the style loss.

Original: Then, we can optimize our latent code to achieve the sophisticated photo-realistic effect described in
Revised: Then, we can optimize our latent code to achieve the sophisticated photo-realistic effect described in

Original: Sajjadi et al.
Revised: Sajjadi et al.

Original: And it is so flexible that we do not even need to know what those loss functions are during training.
Revised: And it is so flexible that we do not even need to know what those loss functions are during training.

Original: However, we have not conducted empirical study on L MUL y , ∆ as it is a much less common scenario in practice.
Revised: However, we have not conducted empirical study on L MUL y , ∆ as it is a much less common scenario in practice.

Original: [Equation]
Revised: [Equation]

Original: 2.6 Hierarchical Latent Case
Revised: 2.6 Hierarchical Latent Case

Original: Most of the sota NIC methods [Minnen et al., 2018, Minnen and Singh, 2020, Cheng et al., 2020, He et al., 2022] are based on the hierarchical latent framework proposed by Ballé et al.
Revised: Most of the sota NIC methods [Minnen et al., 2018, Minnen and Singh, 2020, Cheng et al., 2020, He et al., 2022] are based on the hierarchical latent framework proposed by Ballé et al.

Original: [2018], which has graphic model ˜ z
Revised: [2018], which has graphic model ˜ z

Original: → ˜ y
Revised: → ˜ y

Original: → x .
Revised: → x .

Original: In this framework, ˜ y , ˜ z are relaxed latent code and x is image.
Revised: In this framework, ˜ y , ˜ z are relaxed latent code and x is image.

Original: To simplify notation, in this section we base our analysis upon 1 level latent framework by Ballé et al.
Revised: To simplify notation, in this section we base our analysis upon 1 level latent framework by Ballé et al.

Original: [2017] with graphic model ˜ y
Revised: [2017] with graphic model ˜ y

Original: → x .
Revised: → x .

Original: We note that all the formulas in this section can be easily extended to the hierarchical latent framework.
Revised: We note that all the formulas in this section can be easily extended to the hierarchical latent framework.

Original: We also note that for hierarchical latent case, the y ’s quantization stepsize ∆ y is directly optimized by gradient descent, and the z ’s quantization stepsize ∆ z is optimized by grid search (See Appendix A.2 and A.3).
Revised: We also note that for hierarchical latent case, the y ’s quantization stepsize ∆ y is directly optimized by gradient descent, and the z ’s quantization stepsize ∆ z is optimized by grid search (See Appendix A.2 and A.3).

Original:  Related Works 3.1 Variable Bitrate and ROI-based control for NIC
Revised:  Related Works 3.1 Variable Bitrate and ROI-based control for NIC

Original: The pioneers of NIC [Toderici et al., 2016, 2017, Johnston et al., 2018] achieve coarse rate control by growing the number of residuals.
Revised: The pioneers of NIC [Toderici et al., 2016, 2017, Johnston et al., 2018] achieve coarse rate control by growing the number of residuals.

Original: On the other hand, Theis et al.
Revised: On the other hand, Theis et al.

Original: [2017] learns the scaling vector of latent code for each bitrate to control the bitrate.
Revised: [2017] learns the scaling vector of latent code for each bitrate to control the bitrate.

Original: However, their R-D performance is surpassed by
Revised: However, their R-D performance is surpassed by

Original: Ballé et al.
Revised: Ballé et al.

Original: More recently, based on the framework by Ballé et al.
Revised: More recently, based on the framework by Ballé et al.

Original: [2017], Choi et al.
Revised: [2017], Choi et al.

Original: Cui et al.
Revised: Cui et al.

Original: [2021], Sun et al.
Revised: [2021], Sun et al.

Original: [2021] uses the one-hot encoded λ
Revised: [2021] uses the one-hot encoded λ

Original: as conditional input to the auto-encoder where λ is sampled from uniform discrete prior during training.
Revised: as conditional input to the auto-encoder where λ is sampled from uniform discrete prior during training.

Original: And additional efforts are required to interpolate the R-D curve for continuous rate control.
Revised: And additional efforts are required to interpolate the R-D curve for continuous rate control.

Original: Cui et al.
Revised: Cui et al.

Original: [2021] improves Theis et al.
Revised: [2021] improves Theis et al.

Original: [2017] with asymmetrically scaled vectors.
Revised: [2017] with asymmetrically scaled vectors.

Original: More recently, Song et al.
Revised: More recently, Song et al.

Original: [2021] inserts Spatial Feature
Revised: [2021] inserts Spatial Feature

Original: Transform (SFT) module into the auto-encoder to modulate the intermediate feature.
Revised: Transform (SFT) module into the auto-encoder to modulate the intermediate feature.

Original: However, to the best of knowledge, we are the first to explore semi-amortized inference based variable bitrate model.
Revised: However, to the best of knowledge, we are the first to explore semi-amortized inference based variable bitrate model.

Original: For ROI-based control, Agustsson et al.
Revised: For ROI-based control, Agustsson et al.

Original: [2019] distinguishes images into important and unimportant regions, and use a GAN to generate the unimportant regions.
Revised: [2019] distinguishes images into important and unimportant regions, and use a GAN to generate the unimportant regions.

Original: Cai et al.
Revised: Cai et al.

Original: [2020] supports bit allocation by applying different losses to ROI and non-ROI regions.
Revised: [2020] supports bit allocation by applying different losses to ROI and non-ROI regions.

Original: Song et al.
Revised: Song et al.

Original: [2021] uses ROI to modulate the intermediate features and recover the ROI implicitly from the hyperprior, and it claims that it support flexible ROI control.
Revised: [2021] uses ROI to modulate the intermediate features and recover the ROI implicitly from the hyperprior, and it claims that it support flexible ROI control.

Original: However, it requires complicated handcrafted ROI prior during training.
Revised: However, it requires complicated handcrafted ROI prior during training.

Original: Compared with above-mentioned methods, our Code Editing does not require any prior of ROI, nor data contains segmentation map during training.
Revised: Compared with above-mentioned methods, our Code Editing does not require any prior of ROI, nor data contains segmentation map during training.

Revised: After the submission of this manuscript, Shi et al.

Revised: [2022], Fathima et al.

Revised: [2023] also achieve pixel level variable rate coding for neural codec.

Revised: On the other hand, Agustsson et al.

Revised: [2022] also implement promising perception-distortion trade-off with a conditional decoder.

Original: 3.2 Semi-amortized Inference for NIC
Revised: 3.2 Semi-amortized Inference for NIC

Original: The semi-amortized variational inference is concurrently invented by Kim et al.
Revised: The semi-amortized variational inference is concurrently invented by Kim et al.

Original: [2018] and Marino et al.
Revised: [2018] and Marino et al.

Original: The basic idea is that works following Kingma and Welling [2013] lose the good tradition of learning variational posterior parameters per datapoint.
Revised: The basic idea is that works following Kingma and Welling [2013] lose the good tradition of learning variational posterior parameters per datapoint.

Original: Instead they learn fully amortized parameters for the whole dataset.
Revised: Instead they learn fully amortized parameters for the whole dataset.

Original: And this fully amortized inference might lead to sub-optimal posterior distribution.
Revised: And this fully amortized inference might lead to sub-optimal posterior distribution.

Original: Cremer et al.
Revised: Cremer et al.

Original: [2018] refers to this phenomena as amortization gap and argues it is a major contributor to inference sub-optimality of VAE.
Revised: [2018] refers to this phenomena as amortization gap and argues it is a major contributor to inference sub-optimality of VAE.

Original: The semi-amortized inference is to initialize the variational posterior’s parameter with fully amortized parameters, and conduct stochastic optimization per data-point.
Revised: The semi-amortized inference is to initialize the variational posterior’s parameter with fully amortized parameters, and conduct stochastic optimization per data-point.

Original: Although Kim et al.
Revised: Although Kim et al.

Original: [2018], Marino et al.
Revised: [2018], Marino et al.

Original: [2018] are alluring to apply, they require nested iterative optimization within the stochastic gradient descent loop.
Revised: [2018] are alluring to apply, they require nested iterative optimization within the stochastic gradient descent loop.

Original: When adopting semi-amortized inference to NIC, Djelouah and Schroers [2019], Yang et al.
Revised: When adopting semi-amortized inference to NIC, Djelouah and Schroers [2019], Yang et al.

Original: [2020] simplify it into two stages: 1) training a fully amortized encoder-decoder pair 2) iteratively optimize latent which is initialized by amortized encoder.
Revised: [2020] simplify it into two stages: 1) training a fully amortized encoder-decoder pair 2) iteratively optimize latent which is initialized by amortized encoder.

Original: Although this simplification can lead to sub-optimal generative model (recall that now the generative model is paired with sub-optimal inference model), it indeed makes semi-amortized inference practical for NIC.
Revised: Although this simplification can lead to sub-optimal generative model (recall that now the generative model is paired with sub-optimal inference model), it indeed makes semi-amortized inference practical for NIC.

Original: Our method follows Yang et al.
Revised: Our method follows Yang et al.

Original: [2020] as we explore semi-amortized inference for R-D trade-off instead of R-D performance.
Revised: [2020] as we explore semi-amortized inference for R-D trade-off instead of R-D performance.

Original:  Experimental Results 4.1 Experimental Settings
Revised:  Experimental Results 4.1 Experimental Settings

Original: Following He et al.
Revised: Following He et al.

Original: [2021], we train the baseline models on a subset of 8,000 images from ImageNet.
Revised: [2021], we train the baseline models on a subset of 8,000 images from ImageNet.

Original: For training, we use MSE as distortion.
Revised: For training, we use MSE as distortion.

Original: For each baseline method, we train models with 7 fixed R-D trade-off points, with λ ∈ { 0 . 0016 , 0 . 0032 , 0 . 0075 , 0 . 015 , 0 .
Revised: For each baseline method, we train models with 7 fixed R-D trade-off points, with λ ∈ { 0 . 0016 , 0 . 0032 , 0 . 0075 , 0 . 015 , 0 .

Original: This setup is aligned with Cheng et al.
Revised: This setup is aligned with Cheng et al.

Original: The baseline models’ bitrate ranges from 0.1 to 1.0 bpp on Kodak dataset [Kodak, 1993].
Revised: The baseline models’ bitrate ranges from 0.1 to 1.0 bpp on Kodak dataset [Kodak, 1993].

Original: All baseline models are trained using the Adam optimizer for 2000 epochs.
Revised: All baseline models are trained using the Adam optimizer for 2000 epochs.

Original: Batchsize is set to 16 and the initial learning rate is set to 10 − 4 .
Revised: Batchsize is set to 16 and the initial learning rate is set to 10 − 4 .

Original: For all the results reported in the R-D curve, the bitrate is measured by actual bits of range encoder, and the reconstruction is computed by the latent coded from the actual the range decoder.
Revised: For all the results reported in the R-D curve, the bitrate is measured by actual bits of range encoder, and the reconstruction is computed by the latent coded from the actual the range decoder.

Original: For all visualization involves spatial bitrate distribution, the theoretical bitrate is used.
Revised: For all visualization involves spatial bitrate distribution, the theoretical bitrate is used.

Original: More detailed setup can be found in Appendix C.
Revised: More detailed setup can be found in Appendix C.

Original: 4.2 Ablation Study
Revised: 4.2 Ablation Study

Original: We set Ballé et al.
Revised: We set Ballé et al.

Original: [2018] as the baseline of the ablation study.
Revised: [2018] as the baseline of the ablation study.

Original: All experiments involving Code
Revised: All experiments involving Code

Original: Editing are based on the model trained with λ 0 = 0 . 015 .
Revised: Editing are based on the model trained with λ 0 = 0 . 015 .

Original: In ablation study, all the results are tested on Kodak dataset.
Revised: In ablation study, all the results are tested on Kodak dataset.

Original: Adaptive ∆ .
Revised: Adaptive ∆ .

Original: To reduce the prior mismatch, we propose to control the quantization step ∆ in addition to optimizing y .
Revised: To reduce the prior mismatch, we propose to control the quantization step ∆ in addition to optimizing y .

Original: To verify its effectiveness, we compare Code Editing Enhnanced (w/ adaptive ∆ ) with Code Editing Naïve (w/o adaptive ∆ ).
Revised: To verify its effectiveness, we compare Code Editing Enhnanced (w/ adaptive ∆ ) with Code Editing Naïve (w/o adaptive ∆ ).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, the R-D performance of Code Editing
Revised: 1, the R-D performance of Code Editing

Original: Naïve drops evidently when the bitrate changes out of a small range ( ± 0.1 bpp).
Revised: Naïve drops evidently when the bitrate changes out of a small range ( ± 0.1 bpp).

Original: Further, we provide statistics of normalized dequantized results of quantized y ( i ) before and after Code Editing on the
Revised: Further, we provide statistics of normalized dequantized results of quantized y ( i ) before and after Code Editing on the

Original: Kodak dataset in Fig.
Revised: Kodak dataset in Fig.

Original: It is shown that the distribution mismatch of Code Editing Naïve is more evident than Code Editing Enhanced.
Revised: It is shown that the distribution mismatch of Code Editing Naïve is more evident than Code Editing Enhanced.

Original: On the other hand, we also show the result of controlling ∆ alone.
Revised: On the other hand, we also show the result of controlling ∆ alone.

Original: As Choi et al.
Revised: As Choi et al.

Original: [2019], we find that adjusting ∆ can shift the bitrate in a limited range ( ± 0.
Revised: [2019], we find that adjusting ∆ can shift the bitrate in a limited range ( ± 0.

Original: bpp).
Revised: bpp).

Original: But out of this range, the R-D performance drops rapidly.
Revised: But out of this range, the R-D performance drops rapidly.

Original: SGA vs. AUN.
Revised: SGA vs. AUN.

Original: In Code Editing, we adopt SGA instead of additive uniform noise (AUN) as relaxation of discrete latent.
Revised: In Code Editing, we adopt SGA instead of additive uniform noise (AUN) as relaxation of discrete latent.

Original: To verify the effect of SGA, we optimize y for 2,000 iterations using SGA and
Revised: To verify the effect of SGA, we optimize y for 2,000 iterations using SGA and

Original: AUN, respectively.
Revised: AUN, respectively.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, although Code Editing with AUN also outperforms the baseline, the SGA’s R-D performance is significantly higher than AUN especially in high bitrate region.
Revised: 1, although Code Editing with AUN also outperforms the baseline, the SGA’s R-D performance is significantly higher than AUN especially in high bitrate region.

Original: This is due to the fact that SGA closes the discretization gap
Revised: This is due to the fact that SGA closes the discretization gap

Original: [Yang et al., 2020] via soft to hard annealing.
Revised: [Yang et al., 2020] via soft to hard annealing.

Original: Encoder Fine-tuning for Fast Inference.
Revised: Encoder Fine-tuning for Fast Inference.

Original: We show that we can control the computational complexity of Code Editing by early termination.
Revised: We show that we can control the computational complexity of Code Editing by early termination.

Original: We terminate Code Editing with { 50 , 100 , 200 } iterations, which correspond to { 2 . 5% , 5% , 10% } of full iterations.
Revised: We terminate Code Editing with { 50 , 100 , 200 } iterations, which correspond to { 2 . 5% , 5% , 10% } of full iterations.

Original: The results on the Kodak dataset are shown in Fig.
Revised: The results on the Kodak dataset are shown in Fig.

Original: In low bitrate range, the R-D performance achieves a reasonable point within 10% of iterations.
Revised: In low bitrate range, the R-D performance achieves a reasonable point within 10% of iterations.

Original: However, the R-D performance in the high bitrate region degrades severely.
Revised: However, the R-D performance in the high bitrate region degrades severely.

Original: To solve this problem, we finetune encoder with λ = 0 .
Revised: To solve this problem, we finetune encoder with λ = 0 .

Original: 052 under ∆ y = 0 .
Revised: 052 under ∆ y = 0 .

Original: The latent code predicted by the finetuned encoder is used as the initial value for Code Editing in the high bitrate range.
Revised: The latent code predicted by the finetuned encoder is used as the initial value for Code Editing in the high bitrate range.

Original: And a stronger encoder achieves better R-D performance with very few iterations (marked with "Encoder
Revised: And a stronger encoder achieves better R-D performance with very few iterations (marked with "Encoder

Original: FT.
Revised: FT.

Original: in the Fig. 1).
Revised: in the Fig. 1).

Original: 4.3 Variable Rate Coding
Revised: 4.3 Variable Rate Coding

Original: First, we evaluate Code Editing Enhanced for continuous variable bitrate coding.
Revised: First, we evaluate Code Editing Enhanced for continuous variable bitrate coding.

Original: We choose three established works [Ballé et al., 2018, Minnen et al., 2018, Cheng et al., 2020] as baselines.
Revised: We choose three established works [Ballé et al., 2018, Minnen et al., 2018, Cheng et al., 2020] as baselines.

Original: For each of the baselines, we set base model’s R-D trade-off parameter λ 0 = 0 .
Revised: For each of the baselines, we set base model’s R-D trade-off parameter λ 0 = 0 .

Original: 015 , which results in a mid-range bitrate (around 0.5 bpp).
Revised: 015 , which results in a mid-range bitrate (around 0.5 bpp).

Original: For each image x to be compressed, we initialize y ← [Table] [Equation] ) and optimize y for 2,000 iterations using SGA, with the learning rate of 5 × 10 − 3 .
Revised: For each image x to be compressed, we initialize y ← f ϕ λ 0 ( x ) and optimize y for 2,000 iterations using SGA, with the learning rate of 5 × 10 − 3 .

Original: This setting is aligned with Yang et al.
Revised: This setting is aligned with Yang et al.

Original: ∆ y is directly optimized by gradient descent and ∆ z is optimized by grid search.
Revised: ∆ y is directly optimized by gradient descent and ∆ z is optimized by grid search.

Original: Fig.
Revised: Fig.

Original: 3 shows the R-D performance of variable bitrate via Code Editing.
Revised: 3 shows the R-D performance of variable bitrate via Code Editing.

Original: It can be seen  that Code Editing Enhanced can effectively achieve wide range continuous R-D trade-off with fixed decoder and entropy model trained at λ 0 .
Revised: It can be seen [Figure] that Code Editing Enhanced can effectively achieve wide range continuous R-D trade-off with fixed decoder and entropy model trained at λ 0 .

Original: Then, we compared Code Editing Enhanced with the sota variable code-rate methods [Theis et al., 2017, Cui et al., 2021, Song et al., 2021].
Revised: Then, we compared Code Editing Enhanced with the sota variable code-rate methods [Theis et al., 2017, Cui et al., 2021, Song et al., 2021].

Original: For fairness, we re-implement these methods on the[Figure] baselines with the same setting as original papers.
Revised: For fairness, we re-implement these methods on the baselines with the same setting as original papers.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: Note that we only show the results of Song et al.
Revised: Note that we only show the results of Song et al.

Original: [2021] on Ballé et al.
Revised: [2021] on Ballé et al.

Original: [2018] as it produces NaNs on other models.
Revised: [2018] as it produces NaNs on other models.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, Our proposed Code Editing outperforms existing variable bitrate methods with all three baseline.
Revised: 3, Our proposed Code Editing outperforms existing variable bitrate methods with all three baseline.

Original: Theis et al.
Revised: Theis et al.

Original: [2017] brings larger degradation in R-D performance, especially in the high bitrate region.
Revised: [2017] brings larger degradation in R-D performance, especially in the high bitrate region.

Original: Cui et al.
Revised: Cui et al.

Original: [2021] achieves similar R-D performance as the original fix-bitrate model.
Revised: [2021] achieves similar R-D performance as the original fix-bitrate model.

Original: The R-D performance of Song et al.
Revised: The R-D performance of Song et al.

Original: [2021] on Ballé et al.
Revised: [2021] on Ballé et al.

Original: [2018] exceeds the fix-bitrate model in the low bitrate range but is inferior to Code Editing Enhanced.
Revised: [2018] exceeds the fix-bitrate model in the low bitrate range but is inferior to Code Editing Enhanced.

Original: 4.4 Multi-Distortion Trade-off
Revised: 4.4 Multi-Distortion Trade-off

Original: Next, we explore the potential of Code Editing Enhanced in multi-distortion trade-off.
Revised: Next, we explore the potential of Code Editing Enhanced in multi-distortion trade-off.

Original: For experiments, we choose to balance between MSE and LPIPS [Zhang et al., 2018].
Revised: For experiments, we choose to balance between MSE and LPIPS [Zhang et al., 2018].

Original: MSE is adopted to measure fidelity, while LPIPS is widely adopted for perceptual quality [Mentzer et al., 2020, Bhat et al.
Revised: MSE is adopted to measure fidelity, while LPIPS is widely adopted for perceptual quality [Mentzer et al., 2020, Bhat et al.

Original: , 2021].The lower the LPIPS is, the better perceptual quality an image has.
Revised: , 2021].The lower the LPIPS is, the better perceptual quality an image has.

Original: In our experiment, we select 3 trade-off points by controlling the LPIPS weight λ p ∈ { 0 . 1 , 0 . 5 , 1 . 0 } .
Revised: In our experiment, we select 3 trade-off points by controlling the LPIPS weight λ p ∈ { 0 . 1 , 0 . 5 , 1 . 0 } .

Original: We use the bitrate control scheme in Mentzer et al.
Revised: We use the bitrate control scheme in Mentzer et al.

Original: [2020] to stabilize bitrate (See Appendix C for detail).
Revised: [2020] to stabilize bitrate (See Appendix C for detail).

Original: Fig.shows the multi-distortion trade-off results based on Ballé et al.
Revised: Fig.shows the multi-distortion trade-off results based on Ballé et al.

Original: [2018] on the Kodak dataset.
Revised: [2018] on the Kodak dataset.

Original: As λ p increases, both LPIPS and PSNR decrease, implying better perceptual quality and worse distortion is achieved.
Revised: As λ p increases, both LPIPS and PSNR decrease, implying better perceptual quality and worse distortion is achieved.

Original: Further, we show qualitative results based on Ballé et al.
Revised: Further, we show qualitative results based on Ballé et al.

Original: [2018] on the CLIC2022 dataset [CLIC, 2022] in Fig.
Revised: [2018] on the CLIC2022 dataset [CLIC, 2022] in Fig.

Original: We can see that as λ p increases, the detail level of image increases, which indicates a better perceptual quality.
Revised: We can see that as λ p increases, the detail level of image increases, which indicates a better perceptual quality.

Original: Both quantitative and qualitative results show that our Code
Revised: Both quantitative and qualitative results show that our Code

Original: Editing can achieve flexible multi-distortion trade-off.[Figure] [Figure] [Figure]
Revised: Editing can achieve flexible multi-distortion trade-off.

Original: [Figure]

Original: [Figure]

Original: [Figure] 4.5 ROI-based Coding
Revised: 4.5 ROI-based Coding

Original: This section shows the performance of Code Editing in terms of ROI-based coding.
Revised: This section shows the performance of Code Editing in terms of ROI-based coding.

Original: In order to verify the effectiveness of spatially adaptive R-D trade-off, we use various quality maps.
Revised: In order to verify the effectiveness of spatially adaptive R-D trade-off, we use various quality maps.

Original: We use Ballé et al.
Revised: We use Ballé et al.

Original: [2018] as the baseline model to verify the effectiveness of Code Editing in ROI-based coding on
Revised: [2018] as the baseline model to verify the effectiveness of Code Editing in ROI-based coding on

Original: Kodak and CLIC2022 datasets.
Revised: Kodak and CLIC2022 datasets.

Original: Fig.
Revised: Fig.

Original: 6 illustrates these results.
Revised: 6 illustrates these results.

Original: And our Code Editing can allocate bitrate spatially according to arbitrary-shape ROI maps, including checkerboard and alphabet shapes.
Revised: And our Code Editing can allocate bitrate spatially according to arbitrary-shape ROI maps, including checkerboard and alphabet shapes.

Original: Moreover, we compared our Code Editing ROI results with Song et al.
Revised: Moreover, we compared our Code Editing ROI results with Song et al.

Original: [2021] which is trained with sophisticated prior in Fig.
Revised: [2021] which is trained with sophisticated prior in Fig.

Original: It can be seen that Song et al.
Revised: It can be seen that Song et al.

Original: [2021]’s result suffers from ROI map diminishing issue.
Revised: [2021]’s result suffers from ROI map diminishing issue.

Original: We can barely sense the effect of ROI map inside the region indicated by the white rectangle.
Revised: We can barely sense the effect of ROI map inside the region indicated by the white rectangle.

Original: In Appendix B.2, we provide more ROI results on different types of masks, including segmentation mask.
Revised: In Appendix B.2, we provide more ROI results on different types of masks, including segmentation mask.

Original: 4.6 Gap to SAVI with Multiple Models
Revised: 4.6 Gap to SAVI with Multiple Models

Original: We present the results of direct performing semi-amortized inference on baseline models on Kodak dataset.
Revised: We present the results of direct performing semi-amortized inference on baseline models on Kodak dataset.

Original: To be specific, we directly optimize y using SGA based on multiple models, which is equivalent to Yang et al.
Revised: To be specific, we directly optimize y using SGA based on multiple models, which is equivalent to Yang et al.

Original: [2020] without bits-back coding.
Revised: [2020] without bits-back coding.

Original: From Fig.
Revised: From Fig.

Original: 8 we can see that when baselines are Ballé et al.
Revised: 8 we can see that when baselines are Ballé et al.

Original: [2018], Minnen et al.
Revised: [2018], Minnen et al.

Original: [2018], our Code Editing Enhanced is comparable to SGA based on multiple models.
Revised: [2018], our Code Editing Enhanced is comparable to SGA based on multiple models.

Original: However, when baseline is Cheng et al.
Revised: However, when baseline is Cheng et al.

Original: [2020], our Code Editing Enhanced indeed suffers from R-D performance decay in high bitrate region.
Revised: [2020], our Code Editing Enhanced indeed suffers from R-D performance decay in high bitrate region.

Original: We use a single decoder to achieve this while Yang et al.
Revised: We use a single decoder to achieve this while Yang et al.

Original: [2020] require multiple decoder.
Revised: [2020] require multiple decoder.

Original: We are neither superior nor inferior to Yang et al.
Revised: We are neither superior nor inferior to Yang et al.

Original: [2020], as the task is very different. 
Revised: [2020], as the task is very different. [Figure] [Figure]

Original: Note that the major difference between our work and Yang et al.
Revised: Note that the major difference between our work and Yang et al.

Original: [2020] is that our approach requires only one decoder for continuous bitrate control, ROI and perception-distortion trade-off.
Revised: [2020] is that our approach requires only one decoder for continuous bitrate control, ROI and perception-distortion trade-off.

Original: And Yang et al.
Revised: And Yang et al.

Original: [2020] require multiple decoders for them.
Revised: [2020] require multiple decoders for them.

Original: Yang et al.
Revised: Yang et al.

Original: [2020] adopt SAVI Kim et al.
Revised: [2020] adopt SAVI Kim et al.

Original: [2018] to improve the R-D performance of a pair of encoder-decoder.
Revised: [2018] to improve the R-D performance of a pair of encoder-decoder.

Original: We find SAVI can also be adopted to achieve bitrate control, ROI and multi-distortion with only a single decoder.
Revised: We find SAVI can also be adopted to achieve bitrate control, ROI and multi-distortion with only a single decoder.

Original: In fact, even without the
Revised: In fact, even without the

Original: SGA of Yang et al. [2020], the semi-amortized inference of the simple AUN implementation can still achieve flexible bit rate control (See Sec. 4.2).
Revised: [Figure]

Original:  Conclusions
Revised:  Conclusions

Original: We propose Code Editing, a new paradigm for continuous variable bitrate NIC based on semiamortized inference.
Revised: We propose Code Editing, a new paradigm for continuous variable bitrate NIC based on semiamortized inference.

Original: It outperforms current variable bitrate NIC methods.
Revised: It outperforms current variable bitrate NIC methods.

Original: Moreover, it achieves flexible spatial bits allocation and multi-distortion trade-off, all with one decoder.
Revised: Moreover, it achieves flexible spatial bits allocation and multi-distortion trade-off, all with one decoder.

Original: Limitations include that the efficiency of Code Editing could be improved, and that the empirical study on more compounded target mentioned as Eq.
Revised: Limitations include that the efficiency of Code Editing could be improved, and that the empirical study on more compounded target mentioned as Eq.

Original: 10 should be addressed.
Revised: 10 should be addressed.

Original: Limitation
Revised: Limitation

Original: In general, our method does not work for the cases where encoding time matters, such as real-time communication.
Revised: In general, our method does not work for the cases where encoding time matters, such as real-time communication.

Original: Our method is extremely useful for the cases where we encode just once but decode/view plural number of times, such as content delivery network.
Revised: Our method is extremely useful for the cases where we encode just once but decode/view plural number of times, such as content delivery network.

Original: Another limitation of our work is that we limit the scope of discussion to scalar ∆ instead of vector ∆ .
Revised: Another limitation of our work is that we limit the scope of discussion to scalar ∆ instead of vector ∆ .

Original: Adopting a vector ∆ increase the range of bitrate in ROI-based coding.
Revised: Adopting a vector ∆ increase the range of bitrate in ROI-based coding.

Original: For Code Editing Enhanced, all pixels share one ∆ .
Revised: For Code Editing Enhanced, all pixels share one ∆ .

Original: And this means in ROI-based coding, the overall ∆ might not be suitable for all pixels with different quality map value m .
Revised: And this means in ROI-based coding, the overall ∆ might not be suitable for all pixels with different quality map value m .

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: This work is supported by SenseTime Research.
Revised: This work is supported by SenseTime Research.

Original: The content is solely the responsibility of the authors and does not necessarily represent the official views of SenseTime Research Checklist
Revised: The content is solely the responsibility of the authors and does not necessarily represent the official views of SenseTime Research.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: Appendix

Original: In Appendix A, extra quantitative results, including the results related to optimizing ∆ and quantitative results on CLIC2020 [CLIC, 2022] are presented.

Original: In Appendix B, extra qualitative results are presented.

Original: And in Appendix C, we present the implementation details, including the detailed guidance to reproduce main results.

Original: A More Quantitative Results A.1 Experimental Results on CLIC

Original: The results in Sec.

Original: 4.3 are based on Kodak dataset.

Original: In this section we present the results comparing our Code Editing Enhance with other methods based on CLIC2022 dataset.

Original: From Fig .

Original: A1, we can see that the main conclusion remains unchanged.

Original: A.2 Optimizing and Transferring ∆

Original: As shown in Fig.

Original: A2, using gradient method to optimize ∆ of z brings significant performance decay, while using gradient method to optimize ∆ of y is fine.

Original: So for Code Editing Enhanced, we use gradient based method to optimize ∆ y , and grid search to optimize ∆ z .

Original: We set the grid search range as ∆ z ∈ { 2 − 1 .

Original: 5 , 2 − 1 . 0 , ..., 2 1 . 5 } , and we pick ∆ z with the smallest R-D cost ( −L y , z , ∆ y , ∆ z ).

Original: The result of searching ∆ z is presented in Tab.

Original: A1.

Original: Note that as the value of ∆ z is discrete and has only 7 possibilities, we can simply use uniform prior, and we need only 3 bits

Original: [Table]

Original: to encode it.

Original: And the value of ∆ y is a single-precision float which takes another 32 bits to encode.

Original: On Kodak dataset with 512 × 768 pixels, compressing both of the ∆ s requires around 8 × 10 − 5 bpp.

Original: And on CLIC2022 which is even larger, the bpp of ∆ s is even less.

Original: So when presenting experimental results, we simply ignore the bpp of ∆ s.

Original: Choi et al.

Original: [2019] also adopt the quantization stepsize control.

Original: However, our approach is very different from it.

Original: Specifically, we find there is train-test mismatch of the entropy model in Code Editing Naïve, which damages R-D performance.

Original: And then we propose adaptive quantization step to alleviate this problem (See Sec. 2.3 and Sec. 4.2).

Original: On the other hand, Choi et al.

Original: [2019] adjust the quantization step to fine-tune the bitrate.

Original: From the perspective of results, our proposed adaptive quantization works in the a wide bitrate region while the quantization step adjustment of Choi et al.

Original: [2019] works in a narrow bit rate region.

Original: Moreover, Choi et al.

Original: [2019] sample ∆ during training, which requires a carefully designed prior on ∆ .

Original: According to the original paper, training ∆ ∈ [0 . 5 , 2] brings best performance, and making it larger or narrower brings performance decay.

Original: While for us, the ∆ is learned during SAVI stage and no deliberate prior is required.

Original: And during training we keep ∆ = 1 like a normal model.

Original: The advantage is that our method can be directly applied to any pre-trained neural compression model, while Choi et al.

Original: [2019] can not.

Original: Furthermore, we study the effect of optimizing ∆ jointly with SAVI, which is never studied before.

Original: Moreover, we provide non-trivial extra insights into why this approach might work by theoretical analysis (Sec. 2.3) and empirical study (Sec. 4.2).

Original: We note that it is also possible to treat ∆ s as a vector ∆ representing per dimension quantization step.

Original: In that case, the bpp of ∆ is unneglectable and has to be compressed separately.

Original: However, the vector ∆ is out of the scope of this paper.

Original: A

Original: 3 Go without Grid Search

Original: Fig.

Original: A3 shows the result of fixing ∆ z = 1 . 0 and performing no grid search at all.

Original: It can be seen that the R-D performance drop is only marginal compared with the grid search scheme in Sec.

Original: A.

Original: Therefore, it is possible to abandon the grid search and trade R-D performance for speed.

Original: [Figure]

Original: Fig.

Original: A3 shows the result of bpp beyond 1 . 0 .

Original: It can be observed that when we stick to a decoder trained with bpp ≈ 0 .

Original: 5 ( λ = 0 . 015 ), its R-D performance drops below the baseline when bpp approaches 1.5.

Original: However, if we adopt a decoder trained with bpp ≈ 1 . 0 ( λ = 0 . 045 ), the R-D performance of our approach in high bitrate is significantly enhanced.

Original: We can achieve variable bitrate control without

Original: R-D performance loss in 0 .

Original: 1 ≤ bpp ≤ 1 . 9 .

Original: B More Qualitative Results B.1 Multi-Distortion Trade-off

Original: We present more qualitative results of multi-distortion trade-off in Fig.

Original: B1-B3.

Original: Both Kodak and

Original: CLIC2022 dataset are used.

Original: B.2 ROI-based Coding

Original: We present more qualitative results of ROI-based coding in Fig.

Original: B4-B6.

Original: The ROI maps are shown in the main paper.

Original: Both Kodak and CLIC2022 dataset are used.

Original: For segmentation based ROI, We selected the image 13e9b6 from the CLIC2022 dataset to test the segmentation ROI.

Original: There are 4 people in this image.

Original: We use separate segmentation for each person.

Original: Unlike the high contrast ROI shown in the main paper, we give the background a weight of 0.04 instead of 0.

Original: We define PSNR region to evaluate the image quality when ROI is used.

Original: Given an m × n image I and the reconstructed image K , MSE region is defined as:

Original: [Equation]

Original: where M ( i, j ) is 1(0) if pixel ( i, j ) is inside(outside) the ROI region.

Original: PSNR region (in dB) is defined as:

Original: [Equation]

Original: These results can be found at Fig.

Original: B7-B9.

Original: We can see that code editing is effective for complex semantic ROI.

Original: And the visual quality of each person is improved accordingly as the ROI masks shift.

Original: λ p = 0.

Original: Bitrate:0.1756bpp

Original: PSNR:26.30dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1772bpp

Original: PSNR:26.10dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1764bpp

Original: PSNR:25.64dB

Original: LPIPS:0.2194 Original Image λ p = 0.

Original: Bitrate:0.1264bpp

Original: PSNR:27.67dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1305bpp

Original: PSNR:27.54dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1330bpp

Original: PSNR:27.08dB

Original: LPIPS:0.2792 Original Image λ p = 0.

Original: Bitrate:0.1519bpp

Original: PSNR:26.92dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1544bpp

Original: PSNR:26.79dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1519bpp

Original: PSNR:26.20dB

Original: LPIPS:0.2400 Original Image λ p = 0.

Original: Bitrate:0.1194bpp

Original: PSNR:28.33dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1226bpp

Original: PSNR:28.13dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1222bpp

Original: PSNR:27.56dB

Original: LPIPS:0.2877 Original Image λ p = 0.

Original: Bitrate:0.0861bpp

Original: PSNR:29.56dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.0886bpp

Original: PSNR:29.09dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.0888bpp

Original: PSNR:28.87dB

Original: LPIPS:0.3424 Original Image λ p = 0.

Original: Bitrate:0.0892bpp

Original: PSNR:29.62dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.0913bpp

Original: PSNR:29.13dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.0915bpp

Original: PSNR:28.88dB

Original: LPIPS:0.3292 Original Image λ p = 0.

Original: Bitrate:0.1269bpp

Original: PSNR:27.82dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1276bpp

Original: PSNR:27.26dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1277bpp

Original: PSNR:27.02dB

Original: LPIPS:0.3139 Original Image λ p = 0.

Original: Bitrate:0.1014bpp

Original: PSNR:29.71dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1016bpp

Original: PSNR:29.00dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1014bpp

Original: PSNR:28.72dB

Original: LPIPS:0.3171 Original Image λ p = 0.

Original: Bitrate:0.1574bpp

Original: PSNR:27.08dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1599bpp

Original: PSNR:25.89dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1599bpp

Original: PSNR:26.37dB

Original: LPIPS:0.2489 Original Image λ p = 0.

Original: Bitrate:0.1385bpp

Original: PSNR:27.56dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1409bpp

Original: PSNR:27.40dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1408bpp

Original: PSNR:26.81dB

Original: LPIPS:0.3368 Original Image λ p = 0.

Original: Bitrate:0.1098bpp

Original: PSNR:28.65dB

Original: LPIPS:0.

Original: λ p = 0.

Original: Bitrate:0.1106bpp

Original: PSNR:28.12dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1102bpp

Original: PSNR:27.89dB

Original: LPIPS:0.3807 Original Image λ p = 0.

Original: Bitrate:0.1955bpp

Original: PSNR:25.29dB

Original: LPIPS: 0.

Original: λ p = 0.

Original: Bitrate:0.1974bpp

Original: PSNR:25.16dB

Original: LPIPS:0.

Original: λ p = 1.

Original: Bitrate:0.1973bpp

Original: PSNR:24.71dB

Original: LPIPS:0.2448 Original Image

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Reconstructed

Original: ROI

Original: [Figure]

Original: (#1), black=0.04 (least bits)

Original: [Figure]

Original: white=1 (most bits)

Original: [Figure]

Original: 𝑃𝑆𝑁𝑅 !"

Original: #$%

Original: [Figure]

Original: for person #1 : (

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: ) 29.44𝑑𝐵 → 33.17𝑑𝐵

Original: [Figure]

Original: [Figure]

Original: =0.

Original: [Figure]

Original: (least bits)

Original: [Figure]

Original: (most bits)

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Reconstructed Image

Original: [Figure]

Original: C Experiment Details C.1 Detailed Experimental Settings

Original: All the experiments are conducted on a computer with Intel(R)

Original: Xeon(R) CPU E5-2620 v4 @ 2.10GHz and 8 × Nvidia(R)

Original: TitanXp.

Original: All the experiments are implemented in Pytorch 1.7 and CUDA 9.0.

Original: The

Original: Hyper-parameters are aligned with Yang et al.

Original: Specifically, we optimize ∆ y , y , z with SGA for 2 , 000 iterations and learning rate 5 × 10 − 3 .

Original: More specifically for SGA, the temperature is set to 0 .

Original: 5 × e − 10 − 3 ( k − 700) , where k is current epoch index.

Original: For Code Editing Enhanced, we search ∆ z to be { 2 1 . 5 , 2 1 . 0 , 2 0 . 5 , 2 0 . 0 , 2 − 0 .

Original: 5 , 2 − 1 . 0 , 2 − 1 . 5 } .

Original: C.2 Guidance for Reproducing Main Results

Original: Baseline Models.

Original: To reproduce the results in the main paper, we train the baseline models [Ballé et al., 2018, Minnen et al., 2018, Cheng et al., 2020].

Original: We follow the settings of He et al.

Original: These models are trained on the subset of 8,000 images from ImageNet for 2,000 epochs.

Original: We use

Original: Adam [Kingma and Ba, 2014] with learning rate of 10 − 4 and batch-size 16.

Original: Fig.

Original: 1 Left.

Original: For Code Editing Enhanced, we use SGA[Yang et al., 2020] to optimize y , z .

Original: For

Original: Code Editing Enhanced(AUN)

Original: , we use additive uniform noise as [Ballé et al., 2017, 2018, Cheng et al., 2020].

Original: For Code Editing Naïve, we fix ∆ y and ∆ z to 1 . 0 .

Original: For ∆ z optimization, we set ∆ z to { 2 1 . 5 , 2 1 . 0 , 2 0 . 5 , 2 0 . 0 , 2 − 0 .

Original: 5 , 2 − 1 . 0 , 2 − 1 . 5 } .

Original: Fig.

Original: 1 Right.

Original: We set iterations to 0, 50, 100, 150 and 200, respectively.

Original: Due to early termination, we adjust SGA temperature from original 0 .

Original: 5 × e − 10 − 3 ( k

Original: Fig.

Original: For Song et al.

Original: [2021] and Cui et al.

Original: [2021], we add their proposed Spatial Feature Transform and Gain Unit to the the baseline models.

Original: We train these models using the same settings as training the baseline models.

Original: For Theis et al.

Original: [2017], we add the scale vectors to the baseline models trained under λ 0 = 0 .

Original: 015 and finetune the scale vectors for 100 epochs, with a learning rate of 10 − 4 .

Original: Fig.

Original: These curves are obtained by optimizing y and ∆ to maximize L MD y , ∆ (see Eq. 9).

Original: We adopt a bitrate control scheme from Mentzer et al.

Original: [2020] to make the bitrate roughly same.

Original: First, we set R target to the bitrate before editing for each image.

Original: Then, we add a weight λ r to the left term log P θ λ 0 ( ⌊ y / ∆ ⌉ ; ∆) in Eq. 9.

Original: [-λ r is set to λ ( a ) and λ ( b ) for − log P θ λ 
-]
Revised: [-λ r is set to λ ( a ) and λ ( b ) for − log P θ λ 
-]

Original: [-⌊ y / ∆ ⌉ ; ∆) > R target and − log P θ λ 
-]
Revised: [-⌊ y / ∆ ⌉ ; ∆) > R target and − log P θ λ 
-]

Original: ⌊ y / ∆ ⌉ ; ∆) <

Original: R target , respectively, where λ ( a ) ≫ λ ( b ) .

Original: [-Here, we set λ ( a ) to 4 . 
-]
Revised: [-Here, we set λ ( a ) to 4 . 
-]

Original: and λ ( b ) to 0.25.

Original: Fig.

Original: We optimize the target L ROI y , ∆ to obtain the ROI-based coding results.

Original: Here, we adopt the baseline model Ballé et al.

Original: [2018] trained under λ 0 = 0 . 0016 .

Original: We conduct the experiment on Kodak and CLIC2022 datasets.

Original: We sample 3 images and show them in Fig.

Original: They are from CLIC dataset.

Original: Their IDs are 76e721, 2ff706 and 631791, respectively.

Original: Broader Impact

Original: In CLIC (Challenge on Learned Image Compression) 2021, the champion solution Gao et al.

Original: [2021]’s decoder size is 230MB.

Original: On the other hand, Savinaud et al.

Original: [2013] as a JPEG2000 codec is only around 1.4MB.

Original: Thus, a flexible neural codec with one decoder for all conditions is highly valuable in terms of disk storage reduction, which in turn benefits carbon footprint reduction.

Original: On the other hand, our solution enables continuous control of bitrate with flexible ROI.

Original: This makes NIC as flexible as traditional codecs, and thus prompts the practical deployment of NIC.

Original: Moreover, the distortionperception trade-off also attracts attentions of contemporary traditional codecs, such as H.266 [Bross et al., 2021].