V ECTOR Q UANTIZATION AND S HIFTING : E XPLOITING L ATENT P ROPERTIES TO O PTIMIZE N EURAL C ODECS
Anonymous authors
Paper under double-blind review
A BSTRACT
End-to-end image/video codecs are getting competitive compared to traditional compression techniques that have been developed through decades of manual engineering efforts.
These trainable codecs have many advantages over traditional techniques such as easy adaptation on perceptual distortion metrics and high performance on specific domains thanks to their learning ability.
However, state of the art neural codecs do not take advantage of vector quantization technique and existence of gradient of entropy in decoding device.
In this research, we propose some theoretical insights about these two properties (quantization and entropy gradient) and show that this can improve the performances of many off-the-shelf codecs.
First, we prove that non-uniform quantization map on neural codec’s latent is not necessary.
Thus, we improve the performance by using a predefined optimal uniform vector quantization map.
Secondly, we theoretically show that gradient of entropy (available at decoder side) is correlated with the gradient of the reconstruction error (which is not available at decoder side).
Thus, we use the former as a proxy in order to improve the compression performance.
According to our results, we save between 2-4% of rate for the same quality with this proposal, for various pre-trained methods.
I NTRODUCTION
Lossy Image and video compression is a fundamental task in image processing which has become crucial in the time of pandemic and increasing video streaming volume.
Thanks to the community’s decades long efforts, traditional methods (e.g. VVC) have reached current state of the art ratedistortion (RD) performance and dominate current codecs market.
Recently, end-to-end trainable deep models have emerged with promising RD performances by learning the informative latents and modeling the latent distribution.
Even though deep learning based models clearly exceed many traditional techniques and surpass human capability for some general computer vision tasks, they are only almost on par with the best traditional image compressing methods when trained to optimized the peak signal-to-noise ratio (PSNR) according to our knowledge.
End-to-end deep compression methods are generally rate-distortion autoencoders (Habibian et al., 2019), where the latents are optimized using a rate-distortion loss function.
For perceptual friendly compression, distortion based on a perceptual metric can also be used in the loss function (Blau &
Michaeli, 2019).
These methods can be seen as a special case of Variational Autoencoder (VAE) models as described in (Kingma & Welling, 2013), where the approximate posterior distribution is a uniform distribution centered on the encoder’s outputs (latents) at training time and has a fixed variance output distribution and trainable priors (Theis et al., 2017; Ball´e et al., 2017).
It was shown that minimizing the evidence lower bound (ELBO) of this special VAE is equivalent to minimizing jointly the mean square error (MSE) of the reconstruction and the entropy of latents w.r.t the priors (Ball´e et al., 2018).
All proposed models differ mainly by the modelling of priors: using either fully-factorized (Ball ´ e et al., 2017), zero-mean gaussian (Ball ´ e et al., 2018), gaussian (Minnen et al., 2018; Minnen & Singh, 2020) or mixture of gaussian (Cheng et al., 2020), where some methods predict the priors using an autoregressive schema (Minnen et al., 2018; Minnen & Singh, 2020;
Cheng et al., 2020; Xie et al., 2021;
He et al., 2021) and some improve the priors by global and local context modeling (Qian et al., 2021; Kim et al., 2022).
These neural image codecs were extended to the video compression domain by using two VAEs, one for encoding motion information, another one for encoding residual information in end-to-end video compression (Lu et al., 2019;
Agustsson et al., 2020; Ladune & Philippe, 2022; Yılmaz & Tekalp, 2021; Pourreza & Cohen, 2021; Li et al., 2021; 2022)
An important step in the building of a neural codec is the quantization of the latent before the entropy coding.
All mentioned prior state of the art models use fixed bin-width uniform Scalar Quantization (SQ).
Even though Vector Quantization (VQ) is theoretically better (Gersho & Gray, 2012), very limited VQ attempt was done on neural codecs such as in (Agustsson et al., 2017; Zhu et al., 2022).
However, VQ does not seem to bring improvement over SQ, which can be explained by various reasons.
First, VQ introduces extra trainable parameters to be learned: the codeword centroids.
Learning these parameters jointly with all model parameters can introduce additional complexity at training time.
Second, since quantization is non-differentiable, a relaxation strategy (Bengio et al., 2013; Agustsson et al., 2017; Zhu et al., 2022) needs to be used at training time, and the continuous relaxation of VQ is less efficient than SQ one.
One of the main principle in general compression is to exploit all information available at the decoder to reconstruct the data.
However, even though the gradients of the entropy w.r.t latents are available at decoder side, this information remains unused so far in the literature.
Some similar work in the literature have tried to improve the performance of codec, for example by using specific parameterization (Balcilar et al., 2022), or computationally heavy finetuning solutions (Yang et al., 2020;
Guo et al., 2021), either partially (Campos et al., 2019; Lu et al., 2020) or entirely (van
Rozendaal et al., 2021).
However, all these methods ignore the gradients of the entropy.
In this paper, we propose two original contributions to leverage the properties of the learned latent representation for compression.
First, we prove that non-uniform quantization over learned latents is not necessary, even for VQ, contrarily to the usual case.
Thus, in order to use the theoretical power of VQ, we propose to apply uniform VQ map over the latents.
Since the optimal uniform VQ map is known up to some certain dimensions, we do not need to optimize these quantization centers.
This contribution can be applied even without re-training the model if the original model is trained for uniform SQ which is often the case for neural codecs.
Second, we apply Karush–Kuhn–Tucker (KKT) conditions on the neural codec which has not been done to the best of our knowledge.
These conditions show that the gradient of the reconstruction error (unavailable at decoder side) is actually correlated with the gradient of entropy (available at decoder side) w.r.t latents.
This motivates us to use the available gradient as a proxy for the unavailable gradient in order to improve the performance of the neural codecs without re-training.
Our two contributions are generic enough (they do not depend on the encoder-decoder architecture) to bring a rate saving of 2 − 4% at same quality on several neural codecs architectures.
P ROBLEM STATEMENT AND S TATE OF THE A RT
In this section, we recall the basic principles of neural compression, including the training, inference and quantization steps.
For an input color image x ∈ R n × n × 3 to be compressed (the image can be considered square without any loss of generality), the neural codec learns a non-linear encoder g a ( x ; ϕ ) , parameterized by the weights ϕ .
The output of the encoder, y ∈ R m × m × o , is called the main embedding (or main latents) of the image.
The latent representation is then quantized as ˜y =
Q ( y ) to obtain the main codes of the image (the dequantization block ˆy =
Q − 1 ( ˜y ) is used to obtain reconstructed main latents ˆy at decoding side 1 ).
The decompressed image ˆx ∈ R n × n × 3 is obtained by the learned deep decoder with ˆx =
g s ( ˆy ; θ ) .
The neural codec is learned so as to minimize two objectives simultaneously, namely the distortion between x and ˆx , and the length of the bitstream ˜y .
The codes are encoded into a bitstream in a lossless manner by any entropy encoder such as range asymmetric numeral systems (RANS)
Duda (2009).
RANS needs probability mass function (PMF) of each code, and this information is also learned at training time.
Since RANS is asymptotically optimal, the lower bound of bitlength according to Shanon’s entropy theorem can be used instead of experimental bitlength from RANS in order to make bitlength objective differentiable.
Thus, neural codecs use the entropy model to learn the PMF of each codes under determined quantization that allows us to know lower bound of bitlength.
Current state of the art neural codecs use the hyperprior entropy model, where the side embedding (or side latents) z ∈ R t × t × s is learned by another deep neural network by z
= h a ( y ; Φ) .
The side embeddings are quantized as ˜z = Q ( z ) to obtain side codes ˜z followed by dequantization ˆz = Q − 1 ( ˜z ) to obtain the reconstructed side latents ˆz .
The main motivation is to remove from the side information any image structure that would be left in the main latent representation y .
The hyperprior entropy model first obtains probability density function (PDF) of each scalar latent, assuming it follows a Gaussian distribution where the parameters are obtained by another deep network such as µ, σ = h s ( ˆz ; Θ) ) 2 .
Thus, the hyperprior model’s prediction can be defined by p ˆy ijk ( . ) :
= N ( .
| µ ijk , σ ijk ) .
The PMF of latent code P ( ˜y ijk ) can be written under determined quantization as a function of p ˆy ijk ( . ) .
Using this PMFs, the lower bound of the main codes’ bitlength can be defined by − log ( p h ( ˆy ; ˆz , Θ)) :
= − (cid:80) ijk log ( P ( ˜y ijk )) .
The Factorized entropy model learns the PDF for each t × t slice of latent defined as p ˆz : , : ,s ( . ) .
This PDF is enough to have PMF of side codes P ( ˆz ijk ) under determined quantization.
Thus, the lower bound of side codes’ bitlength can be defined by − log ( p f ( ˆz ; Ψ)) :
= − (cid:80) ijk log ( P ( ˜z ijk )) .
In this setting, the deep encoder ( g a ( . ; ϕ ) ), the deep decoder ( g s ( . ; θ ) ), the hyperprior entropy p h ( . ; ˆz , Θ) (composed of the deep hyperprior encoder ( h a ( . ; Φ) ), the deep hyperprior decoder ( h s ( . ; Θ) ), and the factorized entropy model p f ( . ; Ψ) ) are the trainable blocks implemented by neural networks.
The optimal values of the parameters ϕ, θ, Φ , Θ and Ψ are found by minimizing following loss function that train sample x comes from distribution of training dataset p x .
[Equation]
where d ( ., . ) is a distortion measure between the original and the reconstructed image (for example the mean square error).
The rate term is sum of the lower bound of bitlength of side information ( − log ( p f ( ˆz ; Ψ)) ) and main information ( − log ( p h ( ˆy ; ˆz , Θ)) ).
Hyperparameter λ controls the tradeoff between the rate (r) and distortion (d) terms.
After the training phase, the learned codec can be used at the encoder and decoder side.
The image is transformed to first obtain ˜z and that information is processed by RANS encoder using factorized entropy’s PMF table.
Then, ˜y is computed and encoded using RANS with hyperprior entropy’s
PMF table, obtained by ˜z .
At the receiver side, first ˜z is decoded by RANS decoder using factorized entropy’s PMF table.
Using ˜z into the hyperprior model, the PMF table of ˜y can be computed.
Finally, ˜y is decoded from the bitstream, leading to the reconstructed image using dequantization and deep decoder.
Both quantization step Q ( . ) and its counterpart dequantization Q − 1 ( . ) need to be applied for main and side information.
In addition, both factorized and hyperprior entropy models should know the determined quantization technique to determine the used PMF.
To the best of our knowledge, all current methods implement quantization as a 1-bin width uniform Scalar Quantization (SQ).
This quantization step is implemented by element-wise nearest integer rounding Q ( x ) = round ( x ) and its dequantization Q − 1 ( x ) = x .
Thus, the PMF of ˜y ijk ∈ R can be calculated by P ( ˜y ijk ) =
(cid:82) ˆy ijk +0 .
5 ˆy ijk − 0 .
5 p ˆy ijk ( x ) dx where p ˆy ijk ( . ) is the PDF of latent ˆy ijk learnt by entropy model 3 .
Since the nearest integer rounding operation has non-informative gradients, continuous relaxation must be applied at training as Q ( x ) = x + ϵ , where ϵ is randomly sampled from the uniform distribution ϵ ∼ U ( − 0 .
However, for the Vector Quantization (VQ) case, latents can be packed into v -dimensional vector u ∈ R v
and each u is assigned to a single code.
Quantization centers C j ∈ R v , j = 1 . . .
M are learned by entropy model.
Quantization step thus amounts to finding nearest center’s index as Q ( x ) =
arg min i || x
− C
i || , while dequantization returns the quanta center as Q − 1 ( i ) =
C i where i ∈ { 1 . . .
M } .
In this case, there are M different quantization centers, and M unique codes.
Since argmin operator applies hard assignment, it has non-informative gradients and continuous relaxation must also be applied during training.
This is generally achieved by softmax operator calculate
[Figure]
that assigns all codes to the latent vector with different probabilities, depending on the distances to the centers.
These probabilities are used by the entropy model to learn the PMF of each quanta center (codes), and also for dequantization which is the expectation of quanta centers under these probabilities during training.
U NIFORM V ECTOR Q UANTIZATION
Vector quantization (VQ) is theoretically optimal than SQ even if the dimensions are i.i.d (Gersho, 1982), but so far it could not perform significantly better than uniform SQ in neural codecs.
The reasons could be that (i) VQ learns non-uniform quantization center over the neural codec’s latents which is doubly necessary, (ii) the convergence of these centers might encounter training difficulties and can take longer training time whereas in uniform quantization, the quantization centers are predefined and fixed and (iii) since the dequantization of VQ averages these variable quantization centers in training phase, the reconstructed latents in training phase may be far away from their hard assignment values (values in test phase).
This mismatch can be higher than mismatch between nearest integer rounding and its continuous relaxation by additive uniform noise.
Here we improve the performance of vector quantization in neural codec by proposing uniform vector quantization.
Our main motivation is to use VQ with uniform grid is based on the sufficiency of uniform SQ among all SQs in the neural codec that we state in the following theorem.
Theorem 1.
If a neural codec has an encoder block g a : R n × n × 3 → R m × m × o , an decoder block g s : R m × m × o → R n × n ×
and it needs non-uniform SQ map for the optimal rate-distortion performance, there exists another neural codec that gives the same rate-distortion performance with 1-bin width uniform SQ (
nearest integer
rounding
quantization) whose encoder block is f ◦ g a and decoder block is g s ◦ f − 1 where f : R m × m × o
→ R m × m × o is an invertible transformation
The proof can be found in Appendix A and is based on modelling the one-dimensional quantizer using a memoryless monotonically increasing nonlinearity followed by a uniform fixed-point quantizer as in (Bennett, 1948).
We show that an invertible function that can be implemented by neural networks can transform borders and grid centers of the non-uniform quantization map to the uniform quantization map.
A simple illustration of this is shown in Figure 1(a).
According to Theoremneural codecs with nearest integer rounding quantization is fine among all scalar quantizations as long as it is enough expressive, that its encoder has invertible layers at the bottom of the encoder block and also the inverse of this layers should be at the beginning of the decoder block.
Most of the neural codecs do not have invertible blocks, but since the decoder is almost inverse of the encoder (in a sense that it is able to revert back image with negligible distortion) and usually they are enough deep with millions of parameters, we can assume that they are enough expressive.
The advantage of using invertible layers between less powerful encoder and decoder block were experimentally found recently in (Shukor et al., 2022).
Also, the sufficiency of uniform quantization in neural codecs was discussed in (Ball´e et al., 2021).
Our theorem verifies these two prior contributions.
In the following subsection we make connection between the space tessellation and uniform VQ.
[Figure]
3. S PACE TESSELLATION GRIDS
We propose uniform vector quantization based on the space tessellation in the latent space of neural codes, where the predefined shape covers the entire high dimensional space.
Thus, the quantization grids are fixed instead of learning it.
The nearest integer quantization grid in 1D is equivalent to unit square grid in 2D, as shown in Figure 1(b) for zero-mean Gaussian source, however it might not be the best partition of the space with uniform grids, thus using nearest integer quantization (i.e square grids in 2D) directly in the neural codec might not yield better RD performance.
Following remark shows the existence of entropy constrained optimal space tesellation and can be used for uniform vector quantization.
Remark 1.
VQ constraints with uniform grid produces the optimal space tessellation which refers to a pattern of v -dimensional shapes that fit perfectly together without any gaps and have minimum inertia.
The optimal shapes are regular hexagon for 2D, truncated octohedron for 3D and polytope
D 4 in 4D case (Gersho, 1979).
The above remark shows that instead of learning the grids centre and optimal partition, we can use the optimal space tessellation grids at the corresponding dimensions.
In this paper our proposal for 2D space is to use regular hexagon grid that has unitary volume as shown in Figure 1(c).
Theoretical advantage of space tessellation grid compared to the nearest integer quantization is shown for uniform distribution in Appendix B.
We also represent the simulation results under uniform and
Gaussian sources in Figure 2a-b. Figure 2c shows RD performance of unitary grids under different zero-mean Gaussian source is particularly important.
Because the main information in modern neural codec is zero mean different scale Gaussian distribution and quantized with unitary grids.
Figure 2c shows that when the scale parameter of Gaussian increases, it approximates the theoretical known RD performance of uniform source.
Now we show that how our proposed uniform vector quantization can be applied in the off-the-shelf neural codec without re-training.
3. U NIFORM VQ WITH OFF - THE - SELF NEURAL CODEC
Let y ∈
R m × m × o be the latents of the deep encoder, v be the dimension where the optimal space tessellation is performed,
c ( i ) ∈
R v , i = 1 . . .
M be the M grid centers of the v -dimensional shape and c ( i ) j ∈ R is the scalar value on j -th dimension of the center c ( i ) .
In order to quantize the latents, we reshape the latents y into pseudo v -dimensional latents such that y ∈
R m × m × o → ˇy ∈ R b × v , where b = m.n.ov .
The quantization is performed by assigning the nearest neighbours grid centers, as ˜y j = Q ( ˇy j ) =
arg min
i || ˇy j − c
( i ) || , where ˜y j ∈ { 1 . . .
M } , j = 1 . . .
b are the codes to be encoded into bitstream.
In practice, instead of reshaping all latents together into v dimension, we can reshape the latents whose learned PDFs are the same (i.e coming from the same distribution).
This approaches does not have significantly different results but needs smaller PMF table to be kept in decoding device.
To encode the codes ( ˜y ) into bit-stream, we cannot use the off-the-shelf neural codec
’s 1D entropy model’s PMF calculation, as our latents are v -dimensional latents and domain is different.
To this end, we propose to create the PMF by integrating the pseudo- v dimensional PDF ( (cid:81) vj =1 p c ( i ) j ( u j ) ) inside the grid G whose center is located on c ( i ) :
[Equation]
Here P ( c ( i ) ) is the PMF of i -th symbol in the dictionary and p c ( i ) j : R
→ R is PDF of corresponding latent, learned by entropy model in the baseline model.
Since there is no closed form solution of integral of multi dimensional Gaussian distribution over hexagonal and truncated octahedron domain (Savaux & Le Magoarou, 2020), we use numeric solvers to approximate the solution of equation 2.
We can reach numeric solution for main codes PMFs where p c ( i ) j ( . ) is the Gaussian distribution and also non
-parametric p c ( i ) j ( . ) for side code’s PMF.
In Appendix E, we show how to calculate equation 2 for hexagonal domain.
Another way to calculate those integral by monte-carlo simulation for known PDFs p c ( i ) j ( . ) before deployment which we use it for truncated octahedron domain.
F ORGOTTEN I NFORMATION : G RADIENT OF E NTROPY
In this section, we describe how the gradients of the entropy in the receiver side is used to improve the reconstruction performance.
When receiver decodes side latent, it can also know the gradients of side entropy w.r.t side latent.
After decoding main latent, it can also compute the gradients of main entropy w.r.t main latent.
Since these gradients are only available after decoding the latent codes, they seem not useful in the first sight.
This could be the reason why so far these gradients are never used in test phase.
Here, we propose to use the gradients through the analysis of KarushKuhn-Tucker conditions, and we claim that first gradient is correlated with gradient of main entropy w.r.t side latent and the second one is correlated with gradient of reconstruction error w.r.t main latent.
Thus, using the first one as in its correlated gradient, we can decrease the bitlength of main information and using the second one as in its correlated gradient, we can decrease the reconstruction error.
We can see the neural codec’s loss function in equation 1 as an unconstrained multi-objectives optimization problem, where the objectives are minimum bitlength of side information, minimum bitlength of main information and minimum reconstruction error.
The optimal solution of multiobjective problem is called Pareto Optimal which is a solution where no objective can be made better off without making at least one objective worse off (Miettinen, 2012).
The following remark shows a useful property of a solution of unconstrained multi-objective optimization problems.
Remark 2.
A solution of the multi-objective optimization problem is Pareto Optimal, if and only if it satisfies Karush–Kuhn–Tucker (KKT) conditions.
More specifically, in unconstrained multiobjective optimization problems case, if the aim of the problem is w ∗ =
arg min w ( (cid:80) i α i .
L i ( w )) ; where α i ≥ 0 , (cid:80) α i = 1 and L i is the i -th objective to be minimized, the solution w ∗ is Pareto
Optimal if and only if it satisfies the following (D ´ esid ´ eri, 2012).
[Equation]
In plain words, this shows that on the optimal solution point, all forces driven by gradients cancel each other out and the solution reaches saddle point.
This property is used to test the optimality of the candidate solutions.
This remark is also valid for end-to-end compression models.
Following theorem shows how to use KKT conditions for the end-to-end image compression models.
Theorem 2.
An end-to-end compression model optimized with λ trade-off is Pareto Optimal, if and only if the following two conditions are met.
[Equation]
The proof can be found in Appendix C. It is straight-forward, but the result of this theorem is significant.
One can interpret the theorem that if the existed end-to-end models is optimal (atleast in terms of training performance, not in terms of compressing performance), gradient of side information’s entropy w.r.t side latents ∇ ˆz ( − log ( p f ( ˆz ; Ψ))) and gradient of main information’s entropy w.r.t side latents ∇ ˆz ( − log ( p h ( ˆy ; ˆz , Θ))) cancel themselves out in expectation.
It is the same for the second condition as well.
We can claim that main information’s entropy w.r.t main latents ∇ ˆy ( − log ( p h ( ˆy ; ˆz , Θ))) and weighted gradient of reconstruction error w.r.t main latents λ ∇ ˆy ( d ( x , g s ( ˆy ; θ )))
cancels themselves out in expectation.
If these conditions were valid for every single image, given test image’s available gradient would be negative of unavailable gradient, thus they would have − 1 correlation coefficient.
However, these conditions are valid in average of images in train set and cannot guarantee the correlation for every single test image.
On the other hand, we can still assume that since the train set somehow represents the test image, at least these two gradients are correlated in test image, even though they are not exact negative of themselves.
This assumption is validated for different neural codecs with different test set, and presented in Appendix D and section 5.
The scatter plot of a sample image’s gradient of main latents and its correlated one are shown on Figure 3.
These gradients were calculated on a neural codec described in Minnen et al.
(2018) and implemented in B ´ egaint et al.
According to our test on widespread neural codecs, we have found quite strong correlation on main latent’s gradient (between − 0 .
1 to − 0 . 5 ), but weak correlation (between − 0 .
15 to 0 .
1 ) on side latent’s gradient.
Latent Shift wrt Gradients.
By definition of gradient based optimization, ˆz needs to take a step in negative direction of ∇ ˆz ( − log ( p h ( ˆy ; ˆz , Θ)))
in order to decrease main information bitlength − log ( p h ( ˆy ; ˆz , Θ) .
However ∇ ˆz ( − log ( p h ( ˆy ; ˆz , Θ))) is not available before decoding ˆy in decoding device, but at least the correlated gradient ∇ ˆz ( − log ( p f ( ˆz ; Ψ))) is known after decoding ˆz .
We claim that there is a real number step size ρ ∗ f that decrease the bitlength of main information such that
[Equation]
ρ ∗ f can be obtained by brutal force out of handful candidates or any optimization method to find optimal one such that;
[Equation]
For the second condition, ˆy needs to take a step in negative direction of ∇ ˆy ( d ( x , g s ( ˆy ; θ )))
in order to decrease reconstruction error d ( x , g s ( ˆy ; θ )) .
Since ∇ ˆy ( d ( x , g s ( ˆy ; θ ))) is never available in decoding device, but at least the correlated gradient ∇ ˆy ( − log ( p h ( ˆy ; ˆz , Θ))) is known after decoding ˆy and ˆz .
We claim that there is a real number step size ρ ∗ h that decrease the reconstruction error such that
[Equation]
ρ
[Equation]
h can be found by brute-force search out of handful candidates or any optimization method to find optimal one such that;
[Equation]
; θ ))) .
In summary, our proposal can be seen as shifting the side latent by ˆz ← ˆz + ρ ∗ f ∇ ˆz ( − log ( p f ( ˆz ; Ψ))) after decoding ˆz and shifting the main latent by ˆy ← ˆy + ρ ∗
h ∇ ˆy ( − log ( p h ( ˆy ; ˆz , Θ)))
after decoding ˆy while the best step sizes ρ ∗ f , ρ ∗ h ∈ R are to be found in encoding time and added to the bitstream explicitly with fixed bitlength.
It is noted that during the encoding, the main latents ˆy are encoded wrt entropy model predicted by the shifted side latent.
We show that the shifted side latents shortens the main information’s bitlength and shifted main latents results in the better reconstruction performance.
E XPERIMENTAL R ESULTS
We use CompressAI library (B´egaint et al., 2020) to test our contributions on 4 pre-trained neural image codecs named bmshj2018-factorized in Ball´e et al.
(2017), mbt2018-mean and mbt in Minnen et al.
(2018), cheng2020-attn in Cheng et al.
(2020), and 2 additional pre-trained codec named invcompress in Xie et al.
and DCVC-intra in Li et al.
(2022) with all intra mode and two neural video codec named SSF in Agustsson et al.
(2020) and DCVC in Li et al.
For the evaluation, we use Kodak dataset (Kodak) and Clic-2021 Challenge’s Professional dataset (CLI).
The codecs are taken off the shelf and are not retrained.
The rate is calculated from the final length of the compressed data and RGB PSNR is used for distortion.
We asses the performance using the bdrate (Bjontegaard, 2001) on the rate range used in compressAI: 0.1bpp to 1.6bpp for image codecs and 0.03bpp to 0.35bpp for video codecs.
We report our uniform VQ proposals under the name of
Hex-Quant for regular hexagonal.
We denote by Latent Shift the gradient based improvement and
Join the results using both methods.
[Table]
Results on Image Codecs:
In Table 1, we show the results of the methods for different datasets and codecs.
Since our two proposals are orthogonal to each other, the gain of the joined methods is almost the sum of the two.
However, after uniform VQ, the reconstructed latents change compare to uniform SQ, thus, gradients wrt this latents change as well.
This variation creates slightly different results compared to Latent Shift only, thus Join results are not the exact sum.
Since uniform VQ’s reconstructed latent is much more closer to the latents before quantization, nearly in all cases, Latent Shift with uniform VQ gives better contribution than only Latent Shift .
The bd-rate (gain in %) compared to mbt2018-mean for different quality is shown in Figure 4a on the Kodak dataset.
Even though our proposal has a 2 .
24% gain, the gain on lower quality (thus lower rate) is bigger ( 3 . 5% ).
More detailed results of the gain w.r.t.
to reconstruction quality are shown in
Appendix F.
According to our tests, since the side latent’s correlation’s is weaker, the most important gain of Latent Shift comes from the gradient of the main latent.
The analysis of the source of the gain of Latent Shift and its performance against some alternatives are presented in Appendix D.
Join refers using Oct-Quant and Latent Shift
[Table]
Results on Video Codec: In Table 2, we show the results of the SSF and DCVC video codecs for different sequences.
The SSF (Agustsson et al., 2020) was proposed for low-delay mode: frames are coded sequentially using the previously reconstructed frame and an intra frame is inserted every frames (UVG videos are divided into 75 Intra periods).
I frame encoding uses the same VAE method, so our implementation on I frame is the same as the one described above.
However, each P frame consists of motion information and residual information encoded by two different VAE models.
Since the motion information represents only a fraction of the total bitrate ( 3 − 4% ), we apply our proposal on the residual coding only.
When we apply Oct-Quant and Latent Shift together, it gains 2 − 2 .
7% of bitrate for the same quality on UVG dataset on average.
Again, the gain at lower rate is larger (more than 4% ) as shown in Figure 4b.
More analysis on quantization effects can be found in
Appendix D.
[Figure]
Gradients correlation: The Latent Shift ’s gain depends on how the gradients are correlated.
To assess this correlation, we show the scatter plot between Latent Shift ’s individual gains in dB and actual correlation coefficient between gradients for all test images and for all quality level in Figure 4c.
We found a correlation of − 0 .
76 where this correlation is independent from datasets, reconstruction quality or model.
Since the neural video codec was optimized for the loss of all frames in a GOP, the gradients’ sum in equation 3 and equation 4 are zero in expectation for both the frames in the GOP and over the datasets.
This smooths the sum of gradients even further and decrease the correlations compare to the image codecs.
It explains why the the correlation of the gradients are lower and also why Latent Shift ’s individual gain is lower in video codecs.
Complexity analysis: We have evaluated the complexity of our proposed over the baseline and the detailed analysis are presented in Appendix G. Our uniform VQ introduces about 3-5% additional complexity during encoding and 2% during decoding, and the Latent Shift introduces about 5-10% additional complexity during encoding and about 1% during decoding over the baseline method.
C ONCLUSION
In this work, we have proposed two orthogonal methods to improve further the latent representation of generic compressive variational auto-encoders (VAE).
Firstly, we exploit further the remaining redundancy in the latent during the quantization stage.
To do so, we demonstrate that a uniform
VQ method improves a VAE trained using with uniform scalar quantization.
Secondly, we exploit the correlation of the gradient of the entropy and the reconstruction error to improve the latent representation.
The combination of these two methods improve the latent representation, and bring significant gains on top of several state-of-the-art compressive auto-encoders, without any need of retraining.
From these results, several improvements can be foreseen.
First, the correlation of the gradients depends on the training set according to the definition of the KKT conditions.
However, even though different models use the same training set and training procedure, their gradients’ correlation coefficient may be different.
Thus, it might be interesting to explore the connection of certain type of model architectures with the correlation of the gradients.
Secondly, the uniform VQ process was applied without retraining.
Even though VQ quantization error is lower on average, the maximum quantization error can be higher.
Since the decoder block cannot see that individual high errors during the training, the model becomes sub-optimal.
This can be solved by fine-tuning the decoder block or the entire model using a continuous relaxation of the uniform VQ grid.
A P ROOF OF T HEOREM
Proof.
Scalar quantization map can be see as n + 1 border values with b 0 < b 1 < · · · < b n and n quantization center with c 1 < c 2 < · · · < c n .
Any latent y with b i − 1 ≤ y <
b i should be quantized to the point c i .
Quantization map can be represented by an ordered set consisting of all borders and centers such as M = { b 0 < c 1 < b 1 < c 2 < b 2 < · · · < c n < b n } where | M | = 2 n + 1 .
When the quantization map is non-uniform, the difference between consecutive elements in the set are not necessarily equal, i.e. ∃ ( i, j ) , M i − M i − 1 ̸ =
M j − M j − 1 .
Nearest integer quantization map can be denoted as M ( u ) = { 0 . 5 , 1 , 1 . 5 , 2 , 2 . 5 , . . .
n, n + 0 . 5 } or simply
M ( u ) i = 0 .
5 i thus, ∀ i, M ( u ) i − M ( u ) i − 1 = 0 .
Let’s assume than any arbitrary M is the optimal scalar quantization map of a neural codec
’s latent
y obtained by y
= g a ( x ) .
Since M and M ( u ) are both monotonic increasing set, there exists a bijective function f ( . ) that maps ∀ i, M i to M ( u ) i , and f − 1 ( . )
maps M ( u ) i to M i , ∀ i .
According to universality theorem Hornik et al.
(1989), the function f ( . ) can be implemented by a multi layer neural network.
Two codecs’ performances are equal if and only if entropy of their latents are equal and their reconstructions are the same.
First, we start by showing that their entropy’s are the same by showing that the corresponding center’s PMF are equal in both space, i.e. ∀ i, P ( i ) =
P ( u ) ( i ) .
We can write the i -th quantization center’s PMF as P ( i ) =
(cid:82) b i b i − 1 p ( y ) dy where y is a point in g a ’s output space.
Any point y can be transformed to a new space by z = f ( y ) .
We can write the i -th quantization center’s PMF as P ( u ) ( i ) =
(cid:82) i +0 .
5 i − 0 .
5 p ( z ) dz in this space.
We can rewrite it as
P ( u ) ( i ) =
(cid:82) f − 1 ( i +0 . 5) f − 1 ( i − 0 .
5) p ( f − 1 ( z ))
df − 1 ( z ) .
Since f − 1 ( i + 0 . 5)
=
b i , f − 1 ( i − 0 . 5) =
b i − 1 and f − 1 ( z ) =
y , we can write ∀ i, P ( u ) ( i ) =
(cid:82) b i b i − 1 p ( y ) dy
=
P ( i ) .
The output of a deep decoder is g s ( c i ) if the latent y
=
g a ( x ) meets b i − 1 ≤ y <
b i .
Any latent b i − 1 ≤ y <
b i is mapped to f ( b i − 1 ) ≤ f ( y ) <
f
( b i ) thus i − 0 .
5 ≤ z < i +0 .
The latent z which lies between i − 0 .
5 to i + 0 . 5 , can be quantized to i in the new space.
Since the decoder applies g s ( f − 1 ( z )) , its output should be g s ( f − 1 ( i )) .
Since f − 1 ( i ) =
c i , it gives g s ( c i ) .
B A DVANTAGE OF S PACE T ESSELLATION G RID ON Q UANTIZATION
When the source has a uniform distribution, quantization using a truncated octahedron gives better
RD performance compared to regular hexagonal grids, and regular hexagonal grid gives better RD performance than uniform SQ grids (nearest integer rounding).
In order to show the superiority of a method over another in terms of RD performance, it is enough to compare the reconstruction error at equal bit-rate.
Since the equal volume grids have the same probability under uniform distribution, the rate is equal for the three cases.
Since the distributions are identical, we just need to compute the mean square error for each types of grid at some position, for example the origin for 1D, 2D and 3D cases respectively.
MSE of uniform SQ grid can be written as the integral of square error normalized by the grid size:
[Equation]
It gives MSE ( s ) ( u ) =
u 2 / 12 ≈ 0 .
0833 u 2 .
Hexagonal grid case: For hexagonal grid, we need to double integrate over the hexagonal domain.
Figure 5a shows an hexagon with a side a length a located at the origin.
We also show each side’s functions in 2D space.
We divide the hexagon into two parts for positive and negative y and calculate the analytic integral for these two region separately.
We then normalize the sum of squares by area of the area of the hexagon, which is 3 √ 3 a 2 / 2 .
[Equation]
[Figure]
When we first integrate the positive part of integral over x between − a + y √ 3 and a − y √ 3 , followed by the integral over y between 0 and a √ 32 .
The integral for the positive region is is then 5 √ 3 a 4 32 .
The same can be done for the negative region, but since the distribution is uniform, it gives the exact same result, thus we conclude that MSE ( h ) ( a )
= 5 a 2 24 .
In order to obtain the same volume for the grid, hexagon’s area should be u 2 .
Since the hexagon’s area is 3 √ 3 a 2 / 2 for a side length if a , we find that a = √ 2 u √ 3 √ 3 .
Thus MSE ( h ) ( u ) = 5 √ 3 u 2 108 ≈ 0 .
u 2 .
Truncated octahedron case: For the MSE of the truncated octahedron grid, we need to integrate the mean square error of each 3 dimension over the truncated octahedron domain.
In figure 5b, a truncated octahedron with a side length of a is shown.
The solution can be obtained by integrating one octant, multiply it by 8 and normalize by its volume which is 8 √ 2 a 3 as follows:
[Equation]
First, we integrate over z , y and then x , which gives the solution MSE ( o ) ( a )
= 1948 a 2 .
To compare with the same grid volume, the truncated octahedron should have a volume of u 3 .
Since the volume is 8 √ 2 a 3 when one side length is a , we can find that a =
[Equation]
3 , thus MSE ( o ) ( u ) = 19 48(8 √ 2) 2 / 3 u 2 ≈ 0 .
u 2 .
As a result, MSE ( s ) ≈ 0 .
0833 u 2 , MSE ( h ) ≈ 0 .
u 2 and MSE ( o ) ≈ 0 .
0785 u 2 , where the volume of the grid is u .
We can thus conclude that ∀ u ∈ R + , MSE ( o ) ( u ) <
MSE ( h ) ( u ) <
MSE ( s ) ( u ) .
C P ROOF OF T HEOREM
Proof.
Let us rewrite the end-to-end loss in equation 1 as an unconstrainted multi-objective optimization:
[Equation]
Where α 1 = 1 / (2+ λ ) , α 2 = 1 / (2+ λ ) and α 3 = λ/ (2+ λ ) .
Since λ > 0 , ∀ i, α i > 0 and (cid:80) i α i
= 1 , the set of α
i s corresponds to an unconstrained multi-objective optimization’s coefficients set.
Since we do not target to update the parameters of the end-to-end model, but just target to adjust main and side latents ˆy and ˆz , we keep all the parameter fixed, but ˆy and ˆz as variable.
Thus, we can represent side information’s bitlength objective with L 1 ( ˆz ) :
= − log ( p f ( ˆz ; Ψ)) , main information’s bitlength objective with L 2 ( ˆy , ˆz ) :
= − log ( log ( p h ( ˆy ; ˆz , Θ))) and finally distortion objective with L 3 ( x , ˆy ) := − d ( x , g s ( ˆy ; θ )) .
With these notations, training the model can be written as an unconstrained multi objective optimization problem:
[Equation]
This problem has two set of variables to be optimized, and both variables should meet KKT conditions.
The variable ˆz is decorrelated from L 3 ( x , ˆy ) , thus ∇ ˆz L 3 ( x , ˆy )
= 0 .
We can write KKT condition for ˆz as follows.
[Equation]
Since α 1 = α 2 , we obtain the first condition in equation 3 by simply replacing L 1 ( ˆz ) and L 2 ( ˆy , ˆz ) with their definitions.
The second variable ˆy does not impact L 1 ( ˆz ) , thus ∇ ˆy L 1 ( ˆz ) = 0 .
We can write KKT condition for ˆy as follows.
[Equation]
If we replace the objectives by their definitions and divide the two-hand side by α 2 , and since α 3 /α 2 = λ , we reach the second condition in the theorem in equation 4.
D A BLATION S TUDIES
In this section, we present several ablation studies for uniform VQ and latent shifting.
D. G AIN A NALYSIS OF L ATENT S HIFT
To understand the upper limits of gradient based latent shifting, we use the true gradients instead of proxy one and measure the performance.
Simply, we shift the side latents by ˆz ← ˆz + ρ ∗ f ∇ ˆz ( − log ( p h ( ˆy ; ˆz , Θ))) after decoding ˆz .
Later, we shift the main latent by ˆy ← ˆy + ρ ∗ h ∇ ˆy ( d ( x , g s ( ˆy ; θ ))) after decoding ˆy .
We can see this hypothetical case as if the correlations are − 1 .
This case is mentioned by True Gradients in the results on Table 3 while our proposal is
Latent Shift .
The results are taken with mbt2018-mean image codec on Kodak dataset.
[Table]
According to this results, we can see that even if the side latent’s gradients were perfectly correlated, our maximum gain would be around 1% .
Since the correlation of gradients wrt the side latent is weak ( r 2 ≈ − 0 . 07 ), our gain is negligible.
As a consequence, in practice, shifting side latent may not be neglected.
On the other hand, main latents true gradients increase
PSNR by 1 . 40 dB in average which is equivalent of saving around 25% of the bitstream.
Our proposal could increase the PSNR by 0 .
07 dB in average which is equivalent to saving 1 .
27% of the bitstream for the same quality thanks to the existing correlation between gradients wrt main latents as shown in Figure 6.
The gain is of course smaller than the upper limit, but significant still.
Since keeping these gradients are costly (nearly the same cost of saving image itself), searching more effective way of using those gradient is meaningful.
D. L ATENT S HIFT VERSUS A LTERNATIVES
In order to show how proposed latent shifting is better than alternatives, we shift the latent in random direction at encoding time such that ˆy ← ˆy + ϵ ( ρ h ) , where ϵ ( ρ h ) ∈
R m × m × o ∼ N (0 , 1) and ρ h is a random seed number to be signalled.
The best random seed and gradients, in terms of PSNR improvement, should be found at encoding time.
We generate 1024 random gradient and encode the best random seed with 10 bits as an extra information.
Even though this approach is costly in terms of computational complexity in encoding time, (it needs 1024 times forward pass), we think this is a natural baseline to our proposal, and we refer to it as Random Shift .
Another alternative would be to use constant gradient for all latents in a given image.
Thus, at encoding time we need to test a large set of values and assume all latents should be shifted by this number such that ˆy ← ˆy + ρ h where ρ h ∈ R .
The best value of ρ h should be signaled to the decoder with 10 bit extra cost.
We refer to this as Scalar Shift approach.
Our last alternative does not use gradients directly, but the fact that they may be correlated as prior knowledge.
Particularly, this approach shifts the latent in the opposite direction of the distribution’s center such that ˆy ← ˆy − ρ h sign( ˆy − µ ) .
In hyperprior entropy models, the latent is assumed to obey a Gaussian distribution.
thus, the latent’s entropy gets smaller if it moves towards the center of the distribution.
By shifting the latent in opposite of this direction, the entropy increases.
Since the factorized entropy model does not use Gaussian distribution, we shift the latent to the opposite direction of zero center means assuming µ = 0 in factorized entropy.
The best value of ρ h should be signaled to the decoder with 10 bit extra cost.
We refer to this approach as Sign Shift .
[Table]
In Table 4, we compare bmshj2018-factorized (lowest correlation between gradients) and mbt2018-mean (highest correlation between gradients).
Since all alternatives need extra 10 bits signaling cost, we neglect it assume the bitlengths are the same as the baselines, and we report the
BD-PSNR defined in Bjontegaard (2001).
These results show that our proposal is significantly better than all alternatives.
The closest one ( Sign Shift ) reaches half of Latent Shift ’s performance.
The random alternatives could not improve the baseline significantly, even though their encoder is almost 1000 times computational demanding.
D. T RUNCATED O CTAHEDRON VERSUS H EXAGON
Concerning uniform VQ quantizations, either hexagonal grid quantization Hex-Quant or truncated octahedron grid quantization Hex-Quant give different results.
Better quantization leads to the smaller quantization error on latent and better reconstruction.
However, bigger symbol dictionary is needed (square of SQ’s dictionary size in hexagonal grid).
Since the arithmetic encoder has a fixed bit resolution, it has representation limitations (for 16-bit resolution, the minimum probability is 1 / 65536 ).
Thus, in practice, we can assign 1 / 65536 probability to the symbol whose probability is lower than 1 / 65536 , what makes encoding less efficient and increase the rate.
Another alternative is to remove those symbols, what increases the quantization error of latents but decreases the rate.
In practice, we have chosen to remove the symbol if its probability is less than 10 − 7 , and we found out that it always gives better RD performance.
In this ablation study, we analyse further the quantization effects on reconstruction error (PSNR), as well as the impact of latent quantization error in terms of Signal Noise Ration (SNR) w.r.t.
the rate (under the assumption that arithmetic encoder has infinite bit resolution).
To this end, we do not encode the symbols into bitstream, but calculate the lower bound of bitlength according to
Shanon’s entropy theorem without limits of integer resolution that dictates some certain probabilities on symbols.
We calculate SNR of latents by SNR = − 10 log 10 ( d ( ˆy , y )) where y is latent, ˆy is reconstructed latent by certain quantization techniques and d ( ., . )
measures MSE error between two inputs.
We use mbt2018-mean neural codec in Minnen et al.
(2018) on Kodak dataset.
Results are presented in Table 5 and show that Oct-Quant gives lower reconstruction error (higher PSNR improvement) and lower quantization error over latent (higher SNR improvement) than Hex-Quant for the same rate.
In addition, Oct-Quant saves more bitlength than Hex-Quant for the same reconstruction quality.
Figure 7 compares the performances of Oct-Quant and Hex-Quant to the uniform SQ baseline, for different rate and reconstruction quality.
[Table]
[Figure]
E N UMERIC PMF C ALCULATION FOR H EXAGONAL D OMAIN
In this section, we give some implementation details.
The closed form solution for the integrals of multidimensional known densities over any domain has very specific solutions, and generally no tractable form (Savaux & Le Magoarou, 2020).
That holds even for Gaussian distribution of dimension 2 on the regular hexagon domain.
Thus, there is no closed form solution of equation 2, where the probabilities are independent Gaussian on the domain of regular hexagonal.
However, we find the solution using the combination of both analytic and numerical integration as detailed below.
The integral centred around ( x, y ) of hexagonal domain G for the independent Gaussian distribution is given by
[Equation]
For sake of simplicity, let ( x, y )
=
(0 , 0) and a be the one side length of the hexagonal.
Then, the integral is defined as
[Equation]
where the hexagon is divided into lower and upper half parts, and we sum the two integrals.
It is noted that the integral does not admit a closed form solution, but the inner integral has the closed form and outer integral does not have the analytic solution.
The solution of the inner integral in the first part is given by
[Equation]
(12) by substituting into equation 11, we obtain:
[Equation]
This is finally solved by the numerical integration 4 .
Similarly, we can also obtain the integral of the second part.
F A DDITIONAL R ESULTS
In this section, we present additional results of gain evaluated over different methods with Kodak and Clic-2021 dataset.
Figure 8-11 shows th BD-rate gain for bmshj2018-factorized in Ball ´ e et al.
(2017), mbt2018-mean and mbt2018 in Minnen et al.
and cheng2020-attn in Cheng et al.
[Figure]
[Figure] [Figure]
G C OMPLEXITY A NALYSIS
Here we describe the computational complexity of proposed method and provide detailed analysis of source of the additional computational costs and the run time results of proposals with respect to the baseline models.
G. S OURCE OF COMPLEXITY
The complexity of our proposed method includes encoding and decoding complexity of uniform
VQ, and encoding and decoding complexity of Latent shift, and they are detailed below
[Figure] [Figure]
G.1. E NCODING COMPLEXITY OF UNIFORM VQ
First, we should define our grid centers as codebook such that c ( i )
∈ R v , i = 1 . . .
M be the M grid centers of the v -dimensional shape (e.g hexagonal grids center for v = 2 ) and PMF of our v -dimensional grid using learned 1 D PMF by equation 2 in the main paper.
The additional steps involved in the encoding time over the baseline approach are as follows:
Reshape the latents into pseudo v -dimensional vector y ∈ R m × m × o → ˇy ∈ R b × v , where b = m.n.ov (before reshaping, the latent can be sorted by their distribution’s σ parameters.
In this way, the latents that belongs to a similar distribution can be dropped into the same latent vector).
Find closest codebook from our initial quantization grids for each b vector such ˜y j = argmin
i ||
ˇy j − c
( i ) || , where ˜y j ∈ { 1 . . .
M } , j = 1 . . .
b are the codes to be encoded into bitstream.
In the shared step (with baseline method), ˜y j , j = 1 . . .
b should be encoded into bitstream by given PMF.
Thus, in encoding time above 2 step’s complexity is the reason for the extra complexity introduced by our proposed method.
[Figure]
G.1. D ECODING COMPLEXITY OF UNIFORM VQ
In decoding time, ˜y j , j = 1 . . .
b should be decoded from bitstream by provided PMF table.
The source of extra complexity of our method (over the baseline) in the decoding time is the following two steps:
De-quantizate the decoded codes: find the center of selected quanta center such that ¯y j = c
˜y j , j = 1 . . .
b 2.
Reshape dequantized latent into the original dimensions such that ¯y ∈ R
b × v → ˆy ∈ R m × m × o , (if the ordering wrt σ is applied, we need to revert the order back in order to have the latents in original order)
In shared step (with the baseline method), these dequantized and reshaped latents are fed to the decoder network.
G.1. E NCODING COMPLEXITY OF L ATENT S HIFT
In encoding time, the complexity introduced by the Latent Shift are as follows:
The calculation of the gradients of the entropy ∇ ˆy ( − log ( p h ( ˆy ; ˆz , Θ)); θ ) 2.
finding the best step size ρ h that maximize the reconstruction quality
when
the latent is shifted as ˆy ← ˆy + ρ ∗ h ∇ ˆy ( − log ( p h ( ˆy ; ˆz , Θ)))
It is noted that calculating gradient of the entropy has a negligible complexity, because we do not need the forward pass and there is closed form solution.
When the entropy model uses Gaussian distribution such as p h ( ˆy ; ˆz , Θ) :
= N ( ˆy ; µ, σ ) , the gradient of the entropy becomes the derivative of − log ( N ( ˆy ; µ, σ ))
wrt ˆy which has closed form solution, where N ( . ; µ, σ ) is PDF of gaussian distribution by given fixed µ, σ parameters.
However, the second step to find the best step size ρ h is moderately demanding process.
In the experiments, we tested 8 different choices for p h and selected the best one.
Thus, it needs 8 forward passes of decoding and calculation of error between input and reconstruction.
G.1. D ECODING COMPLEXITY OF LATENT SHIFT
In decoding time, the complexity of decoding the latents from the bitstream is already included in the decoding step of uniform VQ.
So, the decoding complexity of the Latent Shift are: (1) decoding a single scalar ρ h from bitstream and calculating the gradient entropy wrt latents, which are both negligible; (2) Shifting the latent by using gradients and decoded step size which is actually adding two tensor, and it is negligible compared to the decoding complexity of the baseline model.
G. R UNTIME COMPLEXITY ANALYSIS
For experimental computational complexity, we run the test on single core of Intel(R) Core(TM) i78850H CPU @ 2.60GHz with preventing multi-thread operations.
We measured the computational time of cheng2020 attn , mbt2018-mean on Kodak dataset and SSF on bunny dataset.
In order to have the same resolution image, we just crop the top-left part of the frames in bunny frame set which has the same resolution as kodak dataset (512x768).
In this way, all methods complexity can be comparable between each other.
The results show average encoding/decoding time in seconds for the provided lowest bitrate of baseline model and our Hex-Quant, Oct-Quant and Latent Shift.
The reported encoder running times is the duration from the input image is read to the memory to the producing the bitstream including arithmetic encoder time.
It is the same for decoding.
The duration is from the bitstream in the memory to obtain reconstructed image including arithmetic decoder process.
The results are reported in the table 6.
[Table]
The main source of additional encoding complexity of Hex-Quant and Oct-Quant is to find the closest codebook.
The naive search gives around %20 overhead complexity for 3 -dimensional VQ.
But there are algorithms that find nearest quanta center very efficiently as defined in (Agrell et al., 2002;
Conway & Sloane, 1982).
We have adapted this solution to our model and decreased additional cost to 3%-5% in encoding time for mbt2018-mean model.
Though cheng2020 attn model is the most complex model in our studied neural codec, our extra computation cost is almost the same with mbt2018-mean model (our Hex-Quant and Oct-Quant runtime depends on the resolution of the latent not the architecture), Hex-Quant and Oct-Quant’s relative overhead is between 1.1%-0.7% in encoding time.
For SSF, we neglect the I frame compression runtime (because it is the same with mbt2018-mean ) but measured the average runtime for encoding and decoding of P frames which is done by two VAEs, one for motion information another for residual information.
Thus, there are two main information (main motion and main residual) and our quantization should be done on these latent.
It explains our extra absolute runtime is more than absolute extra runtime in mbt2018-mean .
Since the P frame compression model is more complex than mbt2018-mean , Hex-Quant and Oct-Quant’s extra relative runtime is between 2.4%-3.2%.
Since the differences between dimension of VQ is just reshaping tensor and number of indices to be found from dictionary in decoding time, the decoding time extra complexity of Hex-Quant and Oct-Quant is negligible and almost the same with hex and oct quantization as it can be seen in the table
Regarding the Latent Shift , its decoding complexity is less than 1% as it can be seen in Table 6.
However, Latent Shift ’s encoding complexity is between 5-10 magnitude of baseline encoding running time.
Almost all complexity comes from finding the best shift step size by brutal force out ofcandidates.